Project

General

Profile

Feature #3628

Add pretty reports to rdiff-handler

Added by davekempe about 6 years ago. Updated over 4 years ago.

Status:
Confirmed
Priority:
Low
Assignee:
-
Category:
rdiff-backup handler
Target version:
-
Start date:
11/14/2011
Due date:
% Done:

10%

QA Check:

Description

We use backupninja for backing up client's data, and the clients want a pretty report that says what got backed up and what didn't and what failed.
I am going to put in our whole handler for rdiff-backup here, as I am not sure of the changes. Please feel free to run a diff and merge in the changes you see fit.
The other major feature is that this handler can submit the job result via nsca to a nagios host via a passive check. nsca relies on send_nsca being installed in /usr/sbin/.

I have also attached the sample reports, so you can see how it looks.
Anyone is welcome to take these reports and modify them for backupninja use, of course having our logo on the reports doesn't make sense for anyone else.
All this stuff is released under the GPLv2 of course.
Customising the reports is left up as an exercise to you guys. Hope thats OK.

To make them work, you need to add these variables to backupninja.conf:
company = Company Name

  1. Customer's contact email
  2. used to send backup reports
  3. Can be left blank if you don't want to send reports
    clientemail =
  1. Nagios server the passive checks results
  2. will be sent to
    nagioshost = nagioshost.domain.com

And you need a separate /etc/backupninja_nsca.cfg like so:
password=nscapassword
encryption_method=1

First rdiff.in


$ cat rdiff.in
# -*- mode: sh; sh-basic-offset: 3; indent-tabs-mode: nil; -*-
#
# rdiff-backup handler script for backupninja
# requires rdiff-backup
#

### FUNCTIONS ###

function test_connection() {
        # given a user and host,
        # tests the connection.
        # if user or host is missing, returns 0
        # (ie, assume it's a local connection).
        if [ $# -lt 2 ]; then
                debug "(local is assumed to be a good connection)" 
                return 0
        fi
        local user=$1
        local host=$2
        debug "ssh $sshoptions -o PasswordAuthentication=no $host -l $user 'echo -n 1'" 
        local ret=`ssh $sshoptions -o PasswordAuthentication=no $host -l $user 'echo -n host is alive'`
        if echo $ret | grep "host is alive"; then
                debug "Connected to $host as $user successfully" 
        else
                fatal "Can't connect to $host as $user." 
        fi
}

function get_version() {
        # given no arguments, returns the local version.
        # given a user and host, returns the remote version.
        # if user or host is missing, returns the local version.
        if [ "$#" -lt 2 ]; then
                debug "$RDIFFBACKUP -V" 
                echo `$RDIFFBACKUP -V`
        else
                local user=$1
                local host=$2
                debug "ssh $sshoptions $host -l $user '$RDIFFBACKUP -V'" 
                echo `ssh $sshoptions $host -l $user "$RDIFFBACKUP -V | grep rdiff-backup"`
        fi
}

function check_consistency() {
        local section=$1
        local type=$2
        local user=$3
        local host=$4
        if [ "$type" == "local" ]; then
                if [ "$user" != "" ]; then
                        warning "User should not be specified for local $section." 
                fi
                if [ "$host" != "" ]; then
                        warning "Host should not be specified for local $section." 
                fi
        fi
        if [ "$type" == "remote" ]; then
                if [ "$user" == "" ]; then
                        fatal "User must be specified for remote $section." 
                fi
                if [ "host" == "" ]; then
                        fatal "Host must be specifed for remote $section." 
                fi
        fi
}

function check_cstream() {
        local cstream=$1
        if [ ! -x $cstream ]; then
                fatal "Can't find your cstream binary (trying: $cstream). If you use bwlimit you must have cstream installed." 
        fi
}

function send_nagios_check_result() {

        directory=$destdir/$label

        if [ "$desttype" == "remote" ]; then
                mkdir /tmp/rdiffcheck
                directory=/tmp/rdiffcheck
                sshfs $destuser@$desthost:$destdir/$label $directory
        fi

        result="`/usr/sbin/rdiffcheck -r $directory -w 8 -c 10 -l 1200 -p 24`" 
        exit_result=$?

        # Create the service name string
        service="`echo $company | sed 's/[-_,.\t]/\ /g' | sed 's/\<\(.\)\([^ ]*\)/\u\1\L\2/g'` [$label]" 

        nsca_out=`echo "Remote  $service        $exit_result    $result" | /usr/sbin/send_nsca $nagioshost -c /etc/backupninja_nsca.cfg 2>&1`
        if [[ $? -ne 0 ]]; then
                warning $nsca_out
                warning "Nagios check result sending failed" 
        fi

        if [ "$desttype" == "remote" ]; then
                fusermount -u $directory
                [ -n "`grep $directory /proc/mounts`" ] && fusermount -uz $directory
                rm -rf /tmp/rdiffcheck
        fi

}

function send_rdiff_report() {

        declare -a sessionstats

        rdiff_output="$1" 
        report_company="$2" 
        report_label="$3" 
        mail_to="$4" 

        eml=/tmp/rdiff_report.eml
        staff_eml=/tmp/rdiff_report_staff.eml
        attachment=/tmp/$report_label-backup-`date +%F-%X`.log

        # Check the log for statistics
        if [ -n "`echo \"$rdiff_output\" | grep \"\[\ Session\ statistics\ \]\"`" ]; then

                # Successfull backup
                sessionstats[0]=`echo "$rdiff_output" |grep StartTime|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[1]=`echo "$rdiff_output" |grep EndTime|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[2]=`echo "$rdiff_output" |grep ElapsedTime|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[3]=`echo "$rdiff_output" |grep SourceFiles|cut -d ' ' -f 2`
                sessionstats[4]=`echo "$rdiff_output" |grep SourceFileSize|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[5]=`echo "$rdiff_output" |grep MirrorFiles|cut -d ' ' -f 2`
                sessionstats[6]=`echo "$rdiff_output" |grep MirrorFileSize|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[7]=`echo "$rdiff_output" |grep NewFiles|cut -d ' ' -f 2`
                sessionstats[8]=`echo "$rdiff_output" |grep NewFileSize|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[9]=`echo "$rdiff_output" |grep DeletedFiles|cut -d ' ' -f 2`
                sessionstats[10]=`echo "$rdiff_output" |grep DeletedFileSize|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[11]=`echo "$rdiff_output" |grep ChangedFiles|cut -d ' ' -f 2`
                sessionstats[12]=`echo "$rdiff_output" |grep ChangedSourceSize|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[13]=`echo "$rdiff_output" |grep ChangedMirrorSize|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[14]=`echo "$rdiff_output" |grep IncrementFiles|cut -d ' ' -f 2`
                sessionstats[15]=`echo "$rdiff_output" |grep IncrementFileSize|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[16]=`echo "$rdiff_output" |grep TotalDestinationSizeChange|cut -d '(' -f 2|cut -d ')' -f 1`
                sessionstats[17]=`echo "$rdiff_output" |grep Errors|cut -d ' ' -f 2`

                if [ "${sessionstats[4]}" != "0 bytes" ]; then
                        # We need to change the IFS to split includes and escludes
                        template=/usr/share/backupninja/reports/rdiff_success.html
                        old_ifs=$IFS
                        IFS='"'
                        backup_status=1

                        sed -e "s/%BACKUPLABEL%/$report_label/" $template > $eml

                        sed -i -e "s/%DURATION%/${sessionstats[2]}/" $eml
                        sed -i -e "s/%STARTED%/${sessionstats[0]}/" $eml
                        sed -i -e "s/%ENDED%/${sessionstats[1]}/" $eml
                        sed -i -e "s/%SRC%/${sessionstats[3]}/" $eml
                        sed -i -e "s/%SRCSIZE%/${sessionstats[4]}/" $eml
                        sed -i -e "s/%DST%/${sessionstats[5]}/" $eml
                        sed -i -e "s/%DSTSIZE%/${sessionstats[6]}/" $eml
                        sed -i -e "s/%NEW%/${sessionstats[7]}/" $eml
                        sed -i -e "s/%NEWSIZE%/${sessionstats[8]}/" $eml
                        sed -i -e "s/%DEL%/${sessionstats[9]}/" $eml
                        sed -i -e "s/%DELSIZE%/${sessionstats[10]}/" $eml
                        sed -i -e "s/%CHG%/${sessionstats[11]}/" $eml
                        sed -i -e "s/%CHGSIZE%/${sessionstats[12]}/" $eml

                        if [ ${#include} == 0 ]; then
                                sed -i -e "s/%INCLUDE%/Entire\ system/" $eml
                        else
                                for inc in $include; do
                                        sed -i -e "s#%INCLUDE%#$inc\<br\/\>\n-\ %INCLUDE%#" $eml
                                done
                                sed -i -e "/%INCLUDE%/d" $eml
                        fi

                        if [ ${#exclude} == 0 ]; then
                                sed -i -e "/exclude_start/,/exclude_stop/d" $eml
                        else
                                for exc in $exclude; do
                                        sed -i -e "s#%EXCLUDE%#$exc\<br\/\>\n-\ %EXCLUDE%#" $eml
                                done
                                sed -i -e "/%EXCLUDE%/d" $eml
                        fi

                        # Restore the IFS
                        IFS=$old_ifs

                # Backed up an empty directory
                else
                        backup_status=0

                        echo "                  Backup $report_label size was zero.

Check that source is correct / share are mounted.
See attached logs for more info" > $staff_eml

                fi

        # Logs don't have any statistics
        else
                backup_status=0

                echo "                  Backup $report_label failed.

See attached logs for more info" > $staff_eml

        fi

        mail_subject="$report_company Backup [$report_label]: Backup" 

        if [[ $backup_status -eq 1 ]]; then
                mail_subject="$mail_subject OK" 

        else
                mail_subject="$mail_subject Failed" 

                template=/usr/share/backupninja/reports/rdiff_failed.html
                sed -e "s/%BACKUPLABEL%/$report_label/" $template > $eml

                echo "$rdiff_output" >> $attachment
                zip $attachment.zip $attachment
                /bin/cat $staff_eml | nail -s "$mail_subject" -a $attachment.zip -r support@solutionsfirst.com.au $reportemail
                rm -f $staff_eml $attachment $attachment.zip

        fi

        /bin/cat $eml | mail -s "$mail_subject" -a "Content-type: text/html;" -a "From: support@solutionsfirst.com.au" $mail_to
        rm -f $eml
}

### GET CONFIG ###

getconf rdiffreport no
getconf rdiffreport_email $clientemail

getconf nagioscheck_disable no

getconf options
getconf testconnect yes
getconf nicelevel 0
getconf bwlimit
getconf ignore_version no

setsection source
getconf type; sourcetype=$type
getconf user; sourceuser=$user
getconf host; sourcehost=$host
check_consistency "source" "$type" "$user" "$host" 
getconf label backup
getconf keep 60
getconf include
getconf include_regexp
getconf vsnames all
getconf vsinclude
getconf exclude
getconf exclude_regexp

setsection dest
getconf directory; destdir=$directory
# strip trailing /
destdir=${destdir%/}
getconf type; desttype=$type
getconf user; destuser=$user
getconf host; desthost=$host
getconf sshoptions
check_consistency "destination" "$type" "$user" "$host" 

if [ -n "$sshoptions" ] && echo $options | grep -qv "remote-schema"; then
        options="$options --remote-schema 'ssh -C $sshoptions %s rdiff-backup --server'" 
fi

### CHECK CONFIG ###

# If vservers are configured, check that the ones listed in $vsnames do exist.
local usevserver=no
if [ $vservers_are_available = yes ]; then
   if [ "$vsnames" = all ]; then
      vsnames="$found_vservers" 
   else
      if ! vservers_exist "$vsnames" ; then
            fatal "At least one of the vservers listed in vsnames ($vsnames) does not exist." 
      fi
   fi
   if [ -n "$vsinclude" ]; then
      info "Using vservers '$vsnames'" 
      usevserver=yes
   fi
else
   [ -z "$vsinclude" ] || warning 'vservers support disabled in backupninja.conf, vsincludes configuration lines will be ignored'
fi

# check the connection at the source and destination
[ -n "$test" ] || test=0
if [ "$testconnect" = "yes" ] || [ "${test}" -eq 1 ]; then
        test_connection $sourceuser $sourcehost
        test_connection $destuser $desthost
fi

if [ "$ignore_version" != "yes" ]; then
        # see that rdiff-backup has the same version at the source and destination
        sourceversion=`get_version $sourceuser $sourcehost`
        destversion=`get_version $destuser $desthost`
        if [ "$sourceversion" != "$destversion" ]; then
                fatal "rdiff-backup does not have the same version at the source and at the destination." 
        fi
fi

# source specific checks
case $sourcetype in
        remote ) execstr_sourcepart="$sourceuser@$sourcehost::/" ;;
        local  ) execstr_sourcepart="/" ;;
        *      ) fatal "sourcetype '$sourcetype' is neither local nor remote" ;;
esac

# destination specific checks
[ "$destdir" != "" ] || fatal "Destination directory not set" 
case $desttype in
        remote ) execstr_destpart="$destuser@$desthost::$destdir/$label" ;;
        local  ) execstr_destpart="$destdir/$label" ;;
        *      ) fatal "desttype '$desttype' is neither local nor remote" ;;
esac

### REMOVE OLD BACKUPS ###

if [ "$keep" != yes ]; then

   if [ "`echo $keep | tr -d 0-9`" == "" ]; then
        # add D if no other date unit is specified
      keep="${keep}D" 
   fi

   removestr="$RDIFFBACKUP $options --force --remove-older-than $keep " 
   if [ "$desttype" == "remote" ]; then
      removestr="${removestr}${destuser}@${desthost}::" 
   fi
   removestr="${removestr}${destdir}/${label}";

   debug "$removestr" 
   if [ $test = 0 ]; then
      output="`su -c "$removestr" 2>&1`" 
      if [ $? = 0 ]; then
         debug $output
         info "Removing backups older than $keep days succeeded." 
      else
         warning $output
         warning "Failed removing backups older than $keep." 
      fi
   fi

fi

# Add cstream

if [ ! -z $bwlimit ]; then
        check_cstream $CSTREAM;
        if [ "$desttype" = "remote" ]; then
                RDIFFBACKUP="$RDIFFBACKUP --remote-schema 'cstream -t $bwlimit | ssh %s \''rdiff-backup --server\'''" 
        elif [ "$sourcetype" = "remote" ]; then
                RDIFFBACKUP="$RDIFFBACKUP --remote-schema 'ssh %s \''rdiff-backup --server\'' | cstream -t $bwlimit'" 
        else
                fatal "You specified a bandwidth limit but neither your source nor destination types are remote." 
        fi
fi

### EXECUTE ###

execstr="$RDIFFBACKUP $options -v5 --terminal-verbosity=0 --print-statistics " 

set -o noglob

symlinks_warning="Maybe you have mixed symlinks and '*' in this statement, which is not supported." 

# TODO: order the includes and excludes
#

# Regexp includes and exclude have to be put first
# # to take precedence and match files
# regexp excludes
for i in $exclude_regexp; do
   execstr="${execstr}--exclude-regexp '$i' " 
done
# regexp includes
for i in $include_regexp; do
   execstr="${execstr}--include-regexp '$i' " 
done

# Allow includes and excludes with spaces:
# Split values in exclude and include arrays with " character
# And change IFS to " (we backup current IFS to restore it after)
[ ${#exclude} != 0 ] && exclude="`echo \"$exclude\" | tr -s '\n' '\"'`" 
[ ${#include} != 0 ] && include="`echo \"$include\" | tr -s '\n' '\"'`" 
old_ifs=$IFS
IFS='"'
# excludes
for i in $exclude; do
        # Escape spaces (backslashes should not have been in first place...)
        str="${i// /\\\ }" 
   execstr="${execstr}--exclude '$str' " 
done
# includes
for i in $include; do
   [ "$i" != "/" ] || fatal "Sorry, you cannot use 'include = /'" 
        # Escape spaces (backslashes should not have been in first place...)
        str="${i// /\\\ }" 
   execstr="${execstr}--include '$str' " 
done
IFS=$old_ifs
# vsinclude
if [ $usevserver = yes ]; then
   for vserver in $vsnames; do
      for vi in $vsinclude; do
         str="${vi//__star__/*}" 
         str="$VROOTDIR/$vserver$str" 
         if [ -n "$str" ]; then
            execstr="${execstr}--include '$str' " 
         else
            warning "vsinclude statement '${vi//__star__/*}' will be ignored for VServer $vserver. $symlinks_warning" 
         fi
      done
   done
fi

set +o noglob

# exclude everything else
[ "$include" != "" -o "$vsinclude" != "" ] && execstr="${execstr}--exclude '/*' " 

# include client-part and server-part
execstr="${execstr}$execstr_sourcepart $execstr_destpart" 

debug "$execstr" 
if [ $test = 0 ]; then
        output=`nice -n $nicelevel su -c "$execstr" 2>&1`
        if [ $? = 0 ]; then
                debug $output
                info "Successfully finished backing up source $label" 
        else
                warning $output
                warning "Failed backup up source $label" 
        fi
        # Send nagios result
        if [ $nagioscheck_disable == "no" ]; then
                send_nagios_check_result
        fi
        # Generate rdiff-backup-parser report if requested
        if [ $rdiffreport == "yes" ]; then
                send_rdiff_report "$output" "`echo $company | sed 's/[-_,.\t]/\ /g' | sed 's/\<\(.\)\([^ ]*\)/\u\1\L\2/g'`" "`echo $label | sed '1,$s/ /\-/g; s\-/g'`" "$rdiffreport_email" 
        fi
fi

return 0

and rdiff.helper.in in case that has other changes.

cat rdiff.helper.in
# -*- mode: sh; sh-basic-offset: 3; indent-tabs-mode: nil; -*-

HELPERS="$HELPERS rdiff:incremental_remote_filesystem_backup" 

declare -a rdiff_includes
declare -a rdiff_excludes
declare -a rdiff_vsincludes
declare -a rdiff_vsexcludes

# FUNCTIONS

do_rdiff_host_includes() {
   set -o noglob
   # choose the files to backup
   REPLY=
   while [ -z "$REPLY" ]; do
      formBegin "$rdiff_title - host system: includes" 
         for ((i=0; i < ${#rdiff_includes[@]} ; i++)); do
            formItem include ${rdiff_includes[$i]}
         done
         formItem include
         formItem include
         formItem include
      formDisplay
      [ $? = 0 ] || return
      unset rdiff_includes
      rdiff_includes=($REPLY)
   done
   set +o noglob
}

do_rdiff_vserver() {
   # choose the vservers to backup (into $selected_vservers)
   choose_one_or_more_vservers "$rdiff_title" 
   [ $? = 0 ] || return 1

   set -o noglob
   # choose the files to backup
   REPLY=

   while [ -z "$REPLY" ]; do
      formBegin "$rdiff_title - vsincludes (backup these directories from every vserver)" 
         [ -z "$rdiff_vsincludes" ] && rdiff_vsincludes="$rdiff_default_includes" 

         for i in $rdiff_vsincludes; do
            formItem include "$i" 
         done
         formItem include "" 
         formItem include "" 
         formItem include "" 
      formDisplay
      [ $? = 0 ] || return 1
      rdiff_vsincludes=($REPLY)
   done

   set +o noglob
}

do_rdiff_excludes() {
   set -o noglob
   formBegin "$rdiff_title: excludes" 
     for ((i=0; i < ${#rdiff_excludes[@]} ; i++))
     do
       formItem exclude ${rdiff_excludes[$i]}
     done
     formItem exclude
     formItem exclude
   formDisplay
   [ $? = 0 ] || return
   unset rdiff_excludes
   rdiff_excludes=($REPLY)
   set +o noglob
}

do_rdiff_src() {
   choose_host_or_vservers_or_both "$rdiff_title" 
   [ $? = 0 ] || return 1
   case $host_or_vservers in
      'host')
         do_rdiff_host_includes
         [ $? = 0 ] || return 1
         ;;
      'vservers')
         do_rdiff_vserver
         [ $? = 0 ] || return 1
         ;;
      'both')
         do_rdiff_host_includes
         [ $? = 0 ] || return 1
         do_rdiff_vserver
         [ $? = 0 ] || return 1
         ;;
      *)
         return 1
         ;;
   esac
   do_rdiff_excludes
   [ $? = 0 ] || return 1
   _src_done="(DONE)" 
   setDefault dest
}

do_rdiff_dest() {
   declare -a tmp_array

   set -o noglob
   REPLY=
   while [ -z "$REPLY" -o -z "$rdiff_directory" -o -z "$rdiff_host" -o -z "$rdiff_user" ]
   do
     formBegin "$rdiff_title - destination: last three items are required" 
        formItem "keep" "$rdiff_keep" 
        formItem "dest_directory" "$rdiff_directory" 
        formItem "dest_host" "$rdiff_host" 
        formItem "dest_user" "$rdiff_user" 
        formItem "dest_type" "$rdiff_type" 
        formDisplay
     [ $? = 0 ] || return
     tmp_array=($REPLY)
     rdiff_keep=${tmp_array[0]}
     rdiff_directory=${tmp_array[1]}
     rdiff_host=${tmp_array[2]}
     rdiff_user=${tmp_array[3]}
     rdiff_type=${tmp_array[4]}
  done
  set +o noglob

  _dest_done="(DONE)" 
  setDefault conn
}

do_rdiff_ssh_con() {
   local remote_status="ok" 

   IFS=$' \t\n'
   if [ "$_dest_done" = "" ]; then
      msgBox "$rdiff_title: error" "You must first configure the destination." 
      return
   elif [ "$rdiff_type" = "" ]; then
      msgBox "$rdiff_title: error" "You must first configure the destination backup type." 
      return
   elif [ "$rdiff_user" = "" ]; then
      msgBox "$rdiff_title: error" "You must first configure the destination user." 
      return
   elif [ "$rdiff_host" = "" ]; then
      msgBox "$rdiff_title: error" "You must first configure the destination host." 
      return
   else
      booleanBox "$rdiff_title" "This step will create a ssh key for the local root user with no passphrase (if one does not already exist), and attempt to copy root's public ssh key to authorized_keys file of $rdiff_user@$rdiff_host. This will allow the local root to make unattended backups to $rdiff_user@$rdiff_host.\n\n\nAre you sure you want to continue?" 
      [ $? = 0 ] || return
   fi

   if [ ! -f /root/.ssh/id_dsa.pub -a ! -f /root/.ssh/id_rsa.pub ]; then
      echo "Creating local root's ssh key" 
      ssh-keygen -t dsa -f /root/.ssh/id_dsa -N "" 
      echo "Done. hit return to continue" 
      read
   fi

   ssh -o PreferredAuthentications=publickey $rdiff_host -l $rdiff_user "exit" 2> /dev/null
   if [ $? -ne 0 ]; then
      echo "Copying root's public ssh key to authorized_keys of $rdiff_user@$rdiff_host. When prompted, specify the password for user $rdiff_user@$rdiff_host." 
      ssh-copy-id -i /root/.ssh/id_[rd]sa.pub $rdiff_user@$rdiff_host
      if [ $? -ne 0 ]; then
         echo "FAILED: Couldn't copy root's public ssh key to authorized_keys of $rdiff_user@$rdiff_host." 
         ssh $rdiff_user@$rdiff_host 'test -w .ssh || test -w .'
         result=$?
         echo "Hit return to continue." 
         read
         case $result in
            0 )   msgBox "$rdiff_title: error" "Directories are writable: Probably just a typo the first time." ;;
            1 )   msgBox "$rdiff_title: error" "Connected successfully to $rdiff_user@$rdiff_host, but unable to write. Check ownership and modes of ~$rdiff_user on $rdiff_host." ;;
            255 ) msgBox "$rdiff_title: error" "Failed to connect to $rdiff_user@$rdiff_host. Check hostname, username, and password. Also, make sure sshd is running on the destination host." ;;
            * )   msgBox "$rdiff_title: error" "Unexpected error." ;;
         esac
         return
      else
         echo "Done. hit return to continue" 
         read
      fi
   else
      echo "root@localhost is already in authorized_keys of $rdiff_user@$rdiff_host." 
      echo "Hit return to continue." 
      read
   fi

   # test to see if the remote rdiff backup directory exists and is writable
   echo "Testing to see if remote rdiff backup directory exists and is writable" 
   ssh $rdiff_user@$rdiff_host "test -d ${rdiff_directory}" 
   if [ $? = 0 ]; then
      ssh $rdiff_user@$rdiff_host "test -w $rdiff_directory" 
      if [ $? != 0 ]; then
         msgBox "destination directory is not writable!" "The remote destination directory is not writable by the user you specified. Please fix the permissions on the directory and then try again." 
         remote_status=failed
      fi
   else
      booleanBox "Remote directory does not exist" "The destination backup directory does not exist, do you want me to create it for you?" 
      if [ $? = 0 ]; then
         ssh $rdiff_user@$rdiff_host "mkdir -p ${rdiff_directory}" 
         result=$?
         case $result in
            0) msgBox "$rdiff_title: success" "Creation of the remote destination directory was a success!";;
            1) msgBox "$rdiff_title: error" "Connected successfully to $rdiff_user@$rdiff_host, but was unable to create the destination directory, check the directory permissions." 
               remote_status=failed;;
            255) msgBox "$rdiff_title: error" "Failed to connect to $rdiff_user@$rdiff_host. Check hostname, username, and password. Also, make sure sshd is running on the destination host." 
               remote_status=failed;;
            *) msgBox "$rdiff_title: error" "Unexpected error." 
               remote_status=failed;;
         esac
      fi
   fi

   if [ "$remote_status" = "ok" ]; then
      do_rdiff_con
   fi
}

do_rdiff_con() {
   echo "Checking for local install of rdiff-backup" 
   require_packages rdiff-backup

   echo "Testing to make sure destination has rdiff-backup installed and is compatible." 
   remote_result=`/usr/bin/rdiff-backup --test-server $rdiff_user@$rdiff_host::/ 2>&1 >&-`
   if [ $? -ne 0 ]; then
      echo $remote_result | grep -q "command not found" 
      if [ $? -eq 0 ]; then
         if [ "$rdiff_user" = "root" ]; then
            booleanBox "install rdiff-backup?" "It seems like the remote machine does not have rdiff-backup installed, I can attempt to install rdiff-backup on the remote machine.\n\n\nDo you want me to attempt this now?" 
            if [ $? = 0 ]; then
               ssh $rdiff_user@$rdiff_host 'apt-get install rdiff-backup'
               result=$?
               echo "Hit return to continue." 
               read
               case $result in
                  0) msgBox "$rdiff_title: success" "Installation of rdiff-backup was a success!" 
                     do_rdiff_con;;
                  1) msgBox "$rdiff_title: error" "Connected successfully to $rdiff_user@$rdiff_host, but was unable to install the package for some reason.";;
                  255) msgBox "$rdiff_title: error" "Failed to connect to $rdiff_user@$rdiff_host. Check hostname, username, and password. Also, make sure sshd is running on the destination host.";;
                  *) msgBox "$rdiff_title: error" "Unexpected error.";;
               esac
               return
            fi
         else
            booleanBox "install rdiff-backup" "Please install rdiff-backup on the remote machine, this cannot be done automatically, as the remote user in your configuration is not root. \n\nIf you have installed rdiff-backup on the remote machine and you are getting this error, then there is a version incompatibility between that version and the local version.\n\nPlease resolve this problem and then try connecting again.\n\n\n\nTry connecting again?" 
            if [ $? = 0 ]; then
               do_rdiff_con
            else
               return
            fi
         fi
      else
         msgBox "incompatible versions of rdiff-backup" "It looks like rdiff-backup is installed on the remote machine, but it may be an incompatible version with the one installed locally, or something else is amiss.\n\nPlease resolve this problem and then try connecting again.\n\n\nTry connecting again?" 
         if [ $? = 0 ]; then
            do_rdiff_con
         else
            return
         fi
      fi
   else
        echo "SUCCESS: Everything looks good!" 
        echo "Hit return to continue." 
        read
   fi

   _con_done="(DONE)" 
   setDefault finish
}

do_rdiff_finish() {
   get_next_filename $configdirectory/90.rdiff
   cat > $next_filename <<EOF
# options = --force
# when = everyday at 02

[source]
type = local
keep = $rdiff_keep

# A few notes about includes and excludes:
# 1. include, exclude and vsinclude statements support globbing with '*'
# 2. Symlinks are not dereferenced. Moreover, an include line whose path
#    contains, at any level, a symlink to a directory, will only have the
#    symlink backed-up, not the target directory's content. Yes, you have to
#    dereference yourself the symlinks, or to use 'mount --bind' instead.
#    Example: let's say /home is a symlink to /mnt/crypt/home ; the following
#    line will only backup a "/home" symlink ; neither /home/user nor
#    /home/user/Mail will be backed-up :
#      include = /home/user/Mail
#    A workaround is to 'mount --bind /mnt/crypt/home /home' ; another one is to
#    write :
#      include = /mnt/crypt/home/user/Mail
# 3. All the excludes come after all the includes. The order is not otherwise
#    taken into account.

# files to include in the backup
EOF
   ## includes ##
   if [ "$host_or_vservers" == host -o "$host_or_vservers" == both ]; then
      set -o noglob
      for ((i=0; i < ${#rdiff_includes[@]} ; i++)); do
         echo "include = ${rdiff_includes[$i]}" >> $next_filename
      done
      set +o noglob
   fi

   if [ "$host_or_vservers" == vservers -o "$host_or_vservers" == both ]; then
      cat >> $next_filename <<EOF
#
# If vservers = yes in /etc/backupninja.conf then the following variables can
# be used:
# vsnames = all | <vserver1> <vserver2> ... (default = all)
# vsinclude = <path>
# vsinclude = <path>
# ...
# Any path specified in vsinclude is added to the include list for each vserver
# listed in vsnames (or all if vsnames = all, which is the default).
#
# For example, vsinclude = /home will backup the /home directory in every
# vserver listed in vsnames. If you have 'vsnames = foo bar baz', this
# vsinclude will add to the include list /vservers/foo/home, /vservers/bar/home
# and /vservers/baz/home.
# Vservers paths are derived from $VROOTDIR.

EOF
      set -o noglob
      echo -e "vsnames = $selected_vservers\n" >> $next_filename
      for i in $rdiff_vsincludes; do
         echo "vsinclude = $i" >> $next_filename
      done
      set +o noglob
   fi

   ## excludes ##
   set -o noglob
   for ((i=0; i < ${#rdiff_excludes[@]} ; i++)); do
     echo exclude = ${rdiff_excludes[$i]} >> $next_filename
   done
   set +o noglob
   cat >> $next_filename <<EOF

######################################################
## destination section
## (where the files are copied to)

[dest]
type = remote
directory = $rdiff_directory
host = $rdiff_host
user = $rdiff_user
EOF

   chmod 600 $next_filename
}

rdiff_main_menu() {
   while true; do
      srcitem="choose files to include & exclude $_src_done" 
      destitem="configure backup destination $_dest_done" 
      conitem="set up ssh keys and test remote connection $_con_done" 
      advitem="edit advanced settings $_adv_done" 
      menuBox "$rdiff_title" "choose a step:" \
         src "$srcitem" \
         dest "$destitem" \
         conn "$conitem" \
         finish "finish and create config file" 
      [ $? = 0 ] || return
      result="$REPLY" 
      case "$result" in
         "src") do_rdiff_src;;
         "dest") do_rdiff_dest;;
         "conn") do_rdiff_ssh_con;;
         "adv") do_rdiff_adv;;
         "finish")
            if [[ "$_con_done$_dest_done$_src_done" != "(DONE)(DONE)(DONE)" ]]; then
               msgBox "$rdiff_title" "You cannot create the configuration file until the other steps are completed." 
            else
               do_rdiff_finish
               return
            fi
            ;;
      esac
   done
}

rdiff_wizard() {

   # Global variables
   rdiff_title="rdiff-backup action wizard" 
   _src_done=
   _dest_done=
   _con_done=
   _adv_done=
   rdiff_keep=60D
   rdiff_directory=/backup/`hostname`
   rdiff_type=remote
   rdiff_user=
   rdiff_host=

   # Global variables whose '*' shall not be expanded
   set -o noglob
   rdiff_includes=(/var/spool/cron/crontabs /var/backups /etc /root /home /usr/local/*bin /var/lib/dpkg/status*)
   rdiff_excludes=(/home/*/.gnupg /home/*/.local/share/Trash /home/*/.Trash /home/*/.thumbnails /home/*/.beagle /home/*/.aMule /home/*/gtk-gnutella-downloads)
   rdiff_vsincludes=
   set +o noglob

   rdiff_main_menu
}

rdiff_failed.html.in View (2.44 KB) davekempe, 11/14/2011 09:33 PM

rdiff_success.html.in View (5.82 KB) davekempe, 11/14/2011 09:33 PM

History

#1 Updated by davekempe about 6 years ago

oh forgot to mention, the sending of the HTML reports relies on the mail -a feature being the mail header.
like so:

$bsd-mailx a
bsd-mailx: option requires an argument -
'a'
usage: bsd-mailx [-dEIinv] [-a header] [-b bcc-addr] [-c cc-addr] [-s subject] to-addr ...
[-- sendmail-options ...]
bsd-mailx [-dEIiNnv] -f [name]
bsd-mailx [-dEIiNnv] [-u user]

so you need to have bsd-mailx symlinked to mail.
not sure if thats available in lucid, we have a package for it to make this work if people are interested.
Or, there might be an easier way to acheive the same result.
unfortunately in lucid, ubuntu switched to 'nail' as the default for 'mail', which wrecks this particular option.

#2 Updated by davekempe about 6 years ago

And of course, you will need to remove the hardcoded email addresses from the code as well :(

#3 Updated by intrigeri almost 6 years ago

  • Category set to rdiff-backup handler
  • Status changed from New to In Progress
  • Assignee set to davekempe
  • % Done changed from 0 to 10

Hi!

Ok, well, this one is bigger, and also quite further from being mergeable. Let me draft a bunch of initial comments:

  1. I would need to be convinced by the need for a new config file.
  2. Adding such a feature to one handler only would not make me happy at all.
  3. The big regexps after "Generate rdiff-backup-parser report if requested" must be documented, and probably extracted into well named functions.
  4. This is a big patch. Please provide it as a patch against current Git master branch. A Git branch will probably be needed anyway if you want to do what's needed to get this into a mergeable state.
  5. The added dependency on sshfs (and thus FUSE) seems a bit overkill. Probably could be avoided.
  6. Error checking after running external commands is not optional. (e.g. sshfs, fusermount).
  7. same issue as in the other handler wrt. checking if still mounted.
  8. The proposed changes introduce security holes. We've got a maketemp function in lib/tools.in. It should be used to create temporary files in a secure way.
  9. The rdiff-backup output parser has, well, much room for improvements. We'll discuss that one once we know someone will invest the time and efforts needed to get this patch ready.

Happy hacking!

#4 Updated by intrigeri over 5 years ago

  • Priority changed from Normal to Low

Dave: ping?

#5 Updated by wes over 5 years ago

intrigeri wrote:

Dave: ping?

Hi intrigeri,

Dave's asked me to take over the necessary work on bringing this feature up to scratch (and request #3627 as well).
I've scheduled some time to go over your notes and begin working on the required changes.

I think first up though I need to get familiar with the backupninja libraries/api to ensure I don't duplicate functionality needlessly.

#6 Updated by intrigeri over 5 years ago

Hi wes!

wes wrote:

Dave's asked me to take over the necessary work on bringing this feature up to scratch (and request #3627 as well).
I've scheduled some time to go over your notes and begin working on the required changes.

Nice to hear. Welcome aboard!

I think first up though I need to get familiar with the backupninja libraries/api to ensure I don't duplicate functionality needlessly.

Makes sense. Should be quick, as our lib/ is pretty small.

#7 Updated by davekempe over 4 years ago

Hey
I don't think this is getting updated any time soon. Any takers?

#8 Updated by intrigeri over 4 years ago

  • Status changed from In Progress to Confirmed
  • Assignee deleted (davekempe)

Also available in: Atom PDF