Donate with
Categories
Donate with
82uymVXLkvVbB4c4JpTd1tYm1yj1cKPKR2wqmw3XF8YXKTmY7JrTriP4pVwp2EJYBnCFdXhLq4zfFA6ic7VAWCFX5wfQbCC
Backup multiple directories at once automatically - with e-mail logs - using the Linux rsync command
Donate with
82uymVXLkvVbB4c4JpTd1tYm1yj1cKPKR2wqmw3XF8YXKTmY7JrTriP4pVwp2EJYBnCFdXhLq4zfFA6ic7VAWCFX5wfQbCC
Why this script and what is a rsync backup?
Why?
We wanted to be able to create a simple backup or a directory sync using the Linux rsync command. But also we needed a logfile management where we receive an email when the sync did not work properly for what ever reason. Thus we've written a Linux Shell Script that takes care of synchronizing many different folders at once and takes care of sending mails in case of error or even on success.
What is Rsync?
Rsync is a Linux tool that manages to copy files from one location to another. It does that highly efficient by only copying the file differences that have changed and thus it's a great tool to even copy files over low bandwidth connections.
Extensibility
With some simple modifications you can also use the script to make synchronisations via SSH (Rsync over SSH). You will need an Rsync daemon on the target system. The Rsync parameter would need to be adapted so be able to make SSH connections and it would look like this:
rsync -avzle 'ssh -p5555' --rsync-path='/opt/bin/rsync' --stats --exclude-from '/backupconfig/exclusions.lst' --delete ${SAVE_FROM[$SYNCITEMSCOUNT]} -r root@my.ssh.server.com:/backup_target
Parameters explained:
- ssh -p5555 → The port the target SSH server uses to accept connections
- rsync-path → the path where the rsync (daemon) is found on the target system
- exclude-from → the file that contains excluded directories.
- delete → delete files on the target system that don't exist on the source system
In order for this to work Rsync needs to be able to automatically logon to your SSH target server. I might consider to create a HowTo about that. Please check our search function if you're interested in that 😉
How does the script work?
Simple. Like this:
- Modify the variable SOURCE_DIRS and enter the directories you'd like to copy somewhere else
- Modify the variable DESTINATION_DIRS and enter either
- One (1) destination dir → All the SOURCE_DIRS directories will be rsync'ed into the one destination DIR
- Multiple destination dirs → Each SOURCE_DIRS directory will be rsync'ed into an individual destination DIR
- Set the MAIL_RECIPIENTS variable to the desired recipient address
- Create a backupuser as described in the notes
- Install necessary tools as described in the notes
- Make the script executable like this: chmod 755 backup.sh
Download the script
You can download the script here or copy/paste the content below. You might want to rename the script since the .txt extension had to be appended due to security reasons.
Have fun!
#! /bin/bash
#Enable Debugging
#set -v on
#set -x on
## Notes ###############################################################################################################################################
## The MySQL bin logs are excluded by default (if any). Reason: They can be very large and it's not recommended to backup MySQL with this script anyway
## For each directory that you would like to have synchronized you need extend the SOURCE_DIRS array and increment the array counter by 1
## The script takes advantage of the following tools that you might need to install beforehand (installation example in Debian):
## rsync (aptitude install rsync)
## sudo (aptitude install sudo)
## mail (aptitude install mailx)
## You are going to need to create a backupuser that can be used for sending mails like this:
## useradd -G users -s /bin/false backupuser
## passwd -d backupuser
## Please do not forget to adapt the script paths below. I did not want to develop exceptions for automatically handling this
##
## There is no copyright whatsoever
## Use this script at your own risk
##
## Created by Marcus Fleuti, LEXO, Switzerland, http://www.lexo.ch
########################################################################################################################################################
#set the directories you want to have backed up using the SOURCE_DIRS array below (space separated list - use "" to add a directory path containing spaces which is what I always do)
SOURCE_DIRS=("/etc" "/var/log")
#Use this global destition dir variable to set the destination directory. The directory(ies) will be created automatically if not existent.
#If you set only 1 destination path it will be globally used for all sources. If you want to use individual destination paths watch out that you use exactly
#as many destination directories as you are using source directories. It's also a comma seperated list. Example:
# DESTINATION_DIRS=("/tmp/backup" "/tmp/backup2") #use 2 different destinations paths
DESTINATION_DIRS=("/tmp/configbackup")
MAIL_RECIPIENTS="yourmail@yourdomain.yourTLD" #send mail to multiple receipients by overgiving a space-seperated address list
MAIL_SENDER="backupuser" #this defines a system-user without a shell or password. It's used as the e-mail sender name. Please see the notes above for more information
MAIL_SUBJECT="Weekly Config Backup" #as you wish to appear as the mail subject
PATH_TO_RSYNC=/usr/bin/rsync
PATH_TO_MAIL=/usr/bin/mail
BACKUPLOG=/var/log/rsynclog.log
MAILLOG=/var/log/maillog.log
#Count the number of elements in the SOURCE_DIRS-Array in order to loop through all backup sources and create necessary variables for the backup process
ELEMENTCOUNT=${#SOURCE_DIRS[@]}
#Create the necessary variables based on the number of sources used
COUNTER=0;
while [ $COUNTER -lt $ELEMENTCOUNT ]
do
#The name of the sync process.
SAVE_FROM[$COUNTER]=${SOURCE_DIRS[$COUNTER]}
##define global destination folder - with minor effort you could have separate destination DIRs by transforming the $DESTINATION_DIRS into an array and handle it like the $SOURCE_DIRS array above. Example:
if [ ${#DESTINATION_DIRS[@]} -gt 1 ]; then
SAVE_TO[$COUNTER]=${DESTINATION_DIRS[$COUNTER]}
else
#only one global save_to target directory
SAVE_TO[$COUNTER]=${DESTINATION_DIRS[0]}
fi
ERRORSYNC[$COUNTER]=0
let COUNTER=$COUNTER+1
done
#Building up Errorlog
echo "Bitte beachten Sie die folgenden Backup-Informationen:" >$MAILLOG
echo "" >> $MAILLOG
echo "Start des Backups am `date +%d.%B.%Y,%T`." >> $MAILLOG
echo "" >> $MAILLOG
SYNCITEMSCOUNT=0
GLOBAL_ERRORS=0
while [ $SYNCITEMSCOUNT -lt $ELEMENTCOUNT ]
do
$PATH_TO_RSYNC -a -E --delete --numeric-ids --delete-excluded -h -v --no-relative --exclude=mysql-bin.* --exclude=/mysqldata ${SAVE_FROM[$SYNCITEMSCOUNT]} ${SAVE_TO[$SYNCITEMSCOUNT]} > $BACKUPLOG
RETVAL=$?
#Take care of handling the error codes. Those codes are extracted from the Rsync manual and thus in English. You might want to translate them - I was just too lazy :-)
#For RETVAL=0 (everything was OK) you can write down your own success message.
case $RETVAL in
0)
ERRORMESSAGE="Datensynchronisation von ${SAVE_FROM[$SYNCITEMSCOUNT]} erfolgreich abgeschlossen am `date +%d.%B.%Y,%T`."
;;
1)
ERRORMESSAGE="Syntax or usage error"
;;
2)
ERRORMESSAGE="Protocol incompatibility"
;;
3)
ERRORMESSAGE="Errors selecting input/output files, dirs"
;;
4)
ERRORMESSAGE="Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server."
;;
5)
ERRORMESSAGE="Error starting client-server protocol"
;;
6)
ERRORMESSAGE="Daemon unable to append to log-file"
;;
10)
ERRORMESSAGE="Error in socket I/O"
;;
11)
ERRORMESSAGE="Error in file I/O"
;;
12)
ERRORMESSAGE="Error in rsync protocol data stream"
;;
13)
ERRORMESSAGE="Errors with program diagnostics"
;;
14)
ERRORMESSAGE="Error in IPC code"
;;
20)
ERRORMESSAGE="Received SIGUSR1 or SIGINT"
;;
21)
ERRORMESSAGE="Some error returned by waitpid()"
;;
22)
ERRORMESSAGE="Error allocating core memory buffers"
;;
23)
ERRORMESSAGE="Partial transfer due to error"
;;
24)
ERRORMESSAGE="Partial transfer due to vanished source files"
;;
25)
ERRORMESSAGE="The --max-delete limit stopped deletions"
;;
30)
ERRORMESSAGE="Timeout in data send/receive"
;;
137)
ERRORMESSAGE="Fehlercode 137 :: No clear errorcode. Possible reason: The Rsync process might have crashed."
;;
*)
ERRORMESSAGE="An unidentifed error occured ($RETVAL) - maybe a new errorcode due to a new version of rsync"
;;
esac
#if any kind of error occured attach this message to the mail log
if [ ! $RETVAL -eq 0 ]; then
echo "Achtung! Fehler beim Sichern der Daten. Fehlermeldung:" >> $MAILLOG
fi
#the errormessage-variable is always attached to the log because it contains the errormessages as well as the success-message
echo $ERRORMESSAGE >> $MAILLOG
#attach the Rsync log to the maillog in case an error happened. This way
one can see clearly what happened in the log mail while the success message remains small and simple
if [ ! $RETVAL -eq 0 ]; then
echo "" >> $MAILLOG
echo "Fehlerprotokoll:" >> $MAILLOG
echo "-------------------------------------------------------------------------------------------" >> $MAILLOG
cat $BACKUPLOG >> $MAILLOG
echo "-------------------------------------------------------------------------------------------" >> $MAILLOG
echo "" >> $MAILLOG
echo "" >> $MAILLOG
echo "" >> $MAILLOG
ERRORSYNC[$SYNCITEMSCOUNT]=1
GLOBAL_ERRORS=1 #remember that we had at least 1 error
fi
#increment the counter variable
let SYNCITEMSCOUNT=$SYNCITEMSCOUNT+1
done
#if anyhwere (in any of the given source directories) during the sync an error occured, send an email. You can send success mails as well if you like (in that case - of course - the conditional doesn't really make sense)
if [ $GLOBAL_ERRORS -gt 0 ]; then
MAIL_SUBJECT_SEND="$MAIL_SUBJECT [!!!--- FEHLER ---!!!]"
#Send Log-Mail...
sudo -u $MAIL_SENDER $PATH_TO_MAIL -s "$MAIL_SUBJECT_SEND" $MAIL_RECIPIENTS < $MAILLOG
exit 1 #exit with errorlevel 1
else
MAIL_SUBJECT_SEND="$MAIL_SUBJECT [ERFOLG]"
#uncomment the next line if you want to have mails sent on success as well
#sudo -u $MAIL_SENDER $PATH_TO_MAIL -s "$MAIL_SUBJECT_SEND" $MAIL_RECIPIENTS < $MAILLOG
fi
exit 0 #exit with errorlevel 0 (everything OK)