Null Disquisition

Addendum to two previoius posts.

The other day, I noticed my 8GB USB volume that I use for temporary incremental backups was quite full. Curious, since the folders I back up to that volume do not total but 200MB or so, and rsync was supposed to be doing incremental backups (link-dest ftw).

After a little searching around, I found someone who had a similar problem (and a solution). When you format a volume with OS X it will, by default, ignore file ownership (the linked article explores why this is perhaps). This proves to be a problem for rsync which considers file permissions and ownership as part of the file stat (as it should). Luckily the fix is easy – “Get Info” for the volume in question, then at the bottom unselect “Ignore ownership on this volume”

You will probably want to delete any backups that have been created (since they won’t have the correct file ownership). Source: Terminalapp.net

A close friend of mine was recently subject of home invasion. Aside from the regular pickings (TV, computers) he had his external harddrives stolen. I could only imagine how that felt – just that extra twist of the dagger in your chest. After hearing about this, I reevaluated my personal backup proceedures – specifically, how could one avoid loosing everything if something like this happens.

I’ve been using Time Machine on my MacBook Pro for the past several months. TM is certainly a great program, but we all know it has certain shortcomings. One of those being the lack of a whitelist (you can only blacklist directories from the backup schedule). Say for instance, I only want to backup a single directory (like all of my code). This is effectively impossible with Time Machine.

So until Apple gets their shit together, we Unix folk have rsync. Michael over at IMHO has a great writeup on how to acheive incremental backups in a Time Machine fashion using rsync and cron. I threw together the following script to add in a little OS X sexiness and so that it uses a USB thumb drive.

#!/bin/bash
if [ -d /Volumes/PNY8GB ]
then
    HOME=/Volumes/PNY8GB/Backups.backupdb/TM
    date=`date "+%Y-%m-%d-%H%M%S"`
    n=`rsync --stats -aR --exclude "*.swp" --exclude "*.bak" --exclude "*.pyc"
--exclude "*~" --exclude ".svn" --delete-excluded --link-dest=$HOME/Latest
/Users/davidarthur/Code/Loud3r $HOME/$date |
sed -n 's/Number of files transferred: \([^0]\)/\1/p'`
    rm $HOME/Latest
    ln -s $date $HOME/Latest
else
    exit
fi
if [ $n ]
then
    /usr/local/bin/growlnotify -m 'rsync complete,
number of files: '$n &> /dev/null
fi

This first looks for the target thumb drive (PNY8GB in my case), then runs the rsync command, and finally pops up a nice little growl notification showing how many files were backed up. For non-OS X people, just ignore the growlnotify part.

#!/bin/bash
if [ -d /Volumes/PNY8GB ]
then
    HOME=/Volumes/PNY8GB/Backups.backupdb/TM
    date=`date "+%Y-%m-%d-%H%M%S"`
    rsync --stats -aR --exclude "*.swp" --exclude "*.bak" --exclude "*.pyc"
--exclude "*~" --exclude ".svn" --delete-excluded --link-dest=$HOME/Latest
/Users/davidarthur/Code/Loud3r $HOME/$date
    rm $HOME/Latest
    ln -s $date $HOME/Latest
else
    exit
fi

I’m still working on getting the directory structure to match Time Machine’s so I can actually use Time Machine to explore/restore files from the backups.

This gives me a little piece of mind, and hopefully will you too.

-David

Edit: OS X has some issues with preserving permissions and ownerships with HFS+ volumes – so the incremental part doesn’t really work. Should work on other *nix systems though.

2nd Edit: Got it working finally. Here is my complete script (running every 15mins). N.B. must be run by root (so put it in root’s crontab).

#!/bin/bash -x

DEST="/Volumes/PNY8GB/Backups"
LATEST="Latest"
EXCLUDES_FILE="/Users/davidarthur/.rsyncexcludes"
RSYNC="/usr/bin/rsync "

# Make sure user is root
if (( `id -u` != 0 )); then
    { echo "Sorry, must be root. Exiting..."; exit; }
fi;

# Make sure backup device is attached
! test -d "$DEST" && echo "Please mount the backup drive!" && exit

# Run rsync
DATE=`date "+%Y-%m-%d-%H%M%S"`
n=`$RSYNC -a -x -S --stats --delete --link-dest=$DEST/$LATEST \
    --exclude-from $EXCLUDES_FILE $* /Users/davidarthur/Code/Loud3r $DEST/$DATE
| sed -n 's/Number of files transferred: \([^0]\)/\1/p'`
# Update 'Latest' link
rm $DEST/$LATEST
ln -s $DEST/$DATE $DEST/$LATEST

# Send a growl notification
if [ $n ]
then
    /usr/local/bin/growlnotify -m 'rsync complete,
number of files: '$n &> /dev/null
fi