Null Disquisition

New toys bring new adventures. My awesome wife got me a PS3 for my birthday recently and I’ve been tinkering around with getting some non-Bluray HD content to play on it. My initial attempts to stream stuff over my network proved unsatisfying. Since the PS3 is only capable of 802.11g, I gave up on the high bitrate stuff (it’s perfectly capable of DVD quality content (1.5~2.5 Mb/s).

Software used (on Mac OS X 10.5):

• tsMuxer
• Disk Utility
• newfs_udf
• hdiutil

Hardware used:

• Macbook Pro
• Blank CD/DVD Media
• PS3

Files used: * MKV file with AC3 audio stream and H264 video stream

Attempt 1 (successful): Load the MKV with tsMuxerGUI, select M2TS muxing. If the video profile is above 4.1, lower it to 4.1 (as the PS3 cannot support higher than an H264 level 4.1). Generate the m2ts and meta file, burn them both to a CD/DVD. This will be readable by the PS3 as a data disk – it will not autoplay, but you can access it and play it. To me, this is not an ideal solution as it does not support menus, chapters, or seeking.

Attempt 2 (unsuccessful): Same deal as before, but select “AVCHD disk”. This option will create a BD friendly file structure (folders named BDMV and CERTIFICATE). The trick here is to burn the disk as UDF 2.5 (this is not super easy on OS X or Linux). I wasted many CDs trying to regular ISO9660. Following the instructions here, you must create the image and format it as UDF 2.5.

dd if=/dev/zero of=myfile.img bs=1k count=716800
newfs_udf -r 2.5 myfile.img -v volume_label
hdiutil mount -nobrowse myfile.img
cp -R /path/to/avcdh-files/ /Volume/volume_label/
hdiutil unmount /Volume/volume_label

In plain-speak, create an empty (large) image, format it to UDF 2.5, mount it, copy the BD-compatible files into the volume, and unmount. You then use Disk Utility to burn the resulting image. No success here, but at this point I was so close, I could taste it.

Attempt 3 (successful): Identical procedure as Attempt 2, with one important exception – the image you create with dd must be sized in even increments of 1GB – that is, count==N*1024*1024.

A few things to try next:

• Menus
• DTS audio stream
• Subtitles

Anyone wanting to test this out with a super high quality 1080p rip, I recommend Big Buck Bunny. You’ll need Handbrake to convert it to a compatible container format if you download the AVI (tsMuxer doesn’t like AVIs).

Addendum to two previoius posts.

The other day, I noticed my 8GB USB volume that I use for temporary incremental backups was quite full. Curious, since the folders I back up to that volume do not total but 200MB or so, and rsync was supposed to be doing incremental backups (link-dest ftw).

After a little searching around, I found someone who had a similar problem (and a solution). When you format a volume with OS X it will, by default, ignore file ownership (the linked article explores why this is perhaps). This proves to be a problem for rsync which considers file permissions and ownership as part of the file stat (as it should). Luckily the fix is easy – “Get Info” for the volume in question, then at the bottom unselect “Ignore ownership on this volume”

You will probably want to delete any backups that have been created (since they won’t have the correct file ownership). Source: Terminalapp.net

On my personal computer, I have three sets of x509 certificates/private keys. This makes using the EC2-API-tools quite the hassle. Echoes of EC2_CERT and EC2_PRIVATE_KEY haunt my dreams.

So, like you do with these sort of things, I wrote a bash script to work some magic.

    #!/bin/bash
    echo "Choose Account:"
    read account
    base=grep $account ~/.ec2/README -i | awk '{print $1}'
    if [ ! -n "$base" ]; then
        echo "Sorry, that account does not exist"
        return
    fi
    declare -x EC2_CERT="~/.ec2/cert-$base.pem"
    declare -x EC2_PRIVATE_KEY="~/.ec2/pk-$base.pem"
    echo "EC2 environment updated"

Requires that you your private keys/certs in ~/.ec2, and they are named cert-{something}.pem and pk-{something}.pem. Also, you need a README file in ~/.ec2 that looks like

    something account1
    something-else account2

I setup an alias so I just run “ec2-account personal” to switch to my personal credentials, and “ec2-account work” to switch to my work account.

-David

After a little tinkering here, a little tinkering there, I’ve finally settled on a good solution for my portable backup drive (8GB usb thumb drive). As outlined in my previous post, I wanted a portable backup solution that could do incremental backups (like Apple’s TimeMachine does). I looked, of course, to the wonderful unix utility rsync. Here’s my latest version.

#!/bin/bash -x

DEST="/Volumes/PNY8GB/Backups"
LATEST="Latest"
EXCLUDES_FILE="$HOME/.rsyncexcludes"
FILES_FROM="$HOME/.rsyncfiles"
RSYNC="/usr/bin/rsync --max-size 10m"

# Make sure user is root
if (( `id -u` != 0 )); then
    { echo "Sorry, must be root. Exiting..."; exit; }
fi;

# Make sure backup device is attached
! test -d "$DEST" && echo "Please mount the backup drive!" && exit

# Run rsync
DATE=`date "+%Y-%m-%d-%H%M%S"`
n=`$RSYNC -r -a -x -S -R --stats --delete --link-dest=$DEST/$LATEST \
 --exclude-from $EXCLUDES_FILE --files-from $FILES_FROM $* $HOME \
 $DEST/$DATE | sed -n 's/Number of files transferred: \([^0]\)/\1/p'`

# Update 'Latest' link
rm $DEST/$LATEST
ln -s $DEST/$DATE $DEST/$LATEST

# Send a growl notification
if [ $n ]
then
    /usr/local/bin/growlnotify -m 'rsync complete,
number of files: '$n
fi

By using ––exlude-from and ––files-from, you get more fine grained control of what gets backed up. My Code folder is ~1GB, and my School folder is about 3GB. When I exclude all of my compiled code, data files, images, .git and .svn folders, and other various annoying swap files my base backup footprint is less than 500MB (for both Code and School).

Here’s my excludes file – it’s just one line per exclude filter

*.sql
*.bak
*.swp
.svn
*.pyc
*.log
*.tar.gz
*.dvi
*.o
*.out
*.d
*.tmp
.git

Similarly, the files-from file is one file path per line (remember the trailing slash!). An important note (found in the rsync manual) is that when you specify ––files-from, -r is no longer implied with -a. So make sure to add -r to your argument list.

And yet again, I leave the scheduling to you.

-David

A close friend of mine was recently subject of home invasion. Aside from the regular pickings (TV, computers) he had his external harddrives stolen. I could only imagine how that felt – just that extra twist of the dagger in your chest. After hearing about this, I reevaluated my personal backup proceedures – specifically, how could one avoid loosing everything if something like this happens.

I’ve been using Time Machine on my MacBook Pro for the past several months. TM is certainly a great program, but we all know it has certain shortcomings. One of those being the lack of a whitelist (you can only blacklist directories from the backup schedule). Say for instance, I only want to backup a single directory (like all of my code). This is effectively impossible with Time Machine.

So until Apple gets their shit together, we Unix folk have rsync. Michael over at IMHO has a great writeup on how to acheive incremental backups in a Time Machine fashion using rsync and cron. I threw together the following script to add in a little OS X sexiness and so that it uses a USB thumb drive.

#!/bin/bash
if [ -d /Volumes/PNY8GB ]
then
    HOME=/Volumes/PNY8GB/Backups.backupdb/TM
    date=`date "+%Y-%m-%d-%H%M%S"`
    n=`rsync --stats -aR --exclude "*.swp" --exclude "*.bak" --exclude "*.pyc"
--exclude "*~" --exclude ".svn" --delete-excluded --link-dest=$HOME/Latest
/Users/davidarthur/Code/Loud3r $HOME/$date |
sed -n 's/Number of files transferred: \([^0]\)/\1/p'`
    rm $HOME/Latest
    ln -s $date $HOME/Latest
else
    exit
fi
if [ $n ]
then
    /usr/local/bin/growlnotify -m 'rsync complete,
number of files: '$n &> /dev/null
fi

This first looks for the target thumb drive (PNY8GB in my case), then runs the rsync command, and finally pops up a nice little growl notification showing how many files were backed up. For non-OS X people, just ignore the growlnotify part.

#!/bin/bash
if [ -d /Volumes/PNY8GB ]
then
    HOME=/Volumes/PNY8GB/Backups.backupdb/TM
    date=`date "+%Y-%m-%d-%H%M%S"`
    rsync --stats -aR --exclude "*.swp" --exclude "*.bak" --exclude "*.pyc"
--exclude "*~" --exclude ".svn" --delete-excluded --link-dest=$HOME/Latest
/Users/davidarthur/Code/Loud3r $HOME/$date
    rm $HOME/Latest
    ln -s $date $HOME/Latest
else
    exit
fi

I’m still working on getting the directory structure to match Time Machine’s so I can actually use Time Machine to explore/restore files from the backups.

This gives me a little piece of mind, and hopefully will you too.

-David

Edit: OS X has some issues with preserving permissions and ownerships with HFS+ volumes – so the incremental part doesn’t really work. Should work on other *nix systems though.

2nd Edit: Got it working finally. Here is my complete script (running every 15mins). N.B. must be run by root (so put it in root’s crontab).

#!/bin/bash -x

DEST="/Volumes/PNY8GB/Backups"
LATEST="Latest"
EXCLUDES_FILE="/Users/davidarthur/.rsyncexcludes"
RSYNC="/usr/bin/rsync "

# Make sure user is root
if (( `id -u` != 0 )); then
    { echo "Sorry, must be root. Exiting..."; exit; }
fi;

# Make sure backup device is attached
! test -d "$DEST" && echo "Please mount the backup drive!" && exit

# Run rsync
DATE=`date "+%Y-%m-%d-%H%M%S"`
n=`$RSYNC -a -x -S --stats --delete --link-dest=$DEST/$LATEST \
    --exclude-from $EXCLUDES_FILE $* /Users/davidarthur/Code/Loud3r $DEST/$DATE
| sed -n 's/Number of files transferred: \([^0]\)/\1/p'`
# Update 'Latest' link
rm $DEST/$LATEST
ln -s $DEST/$DATE $DEST/$LATEST

# Send a growl notification
if [ $n ]
then
    /usr/local/bin/growlnotify -m 'rsync complete,
number of files: '$n &> /dev/null
fi