Amazon S3

Amazon S3 is the service behind cloud storage giant Dropbox and a number of other companies. Once your data are hosted on S3, you can quickly and cheaply import it into your Galaxy instance running on Amazon EC2. For $0.01 per gigabyte per month you can host your data on S3. Compare this to approximately $0.30 – $2.00 per hour for running your Galaxy instance on the cloud, and you will quickly realize that uploading your large data to S3 before running Galaxy will save you big time.

Uploading your data to S3

Go to the S3 link in your AWS dashboard.

You will see a management page for your S3 “buckets”. Create a new bucket. You can think of a bucket like a hard drive – you can place files and folders inside a bucket.

Now select your new bucket. You’ll drop in to a page telling you that your bucket is empty, and allowing you to select “upload” to add files to the bucket.

Add some files using the file selector, and then click “start upload”.

You might want to use the “advanced uploader” if your files are rather large. The length of the uploading process will depend the speed of your internet connection. Don’t try this at your local Starbucks. Once your files appear in S3, you can select a file and view its properties. You’ll want to choose “actions” and “make public” so that we can import this file later into Galaxy. Note this will allow anyone on the internet to see your file. I won’t cover file permissions in this guide, but you may be able to keep your data private and still import them into Galaxy on AWS.

Now that your uploads have finished, let’s initialize your Galaxy cluster on AWS and import your files into your history. You can simply select your file and copy its S3 URL (in blue) to your clipboard.

Then in Galaxy, select the “upload file” tool under “get data” and paste the S3 URL of your file into the text box and click “execute”.

That’s it! The file should almost immediately appear in your Galaxy history and you can start working with your data.

git clone https://github.com/uskudnik/amazon-glacier-cmd-interface.git 
cd amazon-glacier-cmd-interface 
sudo python setup.py install  

After we install glacier-cmd, let’s start creating our backup script to upload files located in a certain directory, as well as check that we don’t have too many copies of those files stored already on Glacier. We should keep the last three months of Glacier archives, since deletion prior to three months results in a pro-rated charge. First we need to create a SimpleDB domain to store the glacier-cmd cache:

python
>>> import boto
>>> sdb = boto.connect_sdb('your_access_key', 'your_secret_key')
>>> sdb.create_domain('glacier-cmd')
>>> quit()

Now let’s start our script. We want a wrapper script that passes all of our AWS information, as well as some information about logging and AWS region. We’ll set SimpleDB domain we created so we can have a cached inventory of our vaults and their contents. We’ll want a function that creates a new vault (based on the date and name of the backup set) and uploads some data to it. I’ve written two methods: (1) Using tar and (2) using zfs send. Both methods compress the backup using bzip2, although the zfs method uses a more advanced “snapshot” of the filesystem which will accurately reflect the state of the backup set even if data are added or removed during our upload. We will also want a function that cleans up backup sets. I’ve hard-coded this to a value of 3 months old backup sets, since Glacier expects that data will reside for this minimum period.

#!/bin/bash
# Usage:
# glacier_backup.sh backup-job-name action folder-to-backup
# action=(backup,cleanup,zfsbackup)

# AWS access keys
access_key='your_access_key'
secret_key='your_secret_key'

# glacier-cmd configuration
region='us-east-1'
bookkeeping_domain='glacier-cmd'
logfile=/var/log/glacier.log
loglevel=INFO
output=print

# Command line invocation of glacier-cmd with variables passed
glacier_cmd="
/usr/local/bin/glacier-cmd\
 --aws-access-key $access_key\
 --aws-secret-key $secret_key\
 --region $region\
 --bookkeeping\
 --bookkeeping-domain-name $bookkeeping_domain\
 --logfile $logfile\
 --loglevel $loglevel\
 --output $output
"

# Debugging options
if [ "$1" == "debug" ]; then
    $glacier_cmd $2
    exit 0
fi

# The name of the backup job
backup_job=$1
path_folder=$3

if [ "$2" == "backup" ]; then
    hello=`date --date='now' +%b%g`
    vault="$backup_job$hello" # A unique vault name created from today's date plus job name
    $glacier_cmd 'mkvault' $vault # Create this vault
# Tar and compress directory, then upload from stdin
    tar cjf - $path_folder | $glacier_cmd 'upload' $vault --stdin --name ${path_folder#/}.tar.bz2
fi 

if [ "$2" == "zfsbackup" ]; then
    hello=`date --date='now' +%b%g`
    vault="$backup_job$hello" # A unique vault name created from today's date plus job name
    $glacier_cmd 'mkvault' $vault # Create this vault
# Use ZFS send, then upload from stdin
# If using ZFS send, $path_folder must be "pool/filesystem" with no preceding "/"
    pool_filesystem=${path_folder#/}
    /sbin/zfs snapshot "$pool_filesystem@$vault"
    /sbin/zfs send "$pool_filesystem@$vault" | bzip2 | $glacier_cmd 'upload' $vault --stdin --name ${pool_filesystem}.zfs.bz2
    /sbin/zfs destroy "$pool_filesystem@$vault"
fi 

if [ "$2" == "cleanup" ]; then
    goodbye=`date --date='3 months ago' +%b%g`
    vault="$backup_job$goodbye" # The vault from three months ago - may not exist!
    $glacier_cmd 'rmvault' $vault
fi

We can set up two cron jobs for the beginning of every month to upload and cleanup a set of ZFS filesystem backups:

00 02 01 * * /home/matt/scripts/glacier_backup.sh timemachine zfsbackup socrates/TimeMachine
00 01 01 * * /home/matt/scripts/glacier_backup.sh timemachine cleanup

That’s it.

Our paper reported an analysis of a mouse genetic model that deletes the C terminus of Shank3 to mimic human mutations that cause autism spectrum disorder. Figure panels for several polyubiquitination assays were improperly assembled, leading to multiple repetitions of bands in western blots of the lysates. These errors did not affect the quantitative analysis of polyubiquitination because this analysis was performed as described and was not dependent on representative western blot images. In light of the figure preparation issues, we feel that the most responsible course of action is to retract the paper. We sincerely apologize to the scientific community for any misunderstanding that these errors may have caused.

Well done for preventing the spread of misinformation. It must hurt to issue a retraction, but it sounds like the conclusions still stand regardless of the “improper assembly” of the representative Western blots. When are we going to learn to stop trusting Western blots in papers?

Here it is, after a few weeks of waiting!

The actual board itself is very small. So small, in fact, that I wondered whether I needed to buy a project box to stuff it in. While looking around online at different expensive plastic boxes to put this in, I realized that I have the perfect enclosure for the Pi already in my audio stack.

The Edirol UA-5 usb audio interface

My USB audio interface is made of thick steel and aluminum, and will be plugged in to the Pi anyway (since the Pi uses onboard pulse-width modulation to approximate an audio signal at low quality). Why not put the Pi inside the UA-5? Let’s see if there is room for a Pi.

Plenty of room inside

After removing the thick aluminum cover with a few screws, I see that there looks to be room directly over the digital signal processing side of the board. This is good, because I don’t want to stick a little computer spewing RF noise all over any analogue circuits. Let’s remove the main board.

The board comes right out, and now we’re left with a thick steel case. Let’s cut a hole in it! We need a hole to pass the USB ports, power, and WiFi radio through. I have a feeling this case efficiently blocks and RF signals emanating from inside. At this point I should say that I rushed in to cutting the case without proper tools (I used a drill and some snips). I should have had at least a file to smooth out the rough edges, but I can go back and fix this later.

A very rough looking finish. Oh well.

After cutting a hole in the side of my beautiful equipment, let’s figure out how to fit the Pi in. I’ll screw the main board back in and then cut a piece of the packaging cardboard to make a standoff that will insulate the two boards from each other.

Reduce, Reuse…

And secure the Pi to it’s new home. I’ve added an 8GB SD card with Raspbian along with an Edimax USB wireless dongle and a microUSB cable for power. This is all secured to the cardboard standoff with zip ties.

It fits well

Before I screw the lid back on I need to power up the Pi and log in via SSH using the onboard ethernet. After editing my /etc/network/interfaces and adding “auto wlan0″ I just add the proper wpa_supplicant.conf, grab the wlan0 MAC address, add a static DHCP reservation for the Pi on my router, and reboot. Now let’s put the lid on.

Not bad!

It looks pretty good! Let’s log in to our Pi and install Shairport. It’s pretty straightforward.

# Change to our home directory, install all the packages we need.
cd ~
apt-get install build-essential libssl-dev libcrypt-openssl-rsa-perl libao-dev libio-socket-inet6-perl libwww-perl avahi-utils pkg-config git emacs
git clone https://github.com/albertz/shairport.git
cd shairport
make
# Install Net::SDP Perl module
sudo -s
perl -MCPAN -e 'install Net::SDP'
# Demote the onboard soundcard and allow USB soundcards to become default
# Add 'options snd_bcm2835=-2'
# Comment out line 'options snd-usb-audio index=-2'
emacs /etc/modprobe.d/alsa-base.conf 
# Copy the example init script
cp shairport.init.sample /etc/init.d/shairport
# Edit the init script to point to Shairport
# NAME=RecordPlayer
# DAEMON="/home/pi/shairport/shairport.pl"
emacs /etc/init.d/shairport
update-rc.d shairport defaults
reboot

That’s it! Let’s hook it back in to our amplifier. I’ve connected the UA-5 to the remaining USB port on the Pi.

The Raspberry UA-5 sitting on the phono preamp and stereo amplifier.

With everything connected and powered on, we can now select “RecordPlayer” from our iOS or Mac OS devices and stream music to the nicest speakers in the house.

There’s a computer in there.

The UA-5 is capable of 96KHz 24-bit playback, but because the Pi is not too powerful, and I suspect that Airplay only transmits 16-bit resolution, I have it set up for 48KHz 16-bit playback.

UPDATE:

I’ve recently encountered an issue with the default Pi ALSA configuration. It seems like the Raspberry Pi developers worked in a direct memory access mode for the onboard sound, and this was giving me some issues while using a USB sound card. Sometimes, but not always, the audio would cut out for about a second every five to ten seconds. With some configuration changes I’ve been able to eliminate this. Also, to be safe I’m now using 41KHz playback. Here’s the /etc/asound.conf:

pcm.!default {
    type plug
    slave.pcm anticracking
}

pcm.anticracking {
    type dmix
    ipc_key 1024
    ipc_key_add_uid yes
    slave {
        pcm "hw:0,0"
        period_time 0
        buffer_time 0
        #period_size 1024
        #buffer_size 8192
        rate 44100
    }
}

Thanks to [maximeh] for the suggestion.

]]> http://mattshirley.com/adding-airplay-to-an-external-usb-audio-interface/feed/ 18 http://mattshirley.com/wl911-helicopter-battery-charger-fix/?utm_source=rss&utm_medium=rss&utm_campaign=wl911-helicopter-battery-charger-fix http://mattshirley.com/wl911-helicopter-battery-charger-fix/#comments Sun, 06 Jan 2013 17:35:33 +0000 matt http://mattshirley.com/?p=397 I recently bought a really fun mini RC helicopter. It’s made in China by WL Toys, and it packs lots of technology into a ready to fly kit. Flying it has been a blast, but the manufacturing quality control leaves something to be desired. The remote control has a small piece of something rattling around in it, which doesn’t affect the function at all. The battery charger is supposed to charge two of the Li-polymer batteries simultaneously, but it seems like mine only has one working port:

We need more red LEDs please.

Let’s see what we can do about that. The bottom has four tiny screws and a small vent.

Thank god it has screws.

Inside there is a small, roughly finished board with some SMD components. It looks like a few resistors and maybe voltage regulators.

The top of the board looks fine. Reads “Hong Li”.

Underneath the lid.

Looking at the bottom of the tiny board reveals a potential issue with one of the charging leads. It appears that the leads on the left channel are both soldered, but the right channel only has solder on one lead! Let’s fix that right up with some lead-free solder.

Something missing

Something added

After adding a blob of solder to the pin and re-assembling the whole thing, it works! I was thinking about asking the seller to replace this part, but the entire fix took under 5 minutes with a small amount of work. My friend has this same helicopter kit, and he also had this issue with the first charger he received. It seems like someone was asleep on the assembly line.

It works!

Dave Brubeck, the composer and pianist that exposed many decades of young people to jazz, died yesterday at the age of 91. He literally brought jazz to college by integrating classical European composition elements such as alternating time signatures and introspective chord progressions. Some people, including my favorite music professor Michael Budds, hinted that Brubeck couldn’t swing like other contemporary jazz innovators. I think he just swung more subtly. He had to, since the styles of music he integrated were viewed as diametrically opposed in nature. Along with other visionary composers such as Duke Ellington, Brubeck left more than the legacy of his songbooks and great recorded performances; he pushed the concept of jazz as art and intellectual stimulus into homes worldwide. Dave Brubeck (along with Paul Desmond, Joe Morello, and Eugene Wright) introduced me to jazz that’s not “old-timey” – not a caricature of another decade. His death will surely not stop future generations of young people from enjoying this great American art form. Rest in peace.

PS: This morning (Dec 10) Tom Hall aired his thoughts about Brubeck as well as a three year old interview between the two.

Today, the administration’s Office of Budget Management (OMB) released its plan, and it paints a pretty grim picture for biomedical research. The NIH and NSF would each sustain an 8.2% cut to their budgets which would amount to a reduction of roughly $2.5 billion and $460 million, respectively.

An 8.2% budget cut doesn’t mean that we are “trimming the fat”, or keeping expenses under control. I’m also not sure if it means we’ll be losing many jobs either. The scientific community is closely knit, and funding has been stagnant for a few years. We’ll keep doing some research, but not as much, and the size of incoming graduate student cohorts will continue to dwindle as training grants become smaller. I’m not sure whether this last part is overwhelmingly good or bad.

Since I will soon complete graduate school myself, it’s easy to view my experience as “right time, right place”, and write off the funding contraction as necessary, and look forward to some future time where we again have an inflationary budget, and there will be a shortage of trained scientists of a certain age. The NIH seems aware that we are over-training students for the number of academic positions available, so whether or not sequestration hits at 8.2% or not, we are seem to be moving forward under some selective pressure.