I hate backups. Those carefree days of youth where I would risk an entire project on the reliability of a Seagate drive are long gone. I’m now convinced that anything and everything that can go wrong will go wrong. So, I backup data. In many cases, I backup data in triplicate. Because on that fateful day when yet another drive fails or a software vulnerability invites attack, I will love my backups, if only for a fleeting moment. This month is not that moment...
Last week, my backup server failed. It was an old machine with hardware-based RAID that had trashed one of its own drives. The drive was actually still good, merely a victim of yet another buggy RAID card. After a few years languishing in storage, I dusted it off, installed Ubuntu, setup software RAID, and apt-get installed BackupPC.
Previously, I had been running rsnapshot on a Mac to take backups, but the Mac was out of space, and rsnapshot was quite fragile. BackupPC came in and started doing the job very reliably. 23 VPS systems plus a bloated NAS from our Dallas office all crawled down my frustratingly small Internet pipe into this decade old server. Somehow it survived 4 years in the attic in a crude wooden box with way too much humidity and a less than perfect window unit keeping it cool. I also drive a beat up, old Dodge truck if you’re curious.
My dead server left me with a decision, repair or retool. Given that the server was running out of space requiring trimming backup history, I chose to retool.
Retooling meant researching and selecting a new service. I quickly passed the Dallas NAS off to CrashPlanPRO since we already had an account for some of the data in Dallas. With a recent office move, the Dallas office was able to get Verizon FiOS which meant switching backup providers for the 1 TB of data housed there was much more practical. That left me with the 23 VPS systems.
After a lot of research, I decided to try Jungle Disk Server. Though I’m no fan of Jungle Disk’s parent company Rackspace due to the treatment of Slicehost customers following the Rackspace purchase of Slicehost, I did take some comfort in the size and reputation of Rackspace in deciding to try Jungle Disk. Jungle Disk was also one of a very few providers with storage fees that fit my budget.
Their current pricing is $5/month/server with 10 GB of free storage per server. For me, that meant all but 5 of my VPS systems would cost only $5/month, and I would be paying less than $20/month in storage fees on Rackspace Cloud Files for the servers over 10 GB. It also meant I didn’t have to worry about the variability inherent in Amazon’s S3 pricing model since Rackspace Cloud Files does not charge data transfer fees.
So off I went. I setup Jungle Disk Server on what is essentially an archived VPS running an Ubuntu LTS release. With Jungle Disk Server, you install a daemon on the server and access it with the Jungle Disk Server Management client on your desktop (Windows, Mac, or Linux).
The management client reminds me of really poorly designed Java software I wrote in the 90’s. Instead of approaching the design from the perspective of the user, I would look at it from what was convenient for me as a developer or convenient to the data model. It works, but it doesn’t work well.
The management client is also pretty crash happy. I usually encountered crashes on my Mavericks Mac when using the Change Server function. It never appeared to affect the function of the management client nor did it cause it to lose any of the settings it was saving to the servers.
Since I have only a couple dozen servers, I decided I could live with the management client. If you had a lot of servers, though, you would need to look into automating operations on the Jungle Disk Server configuration files themselves. Otherwise, you would spend the rest of your life going into one server, configuring something, going out, logging back in…lather, rinse, repeat. I’m not sure if Jungle Disk documents the xml files they use for configuration, but I doubt they would be difficult to reverse engineer.
The first couple of days testing Jungle Disk Server went off without a hitch. The backups made it to their vault (Jungle Disk lingo) as scheduled. CPU and Memory usage wasn’t low, but it definitely wasn’t high either.
Restore worked, but it wasn’t terribly convenient. Instead of allowing you to restore to an arbitrary location, Jungle Disk Server requires you to restore to the server from which the backup was taken. If you’re moving to a different server or performing disaster recovery, this means you need to rekey the license for the new server. It wasn’t a deal breaker, but it definitely interfered with my normal workflow. During development, I often pull database backups from the backup system both to get data I need for development *and* to test the backup system. Even so, the Jungle Disk approach just meant an extra step.
With a successful test, I embarked on configuring my VPS systems. I made it through 12 servers before problems started appearing.
After their first backup, several servers reported SSL errors in syslog. Investigating this issue further, SSL errors appear to be common in Jungle Disk’s desktop backup application, but they don’t have any support documents specific to Linux or the server application. However, firewalls and certificate file corruption seemed to be the main issues so I set out to correct them. I confirmed iptables was not filtering the traffic, and then I focused on the certificate file. Comparing checksums between working and non-working systems, the files were identical.
In contacting support, they said to delete the certificate file and restart the server. The server did in fact come back up after this process, but the new certificate file it downloads matched the checksums of the “corrupt” file. Needless to say, my confidence was already shaken, but being already halfway to the finish line with all of my backup hate, I optimistically hoped the error would just go away. Besides, I had 7 servers working fine.
The next issue that appeared was a failed file system scan for my mail server. The server holds 70 GB of data, and as you might guess, there are *a lot* of small files. No matter what backup system you throw at it, file scans are not going to be fun. That said, Jungle Disk Server pegged my cpu at 100% for 2 hours and had only found 10 GB out of 70 GB. BackupPC’s rsync had never run longer than 30 minutes on the mail server.
Around the same time I killed Jungle Disk on the mail server, I noticed a server that had been running without incident failed an overnight backup.
And that was the end of it for me, I may hate backups, but I don’t think Jungle Disk Server would have ever given me that fleeting moment of restored data bliss.
Note: I had some other issues with licenses and backup vaults. I’m giving Jungle Disk the benefit of the doubt and accepting Jungle Disk Support’s explanation that I got my wires crossed during installation (create xml license file, copy/paste license key).