Data Protection – Family Photos – Part 1

My life seems to be one of constantly trying to corral pictures. My wife and I both have a few picture taking devices (phones/tablets/etc…), my 4 year old already has a point and shoot and we have the larger family camera. With all these devices it has been a nightmare to get photos in one place, and at the same time secure that one place against the worst.

We’ve all heard it, a friend or family member loses everything to a theft, corrupt hard drive, ransomware etc… So how the heck do you protect some of your most valuable assets, the memories? For me, it’s been a LONG road to get even half of the way there, so I thought I would share with you what I’ve been doing right and so very very wrong…

Before I begin, a little about my setup – I’m an Apple user, we have MBPs in the house, a Synology NAS device and random other hardware/NAS/etc…

In the beginning – There was iPhoto… and then quickly I moved to Aperture. The photo management aspect of iPhoto was quite annoying and it lacked a lot of the more advanced features. But either way I had no issues with file management, one library had everything in it and that was around 10k pictures back in 2009. Very easy to manage, I would copy the library to a USB drive and I was solid.

Then came the kids – Like so many others, when we had our first child in 2010 I went from 10k photos to over 100k that year. It was a crazy, wonderful, beautiful time but my library went from 20G to around 150G in no time and I was off to get a bigger USB drive. it was still manageable but I started looking for a better solution. As the years have gone on I’m now in the range of 700G of photos and 300G of video. At this point I had to separate my albums for sanity sake and went to yearly based albums. The USB drives just don’t cut it anymore, even with rsync or other copy utilities, it just takes a long time to search for changes and sync, although cutting into smaller albums did fix a lot of this. There HAD to be a better way.

Dark times of file sharing programs – Dropbox, gDrive, Box, Skydrive and many others promised me hope in a world of unmanageable files and a spouse wanting to kill me for not having a better solution. So tepidly I stepped into trying them. I started with my 2009 and prior libraries, which seemed to go well. I had the libraries up in a few days and I tried opening them on different computers and it worked! And this was my downfall, I assumed that the larger libraries would be the same and copied in 2010 library and thought nothing of it… until it was 100 days to sync, and then it was 99.4 days and slowly it went and each of the tools was crunching away.

I was shocked to see roughly 400k files needing to be cached. I looked for 2010 and there were a little over 100k pictures, but why were there 300k ‘other’ things?! Turns out it’s the way that Aperture stores it’s files with thumbnails, previews and other stuff for their database. This caused all of the file sharing tools to slow things WAY down because of hashing, copying etc… Long story short, it failed miserably and had to scrap it. I couldn’t let it copy for 2 years all of the albums.

Nirvana – After trying other programs like Lightroom I realized I needed to change the way I store my files and see what I can do to shrink that massive file count size. I didn’t want to move to Lightroom just yet, as my spouse is more comfortable with Aperture. The answer was to move to ‘Referenced’ images. This is essentially storing the files in my own data structure vs. Aperture storing all of the files in their package (directory).  When I moved to this I realized I could solve 50% of my issue. Keep the albums local for now and get the files up and shared.

What does this all look like?

Photo Management Software – Aperture 3.X w/ Referenced Imanges

Aperture Libraries – Stored on local SSD drive

Master images (Referenced) file structure – Filesharing Folder -> Photos Albums -> Year

File sharing tool of choice – Dropbox

File Sync protection – Dropbox sync of Master images to 2+ other computers + Synology NAS

Data backup – Crashplan from primary MBP to Synology NAS + Crashplan Cloud (All photos, libraries, etc…)

When I moved to this method I went from 2ish YEARS of syncing to 10 days! All of my master images are now sync’d and I can look at the next steps.

What’s Next?

Figuring out how to get the Aperture libraries synchronized to all computers in a reasonable time frame AND the bit rot issue…