Image Mechanics Off-line Archiving System
As capture specialists, Image Mechanics generates a huge amount of data on a daily basis. We have photographers who can shoot over 3000 medium format digital images per day and we often have multiple shoots happening simultaneously.
We had to find a solution for archiving all of this data for ourselves and for our clients. Keeping everything live and accessible 24/7 just isn’t practical or necessary. You would need a giant RAID system like Apple’s xRAID which starts at $6000 for 1 Terabyte of redundant storage. Redundancy is the key for safe storage of digital files but this route is very expensive. There are other systems that offer less expensive alternatives in the 1-3 terabyte range but they aren’t as expandable as the xRAID. Also keep in mind that a RAID offers some protection against loss but isn’t an archive solution. You still have to back up a RAID.
Expandability is one of the other problems we had to solve. Storing images on external enclosures lets you add more drives as you need them but you end up with a big pile of hard drives and power supplies. Each time you need to add storage, you end up paying for the shiny aluminum housing, power supply, manufacturer advertising, store markup and finally the bare hard drive itself. It isn’t as convenient or economical as it could be. A quick word of warning. Many of you are buying large two drive enclosures like the LaCie bigger disk extreme. These drives are set up in a RAID 0 configuration. Yes, they do provide a large volume to store images and they are very fast to write to but they aren’t meant to hold your archives. A RAID 0 stripe is two drives acting as one for speed. This configuration is used for HD video capture where the frame rates exceed normal hard disk speeds. If one of the two drives goes down (and all drives will eventually crash) you will loose all the data on both drives. This makes them twice as prone to failure than a single drive. If this is your archiving system you should do a google search and see how many photographers have lost all their data. I found a good explanation at a data recovery website.
With all that in mind, we set out to find an economical way to safely store all of our shoots for an extended period of time. I should mention to those of you that are asking why not just use DVD’s that there aren’t enough hours in the day to burn the amount of data our clients generate. There are several other reasons I don’t like optical media for long term storage and I will post an article about that later.
The solution we developed is based around an external SATA enclosure with removable hard drive trays from FirmTek. SATA is a type of connection just like firewire only much faster. It is the standard used by new Apple towers to connect the internal hard drives. In conjunction with a SATA PCI card, you can have external storage speeds just as fast as your internal hard drive. Another reason for us to use SATA is that most of our shoots are tethered to the computer via firewire. For speed and compatibility issues, we isolate the firewire bus for the transfer of images from the camera to the computer.
Combined with purchasing bare hard drives, the one time purchase of the external enclosure and the SATA card makes more financial sense than a stack of external firewire drives. The two bay enclosure and PCI card cost about $300 when purchased bundled together. 300 Gb drives are currently about $96 each at zipzoomfly and the trays to put them on are $22 each. This means that every 300 gigs of additional storage costs about $118.
The next obstacle to overcome was how to store the hard drives when they weren’t in the enclosure. They need to be safe from the elements as well as static. Since there wasn’t anything commercially available we designed what we wanted and had it manufactured for us. We had anti-static foam water-cut to hold hard drives vertically inside of a fire safe file cabinet.
The low tech method of quickly finding your images stored off-line is to use a labelmaker to label each hard drive with the shoots it contains. You also probably want to use a database program to track individual images. We like Extensis Portfolio for off-line referencing. To find a file with Extensis, you just click on the thumbnail and it tells you which drive the image is on. You then pull that drive out of the drawer and pop it into the FirmTek box and retrieve your image. A couple of seconds of extra work to safely and economically store your images. Watch the video to see the system in action.
There you have it. A low-tech, inexpensive, easy to maintain archiving solution. We sell the anti-static foam file cabinet inserts for $200 each. Please email me if you are interested and include the dimensions of your file cabinet.
Add to:| Digg it | +Google | Slashdot
5 Responses to “Image Mechanics Off-line Archiving System”
Leave a Reply
You must be logged in to post a comment.
November 14th, 2006 at 2:36 pm
Just a few notes:
* Maybe I’m missing something, but I don’t see that redundancy is solved with your setup. It seems like a specific image is only stored once on a single external SATA drive.
* Your solution may be a cheaper, if less flexible storage solution than Apple XServe, but it looks like it is no more expensive than Sun’s “Thumper” device:
http://www.sun.com/servers/x64/x4500/
Obviously the initial cost is higher, but the storage is on-line, redundant, comes with its own server, etc. and probably takes up less space.
Any thoughts, anything I’m missing?
November 15th, 2006 at 8:45 am
Yes drouse, you are missing something ;-)
I never said to only keep one copy of each drive. As a rule with digital, you ALWAYS need at least two copies of your data. We keep an A and a B version of each drive that is an exact copy. For instance, if the drive is labeled Archive0017A there is an exact copy (stored off-site) named Archive0017B.
This system is far more flexible than any hard wired storage solution. It is infinitely expandable by just adding more bare hard drives. It also doesn’t require a rack-mount airconditioned enclosure and it isn’t constantly running your hard drives using up electricity and generating heat. Hard drives in servers that run 24/7 get replaced every 12 months. A constantly running hard drive isn’t a good archiving choice.
Medium sized businesses that have an IT staff and server room are the target market for systems like you linked to. Photographers generate more data than most businesses 10 times their size. You are sadly mistaken if you think managing a Sun server running Solaris OS is a trivial task that the average person/photographer can handle themselves or with a limited staff. BTW, these same companies use robotic tape machines to back up all their live data. Tape is slow and expensive and is optimized for small incremental changes like text files. It isn’t a great medium for backing up large image files. RAID servers still have to be backed up.
Our off-line solution is cost effective in it’s simplicity. Cheap hardware, no learning curve, fast data transfer/retrieval and off-site capability.
Thanks for your comment. Blogging by it’s nature is a kind of short hand format. The goal is to include enough information to get the point across while still letting the readers get in and out quickly. As writers, we also have a limited amount of time to dedicate to the process and it is good to be able to go more in depth by answering questions.
November 15th, 2006 at 6:55 pm
Extensive Portfolio? really? I have not had great luck with that program. Any thoughts on Aperture? I haven’t played with it since it first came out. I’m currently recommending the Bridge/iView Media pro combination Peter Krogh describes. Any thoughts pros/cons on that vs. extensis? should i give it another shot?
November 21st, 2006 at 8:45 am
Extensis handles off-line images better than most other DAM solutions. We haven’t had any problems with it other than it is a little slow for the amount of images we put through it. We use a customized version of Extensis NetPublish server for our client web galleries so it makes since for us to use Portfolio as our DAM client.
Aperture and Lightroom are both interesting possibilities but until they shoot tethered, I can’t see how to fit them into our daily workflow. I am not sure they will work for Image Mechanics as a capture company. There is no advantage for us to put every image shot into a database since we are just moving them through as a service. So far, Adobe Bridge with ACR is the best browser/processor for how we work.
Bridge/iView is a strong combo. We used iView a lot back when the File Browser in CS1 was dog slow. Once CS2 came out and Bridge was 10 times faster we have been able to do most of our work with one piece of software. iView is very fast and capable if it fits your workflow then you should absolutely use it.
Something to consider about using any database is that if keywording and other metadata are rolled into the DNG file, it doesn’t really matter what DAM (Digital Asset Management) software you use to find it. The info is with the file and not just in a database. Spotlight can search image metadata in OSX and Microsoft’s Vista will also have a capable search function. I am curious to see what direction DAM software takes in the future.
July 8th, 2008 at 6:28 am
Great info on what’s becoming an issue for a lot of photographers. I did a quick search on the archival properties of magnetic media and found nothing worthy to post. It’s almost as if people are merely hoping the state of their magnetic media will never decay. Perhaps if I drilled into Hitachi’s or Western Digital’s website I could find more information on the archival properties of their platters, but the companies don’t make their drives for archiving, they make them for constant use. If you find anything out about “archival hard drives”, please post it.
I don’t acquire nearly as much data as you or your clients do (most of my work is still life) so I’ve gone the route of “archival” DVDs and an off-line hard drive in an enclosure. But after large jobs, such as catalogs, it’s a day burning DVDs, with a glimmer of hope that Blu-Ray disk storage will make it easier.