When you have a large amount of data to sift through, it is often good to create an ironclad framework to manage the data. This framework will include a method of inputting new data, modules for importing and cataloging old data, and an interface to wrap around the whole thing. Collectively, it is called a system.

The problem with systems it they are often created to manage a dataset that is expanding rapidly now, but will taper off quite soon. The designer of the system assumes that the expansion will continue at its present rate, so he creates the system to manage a large amount of data and he designs a thorough catalog to expedite searching. The problem is that with more items, more cataloging effort must be spent on each item so that searches can drill down the necessary data over an ever-expanding dataset. This means the cost of maintaining the system increases exponentially. If the expansion rate drops rapidly, this can be the nail in the system’s coffin, as fourteen layers of metadata provides diminishing returns when you are adding three records a day.

When you picture a “system,” think of a photography catalog. You add more photos as you take them with your camera, importing them using a memory card reader. You add tags and keywords. You sort the images into folders or (preferably) virtual folders. There is a search mechanism which lets you search by folder, date, and keyword. You can search thorough a mass of metadata that is generated by your camera automatically upon shooting each photo. If any one of these components is flawed, the whole system crumbles. Being able to find a photo in two seconds is worthless if you have to spend five minutes cataloging each one. Having a stable catalog of photos sorted with tags and keywords is no good if you have to make a separate physical copy of each file for each keyword. You need a complete and well-rounded system that is versatile and low-upkeep.

Unfortunately, you are just as likely to under-estimate the expansion of your system. What if you’re using Picasa to manage 10,000 images, but then you get a job as a professional photographer and start adding 1000 images a day? You may soon find that Picasa slows to a crawl when you get to 50,000 images. All of the sudden, your system is no better than a pile of dirt, and you’re screwed because you can’t export your database. You have to use clunky ICTP embedding to get the data out, you have to find a new system, you have to adapt to its quirks, you must compensate for its short-comings and differences compared to your current system, and you must do this all under the pressure of impending deadlines. Still think systems aren’t important? How long can you get by haphazardly managing your photographs? Amateurs can do it. Professionals, no.

The biggest problem with systems is overhead. If you keep a to-do list, how much time do you spend doing the tasks on the list? How much time do you spend working on the list, including thinking of new tasks, crossing out old ones, setting deadlines, and revising the list? The time spent on maintaining the list is the system’s overhead. If your overhead is more than 10%, you may consider scrapping or greatly simplifying the list. If procrastination is not a big problem for you, try doing things in the order you please. Instead of storing the list on paper, store it in your head.

Other systems may have overhead greatly in excess of 100%. A public library’s catalog may take more time to maintain than the shelving, sorting, and transactions for all the items in the library. New records have to be cataloged to rigid standards by trained librarians with Master’s degrees, who may be paid upwards of $70 per hour. Items must be checked in and checked out and marked as damaged or missing when problems arise. While it would be much easier to have no catalog and just place the books on the shelf by the author’s last name or the Dewey decimal system, the catalog is essential for searching records and tracking items.

A to-do list, however, should have a much smaller overhead, as it is a simple system to a simple job. Always consider the value of the system you are putting in place. If its overhead exceeds its value, scrap it.

I used to keep my mail-in rebates list on a sheet of paper on the fridge. I’d write the date I mailed the rebate, the product and amount, and the expected date. When the rebate came I’d cross it off the list. At the height of my rebate frenzy (back when EVERYTHING was free after rebate), I had to expand the list from one page to eight pages (8.5″ by 11″), covering the whole front of the fridge. This was unwieldy, so I found a program to put the rebates on my computer and I started managing them there. I got rid of the paper list and keyed everything in. It was a disaster. The whole system didn’t work; I couldn’t search by expected date or amount easily; I had to use klutzy titles that relied on alphanumeric sort. I ended up losing $100 in rebates because I was so bad at keeping track of them. Admittedly, I chose the wrong program (I think it was called Rebate Reminder), but the problem was I chose a system with too much overhead that did not provide any benefits over the previous system. In many ways it was even worse; it gave me a fragmented view of the money I had on the line, while the paper lists let me see everything at once. Instead of being in control, I had no control at all. Systems can often be more trouble than their worth. Remember KISS (keep it simple, stupid).

Perfectionists fall prey to complex and weighty systems. When I used to pirate DVDs from the library, I’d have to compress the video from 8.5GB to 4.7GB to fit on a single-layer DVD (most commercial DVDs are double-layer, but the blanks are prohibitively expensive). For some reason I wanted to save the full-quality video for future use, so I’d make a second copy of the original files onto two discs. I have a stack of 100 of these in my closet that I’ll never use. It was a horribly idiotic thing to do. The copies, compressed to 60% of the original size, do not look noticeably worse than the originals on my 19-inch television set. I’ll never have a reason to buy dual-layer blanks and remake the discs; I’m not interested in watching any of the movies again anyway. In fact, if I had to do it over again, I might have just borrowed the movies I wanted to watch, watched them, and returned them. My Dad likes to watch the same movies again and again, but I find, as with books, that one viewing is enough. However, I’ll listen to the same songs over and over again on my MP3 player, especially if I’m programming or writing.

Whenever you create a system, ask yourself: is this useful? Will the data I’m organizing be useful to me in six months? Three? Is it that much slower to have no system and let the chips fall where they may?

I used to keep a meticulously organized accordion folder of all the receipts, rebate forms, school papers, coupons, and paychecks that I’d received for the last six months. Then I’d move to a new accordion folder, keeping the old receipts indefinitely. Now I put everything in a big pile on a shelf in my bedroom. Every few months, I throw out all the old receipts and coupons. I can count on my hands the number of times I’ve had to hunt down a receipt in that pile. With the accordion folder, it took two minutes. With the big pile, it takes ten. It’s still cheaper than sorting everything, because only rarely do I need anything in the pile. The overhead of a system is simply not worth it.

I used to take thousands of photos a month. Most of them were useless. I’d document the position of all the family’s furniture and can goods each month. I thought that would be interesting later. Turns out it isn’t. I had a huge complex catalog of these photos in iMatch which I used to spend a lot of time on. I haven’t accessed the catalog in nearly a year. Now, I take just a few hundred pictures a month—good pictures, and I use a program called Downloader Pro to sort them into folders by date automatically. Since there are so few photos, I don’t need to catalog anything. I just look through each folder by date with FastStone Image Viewer (a wonderful program). Sometimes, yes, it’s hard to find a photo. Without the overhead of a system, however, I spend more time behind the camera and less time behind the computer.

This blog is a system. It took a long time to set up. The overhead was 100% in the beginning, and as high as 50% till recently. Now, I’m adding practically nothing to the system (WordPress). No new plugins, no theme changes, no design changes. The system has matured and the overhead is now about 10%. I could still be fiddling with all the details, but I’d much rather be writing than working on the system.

Use systems. Don’t let them use you.