Yes, the IRS probably did lose those emails

shutterstock_IRS

The story of the lost emails of Lois Lerner and other IRS employees will undoubtedly be in the news for a long time to come, in fact until President Obama has left office if the GOP has its way. After all, everyone must know that the only way that such a massive store of electronic data could be lost in this modern age of cheap data storage and advanced technology is on purpose. Don’t they? Well, if they do know that it’s just another case of people knowing something that just isn’t so. If you want to discuss something technical or scientific, you should go ask the technicians or the scientists, shouldn’t you? This, unfortunately, while eminently sensible is not particularly in vogue with far too many people lately. Let’s ignore them, shall we?

I thought it was interesting when I found an article on ZDNet, a web site that specializes in technology, on the likelihood of the emails having been lost just as the IRS was saying they were. ZDNet, by the way, is the “descendant” of Ziff-Davis, a company that has been publishing magazines about computers since before the internet existed. I’ve been reading their various publications for a long time. Larry Seltzer, the writer of this article, has always struck me as a pretty good writer. His conclusion is that it’s entirely likely that there was no intentional destruction of those emails, just some amazingly bad IT management. He even cites a column by Megan McCardle, who had experience before her career change as a manager of Exchange email systems, who despite her (understandable given her ideology) qualifications on how suspicious it all still looks admits that it’s pretty likely that it was sheer bad technology management that caused this train wreck. Both Seltzer and McCardle studied this document provided to the House Ways and Means Committee by the IRS to come to their conclusions and after reading it myself I agree wholeheartedly. Here’s why.

The IRS, like many other governmental and private organizations, uses an email system based on software from Microsoft. The software on the client computers, like Lois Lerner’s, is called Outlook and the software on the servers is called Microsoft Exchange. One of the things I do in my day job is to run a very small installation like this for my employers. Like, Seltzer, I found it very striking that nowhere in the document the IRS provided was the word Exchange even mentioned. That’s a red flag about the understanding of the system that the person preparing this document possesses, in my opinion. In such a system the Exchange server not only manages the business of knowing everyone’s email address and the information to route it all properly but it is actually where the main storehouse of email is kept in databases as well as other information like the user’s address book, task list, and calendar. The Outlook software keeps a copy of this data on the client computer if it was set up that way when installed. In an organization with as much sensitive information as the IRS, my personal choice would be to not have that option selected. So everything would be on the servers. Or at least it should be. But in a not very brilliant display of badly allocated resources each employee was limited to 500 megabytes of storage in their mailbox. I can tell you that far more space than that is needed for a busy person who keeps their data for any length of time, like 4 times as much storage just for a busy person in a small business. How much more do you think a manager of an agency with 90,000 employees that is answerable to the Senate, the House and the White House needs? Then there are the policies concerning what to do when you run out of space. Seltzer quotes that paragraph and I think I will too. The emphasis is mine.

When a user needs to create space in his or her email box, the user has the option of either deleting emails (that do not qualify as official records) or moving them out of the active email box (inbox, sent items, deleted items) to an archive. In addition, if an email qualifies as an official record, per IRS policy, the email must be printed and placed in the appropriate file by the employee. Archived email is moved off the IRS email server and onto the employee’s hard drive on the employee’s individual computer.

So, yes, if an individual user has a hard drive crash they will lose all of their immediately available archived email. And it’s so very, very unnecessary. See, if you’re running an Exchange Server system and you’re willing or able to spend the money for what Microsoft calls the Enterprise edition of Exchange Server you can add the software and hardware so that those archives are not on someone’s local drive but are instead stored on drives linked to the servers that can in turn be backed up as more data is archived from the users. For the IRS or any other government agency to not have the budget to do this is, to use an old phrase, penny wise and pound foolish.

Auf Stumbleupon zeigen
Auf tumblr zeigen

Author: JIM SATTERFIELD

  • DORIAN DE WIND, Military Affairs Columnist

    Interesting, Jim. Thanks.

  • dduck

    BTW, It is my understanding that the IRS computers/systems are tied in with the ACA computers and programs for calculating subsidies and mandates. Hope they have good backup and stuff for this next round.

  • http://www.liberalvaluesblog.com Ron Chusid

    Back up of email has nothing to do with IRS calculations of subsidies.

  • dduck

    Emails aren’t the only things you back up. I try to back up data files, doesn’t everyone.

  • http://www.coachkp.com Kevin Purcell

    Too funny dduck.

    My life is in the cloud via Carbonite.

    If anyone cares, so is Louis Lerner’s (or some reasonable facsimile).

    I have spent the last 15 years telling my daughters everything you put out onto the Internet is there forever.

    Now we are to believe this woman and the other six are lost forever?

    This is like walking in on a girl or boy friend in bed with a lover and them looking you in the eye and saying it’s not what it looks like.

    STOP!!

  • JIM SATTERFIELD

    Kevin,

    Email is not out on the internet when it comes to anyone who uses Exchange servers or the other systems like it. It is on the servers of the sending organization, the receiving organization and sometimes the systems of the individual senders. Carbonite or other public cloud based backup systems aren’t used by the government. Using something like that would be against the policies of pretty much any large organization with highly sensitive information like the IRS. They might have a contractor who performs the equivalent duties and in fact did have one but for some reason, probably budgetary, cancelled the contract a few years back. I think that was in 2011. I can promise you that once a contract like that is cancelled that the company won’t be wasting storage space keeping a former client’s data. In fact it would probably be part of the contract that on termination all data would have to be securely erased.

    Also, a huge amount of the email that someone like Lerner has is internal to the IRS and never even leaves their servers, just being routed internally to the people it’s addressed to.

  • JIM SATTERFIELD

    dduck,

    Unfortunately backing up Exchange server data isn’t as simple as basic data backup. To do a really thorough job, especially on large systems, you need specialized backup systems that can deal with the specialized software and databases the program uses. It can be done and for the IRS or other organizations should be done otherwise any data retention policies they have are so much rubbish. And based on reading that document that’s exactly what I’d call what the IRS IT department did. Of course given that in the document submitted to Congress it was noted that some of what was done was with the idea of saving $10,000,000 it does show that the government put data safety and security somewhere low on the ladder of budget priorities, something that they often share with the private sector.

    BTW, if you read any of the parts where the IRS official was explaining how hard it is to go around collecting that kind of data from various client computers and going through it, believe it. I had to do something similar once on a much, much smaller scale by myself and without some of the specialized software that’s available out there and it was most definitely a pain in the posterior that ate lots of time. Searching through thousands and thousands of emails for sets of keywords is not fun.

  • http://www.coachkp.com Kevin Purcell

    They might have a contractor who performs the equivalent duties and in fact did have one but for some reason, probably budgetary, cancelled the contract

    Sure they did :)

  • The_Ohioan

    They might have a contractor who performs the equivalent duties and in fact did have one but for some reason, probably budgetary, cancelled the contract

    The downside of privatization. I’m sure this whole thing will convince congress to stop shopping out important duties and to pony up the money it takes for a department to do its job. Pretty sure.

  • dduck

    I am willing to concede that this past event was a, what, technical/ budget/whatever error and not malicious. However looking and preparing ahead, as we should, , do we know that the vast amount of data involving the ACA/IRS implementation of subsidies and mandates/fines will work with the IRS computers correctly and its data well protected. If not, we are in for a spot of trouble, as the British would say. What say you computer experts (I am not.)

  • sheknows

    Thank you Jim, good article. I would imagine that the antiquated systems of our government have finally seen their day.
    One would think that the most powerful country in the world would have the most efficient and fastest computer systems in the world, but as the VA scandal( the first one..where vets couldn’t get their benefits due to backlog) has shown us, that is not even near the case. Apparently, the only government organization that stores info is the NSA.
    Anyway, I don”t even understand the issue here. On record, it shows the Dem organizations were denied status by a slight margin over the Rep ones applying. Who CARES if they were scrutinized slower or more repeatedly. They don’t have the best track record for honesty as the EXISTING so called non-profits will attest to.
    If they are honest, that’s all that matters.
    Right wingers tend to scream “scandal” just to try and create doubt and what is worse…too many people buy into their crap and look for that one piece of hay in the haystack at the respect of our entire political system.

    Aren’t we just sick and tired of faux Repubican outrage? Apparently not…..

  • rudi

    Even Carbonite is finite and expensive.
    http://www.carbonite.com/backup/pricing-plans/business-plans

    Plus
    $49999 per year

    Cloud Storage Space250 GB
    Additional storage available

    Home PC now come with over 1 tetrabyte HD. 250 GB is 25% of a basic HD.

    A majority of PC’s at Walmart are larger than 250GB HD’s.
    http://www.walmart.com/browse/electronics/desktop-pc-towers-only/3944_3951_1224739/?_refineresult=true&mmodule=1&_mm=&_mm=&povid=cat1090104-env430704-moduleB050814-lLinkDesktopsShelf1DesktopswithoutMonitor&_mm=

  • JIM SATTERFIELD

    dduck,

    Exchange servers are dedicated systems unrelated to data storage for other things like the data that might be shared with HHS for implementation of the ACA. This can readily explain part of what happened. My guess is that email servers and their support infrastructure are much lower in priority when allocating funds than those systems that are more directly related to the main purpose of the agency. It’s a really big planning failure that’s shared with the private sector. Look at the numerous data breaches in the private sector. Security is a poor second cousin when it comes to planning the IT budget because the costs associated with a breach aren’t fully appreciated (Especially the potential cost of customers/clients losing trust in the company.) and the “It won’t happen here.” syndrome so common in pretty much any human endeavor.

  • SteveK

    Having my hard drive fail (read no spin dead) earlier this year with the most recent backup 3 months old I lost 12 weeks of eBay sales / tax data. I was able to rebuild it from hard copy but it cost me 4 long days. That’s when I looked into cloud storage. Now I’ll never go back to keeping data on my computer again.

    Starting with Dropbox because it came installed on my tablet (5 GB free storage) and it gave me a place to keep all my business documents, forms, and Outlook data file (email).

    Then last week Microsoft announced upping their FREE OneDrive storage from 7 to 15 GB so I signed up for that too. I’m now using OneDrive for my primary cloud storage with Dropbox as a once a week backup in case something happens to OneDrives servers. Beats the heck out of monthly backups and / or permanent data loss and it’s free.

  • ShannonLeee

    My institute switched Exchange service providers last year and we lost emails, calendar appointments, and then had appointment doubled and tripled in our calenders. I have never been a fan of MS Exchange.

    and yes, 500 MB is an absolute joke of a limit. That forces people to archive emails in their pst files on their local computers. A recipe for disaster imho.