USAF Lost 12 Years Of Investigations Thanks To A Corrupted File

Discussion in 'HardForum Tech News' started by HardOCP News, Jun 15, 2016.

  1. HardOCP News

    HardOCP News [H] News

    Messages:
    0
    Joined:
    Dec 31, 1969
    This story serves as a reminder that you should regularly back up your data. It is hard to believe that the U.S. Air Force Inspector General and Lockheed Martin didn't use common sense when it came to extremely important data like this.

    Defense One and others are reporting that the USAF lost its records of 100,000 investigations in their Automated Case Tracking System due to some kind of file corruption that neither Lockheed Martin or the Air Force have been able to figure out. Apparently Lockheed Martin spent two weeks trying to fix the problem before informing the Air Force, but had to own up to their failure when the issue proved insurmountable. The Air Force was said to have been informed on June 6th, and has since requested the help of the Pentagon’s cybersecurity operators as well as private contractors.
     
    Last edited: Jun 15, 2016
  2. Nenu

    Nenu [H]ardened

    Messages:
    18,601
    Joined:
    Apr 28, 2007
    Thats a lot of complacent workers and managers over 12 years to not realise the lack of data integrity mechanisms.
    Makes you wonder how much "they" need investigating.
     
  3. Rahh

    Rahh [H]ard|Gawd

    Messages:
    1,608
    Joined:
    Jan 14, 2005
  4. NeoNemesis

    NeoNemesis 2[H]4U

    Messages:
    2,385
    Joined:
    Mar 10, 2004
    Bah, it's only taxpayer money.
     
  5. Dead Parrot

    Dead Parrot 2[H]4U

    Messages:
    2,382
    Joined:
    Mar 4, 2013
    Makes those old 8" floppy based systems the ICBM folks use seem like a good idea.
     
    Red Falcon and Deadly Ramon like this.
  6. grtitan

    grtitan Telemetry is Spying on ME!

    Messages:
    1,266
    Joined:
    Mar 18, 2011
    I wonder who was being investigate when this "accident" happened....
     
  7. Gweenz

    Gweenz [H]ard|Gawd

    Messages:
    1,216
    Joined:
    Dec 18, 2003
    Billions and trillions of dollars.....
     
  8. krotch

    krotch [H]ardness Supreme

    Messages:
    4,509
    Joined:
    Aug 12, 2004
    I'm not surprised at all. A lot of places simply don't want spend the funds for a good backup solution. They feel the money is better spent on things like, 50" HDTVs for their conference room or usually what happens is they buy 30 of them. Then 25 of them go sit in a warehouse until they're out of date.
     
    ruffbytes likes this.
  9. DocSavage

    DocSavage 2[H]4U

    Messages:
    2,409
    Joined:
    Dec 18, 2002
    Why would the USAF outsource what sounds to be a fairly simple database anyway? Don't they have internal groups for this?
     
  10. Devilpup

    Devilpup 2[H]4U

    Messages:
    2,048
    Joined:
    Sep 4, 2002
    More like 12 years of lower ranking guys saying "hey, we need to do ____" and upper management saying "we'll get to it when we get to it." DoD has so many different systems and software that the gov spends billions per year just on software licenses, hardware takes a back seat.

    Plus, anyone who's ever worked for the gov knows how difficult it is to get something done. It's not necessarily the fault of the techs and other workers in the trenches, it mostly boils down to budget and approval authorities.
     
  11. Uvaman2

    Uvaman2 2[H]4U

    Messages:
    2,960
    Joined:
    Jan 4, 2016
    Hopefully they will recoup most or all of it, somehow.
    My guess would be the backup issue is probably more about confusion with security measures or something.
    Still, encrypted DVDs in a secured vault maybe?.. I don't know.
    They will get it back, you will see.
     
  12. Decibel

    Decibel 2[H]4U

    Messages:
    3,802
    Joined:
    Nov 10, 2000
    I constantly tell clients: "Your data is only as good as your last backup."

    I don't always follow up with my internal thought: "And yours are shit."
     
  13. wgm3446

    wgm3446 Gawd

    Messages:
    858
    Joined:
    May 8, 2007
  14. amddragonpc

    amddragonpc [H]ard|Gawd

    Messages:
    1,996
    Joined:
    Sep 20, 2012
    I'm sure there is no disaster recover plan either. But, if they do have one, it hasn't been tested in years.
     
  15. Commander Shepard

    Commander Shepard 2[H]4U

    Messages:
    3,501
    Joined:
    Apr 12, 2016
    Exhibit #999,999,999 of government inefficiency, stupidity, laziness, etc. And, that's just for 2016.
     
  16. TwiceOver

    TwiceOver 2[H]4U

    Messages:
    2,401
    Joined:
    Jan 14, 2003
    "Corrupt File" probably means "Ransomware" and they just don't want to admit it.
     
  17. steakman1971

    steakman1971 2[H]4U

    Messages:
    2,433
    Joined:
    Nov 22, 2005
    They should hold someone accountable for this - this is bad stuff.
     
  18. Commander Shepard

    Commander Shepard 2[H]4U

    Messages:
    3,501
    Joined:
    Apr 12, 2016
    PC Matic!!!!!
     
  19. Rustynuts

    Rustynuts [H]ardForum Junkie

    Messages:
    10,370
    Joined:
    Feb 6, 2003
    I bet they didn't want backups due to the extra security needed to protect the backups. Or nobody even noticed the breach so ALL the backups themselves were corrupted.

    The backup system worked perfectly though! Copied the corruption accurately each time.
     
    File likes this.
  20. DocSavage

    DocSavage 2[H]4U

    Messages:
    2,409
    Joined:
    Dec 18, 2002
    Only if you consider using a private company like Lockheed Martin as an example of government stupidity.
     
  21. jardows

    jardows [H]ard|Gawd

    Messages:
    1,478
    Joined:
    Jun 10, 2015
    1. Full backup once a week
    2. Incremental backups other 6 days
    3. Off site storage for backup media

    Simple enough to do, and affordable enough that the small bank I worked for 12 years ago had this strategy implemented. This way the most you lose is 1 weeks worth of data. Not that hard, not that expensive.
     
  22. Vyedmic

    Vyedmic Limp Gawd

    Messages:
    222
    Joined:
    Jul 20, 2007
    I don't know... You can probably find similarities between preferred government contractors like LM and the government itself...
     
  23. BeavermanA

    BeavermanA 2[H]4U

    Messages:
    2,317
    Joined:
    Apr 27, 2006
    Wow. I fret losing even 8 hours of work, let alone 12 fucking years lol. All the software I write, I make a local backup on my hdd, google drive and have emails with it attached. In addition to the office server weekly backup
     
  24. mynamehere

    mynamehere [H]ard|Gawd

    Messages:
    1,764
    Joined:
    Jun 30, 2007
    I had pretty much the same thought.
     
  25. Ur_Mom

    Ur_Mom I'm Not Serious

    Messages:
    19,604
    Joined:
    May 15, 2006
    You need to say that out loud. Some people don't get it. Some people think their 2 year old backup on a DVD is good enough.

    Of course, when people come and go, some things are "just the way they are" or "it's always been done that way". I think some good sysadmins look at backups first thing when they take the job. Because of this reason - you don't want to find out your backups are shit or not happening when you actually NEED them.
     
  26. Krenum

    Krenum [H]ardForum Junkie

    Messages:
    15,314
    Joined:
    Apr 29, 2005
    If these files were so important, why didn't they have hard copies? Idiots.
     
  27. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    10,439
    Joined:
    Jul 16, 2008

    You don't know how right you are.


    I retired in 1998 and immediately went to work for a contractor company that did life cycle support development. Over the many years the names of the software has changes, now software packages have been developed, and now, 18 years later, I am pretty much right back working with many of the same people on the same thing as back before.

    We don't have backups. Not really, not properly. Not ones that would allow the organization to survive a major failure or say a building fire.

    For a long time they didn't have the consolidated storage systems that I manage today. And then these systems were put on line but backups were not really being done. A year ago I started and we identified early on that we had a problem when it came to backups. We ordered tape drives. The drives were not purchased by the customer. We still don't have proper backups and time just marches on and we all know some day there could be a day of reckoning.

    Then again, odds are far more likely that the contract will move off too another company and it won't be my problem any more. Today it is, tomorrow who knows. It's like Russian Roulette.

    But this is indicative of why these things happen.

    The problem isn't really the contractors. The problem is getting the local government guys to make the funds happen and spend the money on the right things.

    It's too tempting to spend it on something that will get you promoted to a different job then to spend it on something like backups and disaster recovery that will not get you promoted unless of course, you roll snake-eyes.
     
    NeoNemesis likes this.
  28. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    10,439
    Joined:
    Jul 16, 2008
    20 years ago the DoD was supposed to go paper-less to save money. In those twenty years they have spent more money on paper then they ever did in the past.
     
    Wizard220 likes this.
  29. Azrak

    Azrak Gawd

    Messages:
    733
    Joined:
    Oct 4, 2015
    Just call up the NSA. I'm sure they have a copy. Or maybe China does, I forget.
     
    Yossarian22 and Wizard220 like this.
  30. Krenum

    Krenum [H]ardForum Junkie

    Messages:
    15,314
    Joined:
    Apr 29, 2005
    lol , zing!
     
  31. krotch

    krotch [H]ardness Supreme

    Messages:
    4,509
    Joined:
    Aug 12, 2004
    The USAF has a lot of very talented Airmen, but none ever get the chance to really work on databases. They usually contract a company to run their databases, so even if the Airmen could learn, they mostly never get a chance to,


    I have been at once location that had crazy contingency. There were backups done and they were stored in a separate building. On top of just that, the SAN had a mirrored SAN next to it. Then all that was replicated to a location 100 miles away using a mirror of that mirrored setup.

    Was definitely not cheap for that setup, nor the support for it.
     
  32. krotch

    krotch [H]ardness Supreme

    Messages:
    4,509
    Joined:
    Aug 12, 2004
    Sounds like my location. No actual backups, but to protect ourselves and our company, we replicate the data to off site locations which have storage. The command seems fine to spend money on storage, but not to spend money on an actual backup solution. We've told them and they were willing to accept the risk that all data could be lost. Obviously, we are trying our best to cover our own asses, cause when the 4 star loses his shit, he'll scream louder at the contractors than the military that actually made the decisions.
     
  33. rigurat

    rigurat Limp Gawd

    Messages:
    262
    Joined:
    Jun 1, 2010
    Someone forgot to run CHKDSK, or dare I say...ScanDisk!
     
  34. krotch

    krotch [H]ardness Supreme

    Messages:
    4,509
    Joined:
    Aug 12, 2004
    Well that's good it got recovered. Hopefully something big like this will lead to them considering better backup and disaster recovery measures. Sadly, it probably won't, except for the location in question.
     
  35. DracoDan

    DracoDan Limp Gawd

    Messages:
    294
    Joined:
    Apr 16, 2002
    Or better yet, dumping Lockheed Martin for a company that isn't top heavy and incompetent as f___!
     
  36. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    10,439
    Joined:
    Jul 16, 2008
    We need terabytes of data backed up. They haven't purchased equipment to do it with and I have additional storage systems sitting that I could set up at another location except that there is no network connectivity out of this building for those development networks meaning they are completely physically isolated. Even local mirroring could be done if they would just tell me how much storage they want for each of the three networks but I can't get these guys do even do that. Simple decisions take months. And the Prime contractor company, (I work for a sub), their leaders are supposed to be helping focus this customer on what we see that the customer needs, but these guys are pretty clueless. If these IT smart businessmen would just focus on fixing all the broken processes like Configuration Control Boards, Policy, etc, everything else fixes itself and get's done. But the processes are broken so the workforce is hamstrung. In stead these fools think they have to get down in our areas and tell us how to do what isn't getting done without realizing that they are the reason we are not doing what they think we should have done already.

    Ask around, this is a wider problem then most realize. You see it less in the business world because businesses that suffer from this don't stay in business. The government on the other hand just creates new business units to do what the other business units are incapable of.
     
  37. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    10,439
    Joined:
    Jul 16, 2008
    It has nothing at all to do with that. Top heavy has nothing to do with it because everything is a contract. The government let's the contract, companies bid, someone wins. The contract has x number of job positions, the company hires new bodies to fill the positions or allows other employees a chance to transfer to them. But essentially each contract is a living isolated thing of it's own. Bigger companies have some corporate overhead, but it's usually pretty minimal. But if you think it's like the company that winds then just starts doing the work with people they keep on hand then you are mistaken. The decision making doesn't go up the ladder very far at all and usually stops at the program manager level. HR has more interaction then the rest of these corporations do within these government contracts.
     
  38. krotch

    krotch [H]ardness Supreme

    Messages:
    4,509
    Joined:
    Aug 12, 2004
    Oh, I understand. I'm a government contractor right now. Luckily our processes aren't broken. Just take long to do anything. Like if I wanted to upgrade an auditing server from 2008 to 2012, I need to write an engineering development plan, an installation development plan. Then I have to get it approved by section leads (most won't have crap to do with what I'm working on). Once that's complete, then I can send in the date I want to do it and that'll need to be approved too. So something that can literally take me 2 days to do, will actually take me a month or more to do.

    Also management here aren't crazy micro-managers. So I'm left alone to get things done.


    Contractors don't tell the government what to do. We give them recommendations and they make the decision. If they don't like the recommendation, we go back and figure out another way of doing it and propose that.

    But yes, each contract has a set number of job positions. To get more job positions requires a contract modification and reason for it. Management positions are few and very rarely are any added on after the original contract win. Don't expect corporate to send more managers, cause without new job positions, then any new ones they send are coming out of corporate money. Not government money. Damn near no contract company is gonna do that. Not without good cause.
     
  39. lcpiper

    lcpiper [H]ardForum Junkie

    Messages:
    10,439
    Joined:
    Jul 16, 2008
    It's the same with me. I can spend days telling them in a paper how I am going to do 30 minutes of work.

    And of course you are correct, the government doesn't work for the contractors, it's the other way around, and even then, the work is supposed to laid out and defined by the Statement of Work, it's not like contractors just have to do whatever it is the customer wants them to do that day. But in cases like ours where the customer doesn't know what to do or even what he wants or needs, it's the contractor leadership that has to guide the customer in the right direction. Ours is not and it's sad. And this is frequently the kind f things that allow events happen, like in this article.

    I mam just wondering which will happen first with me, will we have a catastrophic data loss or will I move on to a new job before it can happen.