USAF Lost 12 Years Of Investigations Thanks To A Corrupted File

H

HardOCP News

[H] News
Joined
Dec 31, 1969
Messages
0
This story serves as a reminder that you should regularly back up your data. It is hard to believe that the U.S. Air Force Inspector General and Lockheed Martin didn't use common sense when it came to extremely important data like this.

Defense One and others are reporting that the USAF lost its records of 100,000 investigations in their Automated Case Tracking System due to some kind of file corruption that neither Lockheed Martin or the Air Force have been able to figure out. Apparently Lockheed Martin spent two weeks trying to fix the problem before informing the Air Force, but had to own up to their failure when the issue proved insurmountable. The Air Force was said to have been informed on June 6th, and has since requested the help of the Pentagon’s cybersecurity operators as well as private contractors.
 
Last edited:
Thats a lot of complacent workers and managers over 12 years to not realise the lack of data integrity mechanisms.
Makes you wonder how much "they" need investigating.
 
hA107BA65.jpg
 
I wonder who was being investigate when this "accident" happened....
 
I'm not surprised at all. A lot of places simply don't want spend the funds for a good backup solution. They feel the money is better spent on things like, 50" HDTVs for their conference room or usually what happens is they buy 30 of them. Then 25 of them go sit in a warehouse until they're out of date.
 
Why would the USAF outsource what sounds to be a fairly simple database anyway? Don't they have internal groups for this?
 
Nenu said:
Thats a lot of complacent workers and managers over 12 years to not realise the lack of data integrity mechanisms.
*snip*
Click to expand...

More like 12 years of lower ranking guys saying "hey, we need to do ____" and upper management saying "we'll get to it when we get to it." DoD has so many different systems and software that the gov spends billions per year just on software licenses, hardware takes a back seat.

Plus, anyone who's ever worked for the gov knows how difficult it is to get something done. It's not necessarily the fault of the techs and other workers in the trenches, it mostly boils down to budget and approval authorities.
 
Hopefully they will recoup most or all of it, somehow.
My guess would be the backup issue is probably more about confusion with security measures or something.
Still, encrypted DVDs in a secured vault maybe?.. I don't know.
They will get it back, you will see.
 
I constantly tell clients: "Your data is only as good as your last backup."

I don't always follow up with my internal thought: "And yours are shit."
 
I'm sure there is no disaster recover plan either. But, if they do have one, it hasn't been tested in years.
 
Exhibit #999,999,999 of government inefficiency, stupidity, laziness, etc. And, that's just for 2016.
 
I bet they didn't want backups due to the extra security needed to protect the backups. Or nobody even noticed the breach so ALL the backups themselves were corrupted.

The backup system worked perfectly though! Copied the corruption accurately each time.
 
  • Like
Reactions: File
like this
1. Full backup once a week
2. Incremental backups other 6 days
3. Off site storage for backup media

Simple enough to do, and affordable enough that the small bank I worked for 12 years ago had this strategy implemented. This way the most you lose is 1 weeks worth of data. Not that hard, not that expensive.
 
DocSavage said:
Only if you consider using a private company like Lockheed Martin as an example of government stupidity.
Click to expand...

I don't know... You can probably find similarities between preferred government contractors like LM and the government itself...
 
Wow. I fret losing even 8 hours of work, let alone 12 fucking years lol. All the software I write, I make a local backup on my hdd, google drive and have emails with it attached. In addition to the office server weekly backup
 
Decibel said:
I constantly tell clients: "Your data is only as good as your last backup."

I don't always follow up with my internal thought: "And yours are shit."
Click to expand...

You need to say that out loud. Some people don't get it. Some people think their 2 year old backup on a DVD is good enough.

Of course, when people come and go, some things are "just the way they are" or "it's always been done that way". I think some good sysadmins look at backups first thing when they take the job. Because of this reason - you don't want to find out your backups are shit or not happening when you actually NEED them.
 
If these files were so important, why didn't they have hard copies? Idiots.
 
NeoNemesis said:
Bah, it's only taxpayer money.
Click to expand...


You don't know how right you are.


I retired in 1998 and immediately went to work for a contractor company that did life cycle support development. Over the many years the names of the software has changes, now software packages have been developed, and now, 18 years later, I am pretty much right back working with many of the same people on the same thing as back before.

We don't have backups. Not really, not properly. Not ones that would allow the organization to survive a major failure or say a building fire.

For a long time they didn't have the consolidated storage systems that I manage today. And then these systems were put on line but backups were not really being done. A year ago I started and we identified early on that we had a problem when it came to backups. We ordered tape drives. The drives were not purchased by the customer. We still don't have proper backups and time just marches on and we all know some day there could be a day of reckoning.

Then again, odds are far more likely that the contract will move off too another company and it won't be my problem any more. Today it is, tomorrow who knows. It's like Russian Roulette.

But this is indicative of why these things happen.

The problem isn't really the contractors. The problem is getting the local government guys to make the funds happen and spend the money on the right things.

It's too tempting to spend it on something that will get you promoted to a different job then to spend it on something like backups and disaster recovery that will not get you promoted unless of course, you roll snake-eyes.
 
DocSavage said:
Why would the USAF outsource what sounds to be a fairly simple database anyway? Don't they have internal groups for this?
Click to expand...

The USAF has a lot of very talented Airmen, but none ever get the chance to really work on databases. They usually contract a company to run their databases, so even if the Airmen could learn, they mostly never get a chance to,


jardows said:
1. Full backup once a week
2. Incremental backups other 6 days
3. Off site storage for backup media

Simple enough to do, and affordable enough that the small bank I worked for 12 years ago had this strategy implemented. This way the most you lose is 1 weeks worth of data. Not that hard, not that expensive.
Click to expand...

I have been at once location that had crazy contingency. There were backups done and they were stored in a separate building. On top of just that, the SAN had a mirrored SAN next to it. Then all that was replicated to a location 100 miles away using a mirror of that mirrored setup.

Was definitely not cheap for that setup, nor the support for it.
 
lcpiper said:
You don't know how right you are.


I retired in 1998 and immediately went to work for a contractor company that did life cycle support development. Over the many years the names of the software has changes, now software packages have been developed, and now, 18 years later, I am pretty much right back working with many of the same people on the same thing as back before.

We don't have backups. Not really, not properly. Not ones that would allow the organization to survive a major failure or say a building fire.

For a long time they didn't have the consolidated storage systems that I manage today. And then these systems were put on line but backups were not really being done. A year ago I started and we identified early on that we had a problem when it came to backups. We ordered tape drives. The drives were not purchased by the customer. We still don't have proper backups and time just marches on and we all know some day there could be a day of reckoning.

Then again, odds are far more likely that the contract will move off too another company and it won't be my problem any more. Today it is, tomorrow who knows. It's like Russian Roulette.

But this is indicative of why these things happen.

The problem isn't really the contractors. The problem is getting the local government guys to make the funds happen and spend the money on the right things.

It's too tempting to spend it on something that will get you promoted to a different job then to spend it on something like backups and disaster recovery that will not get you promoted unless of course, you roll snake-eyes.
Click to expand...

Sounds like my location. No actual backups, but to protect ourselves and our company, we replicate the data to off site locations which have storage. The command seems fine to spend money on storage, but not to spend money on an actual backup solution. We've told them and they were willing to accept the risk that all data could be lost. Obviously, we are trying our best to cover our own asses, cause when the 4 star loses his shit, he'll scream louder at the contractors than the military that actually made the decisions.
 
Well that's good it got recovered. Hopefully something big like this will lead to them considering better backup and disaster recovery measures. Sadly, it probably won't, except for the location in question.
 
krotch said:
Well that's good it got recovered. Hopefully something big like this will lead to them considering better backup and disaster recovery measures. Sadly, it probably won't, except for the location in question.
Click to expand...

Or better yet, dumping Lockheed Martin for a company that isn't top heavy and incompetent as f___!
 
krotch said:
Sounds like my location. No actual backups, but to protect ourselves and our company, we replicate the data to off site locations which have storage. The command seems fine to spend money on storage, but not to spend money on an actual backup solution. We've told them and they were willing to accept the risk that all data could be lost. Obviously, we are trying our best to cover our own asses, cause when the 4 star loses his shit, he'll scream louder at the contractors than the military that actually made the decisions.
Click to expand...

We need terabytes of data backed up. They haven't purchased equipment to do it with and I have additional storage systems sitting that I could set up at another location except that there is no network connectivity out of this building for those development networks meaning they are completely physically isolated. Even local mirroring could be done if they would just tell me how much storage they want for each of the three networks but I can't get these guys do even do that. Simple decisions take months. And the Prime contractor company, (I work for a sub), their leaders are supposed to be helping focus this customer on what we see that the customer needs, but these guys are pretty clueless. If these IT smart businessmen would just focus on fixing all the broken processes like Configuration Control Boards, Policy, etc, everything else fixes itself and get's done. But the processes are broken so the workforce is hamstrung. In stead these fools think they have to get down in our areas and tell us how to do what isn't getting done without realizing that they are the reason we are not doing what they think we should have done already.

Ask around, this is a wider problem then most realize. You see it less in the business world because businesses that suffer from this don't stay in business. The government on the other hand just creates new business units to do what the other business units are incapable of.
 
DracoDan said:
Or better yet, dumping Lockheed Martin for a company that isn't top heavy and incompetent as f___!
Click to expand...

It has nothing at all to do with that. Top heavy has nothing to do with it because everything is a contract. The government let's the contract, companies bid, someone wins. The contract has x number of job positions, the company hires new bodies to fill the positions or allows other employees a chance to transfer to them. But essentially each contract is a living isolated thing of it's own. Bigger companies have some corporate overhead, but it's usually pretty minimal. But if you think it's like the company that winds then just starts doing the work with people they keep on hand then you are mistaken. The decision making doesn't go up the ladder very far at all and usually stops at the program manager level. HR has more interaction then the rest of these corporations do within these government contracts.
 
lcpiper said:
We need terabytes of data backed up. They haven't purchased equipment to do it with and I have additional storage systems sitting that I could set up at another location except that there is no network connectivity out of this building for those development networks meaning they are completely physically isolated. Even local mirroring could be done if they would just tell me how much storage they want for each of the three networks but I can't get these guys do even do that. Simple decisions take months. And the Prime contractor company, (I work for a sub), their leaders are supposed to be helping focus this customer on what we see that the customer needs, but these guys are pretty clueless. If these IT smart businessmen would just focus on fixing all the broken processes like Configuration Control Boards, Policy, etc, everything else fixes itself and get's done. But the processes are broken so the workforce is hamstrung. In stead these fools think they have to get down in our areas and tell us how to do what isn't getting done without realizing that they are the reason we are not doing what they think we should have done already.

Ask around, this is a wider problem then most realize. You see it less in the business world because businesses that suffer from this don't stay in business. The government on the other hand just creates new business units to do what the other business units are incapable of.
Click to expand...

Oh, I understand. I'm a government contractor right now. Luckily our processes aren't broken. Just take long to do anything. Like if I wanted to upgrade an auditing server from 2008 to 2012, I need to write an engineering development plan, an installation development plan. Then I have to get it approved by section leads (most won't have crap to do with what I'm working on). Once that's complete, then I can send in the date I want to do it and that'll need to be approved too. So something that can literally take me 2 days to do, will actually take me a month or more to do.

Also management here aren't crazy micro-managers. So I'm left alone to get things done.


lcpiper said:
It has nothing at all to do with that. Top heavy has nothing to do with it because everything is a contract. The government let's the contract, companies bid, someone wins. The contract has x number of job positions, the company hires new bodies to fill the positions or allows other employees a chance to transfer to them. But essentially each contract is a living isolated thing of it's own. Bigger companies have some corporate overhead, but it's usually pretty minimal. But if you think it's like the company that winds then just starts doing the work with people they keep on hand then you are mistaken. The decision making doesn't go up the ladder very far at all and usually stops at the program manager level. HR has more interaction then the rest of these corporations do within these government contracts.
Click to expand...

Contractors don't tell the government what to do. We give them recommendations and they make the decision. If they don't like the recommendation, we go back and figure out another way of doing it and propose that.

But yes, each contract has a set number of job positions. To get more job positions requires a contract modification and reason for it. Management positions are few and very rarely are any added on after the original contract win. Don't expect corporate to send more managers, cause without new job positions, then any new ones they send are coming out of corporate money. Not government money. Damn near no contract company is gonna do that. Not without good cause.
 
krotch said:
Oh, I understand. I'm a government contractor right now. Luckily our processes aren't broken. Just take long to do anything. Like if I wanted to upgrade an auditing server from 2008 to 2012, I need to write an engineering development plan, an installation development plan. Then I have to get it approved by section leads (most won't have crap to do with what I'm working on). Once that's complete, then I can send in the date I want to do it and that'll need to be approved too. So something that can literally take me 2 days to do, will actually take me a month or more to do.

Also management here aren't crazy micro-managers. So I'm left alone to get things done.




Contractors don't tell the government what to do. We give them recommendations and they make the decision. If they don't like the recommendation, we go back and figure out another way of doing it and propose that.

But yes, each contract has a set number of job positions. To get more job positions requires a contract modification and reason for it. Management positions are few and very rarely are any added on after the original contract win. Don't expect corporate to send more managers, cause without new job positions, then any new ones they send are coming out of corporate money. Not government money. Damn near no contract company is gonna do that. Not without good cause.
Click to expand...

It's the same with me. I can spend days telling them in a paper how I am going to do 30 minutes of work.

And of course you are correct, the government doesn't work for the contractors, it's the other way around, and even then, the work is supposed to laid out and defined by the Statement of Work, it's not like contractors just have to do whatever it is the customer wants them to do that day. But in cases like ours where the customer doesn't know what to do or even what he wants or needs, it's the contractor leadership that has to guide the customer in the right direction. Ours is not and it's sad. And this is frequently the kind f things that allow events happen, like in this article.

I mam just wondering which will happen first with me, will we have a catastrophic data loss or will I move on to a new job before it can happen.
 
You must log in or register to reply here.
Back
Top