Computer Outage Causes Chaos, Backups At Airports

[HardOCP News]

I'm not going to say it was the Russians but, since they are being blamed for every hack in the past 100 years, it was the Russians. In all seriousness, we don't realize how much we depend on computers these days until one breaks down.

“CBP took immediate action to address the issue and CBP officers continued to process international travelers using alternative procedures at airports experiencing the disruption,” CBP said. “Travelers at some ports of entry experienced longer than usual wait times as CBP officers processed travelers as quickly as possible while maintaining the highest levels of security.”

 

[lcpiper]

Someone had a time sync problem maybe?

I can't help it, any time I see a big problem happen in systems that accessed storage systems and databases I always at least have to run it through the "failed time sync" scenario. It's such a simple thing and it can cause such huge problems.

Still, holiday time frame can simply be a failure in routine maintenance or it was just Murphy at work.

 

[amddragonpc]

Probably that extra second that was added to the end of the year in 2016.

 

[Lakados]

It was more likely that the systems they all use are ancient, and many have not had any work done outside emergency maintenance in years. I have a friend who works in IT for an airline and their main server is pushing 15 years old and it is not allowed to be rebooted because if turned off it will cause outages across a dozen airports and in the event it does get turned off it takes an hour or more for systems to come back up. The software they use runs something archaic, I don't remember the exact language but I think it was RPG or something of that nature, they have been begging to upgrade and replace systems but the costs involved are so large due to the complexity of the port the higher ups keep saying no.

 

[lcpiper]

It was more likely that the systems they all use are ancient, and many have not had any work done outside emergency maintenance in years. I have a friend who works in IT for an airline and their main server is pushing 15 years old and it is not allowed to be rebooted because if turned off it will cause outages across a dozen airports and in the event it does get turned off it takes an hour or more for systems to come back up. The software they use runs something archaic, I don't remember the exact language but I think it was RPG or something of that nature, they have been begging to upgrade and replace systems but the costs involved are so large due to the complexity of the port the higher ups keep saying no.

It's certainly possible, but government systems are not always so restricted when it comes to modernization. It's not impossible, but I think it's improbable. But a system like what the CBP uses accesses centralized databases for records and data like finger print data. Having worked for such organizations in the past, when up-time is critical, maintenance downtime is hard to get and usually maintenance activities suffer. It also means that when they do get down-time to do something, sometimes they experience problems, or try to do too much at once, making too many changes. Then when they do have something go wrong, they have a harder time isolating what went wrong, who did it, and what has to happen to correct it.

Currently, we are trying to build up a lab that is a close replica of our production environment so that we can "practice" and work through things in the lab first, instead of trying to do everything directly on the production systems without a chance to find those "things" you didn't expect.

Another thing we keep having to fight is our own corporate "good idea faeries". As the administrators and engineers, we set it all up and make it all work. But we have these pesky manager types that keep trying to force change and the adoption of newer technologies that will add complexity for little value. The newest "flavor of the day" technology is a burden on our back like no other. This is exactly what the company who was the previous contractor company here did so wrong. They would implement new software and systems just because they wanted to work with them cause they were the new thing. They basically took advantage of the customer and sold him what he didn't need, just so they could make their own resumes look better. We spent over a year just un-fucking their bullshit and getting this environment stable and we still don't have all the bull shit weeded out of it.

Such are the nature of IT sins.

 

[Lakados]

It's certainly possible, but government systems are not always so restricted when it comes to modernization. It's not impossible, but I think it's improbable. But a system like what the CBP uses accesses centralized databases for records and data like finger print data. Having worked for such organizations in the past, when up-time is critical, maintenance downtime is hard to get and usually maintenance activities suffer. It also means that when they do get down-time to do something, sometimes they experience problems, or try to do too much at once, making too many changes. Then when they do have something go wrong, they have a harder time isolating what went wrong, who did it, and what has to happen to correct it.

Currently, we are trying to build up a lab that is a close replica of our production environment so that we can "practice" and work through things in the lab first, instead of trying to do everything directly on the production systems without a chance to find those "things" you didn't expect.

Another thing we keep having to fight is our own corporate "good idea faeries". As the administrators and engineers, we set it all up and make it all work. But we have these pesky manager types that keep trying to force change and the adoption of newer technologies that will add complexity for little value. The newest "flavor of the day" technology is a burden on our back like no other. This is exactly what the company who was the previous contractor company here did so wrong. They would implement new software and systems just because they wanted to work with them cause they were the new thing. They basically took advantage of the customer and sold him what he didn't need, just so they could make their own resumes look better. We spent over a year just un-fucking their bullshit and getting this environment stable and we still don't have all the bull shit weeded out of it.

Such are the nature of IT sins.

Maybe, but a little bit of research shows the Computer that failed was purchased and installed in the 1970's, They are trying to replace it but it is taking years and like all government jobs seems to be behind schedule and over budget and prone to failure. But it's Lockheed Martin, working on the new system (and has been since 2002) so I am sure it will be up one day.

The main problem seems to be that the FAA is also in charge of the upgrade but that causes regulatory conflicts so red tape doesn't help the process any either.

 

[lcpiper]

Maybe, but a little bit of research shows the Computer that failed was purchased and installed in the 1970's, They are trying to replace it but it is taking years and like all government jobs seems to be behind schedule and over budget and prone to failure. But it's Lockheed Martin, working on the new system (and has been since 2002) so I am sure it will be up one day.

The main problem seems to be that the FAA is also in charge of the upgrade but that causes regulatory conflicts so red tape doesn't help the process any either.

links, links, I like to read too

 

[Spidey329]

It was more likely that the systems they all use are ancient, and many have not had any work done outside emergency maintenance in years. I have a friend who works in IT for an airline and their main server is pushing 15 years old and it is not allowed to be rebooted because if turned off it will cause outages across a dozen airports and in the event it does get turned off it takes an hour or more for systems to come back up. The software they use runs something archaic, I don't remember the exact language but I think it was RPG or something of that nature, they have been begging to upgrade and replace systems but the costs involved are so large due to the complexity of the port the higher ups keep saying no.

When I worked IT in the casino industry, some of the machines were from the early 90's. We're talking Point of Sale and billboard controllers. When one broke, it became a fiasco trying to scrounge parts from other properties on the strip. It was even worse because store managers would be constantly on you about when their registers would be back up.

Companies are cheap and it takes a large amount of forward thinking / bureaucratic pushing to get management to realize that putting $$ into a replacement will save in the long term. Older systems were designed around their specific hardware. Modern systems are more virtualized. They don't realize that it's better to preemptively act to bring up a replacement in a test environment (before roll out) versus an all-hands situation trying to duck tape the old equipment together when it crashes.

 

[captaindiptoad]

In situations like this i just hang back and relax. In Virginia the tunnels get closed all the time because someone wrecks in them, i jut get off the road and take a 3-4 hour nap.