Delta Computer Outage Is Just The Start, Say Experts

The Delta incident(s) this week have prompted a number of articles examining the technology powering the airline industry, and most are agreeing that the computer systems employed just don’t cut it and similar incidents may be inevitable. Long story short, they’re ancient, and airliners don’t have the luxury of shutting them down for maintenance.

…the airline industry’s legacy computing systems, 24-hour uptime requirements, and difficulties attracting top technology workers could make preventing future similar outages a major challenge. "The airlines are dealing with a hodgepodge of equipment that’s been cobbled together over the years," says George Hobica, the president of Airfarewatchdog. Industry mergers have meant airlines have interlinked systems, sometimes decades old, from a range of legacy carriers, all without the luxury of ever shutting down their systems for maintenance, he says. While other businesses can occasionally shut down their computer systems for scheduled or even emergency maintenance without a public outcry, airlines simply can’t track passengers, baggage, planes, or crew without their technology systems, he says.

The airline industry has been saying that for years. I suspect that they are not willing to spend $ to invest into making new systems. The technology is there, but it will be expensive. It is not like you flip a switch and then switch over to a new system like that. It should be a slow migration with a solid plan. But, no one wants to be the first to spend that money. Instead it is a patchwork of a system that the management figure "hey it still works! Let's patch it up!"

Yeah this definitely seems more like a case of "won't" instead of "can't." When the changes come I'm guessing those baggage fees and such are going up.

This has been a problem for decades. Many of the problems with the airlines today can be traced back to the Reagan administration and his failed domestic policies. The rest can be blamed on the greed of the airlines. They've been experiencing record profits the last decade, yet their greed as resulted in no significant upgrades to their fleets, computer systems, inventory management, etc. Instead, they've merged with smaller airlines to create bigger monopolies, cut staffing, and blame the extreme wait times only America experiences on government regulation (which is barely there for the airlines), and increases fees and pricing even with record low fuel cost. The entire industry needs a shakeup with government involvement delivering a big kick in the rear to force them into action. The cost of even having the system down for a day should be enough reason for these companies to fork up the money to avoid any unwanted interruptions!

Ridiculous. Not the article, but the airlines moaning. A day one, obvious, requirement for any airlines scheduling, reservations, or billing systems is fault tolerance. They might have started off with a backup system in the same location, but should have switched decades ago to a geographically separate, on-line back-up. As for the whining about stovepipe connected special purpose hardware? Are you kidding me. The upgrade/merger plan is to put in place a server (probably at each major airport) that translates the existing terminal equipment's output to a standard format, and vice-versa. While it is at it, this box can send (and receive) data from redundant servers. Yes, it is a lot of work rolling out something like this, but the yearly savings will start rolling in. (If you cut 25% from passenger check-in times, how many agents don't you need? More to the point, how much shorter will check-in lines get?)

In any case, the airlines had a wake-up call 25 years ago with the Y2K problem. If they had sorted this out then, they would have no problem now. Note that all the airlines use standard formats when interfacing to other airlines. Not using those standards for (internal) record keeping is silly.

Maybe if Delta would have a few ... hubs... of computers around the world, yanno so if you leave from San Francisco all the data of who passengers are are at San Francisco and not at their HQ.

You guys all act like this would be as easy as installing new infrastructure and flipping a switch. They need 24/7 uptime on the systems, look what happens from a brief power outage. This is a decade long project and to take an endeavor like that you wait until absolutely necessary. Spending a cash outlay that large for the sake of it would be foolish.

Chunder said:

This has been a problem for decades. Many of the problems with the airlines today can be traced back to the Reagan administration and his failed domestic policies.

Click to expand...

Ah, we have a fiction author on site.

This is what happens when businesses treat IT as a cost center. You should invest in your IT department, because it's critical for keeping your business operating.

wizdum said:

This is what happens when businesses treat IT as a cost center. You should invest in your IT department, because it's critical for keeping your business operating.

Click to expand...

EVERY business outside of the tech industry is like this. IT is a cost center and generally not valued by the bean counters. Ranges from programmers, support, upgrading, software, you name it.

I see it all over. The bigger the company and infrastructure the worse it is.

First step to fixing this problem. Batteries and backup generators. It is amazing how that fixes issues with power loss.

Terpfen said:

Ah, we have a fiction author on site.

Click to expand...

Terpfen, research what Reagan did to the airline industry and his anti-worker policies during the airline strikes. Many of the problems with the industry today go all the way back to the laws and government/corporate overreach he instituted. Once you research what he put forward, and get informed on the effects of those policies, you will see what a terrible president he was. Furthermore, many of todays problems, not just with the airline industry, are a result of his corporate bought policies. Rather the failed war on drugs and for-profit prisons, trickle down fairy tale, the deregulations that resulted in big banks, big oil, big telecomm, big ag, etc forming and buying our government. His policies were at the root of it.

Chunder said:

Terpfen, research what Reagan did to the airline industry and his anti-worker policies during the airline strikes.

Click to expand...

I'm fully aware, which is why I declare you an author of fiction. And take the half-baked political analogies somewhere else.

They bitch and moan but don't pony up the money to the people who can help...

This article was just not correct (in the case of Delta) when it said good techs are hard to find and rather work for Facebook & Google. Delta outsourced their IT to IBM from what I understand. IBM'ers in the know said after a year the senior and experienced techs where let go. Most of the techs for these systems are now in India and the very few onsite techs are probably not as knowledgeable as the techs they replaced. Anyone who works in company with a few dozen servers, mainframes, etc knows people need to be onsite and they need to know the systems inside out. Having someone onsite that doesn't know the system could do more harm than good.

For fux sake IT infrastructure and staffing is not where Delta should be cutting corners. Maybe the execs can decline their ridiculous multi-million dollar bonuses in favor of reinvesting in American workers, generators, UPS, backup systems, mirrored data centers in different parts of the United States. Enough Monday morning quarterbacking.

Outsourced IT probably hurt Delta Airlines when their power went out - I, Cringely

Lemme get this straight...
Airlines don't want to upgrade their systems because of the downtime it would cause,
but...
When their systems go down due to their out of date infrastructure/lack of redundancy, they still say their systems went down because they're out
Hmm...

LFaWolf said:

The airline industry has been saying that for years. I suspect that they are not willing to spend $ to invest into making new systems. The technology is there, but it will be expensive. It is not like you flip a switch and then switch over to a new system like that. It should be a slow migration with a solid plan. But, no one wants to be the first to spend that money. Instead it is a patchwork of a system that the management figure "hey it still works! Let's patch it up!"

Click to expand...

Only to a point. There are different aspects to the whole thing. 1st on the list is the database element. I am sure the current systems rely on perhaps dozens of different databases. The new system would need to have everything the others currently have and some more to bring them up to date. Truth is, if they are running say a dozen databases, imagine how much you could cut out by consolidation, all the common fields you could drop. A dozen databases that all have common fields, those common fields represent a lot of waste. But once you get the "back end" on line, then you just cut over the front end workstations and terminals. After that you can address the front end site by site with full tech refreshes one location at a time. Getting the back end up and running concurrently and making the cut over, that's the tricky part.

I think this is certainly a case for centralized databases and Oracle FTW.

lcpiper said:

Only to a point. There are different aspects to the whole thing. 1st on the list is the database element. I am sure the current systems rely on perhaps dozens of different databases. The new system would need to have everything the others currently have and some more to bring them up to date. Truth is, if they are running say a dozen databases, imagine how much you could cut out by consolidation, all the common fields you could drop. A dozen databases that all have common fields, those common fields represent a lot of waste. But once you get the "back end" on line, then you just cut over the front end workstations and terminals. After that you can address the front end site by site with full tech refreshes one location at a time. Getting the back end up and running concurrently and making the cut over, that's the tricky part.

I think this is certainly a case for centralized databases and Oracle FTW.

Click to expand...

Oracle ftw? no, take that back right now.

Chunder said:

Terpfen, research what Reagan did to the airline industry and his anti-worker policies during the airline strikes. Many of the problems with the industry today go all the way back to the laws and government/corporate overreach he instituted. Once you research what he put forward, and get informed on the effects of those policies, you will see what a terrible president he was. Furthermore, many of todays problems, not just with the airline industry, are a result of his corporate bought policies. Rather the failed war on drugs and for-profit prisons, trickle down fairy tale, the deregulations that resulted in big banks, big oil, big telecomm, big ag, etc forming and buying our government. His policies were at the root of it.

Click to expand...

You mean not sit idly bye while a public sector union held the country's air travel hostage. What they did would be the equivalent to if they UAW went on strike everyone's car stopped running. Imagine the kind of money they could extort so people could get moving again. And you know what, they struck and it didn't affect a fucking thing. They have nothing to do with scheduling, anyway. They were a redundant safety feature if that.

The decline of air travel quality happened in the merger happy Clinton 90's combined with one other thing compounding the Clinton monopolization era, consumers picking the cheapest flight that meets their schedule. You get what you pay for + no choice.

Below is why the experience is shit.