Google Stadia Port Troubles Blamed on the Linux Kernel Scheduler

Auer

[H]ard|Gawd
Joined
Nov 2, 2018
Messages
1,972
https://www.tomshardware.com/news/google-stadia-port-issues-linux-kernel

If anyone cares more about millisecond-long delays than gamers, it's developers. They know a millisecond can make a big difference in how a game plays. That's bad news for Google Stadia because devs recently claimed an issue with the Linux kernel scheduler can lead to issues in games ported to the platform.

A developer named Malte Skarupke publicized the problem on Monday. Skarupke explained how he became aware of the issue and his efforts to address it in a blog post (shout-out to Phoronix for spotting the post).
 
I have no idea what a spinlock or mutex is (can someone explain?) but if something like that is causing issues then hopefully having the information out there will help other developers struggling with Stadia ports.
 
I have no idea what a spinlock or mutex is (can someone explain?) but if something like that is causing issues then hopefully having the information out there will help other developers struggling with Stadia ports.

Probably not, most likely it will result in developers say "Ya this is not worth our time." Stadia does not look to be catching the world on fire, so a Stadia port is not likely to make you a lot of money. That means they port needs to be easy to do, preferably zero effort. The more work it takes to make it work well, the more it costs and thus the less reason there is to do it.

Really had Google been smart, Stadia would have run things on Windows on the back end, at least to start. Yes, that means paying for Windows licenses but given that Windows is the platform with the most PC games, and almost always the primary platform for testing and developing PC games, it would make things much easier. If porting to Stadia was as simple as just adding in some calls to a library, like Steam, then you might see a lot more people hopping on board. Even if they didn't expect to make much money, if they effort was almost nothing then why not? However given that not only do they have to port the game to Linux, which many games don't bother to do, particularly AAA titles, but it sounds like there are additional gremlins to deal with... Well that just makes it much harder to convince people to do and leads to the weak launch lineup.

That also risks creating a feedback loop: Nobody buys Stadia because there are few games to play on it. Since nobody is buying it, developers don't want to bother supporting it because there aren't a lot of users. Since developers aren't supporting it, nobody buys it and so on.
 
Spinlocks and mutexes are ways to make sure only one thread at a time accesses something (might be a piece of memory or a file, for example.) This is a simplification, but a spinlock is s loop that constantly says "can I have it yet?" whereas a mutex usually is more like asking the OS to wake you up when you can have it.

I don't know enough about how you use them in Linux to know if it's easy to swap from one to the other or not--but one reason you have two ways to do something is that they're for different use cases, so it could be that in some cases switching from one to the other could have other side effects.
 
So Stadia is like buying yourself a Steam Machine, except without the fallback to Steam on Windows or OSX.

How did they not see problems culminating with a closed platform like that? Ports are likely to be rushed and broken, resulting in increased latency.
 
Spinlocks and mutexes are used in software design to synchronize access to a piece of critical code. For example, you typically don't want two threads writing to the same memory location (object) at the same time. That's an over simplified example (real world examples are far more complicated), but the easiest to describe and understand.

So a thread acquires a lock/mutex and enters a critical section (piece of code). Until that thread releases the lock, all other threads attempting to acquire the same lock must wait.

The mechanism by how threads wait is often implementation defined. Typically, the thread is put to sleep thus allowing other threads to run on the CPU (typically the OS handles this schedule). But in practice always putting threads to sleep is often not efficient when the critical section itself is short lived (handful of cpu cycles).

So most modern implementors use a combination of spinning and thread sleep. Spinning is just as it sounds, it's a loop that does no work. A thread that wants to acquire a lock already acquired will spin briefly, then if the lock is still unavailable, the thread will sleep. It will periodically be awaken (again typically via the OS) and the lock acquisition process will begin again. Wash rinse repeat until the lock is acquired.

The tldr;

Spinlocks are an implementation. Mutex is an abstraction. In fact, a Mutex can be implemented with a spinlock.

Both are synchronization primitives used in software design.
 
Last edited:
So Stadia is like buying yourself a Steam Machine, except without the fallback to Steam on Windows or OSX.

How did they not see problems culminating with a closed platform like that? Ports are likely to be rushed and broken, resulting in increased latency.

Its not a closed system.... sort of the opposite.

Games in general don't care about the kernel. Really they shouldn't even be fucking talking to the kernel. Man Game developers are just bad.

Having said that... the main reason Stadia will flounder for a long time. (and knowing Google they probably won't wait long enough for it to survive) Is that most of the big game developers have ambitions beyond just being game developers. Lets all be honest all the big game developers have dreams of being the netflix of gaming themselves down the road. It is the reason something like Hulu in the video market is also doomed... all of the partners in that want their own VideoZ+ service. lol
 
Ok. Sure... You know that Azure, the #2 cloud platform in the world, runs on Windows, right?

https://www.zdnet.com/article/linux-now-dominates-azure/

It has windows as an option anyway. (and lets be clear if you believe games will run better on a windows server VM your wrong) Windows server is not built to be customized for something like Stadia, it has shit options for large scale shared file systems.... its a terrible Server OS which is why its dying. And for the most part MS has given up saving it... instead they have are porting as many major MS backend bits to Linux as possible. No doubt they are slowly winding windows server down.
 
Its not a closed system.... sort of the opposite.

Games in general don't care about the kernel. Really they shouldn't even be fucking talking to the kernel. Man Game developers are just bad.

Having said that... the main reason Stadia will flounder for a long time. (and knowing Google they probably won't wait long enough for it to survive) Is that most of the big game developers have ambitions beyond just being game developers. Lets all be honest all the big game developers have dreams of being the netflix of gaming themselves down the road. It is the reason something like Hulu in the video market is also doomed... all of the partners in that want their own VideoZ+ service. lol


No, it's a "closed system" if you're only targeting a Stadia at a single Linux config. Even if it's an open-source kernel, you've locked the platform down just like Android.

I'm simply comparing it to the success of Steam form supporting multiple operating systems, and poking fun. Google is simultaneously redefining what a game server does, and also porting the majority of these games to a new OS; they're guaranteed to have issues.
 
No, it's a "closed system" if you'e only targeting a Stadia at a single Linux config. Even if it's an open-source kernel, you've locked the platform down just like Android.

I'm simply comparing it to the success of Steam form supporting multiple operating systems, and poking fun.

Got ya. :)

I was conflating your post and the one suggesting they should have used windows server on the backend. lol Windows server is closed no way MS would let google poke around their code and build all the needed tools.... never mind that MS is going to try and launch their own game streaming service at some point. (and it will also run on a Linux kernel no doubt)
 
Got ya. :)

I was conflating your post and the one suggesting they should have used windows server on the backend. lol Windows server is closed no way MS would let google poke around their code and build all the needed tools.... never mind that MS is going to try and launch their own game streaming service at some point. (and it will also run on a Linux kernel no doubt)

MS’s xCloud streaming service seems to run on something that runs Xbox games. It’s likely a modified version of the Xbox OS.
 
MS’s xCloud streaming service seems to run on something that runs Xbox games. It’s likely a modified version of the Xbox OS.

The main advantage they have is their closed xbox apis. I have no doubt xcloud is Linux running behind the scenes. (as all MS cloud stuff is... its not like Azure is running windows server on the back end) But unlike google those bits will be better hidden from developers... allowing them to just use their existing xbox ports. MS and Sony both have an advantage, in terms of easy game porting. With Google its like they launched a new console when everyone else is 4 or 5 generations in already.

Ya the streaming wars are just heating up should be fun. Main reason I'm sure google isn't catching fire right now isn't cause its hard to port to... and more developers are thinking about both, their own services. Or that in a few years there will be Sony MS Google and perhaps amazon and a few other players, all of who they will be able to play off and get the big $ exclusive deals again like they did at the height of the console wars. Why commit to anything till you know how much you can squeeze in up front payouts. Same thing happened in consoles years back... the dreamcast comes to mind. They where first to market and had a pretty good system but developers mostly hung back knowing in a year they would have a new Sony box to burn their IPs on.
 
The article is basically full of flaws and has been discussed and largely debunked.

As evidenced by the plethora of benchmarks out there, the Linux scheduler is better than the Windows scheduler, especially where NUMA is concerned. Scheduling policy is set per thread under Linux and if you use spinlocks to essentially push a CPU to saturate CPU cycles halting a thread than you're not following basic principles related to Linux:

https://news.ycombinator.com/item?id=21919988

I've yet to find a single benchmark that actually supports the article's claims, however I can find benchmarks that highlight that in a direct apples to apples comparison the Linux scheduler appears to perform better. In this benchmark the content creator actually had real trouble capturing real time video for their comparative review under Windows as the game keeps 'stalling' in the Windows Vulkan vs Linux Vulkan part of the review - In fact the FPS was all over the place under WIndows, indicative of scheduler issues to me:



The latest Linux DXVK vs native Windows Detroit: Become Human review shows Linux to be running rings around Windows, Windows can't even fully utilize the GPU:

 
Last edited:
The main advantage they have is their closed xbox apis. I have no doubt xcloud is Linux running behind the scenes. (as all MS cloud stuff is... its not like Azure is running windows server on the back end) But unlike google those bits will be better hidden from developers... allowing them to just use their existing xbox ports. MS and Sony both have an advantage, in terms of easy game porting. With Google its like they launched a new console when everyone else is 4 or 5 generations in already.

Ya the streaming wars are just heating up should be fun. Main reason I'm sure google isn't catching fire right now isn't cause its hard to port to... and more developers are thinking about both, their own services. Or that in a few years there will be Sony MS Google and perhaps amazon and a few other players, all of who they will be able to play off and get the big $ exclusive deals again like they did at the height of the console wars. Why commit to anything till you know how much you can squeeze in up front payouts. Same thing happened in consoles years back... the dreamcast comes to mind. They where first to market and had a pretty good system but developers mostly hung back knowing in a year they would have a new Sony box to burn their IPs on.

The streaming wars will be interesting going forward. Sony is even running PS Now off Azure so it looks like MS is in a good position.

The article is basically full of flaws and has been discussed and largely debunked.

As evidenced by the plethora of benchmarks out there, the Linux scheduler is better than the Windows scheduler, especially where NUMA is concerned. Scheduling policy is set per thread under Linux and if you use spinlocks to essentially push a CPU to saturate CPU cycles to halt a thread than you're not following basic principles related to Linux:

https://news.ycombinator.com/item?id=21919988

I've yet to find a single benchmark that actually supports the article's claims, however I can find benchmarks that highlight that in a direct apples to apples comparison the Linux scheduler appears to perform better. In this benchmark the content creator actually had real trouble capturing real time video for their comparative review under Windows as the game keeps 'stalling' in the Windows Vulkan vs Linux Vulkan part of the review - In fact the FPS was all over the place under WIndows, indicative of scheduler issues to me:



The latest Linux DXVK vs native Windows Detroit: Become Human review shows Linux to be running rings around Windows, Windows can't even fully utilize the GPU:



The information comes straight from a developer. I imagine they know what they’re talking about. You don’t need to get so worked up over everything that is mildly negative regarding Linux. Also, we don’t know if Google made any changes to the kernel or anything else. They’re probably using a custom distro so who knows what they messed with. No one was saying this is a problem with Linux in general, chill.
 
The information comes straight from a developer. I imagine they know what they’re talking about. You don’t need to get so worked up over everything that is mildly negative regarding Linux. Also, we don’t know if Google made any changes to the kernel or anything else. They’re probably using a custom distro so who knows what they messed with. No one was saying this is a problem with Linux in general, chill.

Who says I'm getting worked up? I stated there is no evidence the Linux scheduler is worse than the Windows scheduler, in fact there's evidence on the contrary. I provided evidence highlighting that many of this apparent developers claims have been discussed and debunked, and I commented that there are many situations outside gaming where the Linux scheduler is making a mockery of Windows, especially where NUMA is concerned.

The only one getting worked up is yourself by the sounds of things? Do people think Stadia would run better under Windows server? :ROFLMAO:
 
Who says I'm getting worked up? I stated there is no evidence the Linux scheduler is worse than the Windows scheduler, in fact there's evidence on the contrary. I provided evidence highlighting that many of this apparent developers claims have been discussed and debunked, and I commented that there are many situations outside gaming where the Linux scheduler is making a mockery of Windows, especially where NUMA is concerned.

The only one getting worked up is yourself by the sounds of things? Do people think Stadia would run better under Windows server? :ROFLMAO:

You clearly are worked up. This has nothing to do with Windows vs Linux or which has better stuff. It is 100% about Stadia. You just felt the need to turn it into an issue because you can’t stand even the tiniest negative thing being said even tangentially related to Linux.
 
You clearly are worked up. This has nothing to do with Windows vs Linux or which has better stuff. It is 100% about Stadia. You just felt the need to turn it into an issue because you can’t stand even the tiniest negative thing being said even tangentially related to Linux.

Nope, I'm participating in discussion and am not worked up in the slightest. Azure runs primarily under Linux and Azure seems to be performing just fine, this sounds like flawed excuses for Google's poor Stadia implementation or a refusal by developers to port titles efficiently to Stadia under Linux.
 
Last edited:
Does it? Is Azure some kind of mainframe?

It's MS's cloud service. While it pales in comparison in size to AWS, it is still the second largest by a pretty big margin. Amazon has about 48% of the market, Azure has about 15%, Alibaba has about 7% (though their number may be a little suspect) and Google has about 4%. Those are the top 4. So if your company decides they are going to move to "The Cloud" chances are it is one of those 4. On the hardware side underlying Azure is lots and lots of commodity servers of various sizes. Just your typical really big data center, or rather data centers as they have many around the world. On the software side it is Windows Server and Hyper-V. For better or worse Microsoft has long been an "eat your own dog food" company and this is no exception, they use Windows as the OS, Hyper-V as the hypervisor, Storage Spaces to manage the storage and so on.

Now all that said, it wasn't what I was talking about when I said Google should use Windows, I didn't mean as the underlying OS/Hypervisor for Stadia. For that were it me I'd be looking at VMWare since they seem to have more experience with GPU virtualization than anyone else. I was meaning as a guest OS where the game software itself would run. However my post on Azure was just to clear up that, despite what some evangelists will claim, Windows is perfectly capable of operating on a massive, "cloud" scale. People confuse the guest OS with the Hypervisor/infrastructure that runs under it.
 
Physics are still a limitation to this. Period. Not viable for another 5 years at a minimum.
 
It's MS's cloud service. While it pales in comparison in size to AWS, it is still the second largest by a pretty big margin. Amazon has about 48% of the market, Azure has about 15%, Alibaba has about 7% (though their number may be a little suspect) and Google has about 4%. Those are the top 4. So if your company decides they are going to move to "The Cloud" chances are it is one of those 4. On the hardware side underlying Azure is lots and lots of commodity servers of various sizes. Just your typical really big data center, or rather data centers as they have many around the world. On the software side it is Windows Server and Hyper-V. For better or worse Microsoft has long been an "eat your own dog food" company and this is no exception, they use Windows as the OS, Hyper-V as the hypervisor, Storage Spaces to manage the storage and so on.

Now all that said, it wasn't what I was talking about when I said Google should use Windows, I didn't mean as the underlying OS/Hypervisor for Stadia. For that were it me I'd be looking at VMWare since they seem to have more experience with GPU virtualization than anyone else. I was meaning as a guest OS where the game software itself would run. However my post on Azure was just to clear up that, despite what some evangelists will claim, Windows is perfectly capable of operating on a massive, "cloud" scale. People confuse the guest OS with the Hypervisor/infrastructure that runs under it.

Azure has been growing exponentially due to Amazon entering competing markets for many of their customers. They are actually closer to 25% of the Public cloud market, and have been growing massively. Don't discount them at all. They are a legit cloud competitor to AWS now. Google and Alibaba flopping around in the 7-8% range. GCP being mainly for AI/k8 workloads at large.
 
It's MS's cloud service. While it pales in comparison in size to AWS, it is still the second largest by a pretty big margin. Amazon has about 48% of the market, Azure has about 15%, Alibaba has about 7% (though their number may be a little suspect) and Google has about 4%. Those are the top 4. So if your company decides they are going to move to "The Cloud" chances are it is one of those 4. On the hardware side underlying Azure is lots and lots of commodity servers of various sizes. Just your typical really big data center, or rather data centers as they have many around the world. On the software side it is Windows Server and Hyper-V. For better or worse Microsoft has long been an "eat your own dog food" company and this is no exception, they use Windows as the OS, Hyper-V as the hypervisor, Storage Spaces to manage the storage and so on.

Now all that said, it wasn't what I was talking about when I said Google should use Windows, I didn't mean as the underlying OS/Hypervisor for Stadia. For that were it me I'd be looking at VMWare since they seem to have more experience with GPU virtualization than anyone else. I was meaning as a guest OS where the game software itself would run. However my post on Azure was just to clear up that, despite what some evangelists will claim, Windows is perfectly capable of operating on a massive, "cloud" scale. People confuse the guest OS with the Hypervisor/infrastructure that runs under it.

https://www.zdnet.com/article/micro...s-now-more-used-on-azure-than-windows-server/

https://www.digitaltrends.com/computing/more-linux-on-microsoft-azure-than-windows/

I don't believe that's confusing any guest OS with the hypervisor host OS.

"Native Azure services are often running on Linux. Microsoft is building more of these services. For example, Azure's Software Defined Network (SDN) is based on Linux."
 

Ya it is. What these say is that Linux is the most popular guest OS on Azure. Fine and well, and not at all surprising to me. However the claim I was responding to was claiming that Windows wouldn't work for cloud infrastructure, which is patently false. Azure was, and still is, Windows Server and Hyper-V running on the system. This is not about some OS dick-waving contest, it is about the reality of the fact that Windows works just fine for big "cloud" stuff and indeed underlies the second biggest public one in the world.
 
Ya it is. What these say is that Linux is the most popular guest OS on Azure. Fine and well, and not at all surprising to me. However the claim I was responding to was claiming that Windows wouldn't work for cloud infrastructure, which is patently false. Azure was, and still is, Windows Server and Hyper-V running on the system. This is not about some OS dick-waving contest, it is about the reality of the fact that Windows works just fine for big "cloud" stuff and indeed underlies the second biggest public one in the world.

By mentioning an OS dick waving contest, you are in fact the one instigating that direction of argument in a field of large scale cloud infrastructure that's already been dominated by Linux, and not by a Linux hypervisor - By Linux the guest OS. Windows Server under Azure is declining in favor of Linux.

To re quote:

It's not just Microsoft's Azure customers who are turning to Linux. Guthrie explained, "Native Azure services are often running on Linux. Microsoft is building more of these services."

And to add to that quote:

Why is everyone, including Microsoft, switching to Linux and open-source software? Guthrie explained: "It started more than 10 years ago when we open-sourced ASP.NET. We recognized open source is something that every developer can benefit from. It's not nice, it's essential. It's not just code, it's community." Indeed, Guthrie then claimed, "We're now the largest open-source project supporter in the world."

In other words, Microsoft are on their own too small to accommodate such a large and constantly growing industry with the Windows operating system. Hence the reason they not only own Github, but are the largest contributor to Github.

Another article on the subject:

https://www.zdnet.com/article/linux-now-dominates-azure/

That's right. Microsoft's prize cloud, Linux, not Windows Server, is now, at least some of the time, the most popular operating system. Windows Server isn't going to be making a come back.

You see as Guthrie added, "Every month, Linux goes up."

You make a rational argument and apparently you're worked up and dick slinging...? There's no way Google are going to consider Windows Server as the basis of their Stadia infrastructure.
 
Last edited:
It's not that Stadia sucks or that the whole idea of game streaming is flawed. No, the real issue is that a bug in the Linux kernel prevents software from changing the speed of light.
 
It's not that Stadia sucks or that the whole idea of game streaming is flawed. No, the real issue is that a bug in the Linux kernel prevents software from changing the speed of light.

I think, more to the point, they're looking to reduce latency to negative figures by running code optimized for Windows on a Linux based system and blaming the Linux scheduler when they should be blaming the code.

...And the fact that what they're trying to achieve is impossible.
 
It's MS's cloud service. While it pales in comparison in size to AWS, it is still the second largest by a pretty big margin. Amazon has about 48% of the market, Azure has about 15%, Alibaba has about 7% (though their number may be a little suspect) and Google has about 4%. Those are the top 4. So if your company decides they are going to move to "The Cloud" chances are it is one of those 4. On the hardware side underlying Azure is lots and lots of commodity servers of various sizes. Just your typical really big data center, or rather data centers as they have many around the world. On the software side it is Windows Server and Hyper-V. For better or worse Microsoft has long been an "eat your own dog food" company and this is no exception, they use Windows as the OS, Hyper-V as the hypervisor, Storage Spaces to manage the storage and so on.

Now all that said, it wasn't what I was talking about when I said Google should use Windows, I didn't mean as the underlying OS/Hypervisor for Stadia. For that were it me I'd be looking at VMWare since they seem to have more experience with GPU virtualization than anyone else. I was meaning as a guest OS where the game software itself would run. However my post on Azure was just to clear up that, despite what some evangelists will claim, Windows is perfectly capable of operating on a massive, "cloud" scale. People confuse the guest OS with the Hypervisor/infrastructure that runs under it.

Pretty good summary of Azure. And no doubt its true the core of Azure is still MS internal Azure OS that runs modified Windows Server 2008... since they launched it they have been moving more and more of their own systems to Linux. A couple years ago they started using Azure cloud switch... all their network switching for Azure is now done by a custom MS version of Linux. Also many of their hosted bits are now running on Linux as well... again that started a few years ago with HDinsite.

They may never kill the Azure windows server machines completely... I mean MS is a company that kept an internal Xenix email machine running for almost 10 years after they killed Xenix. :)
 
Ya it is. What these say is that Linux is the most popular guest OS on Azure. Fine and well, and not at all surprising to me. However the claim I was responding to was claiming that Windows wouldn't work for cloud infrastructure, which is patently false. Azure was, and still is, Windows Server and Hyper-V running on the system. This is not about some OS dick-waving contest, it is about the reality of the fact that Windows works just fine for big "cloud" stuff and indeed underlies the second biggest public one in the world.

To be clear its not off the shelf windows server they used to build Azure. Its a heavily customized version of windows server 2008.
 
It's MS's cloud service. While it pales in comparison in size to AWS, it is still the second largest by a pretty big margin. Amazon has about 48% of the market, Azure has about 15%, Alibaba has about 7% (though their number may be a little suspect) and Google has about 4%. Those are the top 4. So if your company decides they are going to move to "The Cloud" chances are it is one of those 4. On the hardware side underlying Azure is lots and lots of commodity servers of various sizes. Just your typical really big data center, or rather data centers as they have many around the world. On the software side it is Windows Server and Hyper-V. For better or worse Microsoft has long been an "eat your own dog food" company and this is no exception, they use Windows as the OS, Hyper-V as the hypervisor, Storage Spaces to manage the storage and so on.

Now all that said, it wasn't what I was talking about when I said Google should use Windows, I didn't mean as the underlying OS/Hypervisor for Stadia. For that were it me I'd be looking at VMWare since they seem to have more experience with GPU virtualization than anyone else. I was meaning as a guest OS where the game software itself would run. However my post on Azure was just to clear up that, despite what some evangelists will claim, Windows is perfectly capable of operating on a massive, "cloud" scale. People confuse the guest OS with the Hypervisor/infrastructure that runs under it.

I know what Azure is, and I don't believe the entire infrastructure runs "windows". That would be stupid
 
Azure can be running anything.

Any networking component runs in Linux, doubt MS has any say in that regard.. you'd be crazy not to use Cisco virtual routers (or any competitors virtual router/switch/firewall).

Anything run in a docker... is using Linux. MS released PowerShell for Linux and vice versa updated Linux components in windows optional features.

MS is probably using Windows server 2019 without the GUI on the actual hosts and for Azure AD and exchange/o365 which is where the value of Azure really is. No way it's 2008/2012 based as you cannot run a massive hyper-v host inside of another VM before server 2016. Also 2008 doesn't even support SMB 3.0 and barely functions with TLS 1.2

Ok. Sure... You know that Azure, the #2 cloud platform in the world, runs on Windows, right?
 
There's no way the Windows scheduler is better than the Linux scheduler, there's not a single real world test proving this - In fact there's plenty of real world tests showing Linux running rings around Windows and it's aging scheduler. Good job Linus.

From what I've read today, it would seem scheduler performance is largely based on configuration.
 
Configuration of what? What is your context for comparison? :)

People are talking about kernel parameters netting a lot of performance in certain loads, while slowing down others. Latency vs throughput IIRC
 
Back
Top