RegreSSHion: OpenSSH Server Vulnerability

[Zarathustra[H]]

I missed this one over the holiday week. Maybe many of you did too.

Apparently there is yet another vulnerability in the OpenSSH server.

The kicker is, this one was fixed before, but somehow it came back, thus it has been named RegreSSHion.

https://www.phoronix.com/news/RegreSSHion-CVE-2024-6387

Time to update all the servers again...

 

[uOpt]

Don't update yet. New patches coming today:

https://marc.info/?l=oss-security&m=172045570013195&w=2

 

[Lakados]

I missed this one over the holiday week. Maybe many of you did too.

Apparently there is yet another vulnerability in the OpenSSH server.

The kicker is, this one was fixed before, but somehow it came back, thus it has been named RegreSSHion.

https://www.phoronix.com/news/RegreSSHion-CVE-2024-6387

Time to update all the servers again...

I am starting to think that there are more bad actors out there "contributing" to open-source projects than I care to think about.
Back when that Microsoft employee stumbled across that exploit that was cooked into XZ it was largely written off as an isolated issue that was luckily avoided.
I'm starting to think that it wasn't so isolated, there have been far too many huge breeches in the past 3 months and I fear there is a growing problem about to burst.

 

[uOpt]

I am starting to think that there are more bad actors out there "contributing" to open-source projects than I care to think about.
Back when that Microsoft employee stumbled across that exploit that was cooked into XZ it was largely written off as an isolated issue that was luckily avoided.
I'm starting to think that it wasn't so isolated, there have been far too many huge breeches in the past 3 months and I fear there is a growing problem about to burst.

Yeah, but in the OpenBSD team of all places?

These look like honest mistakes and unlike the xz attack no attempt at obfuscation was made.

 

[Nobu]

I am starting to think that there are more bad actors out there "contributing" to open-source projects than I care to think about.
Back when that Microsoft employee stumbled across that exploit that was cooked into XZ it was largely written off as an isolated issue that was luckily avoided.
I'm starting to think that it wasn't so isolated, there have been far too many huge breeches in the past 3 months and I fear there is a growing problem about to burst.

Software development security and background checks have been too lax for a long time now. Sure, some companies and projects take it very seriously. But many smaller projects, even when part of an important OS component/system or used by a major project/corporation, get overlooked or ignored (afa security and personel go).

 

[uOpt]

Software development security and background checks have been too lax for a long time now. Sure, some companies and projects take it very seriously. But many smaller projects, even when part of an important OS component/system or used by a major project/corporation, get overlooked or ignored (afa security and personel go).

The error rate seems the same in small projects and big companies' big systems, though.

 

[Lakados]

Yeah, but in the OpenBSD team of all places?

These look like honest mistakes and unlike the xz attack no attempt at obfuscation was made.

Maybe I am just getting paranoid in my old age... But you know what they say about wearing a hi-viz..
You walk in with a hi-viz and a ladder and nobody questions you, do what you are going to do in plain sight and nobody asks questions until things have gone horribly wrong.

 

[Nobu]

The error rate seems the same in small projects and big companies' big systems, though.

I think your data set is too broad to make those kinds of conclusions. But I guess otoh, constraining the data might just be cherry picking.

 

[uOpt]

I think your data set is too broad to make those kinds of conclusions. But I guess otoh, constraining the data might just be cherry picking.

All the big breakins seem to be in some POS large enterprise "solution". That stuff really doesn't look any better than smaller projects.

And we learn about the smaller projects's problems in CVEs, for large system that doesn't exist unless an exploit goes public. Otherwise they stuff the hole privately.

Similar error rate I'd say.

 

[Zarathustra[H]]

Don't update yet. New patches coming today:

https://marc.info/?l=oss-security&m=172045570013195&w=2

Looks like the vulnerability only impacts OpenSSH versions 8.5p1 through 9.7p1 on gllbc based Linux distributions. (this time)

9.8p1 fixes it, but that was released on July 1st. I'll have to read about what these new patches do.

A couple of my servers are on Debian Bookworm. I did get new versions of the packages since this vulnerability was announced, but it is still only listed as version 9.2p1. Maybe the fix was backported by the Debian team?

Did some more googling and found this, looks like Debian Bookworm is affected up through 1:9.2p1-2+deb12u3, which is the latest available in my repositories right now. Thats a little disappointing. This has been in the wild and public knowledge now for a week. Usually they are much faster at patching shit than this...

The rest of my servers are running on Ubuntu Server 20.04 LTS. My habit of running on the oldest still supported LTS pays off, as these still use OpenSSH 8.2p1, so it shouldn't be affected. As luck would have it, my only publicly exposed SSH ports are unaffected.

Once 20.04 LTS goes obsolete - however - I am probably migrating all of my servers away from Ubuntu, probably to Debian. I do not want Snapd on my systems, and it appears increasingly integrated in newer Ubuntu releases.

 

[DukenukemX]

Once 20.04 LTS goes obsolete - however - I am probably migrating all of my servers away from Ubuntu, probably to Debian. I do not want Snapd on my systems, and it appears increasingly integrated in newer Ubuntu releases.

Linux Mint removes Snapd by default. It's getting to the point where distros based on Ubuntu may want to switch away from it.

 

[Zarathustra[H]]

Linux Mint removes Snapd by default. It's getting to the point where distros based on Ubuntu may want to switch away from it.

Mint does. But then it goes ahead and installs Flatpak instead. It's not quite as bad, due to the absence of the ties to the Snap Store (Canocicals attempt to own the Linux world) but it still has all the inefficiencies I hate about these binary distribution blobs with statically compiled dependencies. :/

If I have a system with apt, 100% of all packages should be managed and installed using.deb packages.

No manually downloaded binary blob installs, no make && make install from source and definitely no Snap/Flatpak/AppImage.

I use it on my laptops and on my desktop.

But mint isn't really a server OS. (You can turn it into one, but its not the most efficient way to get there)

I wouldn't use Mint as a server install. I wouldn't use the regular Ubuntu as a server install either.

Ubuntu Server edition (essentially a strippe3d down version of the client without a GUI/desktop or desktop apps preinstalled, and some defaults that are better for server use has beenmy go-to for over a decade now, but I am losing my patience with canonical, which is why I am considering making the switch to Debian.

I really like the Apt package manager, and I don't want to use any Enterprise Linux like RHEL. I also would prefer to never run into a binary blob distribution system like Snap, Flatpak or AppImage. I want a distribution where everything is available in the one main package manager without statically linked dependencies.

 

[jardows]

Looks like the vulnerability only impacts OpenSSH versions 8.5p1 through 9.7p1 on gllbc based Linux distributions. (this time)

Ah, all good for me. I use FreeBSD, so no glibc. Interesting that it was hooks into glibc that enabled the xz exploit. It may be high time for the distributions and developers to take a more serious look at musl.

 

[uOpt]

This in

Ah, all good for me. I use FreeBSD, so no glibc.

I am slightly confused, but I think that the example exploit deveolped so far only works on glibc, but that the underlying problem is also in FreeBSD.

In any case, no new patches yesterday which confuses me even more.

 

[Zarathustra[H]]

This in


I am slightly confused, but I think that the example exploit deveolped so far only works on glibc, but that the underlying problem is also in FreeBSD.

In any case, no new patches yesterday which confuses me even more.

Same on both.

It seems as if the bug is in OpenSSH itself (which is maintained by OpenBSD) but that the race condition doesn't happen unless it hooks into glibc, which it only does on Linux.

So its probably better to get a patched OpenSSH regardless of platform, but it is likely much less critical on the BSD's than on Linux.

Yeah, I am really confused why they are not rushing to patch this... Especially since it seems the patch was already released and tested years ago the first time around. It ought to be simpler to just put the code back in, than it would be to patch something like this from scratch.

Luckily the affected SSH daemons for me are all on my local network with no port forwards. I have only two publicly facing SSH ports, and those are both on older OpenSSH versions that are unaffected.


They recommend tempirarily limiting access to affected SSH instances to local networks only. If you can't, a configuration change (something about setting some delay to 0, saw it earlier but I'm on my phone right now and can't find it) might reduce the exposure short term.

 

[Zarathustra[H]]

Also, to be fair, to exploit this vulnerability it takes hammering a vulnerable host for several hours under ideal conditions, so it may not be quite as high priority as it seems at first.

 

[uOpt]

Just a reminder that you can also set `LoginGraceTime 0` in sshd_config to disable the hole. I did that on my Macs.

 

[DukenukemX]

Mint does. But then it goes ahead and installs Flatpak instead. It's not quite as bad, due to the absence of the ties to the Snap Store (Canocicals attempt to own the Linux world) but it still has all the inefficiencies I hate about these binary distribution blobs with statically compiled dependencies. :/

If I have a system with apt, 100% of all packages should be managed and installed using.deb packages.

No manually downloaded binary blob installs, no make && make install from source and definitely no Snap/Flatpak/AppImage.

I use it on my laptops and on my desktop.

But mint isn't really a server OS. (You can turn it into one, but its not the most efficient way to get there)

I wouldn't use Mint as a server install. I wouldn't use the regular Ubuntu as a server install either.

Ubuntu Server edition (essentially a strippe3d down version of the client without a GUI/desktop or desktop apps preinstalled, and some defaults that are better for server use has beenmy go-to for over a decade now, but I am losing my patience with canonical, which is why I am considering making the switch to Debian.

I really like the Apt package manager, and I don't want to use any Enterprise Linux like RHEL. I also would prefer to never run into a binary blob distribution system like Snap, Flatpak or AppImage. I want a distribution where everything is available in the one main package manager without statically linked dependencies.

This is why some people like Arch Linux, because sorta solves these problem. By sorta, I mean that it kinda creates new ones in the process. The main problem comes for AUR where it's a community maintained repository, which sounds about as appealing as you'd imagine. If you need software that isn't part of Arch then you need to enable AUR, but this is where things can break and become a security liability. The community despises Snapd because it's run by Ubuntu, but the community also hates Flatpak because it's inefficient. That being said, Debian looks like a good idea because their lacking of wanting to keep things up to date has really helped them.

 

[Nobu]

This is why some people like Arch Linux, because sorta solves these problem. By sorta, I mean that it kinda creates new ones in the process. The main problem comes for AUR where it's a community maintained repository, which sounds about as appealing as you'd imagine. If you need software that isn't part of Arch then you need to enable AUR, but this is where things can break and become a security liability. The community despises Snapd because it's run by Ubuntu, but the community also hates Flatpak because it's inefficient. That being said, Debian looks like a good idea because their lacking of wanting to keep things up to date has really helped them.

I'm cool with flatpack. Sucks that it uses so much space, but I hardly use it anyway. Nice when it works though.

As for AUR, I just wget the snapshot from the package page, un-package it, then makepkg -cris. If it fails bc of missing deps, I check for aur deps and grab those. If they have AUR deps, I say fuck-it and give up. Not gonna rebuild a whole dep chain when something in AUR has updates, and I'm not using a helper either. I'll find another solution.

 

[socK]

Ah, all good for me. I use FreeBSD, so no glibc. Interesting that it was hooks into glibc that enabled the xz exploit. It may be high time for the distributions and developers to take a more serious look at musl.

Musl is a serious pain in the ass - one of the leading reasons Alpine was also a serious pain in the ass

Super annoying when it broke shit in bizarre ways

Mint does. But then it goes ahead and installs Flatpak instead. It's not quite as bad, due to the absence of the ties to the Snap Store (Canocicals attempt to own the Linux world) but it still has all the inefficiencies I hate about these binary distribution blobs with statically compiled dependencies. :/

If I have a system with apt, 100% of all packages should be managed and installed using.deb packages.

No manually downloaded binary blob installs, no make && make install from source and definitely no Snap/Flatpak/AppImage.

I use it on my laptops and on my desktop.

But mint isn't really a server OS. (You can turn it into one, but its not the most efficient way to get there)

I wouldn't use Mint as a server install. I wouldn't use the regular Ubuntu as a server install either.

Ubuntu Server edition (essentially a strippe3d down version of the client without a GUI/desktop or desktop apps preinstalled, and some defaults that are better for server use has beenmy go-to for over a decade now, but I am losing my patience with canonical, which is why I am considering making the switch to Debian.

I really like the Apt package manager, and I don't want to use any Enterprise Linux like RHEL. I also would prefer to never run into a binary blob distribution system like Snap, Flatpak or AppImage. I want a distribution where everything is available in the one main package manager without statically linked dependencies.

Or you just use containers on your servers and live life on easy street.

You can't even get a shell to our hosts or containers in production. It's about as stripped down as humanly possible. You can effectively patch a host by literally deleting it. There is no downtime.

 

[jardows]

This in


I am slightly confused, but I think that the example exploit deveolped so far only works on glibc, but that the underlying problem is also in FreeBSD.

In any case, no new patches yesterday which confuses me even more.

Don't get me wrong, I keep my systems updated regularly. I just checked the FreeBSD mailing lists, and they already have a patch for the vulnerability. I suppose the developers figured someone with enough motivation could figure out a way to exploit it on FreeBSD even without glibc. The update is included in the latest round of updates for 14.1-RELEASE I ran today, so no fear here!

 

[uOpt]

Don't get me wrong, I keep my systems updated regularly. I just checked the FreeBSD mailing lists, and they already have a patch for the vulnerability. I suppose the developers figured someone with enough motivation could figure out a way to exploit it on FreeBSD even without glibc. The update is included in the latest round of updates for 14.1-RELEASE I ran today, so no fear here!

That is for the first round of regresshion, but what about that second one indicated in the mailing list post?

 

[uOpt]

You can't even get a shell to our hosts or containers in production. It's about as stripped down as humanly possible.

How do you debug anything in there when things go south?

 

[socK]

How do you debug anything in there when things go south?

I still have logs for everything, to start. I don't need a shell to look at them.

Realistically it's basically never a problem with the host. It's beyond exceedingly rare. Nothing is configured or provisioned by hand. If I really need to update or roll back the OS, it's largely trivial.

The hosts literally their OS by committing suicide monthly.

If it's a container that's having issues, I'll just roll back the offending image in production while I look into it. Then I'll run that image either locally or in dev to reproduce. They're generally configured to barf out more information there and you're free to attach an actual debugger to our APIs and whatnot.

 

[uOpt]

FreeBSD patches for the second problem just dropped. No idea about other platforms.