Evidence of Supermicro Infected Hardware Found at U.S. Telecom

cageymaru

[H]ard as it Gets
Joined
Apr 10, 2003
Messages
19,817
Bloomberg says that security expert Yossi Appleboum has found evidence of altered Supermicro hardware in a major U.S. telecom's network. Mr. Appleboum has worked for the Israeli Army Intelligence Corps and is now co-chief executive officer of Sepio Systems in Gaithersburg, Maryland. His company was hired to scan data centers for an undisclosed telecom and discovered that a server was performing unusual communications. An implant was discovered on the server's Ethernet connector. He says that his company has seen other modifications on hardware being imported from China and explained that Supermicro isn't the only victim of the Chinese supply chain.

In the case of the telecommunications company, Sepio's technology detected that the tampered Supermicro server actually appeared on the network as two devices in one. The legitimate server was communicating one way, and the implant another, but all the traffic appeared to be coming from the same trusted server, which allowed it to pass through security filters. Appleboum said one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones. The metal is necessary to diffuse heat from the chip hidden inside, which acts like a mini computer. "The module looks really innocent, high quality and 'original' but it was added as part of a supply chain attack," he said.
 
  • Like
Reactions: erexx
like this

Eickst

[H]ard|Gawd
Joined
Aug 24, 2005
Messages
1,866
Still doesn't add up unless these chips can communicate with wireless from around the globe in order to avoid the network monitoring that happens at these places.
 

Eickst

[H]ard|Gawd
Joined
Aug 24, 2005
Messages
1,866
I have a theory -

National security experts say a key problem is that, in a cybersecurity industry approaching $100 billion in revenue annually, very little of that has been spent on inspecting hardware for tampering. That's allowed intelligence agencies around the world to work relatively unimpeded, with China holding a key advantage.
And the guy 'blowing the whistle' is the Co-CEO of a firm that does security analysis on hardware.........and had no evidence to back up his claim that he found stuff.....

And now -

In the wake of Bloomberg's reporting on the attack against Supermicro products, security experts say that teams around the world, from large banks and cloud computing providers to small research labs and startups, are analyzing their servers and other hardware for modifications, a stark change from normal practices. Their findings won't necessarily be made public, since hardware manipulation is typically designed to access government and corporate secrets, rather than consumer data.
I wonder if his business has increased 100 fold in the last week?
 

mikeo

Gawd
Joined
May 17, 2006
Messages
624
I have a theory -



And the guy 'blowing the whistle' is the Co-CEO of a firm that does security analysis on hardware.........and had no evidence to back up his claim that he found stuff.....

And now -



I wonder if his business has increased 100 fold in the last week?
I like that theory, more believable than all the Bloomberg articles.
 

panhead

Gawd
Joined
Dec 19, 2003
Messages
902
Still doesn't add up unless these chips can communicate with wireless from around the globe in order to avoid the network monitoring that happens at these places.
Read the article

In the case of the telecommunications company, Sepio's technology detected that the tampered Supermicro server actually appeared on the network as two devices in one. The legitimate server was communicating one way, and the implant another, but all the traffic appeared to be coming from the same trusted server, which allowed it to pass through security filters.
 

SixFootDuo

[H]ardness Supreme
Joined
Oct 5, 2004
Messages
5,566
wow ...... what a great calling card for your business. I bet his company will now get a lot of new business because of his "discovery." Sounds almost too good to be true.

See, you don't know who to believe these days.
 

Eickst

[H]ard|Gawd
Joined
Aug 24, 2005
Messages
1,866
Read the article

In the case of the telecommunications company, Sepio's technology detected that the tampered Supermicro server actually appeared on the network as two devices in one. The legitimate server was communicating one way, and the implant another, but all the traffic appeared to be coming from the same trusted server, which allowed it to pass through security filters.
We have 'trusted' servers and they're allowed to communicate to 'trusted' destinations. Anything else gets blocked (and first packet captured ie pcap file. We'd capture more but it's hard to capture more than the first packet if you block it.....)

I tend to think a major telco would not have a server involved in any kind of sensitive communications operating on an 'allow all' filter because it was 'trusted'.
 
D

Deleted member 93354

Guest
Still doesn't add up unless these chips can communicate with wireless from around the globe in order to avoid the network monitoring that happens at these places.
As he said, the traffic came from a trusted server. What they found odd was it was sending packets out to odd locations which gave it away. Assuming this is true and the firewall was trained to ignore outgoing traffic from this device, it would explain a lot.
 
D

Deleted member 93354

Guest
We have 'trusted' servers and they're allowed to communicate to 'trusted' destinations. Anything else gets blocked (and first packet captured ie pcap file. We'd capture more but it's hard to capture more than the first packet if you block it.....)

I tend to think a major telco would not have a server involved in any kind of sensitive communications operating on an 'allow all' filter because it was 'trusted'.
Yes but if they are Tier 3 provider, they probably get so many destinations settings up an outbound firewall would be prohibitive.
 
D

Deleted member 93354

Guest
I like that theory, more believable than all the Bloomberg articles.
That would quite possibly one of the stupidest things he could EVER do as it would get him sued if he made it up.
 

bbenz33

Limp Gawd
Joined
Dec 8, 2004
Messages
382
Read the article

In the case of the telecommunications company, Sepio's technology detected that the tampered Supermicro server actually appeared on the network as two devices in one. The legitimate server was communicating one way, and the implant another, but all the traffic appeared to be coming from the same trusted server, which allowed it to pass through security filters.
The problem I have with this is you either have two "systems" or one. So how do you have the traffic appearing to all come from the same trusted server?
 

JosiahBradley

[H]ard|Gawd
Joined
Mar 19, 2006
Messages
1,766
#fakenews. Same shit pulled on AMD. No firewall worth anything is going to allow ipmi traffic outbound on any port. This is all being blown out of proportion if it is true which I doubt it is.
 
D

Deleted member 93354

Guest
The problem I have with this is you either have two "systems" or one. So how do you have the traffic appearing to all come from the same trusted server?
Different MAC?

It's possible this server was going through qualification testing and the qualification testing included test packets and they noticed that the packets were being duplicated to different target IPs
 

bbenz33

Limp Gawd
Joined
Dec 8, 2004
Messages
382
Different MAC?

It's possible this server was going through qualification testing and the qualification testing included test packets and they noticed that the packets were being duplicated to different target IPs
Different MACs would allow for the two systems scenario but would not allow for the traffic to all come from the trusted server.
 

Schtask

Limp Gawd
Joined
Nov 29, 2011
Messages
436
Caught it on the network as two devices in one! Malicious network comms! Boards have implanted ships!

Oh my... So here's the thing. To look at these from a hardware implant level you have to do some very time consuming and expensive things:

1: This is the most important! ** THERE ARE NO PUBLISHED INDICATORS OF COMPROMISE (IOCS) ** What exactly are these companies going to test? The articles shown list NO TECHNICAL DETAILS....Only representative devices / details. Nothing anyone finds is conclusive.

2: Since no IOCs are present, every analysis has to be EXTREMELY THOROUGH. Invasive. Intrusive. We're talking tearing apart boards here. Every component needs analysis...individually. Guess how expensive that will be.

3: There are easier vectors to accomplish these goals. Why would an attacker invest considerable time / money into this when you accomplish the same tasks with software / firmware attacks.

4: How much time do you think it would take to analyze every component on these boards? Active logic, firmware, and passive. A considerable amount of time. You try to save time by X-Raying these boards and comparing to schematics. You realize you still need to tear the boards down and check components that may be different only to realize later that the schematics don't always cover hardware revisions, profit saving materials, etc...

5: Every single component is removed from these boards and inventoried. They now have to be compared against manufacturer schematics...again.

6: Analyze every input and output on a live system for every component. Compare to identical boards. Is there a difference? Parts that have firmware should be removed / analyzed with firmware dumped.

7: No one has actually ever seen a deployed hardware back door in servers. No one... though MANY forms of hardware implants exist on the malicious end.

8: Where are the IOCs? It's pretty much standard industry practice that if you are going to release a bomshell of a report such as this one, you include technical indicators of compromise. Of which there are none. Not even a hint of one outside of generic board mock ups and a picture of a chip that exists on some impossibly small fab process for the work it is expected to perform.

9: The below is a photo of Sepio's response for more information regarding this "find". It is COMPLETE MARKETING FLUFF! So what...they have a security suite...like everyone else. NOT BUYING IT....... I wonder if this is the info they sent to Bloomberg. This whole situation is starting to go from stupid to reckless to dangerous real quick.

wtf.PNG

 

mikeo

Gawd
Joined
May 17, 2006
Messages
624
If it was on an ethernet rj45 I cant imagine that server hosting anything useful unless it was a lights out management port. I would think a telco would be using fiber.
 

Icon_Charlie

[H]Lite
Joined
Aug 3, 2018
Messages
100
So either Bloomberg is doubling down on something that is probably legit in some form, or Silicone Valley is about to get caught with their pants down. Easy spin either way...
Since Silicon Valley is in my back yard and I hate the ceramic mask of lies and deception that goes on over there is Techie Land, I could believe that someone knew but either choose to ignore it OR was in on the payout.


I have seem some strange shit over the years to the point that with the exception of PayPal all my financial transactions are HARD COPY...

"CASH IS KING! BABY!!!"

Quoted the man with the base ball bat on top of money hill.
 

mikeo

Gawd
Joined
May 17, 2006
Messages
624
Oh my... So here's the thing. To look at these from a hardware implant level you have to do some very time consuming and expensive things:

1: This is the most important! ** THERE ARE NO PUBLISHED INDICATORS OF COMPROMISE (IOCS) ** What exactly are these companies going to test? The articles shown list NO TECHNICAL DETAILS....Only representative devices / details. Nothing anyone finds is conclusive.

2: Since no IOCs are present, every analysis has to be EXTREMELY THOROUGH. Invasive. Intrusive. We're talking tearing apart boards here. Every component needs analysis...individually. Guess how expensive that will be.

3: There are easier vectors to accomplish these goals. Why would an attacker invest considerable time / money into this when you accomplish the same tasks with software / firmware attacks.

4: How much time do you think it would take to analyze every component on these boards? Active logic, firmware, and passive. A considerable amount of time. You try to save time by X-Raying these boards and comparing to schematics. You realize you still need to tear the boards down and check components that may be different only to realize later that the schematics don't always cover hardware revisions, profit saving materials, etc...

5: Every single component is removed from these boards and inventoried. They now have to be compared against manufacturer schematics...again.

6: Analyze every input and output on a live system for every component. Compare to identical boards. Is there a difference? Parts that have firmware should be removed / analyzed with firmware dumped.

7: No one has actually ever seen a deployed hardware back door in servers. No one... though MANY forms of hardware implants exist on the malicious end.

8: Where are the IOCs? It's pretty much standard industry practice that if you are going to release a bomshell of a report such as this one, you include technical indicators of compromise. Of which there are none. Not even a hint of one outside of generic board mock ups and a picture of a chip that exists on some impossibly small fab process for the work it is expected to perform.

9: The below is a photo of Sepio's response for more information regarding this "find". It is COMPLETE MARKETING FLUFF! So what...they have a security suite...like everyone else. NOT BUYING IT....... I wonder if this is the info they sent to Bloomberg. This whole situation is starting to go from stupid to reckless to dangerous real quick.

View attachment 110521
Hahahaha so they are marketing that they help educate people not to pick up a flash drive off the street and insert it into their corporate devices?
 
D

Deleted member 93354

Guest
Different MACs would allow for the two systems scenario but would not allow for the traffic to all come from the trusted server.
If two MAC's took the same IP it would. (And they were filtering on source IP)
 

exiled350

[H]ard|Gawd
Joined
Jun 26, 2013
Messages
1,208
#fakenews. Same shit pulled on AMD. No firewall worth anything is going to allow ipmi traffic outbound on any port. This is all being blown out of proportion if it is true which I doubt it is.
Compromising the network port its self is perfect. Snoop the server and send the data out in packets that look fine but actually have extra info encoded in them. Catch the traffic once it enters the internet proper, volla. Even better, design the packet to be dropped after interception but few enough to be considered acceptable loss and you could go on for years without detection.
 

Bobert

Limp Gawd
Joined
May 22, 2011
Messages
202
"Mr. Appleboum has worked for the Israeli Army Intelligence Corps"

Fake story confirmed.
 

Ski

Gawd
Joined
Jun 21, 2008
Messages
978
I'm still waiting for Bloomberg to provide some actual evidence, rather than just repeating hearsay.

• Where are the pictures of this spy chip on a server, so that people can check their own servers?
• Where are the chips?
• Where are the actual reports?
• Where are the actual company names of the security firms that did these audits?
• Exactly how was this chip discovered (they claim it showed up in network traffic, but don't explain how it showed up, or what it was doing)?
• What percentage of servers have this 'spy chip' (are we talking one server out of thousands, or are we talking hundreds of infected servers)?

Extraordinary claims require extraordinary evidence, instead all Bloomberg's doing is giving us theories from second hand sources, spoon-fed information, and click-bait headlines once again. I'm out of my depth when it comes down to the physical engineering, which is why I read in detail the article to see precisely to understand their claims, but once again I'm left with more questions than answers. Not to mention the timing of this article, the trade war currently going on with US and China, elections coming up right now, and now these accusations, I have no choice but to speculate on the coincidences here (which I loathe doing by the way) and the only conclusion I can come up is a silly conspiracy which has as about as much evidence as Bloomberg is providing sadly.

On a different note can someone more knowledgeable break this quote down from the article:

"Appleboum said one key sign of the implant is that the manipulated Ethernet connector has metal sides instead of the usual plastic ones. The metal is necessary to diffuse heat from the chip hidden inside, which acts like a mini computer."

Correct me if I'm wrong but from my limited understanding metal around Ethernet sockets is not there for cooling it is there for RF shielding and are completely normal. This particularly confused me because I wanted Bloomberg show a video or example of someone taking one of these 'suspicious sockets' apart on a genuine SuperMicro motherboard otherwise it's just a red flag for lack of evidence again. If this is a physical hack then there should be plenty of physical evidence for that.

Additionally, is it physically possible (in theory or no) that a chip the size of a grain of rice with no external power source could generate the kind of heat they're alluding to in the article?

Help a homie here.
 
Last edited:

d3athf1sh

Gawd
Joined
Dec 16, 2015
Messages
529
if you guys watched any of the CCCen videos from around the time when the whole snowden/wiki leaks stuff was going down, it was leaked that the NSA had these capabilities in their arsenal. why do you doubt china would do the same thing? especially after it was public knowledge that the US Gov was doing this exact same thing?????
 

Ski

Gawd
Joined
Jun 21, 2008
Messages
978
if you guys watched any of the CCCen videos from around the time when the whole snowden/wiki leaks stuff was going down, it was leaked that the NSA had these capabilities in their arsenal. why do you doubt china would do the same thing? especially after it was public knowledge that the US Gov was doing this exact same thing?????
Are you referencing Vault 7?
 

Spidey329

[H]ardForum Junkie
Joined
Dec 15, 2003
Messages
8,676
We have 'trusted' servers and they're allowed to communicate to 'trusted' destinations. Anything else gets blocked (and first packet captured ie pcap file. We'd capture more but it's hard to capture more than the first packet if you block it.....)

I tend to think a major telco would not have a server involved in any kind of sensitive communications operating on an 'allow all' filter because it was 'trusted'.
I'd like to think that a major credit reporting agency wouldn't hire a music major to oversea their IT infrastructure when sensitive data is their business.

But we all know how that turned out. Dollars Saved > Common Sense in the corporate world.

I would not be at all surprised to see a misconfigured filter.
 

RealBeast

Gawd
Joined
Aug 4, 2010
Messages
648
Nope, sorry my 85 year old Mom says fuck these Chinese spy efforts, so I can't tell her about NSA and all that shit or no more cookies.

So yeah, tariff the shit out of them. ;)
 

Advil

[H]ard|Gawd
Joined
Jul 16, 2004
Messages
1,874
Why would they invest the time and money to do it this way? Because it's a lot harder to find.

The software/firmware route can be more easily found and mitigated. Or it can be disabled if the software changes.

Lucky for me I havn't built any SuperMicro servers lately but a decade ago I used to. They were a good company for small volume high quality servers. They cared about their firmware and knew how to support RAID properly. All their hardware was validated and tested with all combinations they support of a particular chassis. Good stuff. A little spendy but not when you actually need it to work the first time.

I kind of feel bad for them. Only a little though. They started out as a mostly USA company. But that was way way back.
 

chrispix

Gawd
Joined
Jan 14, 2003
Messages
517
If it was part of the RJ-45 adapter, and tied into it. I find it interesting as the actual compute for the network hardware is usually outside that.. Meaning if all your data on your network is encrypted, then everything it can sniff on the network is encrypted. To me this, as described is a way to get hardware into a facility, and sniff/capture data and relay it back home.. If the data is encrypted, then not so much.. Interesting..
 

N4CR

[H]ardness Supreme
Joined
Oct 17, 2011
Messages
4,449
Still doesn't add up unless these chips can communicate with wireless from around the globe in order to avoid the network monitoring that happens at these places.
S3223_FIREWALK.jpg


The wireless is unspecified - it's likely radar illuminated..
Here is a NSA/CIA version.

They all do it, that's why they covered this shit up really quick and shilled it as nothing here too ;)
 

bbenz33

Limp Gawd
Joined
Dec 8, 2004
Messages
382
If two MAC's took the same IP it would. (And they were filtering on source IP)
Also I'm not sure how many places don't do Sticky MAC filtering because that would easily prevent this from working or being found very quickly.
 

Eickst

[H]ard|Gawd
Joined
Aug 24, 2005
Messages
1,866
Yes in that example they used wireless to extract the data undetected. The Israeli 007 security pro said they were using the ethernet adapter and sending stuff out over the wire.

One, even with the wireless example above, it's going to have to be a super dedicated and targeted attack as someone has to have a transceiver close enough to interact with that thing. Two, if it was in fact going over the wire, I call shens because there's no way that went undetected for a decade, as prolific as supermicro boards are.
 

Ski

Gawd
Joined
Jun 21, 2008
Messages
978
yeah, but CCC had a bunch of good videos that year detailing a lot of stuff from Vault 7, really scary stuff if you're a privacy nut.
CCC? Share me a link to the videos in question, I would like to watch one of them.
 

MRAB54

Gawd
Joined
Sep 9, 2001
Messages
848
Still doesn't add up unless these chips can communicate with wireless from around the globe in order to avoid the network monitoring that happens at these places.
This is all really blown out of proportion, imo. BMC has been suspect for _years_, which is why nobody worth their salt would put that on a network that can get external.

Edit - in fact, in security conscious places I've seen would not plug into a BMC port.
 
Top