It Begins:. The great NAND oversupply disappearance!

ccityinstaller

Supreme [H]ardness
Joined
Feb 23, 2007
Messages
4,236
  • Like
Reactions: c3k
like this
I still can’t figure out how a 13 min outage should be a big deal at a supposedly world class manufacturing facility. You’d think they’d have UPS/rectifier and generator... Batteries only need to be enough to last 10-15 minutes for generator to kick over. At that point you can go forever as long as you have diesel.

Seems like someone just wants an excuse for a price increase.

I just can’t believe the facility design was so incompetent that a short commercial power loss would be a problem.
 
Wrong link ?


My bad. Stupid Google AMP gave me the wrong link. It was about fans though!


I think it's time to stock up on a 1TB SSD if you are in the market. If Toshiba lost even the same as WD (let alone 50% more) expect prices to rise 10-25% IMO. That joint venture supplies NAND for their products and all the major name "rebranders"


It escapes me how this could have truly happened shy of Corp espionage ala TSMC's issue with a few months ago. You would have to have malware interrupt main power, and then cause the battery backup/idiling gennys to go offline as well.


The crazy part is that it was a 15 minute outage. That is like a 2 hr wait for an ambulance responding to a downed Police officer. It just doesn't happen. If they really lost 12-15EXB of NAND that has to be over a quarter billion of damage.
 
Hmm. That stinks, and it may very well have an upward pressure on NAND prices in the short term, but I kind of doubt this was on purpose.

Any company doing this to themselves is going to lose money on it.

Also, the quantity can't be right. A single power outage is not going to lose more than a single lot of parts. 6 exabytes is 6 million terabytes. Are lots really that large?

Something just seems off here.
 
Last edited:
Hmm. That stinks, and it may very well have an upward pressure on NAND prices in the short term, but I kind of doubt this was on purpose.

Any company doing this to themselves is going to lose money on it.

Also, the qua titu can't be right. A single power outage is not going to lose more than a single lot of parts. 6 exabytes is 6 million terabytes. Are lots really that large?

Something just seems off here.

Yes, the plants are huge and have huge batches going through different steps, all which require power. 6 exabytes from a physical foot print standpoint really isn't as much as it sounds like when you are talking everything in production on the plant floor.
 
Hmm. That stinks, and it may very well have an upward pressure on NAND prices in the short term, but I kind of doubt this was on purpose.

Any company doing this to themselves is going to lose money on it.

Also, the qua titu can't be right. A single power outage is not going to lose more than a single lot of parts. 6 exabytes is 6 million terabytes. Are lots really that large?

Something just seems off here.

Are we not assuming there was an actual power outage ?

If we buy the conspiricy theory there was no outage correct ? In which case prices go up... I mean there are enough suppliers to sell too that simply telling them they are lucky to get stock pay up would work just fine. I doubt they would really know if supply was in fact down or not.

You can do some real shaddy things when your only one of only a few MFGs creating a product whos prinicples vaction together. ;) lol
 
Are we not assuming there was an actual power outage ?

If we buy the conspiricy theory there was no outage correct ? In which case prices go up... I mean there are enough suppliers to sell too that simply telling them they are lucky to get stock pay up would work just fine. I doubt they would really know if supply was in fact down or not.

You can do some real shaddy things when your only one of a few MFGs whos prinicples vaction together. ;) lol

Keep the following in mind.

If they wanted to reduce supply to drive up prices, they wouldn't need the excuse of a power outage. They could just start manufacturing less.

There is absolutely no reason they couldn't - say - blame low NAND prices, and cancel 3rd shift and cut their production by a third. This would be fully legit. They wouldn't need some cockamamie story.
 
Are we not assuming there was an actual power outage ?

If we buy the conspiricy theory there was no outage correct ? In which case prices go up... I mean there are enough suppliers to sell too that simply telling them they are lucky to get stock pay up would work just fine. I doubt they would really know if supply was in fact down or not.

You can do some real shaddy things when your only one of only a few MFGs creating a product whos prinicples vaction together. ;) lol

Commercial power outage should have absolutely zero impact on a world class manufacturing facility with tens of billions moving through the floor. You have UPS, DC Rectifier systems, automatic kickover, and generators running diesel/whatever. A 13 minute outage should be nothing. Shit, 13 minutes is short enough that it's within the standard window for the UPS/Rectifier batteries anyways.

Either these shitbags don't build their facilities with power outages in mind, because they know they can just pass the cost onto the world market since it's effectively a monopoly - Or they were sabotaged.
 
As soon as I saw this, I started searching for drives to buy.

It's a shame the 2TB Gen4 based Phison E16 drives are not available yet. I was planning on getting one of those for my Ryzen 3000 build in September when the 3950x becomes available.
 
  • Like
Reactions: mikeo
like this
I still can’t figure out how a 13 min outage should be a big deal at a supposedly world class manufacturing facility. You’d think they’d have UPS/rectifier and generator... Batteries only need to be enough to last 10-15 minutes for generator to kick over. At that point you can go forever as long as you have diesel.
I just can’t believe the facility design was so incompetent that a short commercial power loss would be a problem.
Generators are up in less than 1 minute.
A facility is designed on a cost benefit scale. These plants get tier 1 power like a hospital, they have contracted guaranteed uptime.
 
  • Like
Reactions: N4CR
like this
Generators are up in less than 1 minute.
A facility is designed on a cost benefit scale. These plants get tier 1 power like a hospital, they have contracted guaranteed uptime.

Yes, I realize that.

But even a hospital, a critical manufacturing process, etc, all still have battery for kick over to generator for inevitable outages. You can't guarantee commercial power. Sure, they are the priority of the company, but outages happen.

So either you have incompetence in the plant design, or something else.
 
Any company doing this to themselves is going to lose money on it.

Unless they have business interruption insurance and a seemingly strong enough case so as not to get hit with insurance fraud.

Particularly if the commodity cost of your product dips sufficiently low enough that a claim starts to look attractive.
 
I doubt there's good reason for a conspiracy. If there was a power outage, someone knows about it - power companies are required to log uptime / downtime with the Federal Energy Regulatory Commission. Shweitzer Engineering Laboratories makes most of the power monitoring and distribution switching equipment used by big energy companies (the guy who founded the company is a professor from my region), and their equipment can detect an outage 800 miles away. I'm doubting that the power outages were fake. On the other hand, having a power outage local to the industrial site itself could be possible. Most companies locate their on-site generators at the main power trunk, and things like floods, fires, lightning or even high winds (or a birds nest - happened at an area hospital) can take out your power.

I'm also doubting that Micron has installed generator capacity to continue manufacturing - that's big power, not lights and laptops. I'm sure they had enough power generation and batteries to safe their equipment, but I'd be surprised if they even had enough on-site power to keep their air conditioning running.


P.S. SixFootDuo had a link to a 4TB Micron SATA SSD for $350 in the Hot Deals section. Buy that if you're worried.
 
Commercial power outage should have absolutely zero impact on a world class manufacturing facility with tens of billions moving through the floor. You have UPS, DC Rectifier systems, automatic kickover, and generators running diesel/whatever. A 13 minute outage should be nothing. Shit, 13 minutes is short enough that it's within the standard window for the UPS/Rectifier batteries anyways.

Either these shitbags don't build their facilities with power outages in mind, because they know they can just pass the cost onto the world market since it's effectively a monopoly - Or they were sabotaged.

I know of quite a few cases of "world class manufacturing facilities" having lost power.... Usually it isn't the entire fab, but say ~1/3. Even on the uninterrupted lines. (Heck oddly enough the uninterrupted lines seem to have more issues than the regular lines because of all of those fancy switches they put in place to try to keep the lines stable.)

That said most of the power losses I know of are either split second, or much longer. I don't know of many 10-20 min ones. (And again something like that would usually affect ~1/3 of the lines coming into the factory, so it would be very odd that it hit them all. Must have happened at a relay switch that would have allowed them to swap to a different power line... but then that would take those lines off much longer, ie weeks.)

Also for those not realizing it 13 min power loss = days to recover many of the key tools. (Litho) = basically auto scrap of any key q-time loop lots. I don't know the NAND process flows very well, but I assume q-times apply to nearly every metal interconnect layer. (And 3d stacked NAND might have a bunch of q-time issues due to staircase contacts. Though that is just a guess on my part.) And often any wafer running on a tool would end up scrap levels after recover. (I don't know how much WIP they have vs stuff waiting to run, but I can see how a huge amount of lots would end up just auto scrapped.)
 
Hmm. That stinks, and it may very well have an upward pressure on NAND prices in the short term, but I kind of doubt this was on purpose.

Any company doing this to themselves is going to lose money on it.

Also, the quantity can't be right. A single power outage is not going to lose more than a single lot of parts. 6 exabytes is 6 million terabytes. Are lots really that large?

Something just seems off here.

I work at a large fab that has lost power before. There are no battery backups or generators for the big stuff. Computers might, just to protect against data corruption, but not the big ass multi-kilowatt machines that are lined up in rows up and down aisles. As said, we have guarantees (insurance) of power from the power company or the contractors doing work on the utilities are insured incase they break something (which has happened).

As far the yield loss. You have to consider that there are hundreds of lots moving through the fab at any given moment. They are all being processed at some critical level along the way. Lot A might be at metal mask level 2 while Lot B is at metal mask level 11. But getting a disruption at any mask level could be catastrophic.

I don't know the specifics of this fab or device size, but I could imagine on a 300mm wafer, it wouldn't be unheard of to have 7,000 chips per wafer. 7000 x 25 wafers = 175,000 chips per lot. Assuming these are 256gb die (32GB), thats 175,000 chips x 32GB = 5,600,000GB...or 5.6PB....in one 25 wafer lot. That's just numbers I pulled out of ass, but you can see how they compound very quickly. I have seen excursions like this result in hundreds of thousands to a million chips being scrapped because of some stupid mistake and tiny devices (lots per wafer) being affected.
 
This might be like the big flood in the early 2000's when harddrives were getting ridiculously cheap, for the time. And then after the Tsunami, it felt like we never recovered from the price increase. I seem to remember something similar happening in the PC-100/PC-133 memory days...dirt cheap memory and then it went up (though that might have been actual price fixing back then).
 
Looking to build a new computer when Ryzen launches and I'll be stepping up to nvme m.2 storage. I guess I better go ahead and pick that up. Looking at Samsung 970 evo plus. Only question is 500gb or 1tb.
 
Looking to build a new computer when Ryzen launches and I'll be stepping up to nvme m.2 storage. I guess I better go ahead and pick that up. Looking at Samsung 970 evo plus. Only question is 500gb or 1tb.


You might want to read this thread.

The Phison E12 based Inland Premium drive (not to be mistaken with the Inland Pro which is slower) is a fantastically fast drive and is very affordable.
 
You might want to read this thread.

The Phison E12 based Inland Premium drive (not to be mistaken with the Inland Pro which is slower) is a fantastically fast drive and is very affordable.


This! I went ahead and grabbed another for the $97.99 for 1TB. It's an insane deal, and think this will cost NAND to rise in the coming weeks. Remember this is just the weekend. When the "flood" (that barely had any affect on production at a single plant) happened they jacked the drives up super quick IIRC.

COUPLE this with the back to school season and the new flagship launches next quarter and the NAND makers will be screaming they have no supply.


FYI, if you get the Inland drive, make SURE it's the Premium and not the Professional. The Pro model is a bit cheaper (like $10) but is a fair amount slower since it uses an older controller.
 
As soon as I saw this, I started searching for drives to buy.

It's a shame the 2TB Gen4 based Phison E16 drives are not available yet. I was planning on getting one of those for my Ryzen 3000 build in September when the 3950x becomes available.

Yes they are right now...

Sabrent 1TB Rocket Nvme PCIe 4.0 M.2 2280 Internal SSD Maximum Performance Solid State Drive (SB-ROCKET-NVMe4-1TB) https://www.amazon.com/dp/B07TLYWMYW/ref=cm_sw_r_cp_apa_i_xPAgDbWT608JT
 
As an Amazon Associate, HardForum may earn from qualifying purchases.
... They could just start manufacturing less.

There is absolutely no reason they couldn't - say - ... cancel 3rd shift and cut their production by a third...

If a 15 minute outage is going to take weeks to reset and recover from, they can't just 'cancel 3rd shift'. It likely has to stay going 24/7. Cancelling a shift every night would be worse than this 'outage'.

They could shut down some of the parallel processing lines (guessing that there are some) to cut production, but the operating lines would still be 24/7.
 
If a 15 minute outage is going to take weeks to reset and recover from, they can't just 'cancel 3rd shift'. It likely has to stay going 24/7. Cancelling a shift every night would be worse than this 'outage'.

They could shut down some of the parallel processing lines (guessing that there are some) to cut production, but the operating lines would still be 24/7.

Fair. My comment wasn't specific to their particular production line, it was intended as an example of something simple they could do and scale back volume. The takeway is that they don't need to fabricate stories.
 
Last edited:
I read in one of the articles that this outage destroyed an entire quarter's worth of parts. Sounds fishy
 
I read in one of the articles that this outage destroyed an entire quarter's worth of parts. Sounds fishy

I thought I heard about, maybe it was a HF contamination (Hydroflouric Acid) that took out someones quarter recently. But, in my search, I found an article from Jan 2019 about TSMC having to scrap 10,000 wafers because of a chemical contamination. This stuff happens, usually more often that we like to tell the world. Game of dominoes, this industry.
 
I thought I heard about, maybe it was a HF contamination (Hydroflouric Acid) that took out someones quarter recently. But, in my search, I found an article from Jan 2019 about TSMC having to scrap 10,000 wafers because of a chemical contamination. This stuff happens, usually more often that we like to tell the world. Game of dominoes, this industry.


Agreed...However...fishy.

I work in all areas of industry. Semicon, automotive, chemical, all of it. Not in a single plant have I ever seen an entire FQ's worth of parts all in such a vulnerable stage as what we are being told happened here. Unless... Unless the PLC and database that tracks everything as it is actively moving through the line got completely fucked and there is no way to tell what part is what other than a barcode of some type. Even then....
 
Ok so here is what is fishy to me. Let's say the power outage happened and they had no backups. So that would mean every wafer in progress is trashed and it would take about 2 weeks to spin the plant back up. So they would have 2-3 weeks of lost production. They are claiming to have lost the equivalent of half of an entire QUARTER's worth of output. A quarter is 3 months or about 13 weeks. That math doesn't add up?!
 
Yeah, how long does it take for a wafer to get thru the fab anyway? I don't know the answer, but several months seems ridiculous and too slow.

Everything that was in process, scrapped. But how much is this?

2 ish weeks or production, not produced due to the recovery from the issue. Fine, but how much output was lost?

Neither of the above seem capable of half of a quarters' (6.5 weeks) worth of loss.
 
Back
Top