It Begins:. The great NAND oversupply disappearance!

Discussion in 'HardForum Tech News' started by ccityinstaller, Jun 29, 2019.

  1. ccityinstaller

    ccityinstaller [H]ardness Supreme

    Messages:
    4,232
    Joined:
    Feb 23, 2007
    c3k likes this.
  2. Soulstorm brew

    Soulstorm brew n00b

    Messages:
    60
    Joined:
    Mar 29, 2019
  3. DF-1

    DF-1 2[H]4U

    Messages:
    2,567
    Joined:
    Jun 17, 2011
    Power outage = new floods
     
  4. viivo

    viivo [H]ard|Gawd

    Messages:
    1,265
    Joined:
    Sep 7, 2005
  5. Mchart

    Mchart 2[H]4U

    Messages:
    3,380
    Joined:
    Aug 7, 2004
    I still can’t figure out how a 13 min outage should be a big deal at a supposedly world class manufacturing facility. You’d think they’d have UPS/rectifier and generator... Batteries only need to be enough to last 10-15 minutes for generator to kick over. At that point you can go forever as long as you have diesel.

    Seems like someone just wants an excuse for a price increase.

    I just can’t believe the facility design was so incompetent that a short commercial power loss would be a problem.
     
  6. Brian_B

    Brian_B 2[H]4U

    Messages:
    3,169
    Joined:
    Mar 23, 2012
  7. ccityinstaller

    ccityinstaller [H]ardness Supreme

    Messages:
    4,232
    Joined:
    Feb 23, 2007

    My bad. Stupid Google AMP gave me the wrong link. It was about fans though!


    I think it's time to stock up on a 1TB SSD if you are in the market. If Toshiba lost even the same as WD (let alone 50% more) expect prices to rise 10-25% IMO. That joint venture supplies NAND for their products and all the major name "rebranders"


    It escapes me how this could have truly happened shy of Corp espionage ala TSMC's issue with a few months ago. You would have to have malware interrupt main power, and then cause the battery backup/idiling gennys to go offline as well.


    The crazy part is that it was a 15 minute outage. That is like a 2 hr wait for an ambulance responding to a downed Police officer. It just doesn't happen. If they really lost 12-15EXB of NAND that has to be over a quarter billion of damage.
     
  8. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    28,332
    Joined:
    Oct 29, 2000
    Hmm. That stinks, and it may very well have an upward pressure on NAND prices in the short term, but I kind of doubt this was on purpose.

    Any company doing this to themselves is going to lose money on it.

    Also, the quantity can't be right. A single power outage is not going to lose more than a single lot of parts. 6 exabytes is 6 million terabytes. Are lots really that large?

    Something just seems off here.
     
    Last edited: Jun 29, 2019
  9. Mchart

    Mchart 2[H]4U

    Messages:
    3,380
    Joined:
    Aug 7, 2004
    Yes, the plants are huge and have huge batches going through different steps, all which require power. 6 exabytes from a physical foot print standpoint really isn't as much as it sounds like when you are talking everything in production on the plant floor.
     
    ChoGGi likes this.
  10. owned66

    owned66 n00b

    Messages:
    43
    Joined:
    Aug 8, 2012
  11. ChadD

    ChadD [H]ardness Supreme

    Messages:
    4,136
    Joined:
    Feb 8, 2016
    Are we not assuming there was an actual power outage ?

    If we buy the conspiricy theory there was no outage correct ? In which case prices go up... I mean there are enough suppliers to sell too that simply telling them they are lucky to get stock pay up would work just fine. I doubt they would really know if supply was in fact down or not.

    You can do some real shaddy things when your only one of only a few MFGs creating a product whos prinicples vaction together. ;) lol
     
  12. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    28,332
    Joined:
    Oct 29, 2000
    Keep the following in mind.

    If they wanted to reduce supply to drive up prices, they wouldn't need the excuse of a power outage. They could just start manufacturing less.

    There is absolutely no reason they couldn't - say - blame low NAND prices, and cancel 3rd shift and cut their production by a third. This would be fully legit. They wouldn't need some cockamamie story.
     
  13. Mchart

    Mchart 2[H]4U

    Messages:
    3,380
    Joined:
    Aug 7, 2004
    Commercial power outage should have absolutely zero impact on a world class manufacturing facility with tens of billions moving through the floor. You have UPS, DC Rectifier systems, automatic kickover, and generators running diesel/whatever. A 13 minute outage should be nothing. Shit, 13 minutes is short enough that it's within the standard window for the UPS/Rectifier batteries anyways.

    Either these shitbags don't build their facilities with power outages in mind, because they know they can just pass the cost onto the world market since it's effectively a monopoly - Or they were sabotaged.
     
    drescherjm, N4CR and Skyblue like this.
  14. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    28,332
    Joined:
    Oct 29, 2000
    As soon as I saw this, I started searching for drives to buy.

    It's a shame the 2TB Gen4 based Phison E16 drives are not available yet. I was planning on getting one of those for my Ryzen 3000 build in September when the 3950x becomes available.
     
    mikeo likes this.
  15. Master_shake_

    Master_shake_ [H]ardForum Junkie

    Messages:
    9,193
    Joined:
    Apr 9, 2012
    Illegal activity froma company already busted for price fixing? Doubtful.
     
    Kwaz and pendragon1 like this.
  16. Jim Kim

    Jim Kim 2[H]4U

    Messages:
    3,471
    Joined:
    May 24, 2012
    Generators are up in less than 1 minute.
    A facility is designed on a cost benefit scale. These plants get tier 1 power like a hospital, they have contracted guaranteed uptime.
     
    N4CR likes this.
  17. Mchart

    Mchart 2[H]4U

    Messages:
    3,380
    Joined:
    Aug 7, 2004
    Yes, I realize that.

    But even a hospital, a critical manufacturing process, etc, all still have battery for kick over to generator for inevitable outages. You can't guarantee commercial power. Sure, they are the priority of the company, but outages happen.

    So either you have incompetence in the plant design, or something else.
     
  18. KD5ZXG

    KD5ZXG Limp Gawd

    Messages:
    409
    Joined:
    Mar 24, 2017
    Wahtever you can get away with, therefore clearly not illegal.
     
  19. btgorman

    btgorman Limp Gawd

    Messages:
    231
    Joined:
    Jun 30, 2009
    Corporate conspiracy to fix prices.
     
  20. Brian_B

    Brian_B 2[H]4U

    Messages:
    3,169
    Joined:
    Mar 23, 2012
    Unless they have business interruption insurance and a seemingly strong enough case so as not to get hit with insurance fraud.

    Particularly if the commodity cost of your product dips sufficiently low enough that a claim starts to look attractive.
     
  21. seanreisk

    seanreisk Gawd

    Messages:
    916
    Joined:
    Aug 29, 2011
    I doubt there's good reason for a conspiracy. If there was a power outage, someone knows about it - power companies are required to log uptime / downtime with the Federal Energy Regulatory Commission. Shweitzer Engineering Laboratories makes most of the power monitoring and distribution switching equipment used by big energy companies (the guy who founded the company is a professor from my region), and their equipment can detect an outage 800 miles away. I'm doubting that the power outages were fake. On the other hand, having a power outage local to the industrial site itself could be possible. Most companies locate their on-site generators at the main power trunk, and things like floods, fires, lightning or even high winds (or a birds nest - happened at an area hospital) can take out your power.

    I'm also doubting that Micron has installed generator capacity to continue manufacturing - that's big power, not lights and laptops. I'm sure they had enough power generation and batteries to safe their equipment, but I'd be surprised if they even had enough on-site power to keep their air conditioning running.


    P.S. SixFootDuo had a link to a 4TB Micron SATA SSD for $350 in the Hot Deals section. Buy that if you're worried.
     
  22. ikevi

    ikevi n00b

    Messages:
    57
    Joined:
    Apr 7, 2009
    I know of quite a few cases of "world class manufacturing facilities" having lost power.... Usually it isn't the entire fab, but say ~1/3. Even on the uninterrupted lines. (Heck oddly enough the uninterrupted lines seem to have more issues than the regular lines because of all of those fancy switches they put in place to try to keep the lines stable.)

    That said most of the power losses I know of are either split second, or much longer. I don't know of many 10-20 min ones. (And again something like that would usually affect ~1/3 of the lines coming into the factory, so it would be very odd that it hit them all. Must have happened at a relay switch that would have allowed them to swap to a different power line... but then that would take those lines off much longer, ie weeks.)

    Also for those not realizing it 13 min power loss = days to recover many of the key tools. (Litho) = basically auto scrap of any key q-time loop lots. I don't know the NAND process flows very well, but I assume q-times apply to nearly every metal interconnect layer. (And 3d stacked NAND might have a bunch of q-time issues due to staircase contacts. Though that is just a guess on my part.) And often any wafer running on a tool would end up scrap levels after recover. (I don't know how much WIP they have vs stuff waiting to run, but I can see how a huge amount of lots would end up just auto scrapped.)
     
    Thunderdolt likes this.
  23. Trimlock

    Trimlock [H]ardForum Junkie

    Messages:
    15,150
    Joined:
    Sep 23, 2005
    6 Eleventy Billion is a lot
     
    BoiseTech, c_porter and Mchart like this.
  24. westrock2000

    westrock2000 [H]ardForum Junkie

    Messages:
    9,060
    Joined:
    Jun 3, 2005
    I work at a large fab that has lost power before. There are no battery backups or generators for the big stuff. Computers might, just to protect against data corruption, but not the big ass multi-kilowatt machines that are lined up in rows up and down aisles. As said, we have guarantees (insurance) of power from the power company or the contractors doing work on the utilities are insured incase they break something (which has happened).

    As far the yield loss. You have to consider that there are hundreds of lots moving through the fab at any given moment. They are all being processed at some critical level along the way. Lot A might be at metal mask level 2 while Lot B is at metal mask level 11. But getting a disruption at any mask level could be catastrophic.

    I don't know the specifics of this fab or device size, but I could imagine on a 300mm wafer, it wouldn't be unheard of to have 7,000 chips per wafer. 7000 x 25 wafers = 175,000 chips per lot. Assuming these are 256gb die (32GB), thats 175,000 chips x 32GB = 5,600,000GB...or 5.6PB....in one 25 wafer lot. That's just numbers I pulled out of ass, but you can see how they compound very quickly. I have seen excursions like this result in hundreds of thousands to a million chips being scrapped because of some stupid mistake and tiny devices (lots per wafer) being affected.
     
    dgz, Crackinjahcs, seanreisk and 2 others like this.
  25. westrock2000

    westrock2000 [H]ardForum Junkie

    Messages:
    9,060
    Joined:
    Jun 3, 2005
    This might be like the big flood in the early 2000's when harddrives were getting ridiculously cheap, for the time. And then after the Tsunami, it felt like we never recovered from the price increase. I seem to remember something similar happening in the PC-100/PC-133 memory days...dirt cheap memory and then it went up (though that might have been actual price fixing back then).
     
    Brian_B likes this.
  26. kamikazi

    kamikazi Limp Gawd

    Messages:
    304
    Joined:
    Jan 19, 2006
    Looking to build a new computer when Ryzen launches and I'll be stepping up to nvme m.2 storage. I guess I better go ahead and pick that up. Looking at Samsung 970 evo plus. Only question is 500gb or 1tb.
     
  27. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    28,332
    Joined:
    Oct 29, 2000

    You might want to read this thread.

    The Phison E12 based Inland Premium drive (not to be mistaken with the Inland Pro which is slower) is a fantastically fast drive and is very affordable.
     
    Deleted member 184142 likes this.
  28. ccityinstaller

    ccityinstaller [H]ardness Supreme

    Messages:
    4,232
    Joined:
    Feb 23, 2007

    This! I went ahead and grabbed another for the $97.99 for 1TB. It's an insane deal, and think this will cost NAND to rise in the coming weeks. Remember this is just the weekend. When the "flood" (that barely had any affect on production at a single plant) happened they jacked the drives up super quick IIRC.

    COUPLE this with the back to school season and the new flagship launches next quarter and the NAND makers will be screaming they have no supply.


    FYI, if you get the Inland drive, make SURE it's the Premium and not the Professional. The Pro model is a bit cheaper (like $10) but is a fair amount slower since it uses an older controller.
     
  29. TheOne5

    TheOne5 Limp Gawd

    Messages:
    149
    Joined:
    Oct 26, 2012
  30. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    28,332
    Joined:
    Oct 29, 2000
  31. tangoseal

    tangoseal [H]ardness Supreme

    Messages:
    7,511
    Joined:
    Dec 18, 2010
    Yes they are right now...

    Sabrent 1TB Rocket Nvme PCIe 4.0 M.2 2280 Internal SSD Maximum Performance Solid State Drive (SB-ROCKET-NVMe4-1TB) https://www.amazon.com/dp/B07TLYWMYW/ref=cm_sw_r_cp_apa_i_xPAgDbWT608JT
     
  32. IdiotInCharge

    IdiotInCharge [H]ardForum Junkie

    Messages:
    11,352
    Joined:
    Jun 13, 2003
    Twice the speed, twice the price, that's fair right?
     
    mikeo likes this.
  33. GoodBoy

    GoodBoy [H]ard|Gawd

    Messages:
    1,454
    Joined:
    Nov 29, 2004
    If a 15 minute outage is going to take weeks to reset and recover from, they can't just 'cancel 3rd shift'. It likely has to stay going 24/7. Cancelling a shift every night would be worse than this 'outage'.

    They could shut down some of the parallel processing lines (guessing that there are some) to cut production, but the operating lines would still be 24/7.
     
  34. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    28,332
    Joined:
    Oct 29, 2000
    Fair. My comment wasn't specific to their particular production line, it was intended as an example of something simple they could do and scale back volume. The takeway is that they don't need to fabricate stories.
     
    Last edited: Jul 1, 2019
    workshop35 likes this.
  35. alxlwson

    alxlwson You Know Where I Live

    Messages:
    6,189
    Joined:
    Aug 25, 2013
    I read in one of the articles that this outage destroyed an entire quarter's worth of parts. Sounds fishy
     
  36. Zarathustra[H]

    Zarathustra[H] Official Forum Curmudgeon

    Messages:
    28,332
    Joined:
    Oct 29, 2000
    I'm guessing (or at least hoping) that's just a "first to market" price hike that will come down as the competition launched theirs...

    ...provided NAND doesn't explode in price by then, of course.
     
    IdiotInCharge likes this.
  37. westrock2000

    westrock2000 [H]ardForum Junkie

    Messages:
    9,060
    Joined:
    Jun 3, 2005
    I thought I heard about, maybe it was a HF contamination (Hydroflouric Acid) that took out someones quarter recently. But, in my search, I found an article from Jan 2019 about TSMC having to scrap 10,000 wafers because of a chemical contamination. This stuff happens, usually more often that we like to tell the world. Game of dominoes, this industry.
     
  38. alxlwson

    alxlwson You Know Where I Live

    Messages:
    6,189
    Joined:
    Aug 25, 2013

    Agreed...However...fishy.

    I work in all areas of industry. Semicon, automotive, chemical, all of it. Not in a single plant have I ever seen an entire FQ's worth of parts all in such a vulnerable stage as what we are being told happened here. Unless... Unless the PLC and database that tracks everything as it is actively moving through the line got completely fucked and there is no way to tell what part is what other than a barcode of some type. Even then....
     
  39. extide

    extide 2[H]4U

    Messages:
    3,424
    Joined:
    Dec 19, 2008
    Ok so here is what is fishy to me. Let's say the power outage happened and they had no backups. So that would mean every wafer in progress is trashed and it would take about 2 weeks to spin the plant back up. So they would have 2-3 weeks of lost production. They are claiming to have lost the equivalent of half of an entire QUARTER's worth of output. A quarter is 3 months or about 13 weeks. That math doesn't add up?!
     
  40. GoodBoy

    GoodBoy [H]ard|Gawd

    Messages:
    1,454
    Joined:
    Nov 29, 2004
    Yeah, how long does it take for a wafer to get thru the fab anyway? I don't know the answer, but several months seems ridiculous and too slow.

    Everything that was in process, scrapped. But how much is this?

    2 ish weeks or production, not produced due to the recovery from the issue. Fine, but how much output was lost?

    Neither of the above seem capable of half of a quarters' (6.5 weeks) worth of loss.