Need 2TB SSD for IO intensive work

Discussion in 'SSDs & Data Storage' started by M76, Nov 14, 2015.

  1. Baddreams

    Baddreams [H]Lite

    Messages:
    113
    Joined:
    Aug 4, 2004
    I hear ya. Though the pig known as WSUS might actually benefit here :)
     
  2. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    Unfortunately the offices are located at a site where the only available internet connection is a badly lagging microwave service. So anything offsite is useless. But even if this wasn't the case, transferring the amount of data we're working with to a cloud vm service would take ages. We're already struggling when juggling data between hard drives locally.

    All our hardware purchases must go trough one specific company, which sometimes mean we wait weeks for hardware readily available elsewhere, and of course often pay 20-30% more than the best offer. This is a private owned company, corruption is built into the system, and it's totally legal.

    Of course I can do that, but the point is to speed up the processing. If I ran every task sequentially it would take 24 hours what I can do in 4 now, even with the 840 EVOs.

    We did that out of necessity, 4-5 years ago when we didn't have enough SSD space in the processing machines,

    If I do it sequentially then that means I'm doing it in one thread, so the bottleneck instantly becomes the single thread performance of the CPU. Even with spinning drives I can do 2 tasks simultaneously before the drives starts to become the bottleneck if the input / output is not on the same drive.

    As they say you cook with what you have in the kitchen. Actually the X58 machines were purchased as high end workstations over 4 years ago. We've been using them for this since then. I don't think using server boards would've helped anyway. As the bottleneck was the storage ever since. The H87 computers were only assigned to us when we were doing a big job last year and we asked for more computers, and that's what was readily available. They just stuck with us.

    I Already did my recommendation, now it's time to wait and see what happens. Best case I get P3700, worst case I get 850PRO SSDs, I'll still be ahead.

    It's actually possible to do that right now, but it's not beneficial, as the network connection would become the bottleneck. One work unit is about 1 Gbytes and the output is one magnitude larger. By the time it's transferred over the network it's already processed locally.

    Thanks for the answers everyone, I'll report back what's the decision, when it actually happens.
     
    Last edited: Nov 20, 2015
  3. Lui

    Lui n00b

    Messages:
    15
    Joined:
    Aug 18, 2011
    Hm you said that you process LIDAR data? I'm processing LIDAR data on site and with all reasonable QC it takes approx. 6 hours to obtain georeferenced pointcloud. Most of this time is GNSS/IMU trajectory postprocessing. On site we use some neat network storage (+150TB), processing node and 10GB network all packed into 15U rack.
    You are probably using fullwaveform LIDAR so you have to calculate discrete echos?
     
  4. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    The Trajectory post processing takes the least amount of time in the workflow. 30 minutes at most for a full day of acquisition. But that means the scanner was running for 10-12 hours.

    The waveform processing is done by the scanners on board, but we still need to calculate individual echoes. But georeferencing the points takes more time than calculating echoes. Then we adjust the trajectory for the entire mission based on overlapping point clouds to ensure there is no ghosting. And when that's done the georeferencing is run again from scratch based on the improved trajectory. Then we use the acquired imagery to assign RGB values to each point. Then we export both the LIDAR and the Camera data. Even then we're not finished, as we still need to transform the LIDAR data into a local coordinate system. And a full day's data is roughly 10 billion points. And 300.000 photos.

    A 10GB network would be nice, we were promised that we'll have that last year. Instead they sent us to the back of nowhere, into a crappy old office building, and noone spoke of the 10GB network or the storage we were promised ever since.
     
  5. rsq

    rsq Limp Gawd

    Messages:
    246
    Joined:
    Jan 11, 2010
    Wow, it really does not sound like you are in an agile company. Fixed supply chain, firm hierarchies, you don't even control your own budget...

    Reading Dilbert is probably painful for you because it rings true.

    Your last post seems to indicate a lot of floating point math involved. I would look into GPU assist here. If you cannot get the time used by the disks down, work on the processing time.
     
  6. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    Most of the processing is done with proprietary software. GPU processing is already implemented for waveform processing, exactly the one thing we don't need for our scanners. We can only hope that they'll take it further.
     
  7. BunnyKillBot

    BunnyKillBot n00b

    Messages:
    33
    Joined:
    Mar 25, 2012
    Another random thought, assuming it's the on board controller, what is the disk mode set to in the bios and are you confident it's ahci? If it's a default Ide that may be your issue.
     
  8. westrock2000

    westrock2000 [H]ardForum Junkie

    Messages:
    8,951
    Joined:
    Jun 3, 2005
    Why not just RAID0 two commercially available 1TB drives? That would be about $700-800.

    Use could use a normal HDD to do periodic backups if concerned about dataloss.
     
  9. drescherjm

    drescherjm [H]ardForum Junkie

    Messages:
    14,277
    Joined:
    Nov 19, 2008
    BTW, the 1TB Sandisk Ultra II is on sale for $199 at Amazon and other places. This is an excellent deal.

    http://hardforum.com/showthread.php?t=1882735
     
  10. HammerSandwich

    HammerSandwich [H]ard|Gawd

    Messages:
    1,104
    Joined:
    Nov 18, 2004
    I'd say that's a poor choice for the OP, because it also uses TLC & benches slower than the 840 Evo.
     
  11. drescherjm

    drescherjm [H]ardForum Junkie

    Messages:
    14,277
    Joined:
    Nov 19, 2008
    You are correct. Forgot the original topic of the thread..
     
  12. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    But that's what we're doing right now. Using two commercially available SSDs (840 EVO 1TB) in RAID0. Raid just gives additional overhead that slows down the processing.
     
  13. jyi786

    jyi786 [H]ardness Supreme

    Messages:
    5,460
    Joined:
    Jun 13, 2002
    RAID 0? Are you serious???

    But maybe not an issue because you are not worried about data loss, right?
     
  14. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    OP already covered this in the first page or two
     
  15. Beaflag VonRathburg

    Beaflag VonRathburg Limp Gawd

    Messages:
    399
    Joined:
    Aug 3, 2015
    I know you've already submitted your request, but Intel 750 SSDs sound right up the alley of what you're looking for. The 1.2TB drives are $699.99 on Newegg right now. Two of those in raid 0 would provide you with a giant increase in performance. Your pricing wasn't in USD, but maybe they price match.

    On an unrelated note... From your description of things, it sounds like you might want to think about new employment. Having to deal with the things you're describing sound like an unnecessary amount of stress you're putting yourself through. Use your current employment as the spring board for a new opportunity free of all of the issues you're dealing with. You can take that for what it's worth.
     
  16. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    issue is they ahve to use a single supplier so newegg is not an option. If i worked there for my own sanity i would just buy a damn 750 and use it and say fuck the company
     
  17. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    As long as I get paid while I'm waiting for the computer to finish, they can fool around with consumer level hardware, I don't care. I don't stress about it at all.
     
  18. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    I've just checked and intel 750 SSDs are only warranted for 127TBW and 70GB / day. That's less than what Samsung offers with the 850PRO SSDs.
     
  19. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    219 TBW IIRC which is 70 GB per day but yes their warranty sucks but they should last much longer. Your buying them for the latency or lack of and speed. Kingston predators have an amazing warranty and are also fast but they are not NVMe and only 480GB.
     
  20. evilsofa

    evilsofa [H]ardForum Junkie

    Messages:
    10,078
    Joined:
    Jan 1, 2007
    The datasheet states 70 GB Writes Per Day, Up to 127 TBW. Yes, that math doesn't seem to add up, but they aren't saying "70 GB Writes Per Day Every Day For 5 Years", they're saying "70 GB per day until you hit 127 TBW".
     
  21. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    NM. it was 219 TBW but they updated the documentation. I had it reverse.
     
  22. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    70GB / day is laughable. You install GTA5, and another big game, and you spent 100 GB including the download.

    For what I'm doing even 700GB writes is considered a slow day.

    Anyway we got the green light for two new workstations with P3600 1.2TB SSD.. It's far from ideal, because of the 1.2TB, but it should be enough for most of our work.
     
  23. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    its a great card. getting above 1TB right now is just not practical due to advancements in NAND. You'll see 2TB and larger drives being common by the end of 2016 and 2017 with the new 3D processes.
     
  24. Quartz-1

    Quartz-1 [H]ardness Supreme

    Messages:
    4,257
    Joined:
    May 20, 2011
    Then what you need to do is make the business case, not the technical case, to him. And you need to do it in writing. In particular, you should show that the costs can be recouped within a year.
     
  25. lutjens

    lutjens Gawd

    Messages:
    792
    Joined:
    Feb 18, 2013
    Fortunately, my corporate department has no such issues...the last 2 of those awesome drives are on their way to me now...;)

    Thanks for the head's up...:)
     
  26. HammerSandwich

    HammerSandwich [H]ard|Gawd

    Messages:
    1,104
    Joined:
    Nov 18, 2004
    Please update with results when you have them!
     
  27. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    Still don't have the drive, or new workstation. But got an offer from HP

    HP Z Turbo Drive Quad Pro

    ~1800US for 4x512GB version.

    The only drawback I see is that you have to use software raid to use it as a single 2TB drive.
     
  28. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    that might hamper your CPU if it isnt robust.
     
  29. cbf123

    cbf123 n00b

    Messages:
    50
    Joined:
    Dec 12, 2013
    Just wondering if you could skip the RAID entirely and set things up so that you're generally reading from one disk and writing the results to another?
     
  30. Quartz-1

    Quartz-1 [H]ardness Supreme

    Messages:
    4,257
    Joined:
    May 20, 2011
    Did you make the business case?
     
  31. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    That would work with 2 1TB drives. Definitely not with 4x512. But even 2x1TB could mean trouble processing larger projects.
     
  32. M76

    M76 [H]ardForum Junkie

    Messages:
    9,002
    Joined:
    Jun 12, 2012
    They already gave the go ahead last December to buy a single new workstation as a test case. But they're taking their sweet time actually putting in the order. I gave acquisitions my recommendations back then, and I only saw the first offers this week. One of them was from HP. That OFC contained a bunch of unnecessary shit bloating price while not meeting a few of the requirements I clearly defined as a must.

    The only consolation that we didn't have any projects recently that would have benefited from the new workstation. I'm hoping we can get something before long.
     
  33. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    I loathe that type of management....i loose my minds when stuff like that happens. Such a waste.
     
  34. SomeGuy133

    SomeGuy133 2[H]4U

    Messages:
    3,447
    Joined:
    Apr 12, 2015
    hey so hows the 750 working?

    How much CPU do you atually use. Would tat HP thing work? How much space do you need? I am just curious because your work interests me in that regard because it seems unique in its needs. You could get 4TB RAID 0 going around Q3/Q4 :D

    Would you than be CPU limited? Like how long does this task take? How much time would be saved? curious thats all.