Need 2TB SSD for IO intensive work

I wish we could whitebox servers which would actually utilize this kind of speed, space, and durability... I can only whitebox our non-mission critical things like WSUS and cold storage/archive.... neither of which can be justified in getting hardware like this. :(

I hear ya. Though the pig known as WSUS might actually benefit here :)
 
Have you looked into Tintri's products?

Unfortunately the offices are located at a site where the only available internet connection is a badly lagging microwave service. So anything offsite is useless. But even if this wasn't the case, transferring the amount of data we're working with to a cloud vm service would take ages. We're already struggling when juggling data between hard drives locally.

http://www.ebay.com/itm/New-2015-IN...080693?hash=item3ab68211b5:g:2fIAAOSwnGJWTjOC

Get yourself two of those. Might be tough to get corporate to order something off ebay, but IMO the price is pretty good. It'll obliterate EVO's in the kind of workload you're doing.

All our hardware purchases must go trough one specific company, which sometimes mean we wait weeks for hardware readily available elsewhere, and of course often pay 20-30% more than the best offer. This is a private owned company, corruption is built into the system, and it's totally legal.

I try to offer some non-conventional options:

1.
I would try running the various steps sequentially and timing that.

If the bottleneck is the concurrent access on the SSD's it might be faster to not do multiple things at the same time. This used to be the case with spinning drives.

Of course I can do that, but the point is to speed up the processing. If I ran every task sequentially it would take 24 hours what I can do in 4 now, even with the 840 EVOs.

We did that out of necessity, 4-5 years ago when we didn't have enough SSD space in the processing machines,

It maybe also brings your in-flight volume down enough to use a RAM disk to bring up your sequential speed.

If I do it sequentially then that means I'm doing it in one thread, so the bottleneck instantly becomes the single thread performance of the CPU. Even with spinning drives I can do 2 tasks simultaneously before the drives starts to become the bottleneck if the input / output is not on the same drive.

It does worry me that you mention X58 and H87 mainboards. It indicates you are using regular computers to run this very heavy workload.

As they say you cook with what you have in the kitchen. Actually the X58 machines were purchased as high end workstations over 4 years ago. We've been using them for this since then. I don't think using server boards would've helped anyway. As the bottleneck was the storage ever since. The H87 computers were only assigned to us when we were doing a big job last year and we asked for more computers, and that's what was readily available. They just stuck with us.

2.
Break chain of command and go directly to the boss that said you get whatever you need. Your stingy boss will not like you for doing that.

I Already did my recommendation, now it's time to wait and see what happens. Best case I get P3700, worst case I get 850PRO SSDs, I'll still be ahead.

3.
Use clever programming and farm out work to other workstations. If the parcels are small enough, it can all sit in RAM and be processed there. You would be able to use the resources of the receptionist's computer (at least the resources not used for solitaire). It would involve setting up worker agents on the other computers. There are frameworks to help you do that.

It's actually possible to do that right now, but it's not beneficial, as the network connection would become the bottleneck. One work unit is about 1 Gbytes and the output is one magnitude larger. By the time it's transferred over the network it's already processed locally.

Thanks for the answers everyone, I'll report back what's the decision, when it actually happens.
 
Last edited:
As an eBay Associate, HardForum may earn from qualifying purchases.
Hm you said that you process LIDAR data? I'm processing LIDAR data on site and with all reasonable QC it takes approx. 6 hours to obtain georeferenced pointcloud. Most of this time is GNSS/IMU trajectory postprocessing. On site we use some neat network storage (+150TB), processing node and 10GB network all packed into 15U rack.
You are probably using fullwaveform LIDAR so you have to calculate discrete echos?
 
Hm you said that you process LIDAR data? I'm processing LIDAR data on site and with all reasonable QC it takes approx. 6 hours to obtain georeferenced pointcloud. Most of this time is GNSS/IMU trajectory postprocessing. On site we use some neat network storage (+150TB), processing node and 10GB network all packed into 15U rack.
You are probably using fullwaveform LIDAR so you have to calculate discrete echos?

The Trajectory post processing takes the least amount of time in the workflow. 30 minutes at most for a full day of acquisition. But that means the scanner was running for 10-12 hours.

The waveform processing is done by the scanners on board, but we still need to calculate individual echoes. But georeferencing the points takes more time than calculating echoes. Then we adjust the trajectory for the entire mission based on overlapping point clouds to ensure there is no ghosting. And when that's done the georeferencing is run again from scratch based on the improved trajectory. Then we use the acquired imagery to assign RGB values to each point. Then we export both the LIDAR and the Camera data. Even then we're not finished, as we still need to transform the LIDAR data into a local coordinate system. And a full day's data is roughly 10 billion points. And 300.000 photos.

A 10GB network would be nice, we were promised that we'll have that last year. Instead they sent us to the back of nowhere, into a crappy old office building, and noone spoke of the 10GB network or the storage we were promised ever since.
 
Wow, it really does not sound like you are in an agile company. Fixed supply chain, firm hierarchies, you don't even control your own budget...

Reading Dilbert is probably painful for you because it rings true.

Your last post seems to indicate a lot of floating point math involved. I would look into GPU assist here. If you cannot get the time used by the disks down, work on the processing time.
 
Wow, it really does not sound like you are in an agile company. Fixed supply chain, firm hierarchies, you don't even control your own budget...

Reading Dilbert is probably painful for you because it rings true.

Your last post seems to indicate a lot of floating point math involved. I would look into GPU assist here. If you cannot get the time used by the disks down, work on the processing time.

Most of the processing is done with proprietary software. GPU processing is already implemented for waveform processing, exactly the one thing we don't need for our scanners. We can only hope that they'll take it further.
 
Another random thought, assuming it's the on board controller, what is the disk mode set to in the bios and are you confident it's ahci? If it's a default Ide that may be your issue.
 
Why not just RAID0 two commercially available 1TB drives? That would be about $700-800.

Use could use a normal HDD to do periodic backups if concerned about dataloss.
 
I'd say that's a poor choice for the OP, because it also uses TLC & benches slower than the 840 Evo.

You are correct. Forgot the original topic of the thread..
 
Why not just RAID0 two commercially available 1TB drives? That would be about $700-800.

Use could use a normal HDD to do periodic backups if concerned about dataloss.

But that's what we're doing right now. Using two commercially available SSDs (840 EVO 1TB) in RAID0. Raid just gives additional overhead that slows down the processing.
 
But that's what we're doing right now. Using two commercially available SSDs (840 EVO 1TB) in RAID0. Raid just gives additional overhead that slows down the processing.

RAID 0? Are you serious???

But maybe not an issue because you are not worried about data loss, right?
 
I know you've already submitted your request, but Intel 750 SSDs sound right up the alley of what you're looking for. The 1.2TB drives are $699.99 on Newegg right now. Two of those in raid 0 would provide you with a giant increase in performance. Your pricing wasn't in USD, but maybe they price match.

On an unrelated note... From your description of things, it sounds like you might want to think about new employment. Having to deal with the things you're describing sound like an unnecessary amount of stress you're putting yourself through. Use your current employment as the spring board for a new opportunity free of all of the issues you're dealing with. You can take that for what it's worth.
 
I know you've already submitted your request, but Intel 750 SSDs sound right up the alley of what you're looking for. The 1.2TB drives are $699.99 on Newegg right now. Two of those in raid 0 would provide you with a giant increase in performance. Your pricing wasn't in USD, but maybe they price match.

On an unrelated note... From your description of things, it sounds like you might want to think about new employment. Having to deal with the things you're describing sound like an unnecessary amount of stress you're putting yourself through. Use your current employment as the spring board for a new opportunity free of all of the issues you're dealing with. You can take that for what it's worth.

issue is they ahve to use a single supplier so newegg is not an option. If i worked there for my own sanity i would just buy a damn 750 and use it and say fuck the company
 
On an unrelated note... From your description of things, it sounds like you might want to think about new employment. Having to deal with the things you're describing sound like an unnecessary amount of stress you're putting yourself through. Use your current employment as the spring board for a new opportunity free of all of the issues you're dealing with. You can take that for what it's worth.

As long as I get paid while I'm waiting for the computer to finish, they can fool around with consumer level hardware, I don't care. I don't stress about it at all.
 
I know you've already submitted your request, but Intel 750 SSDs sound right up the alley of what you're looking for. The 1.2TB drives are $699.99 on Newegg right now. Two of those in raid 0 would provide you with a giant increase in performance. Your pricing wasn't in USD, but maybe they price match.

I've just checked and intel 750 SSDs are only warranted for 127TBW and 70GB / day. That's less than what Samsung offers with the 850PRO SSDs.
 
I've just checked and intel 750 SSDs are only warranted for 127TBW and 70GB / day. That's less than what Samsung offers with the 850PRO SSDs.

219 TBW IIRC which is 70 GB per day but yes their warranty sucks but they should last much longer. Your buying them for the latency or lack of and speed. Kingston predators have an amazing warranty and are also fast but they are not NVMe and only 480GB.
 
219 TBW IIRC which is 70 GB per day but yes their warranty sucks but they should last much longer. Your buying them for the latency or lack of and speed. Kingston predators have an amazing warranty and are also fast but they are not NVMe and only 480GB.

The datasheet states 70 GB Writes Per Day, Up to 127 TBW. Yes, that math doesn't seem to add up, but they aren't saying "70 GB Writes Per Day Every Day For 5 Years", they're saying "70 GB per day until you hit 127 TBW".
 
The datasheet states 70 GB Writes Per Day, Up to 127 TBW. Yes, that math doesn't seem to add up, but they aren't saying "70 GB Writes Per Day Every Day For 5 Years", they're saying "70 GB per day until you hit 127 TBW".

NM. it was 219 TBW but they updated the documentation. I had it reverse.
 
70GB / day is laughable. You install GTA5, and another big game, and you spent 100 GB including the download.

For what I'm doing even 700GB writes is considered a slow day.

Anyway we got the green light for two new workstations with P3600 1.2TB SSD.. It's far from ideal, because of the 1.2TB, but it should be enough for most of our work.
 
70GB / day is laughable. You install GTA5, and another big game, and you spent 100 GB including the download.

For what I'm doing even 700GB writes is considered a slow day.

Anyway we got the green light for two new workstations with P3600 1.2TB SSD.. It's far from ideal, because of the 1.2TB, but it should be enough for most of our work.

its a great card. getting above 1TB right now is just not practical due to advancements in NAND. You'll see 2TB and larger drives being common by the end of 2016 and 2017 with the new 3D processes.
 
The problem is the department head, whose reputation depends on the profit/cost ratio of the department, refuses to authorize the purchase of even a pencil unless it's absolutely needed. Typical small mindedness, where he doesn't realize that to produce more, we need tools that we can work faster with.

Then what you need to do is make the business case, not the technical case, to him. And you need to do it in writing. In particular, you should show that the costs can be recouped within a year.
 
As an eBay Associate, HardForum may earn from qualifying purchases.
Anyway we got the green light for two new workstations with P3600 1.2TB SSD.. It's far from ideal, because of the 1.2TB, but it should be enough for most of our work.
Please update with results when you have them!
 
Still don't have the drive, or new workstation. But got an offer from HP

HP Z Turbo Drive Quad Pro

~1800US for 4x512GB version.

The only drawback I see is that you have to use software raid to use it as a single 2TB drive.
 
Just wondering if you could skip the RAID entirely and set things up so that you're generally reading from one disk and writing the results to another?
 
Just wondering if you could skip the RAID entirely and set things up so that you're generally reading from one disk and writing the results to another?

That would work with 2 1TB drives. Definitely not with 4x512. But even 2x1TB could mean trouble processing larger projects.
 
Did you make the business case?
They already gave the go ahead last December to buy a single new workstation as a test case. But they're taking their sweet time actually putting in the order. I gave acquisitions my recommendations back then, and I only saw the first offers this week. One of them was from HP. That OFC contained a bunch of unnecessary shit bloating price while not meeting a few of the requirements I clearly defined as a must.

The only consolation that we didn't have any projects recently that would have benefited from the new workstation. I'm hoping we can get something before long.
 
They already gave the go ahead last December to buy a single new workstation as a test case. But they're taking their sweet time actually putting in the order. I gave acquisitions my recommendations back then, and I only saw the first offers this week. One of them was from HP. That OFC contained a bunch of unnecessary shit bloating price while not meeting a few of the requirements I clearly defined as a must.

The only consolation that we didn't have any projects recently that would have benefited from the new workstation. I'm hoping we can get something before long.

I loathe that type of management....i loose my minds when stuff like that happens. Such a waste.
 
hey so hows the 750 working?

How much CPU do you atually use. Would tat HP thing work? How much space do you need? I am just curious because your work interests me in that regard because it seems unique in its needs. You could get 4TB RAID 0 going around Q3/Q4 :D

Would you than be CPU limited? Like how long does this task take? How much time would be saved? curious thats all.
 
Back
Top