So, I need 24 TB of storage...

RazorWind

Supreme [H]ardness
Joined
Feb 11, 2001
Messages
4,106
Here's my situation:

My employer has recently taken delivery of an airborne LIDAR system that, at full spread, produces over a terabyte of data per day of flying. We'll be taking delivery of an add-on to it in a month or two that produces an additional terabyte of data per day. That's ~2TB for every day that we're in the field. A typical field campaign is about 5 days.

The data comes out of the system on a series of ordinary SATA SSDs. I need to build a computer that can copy the data off of them, and onto a set of large spinning disks so that:
1. We need to have at least two copies of the data - one working copy, and one backup.
2. It is enormously preferable that those two copies be two huge ~12 TB volumes, as opposed to say, four 3TB disks. (We need to automate some of the process of moving the data around, and that becomes much more difficult if the target is more than one disk)

My question is, can anyone recommend either a single RAID controller that will handle ten disks (two arrays of 5 3TB disks, RAID 5) or a model that can peacefully coexist with an identical one? (Two controllers, each with a single array of 5 3TB disks, RAID )

Also, if this needed to be semi-portable, can anyone recommend a case?
 

Dark Shade

[H]ard|Gawd
Joined
May 2, 2006
Messages
1,872
Semi-portable, as in it will be picked up and moved around often, or shuffled around from site-to-site?

You might want to consider a SAN on a small roller-rack unit. We have a SAN tray in production with 12 2TB SATA disks purely for DB backups and it works flawlessly through iSCSI.
 

shabazkilla

Lurker
Joined
Jan 4, 2004
Messages
439
What's your budget?

Once the data is off the LIDAR system and onto this workstation does it stay there, or does it then get moved to a NAS/SAN for long term storage?

Is the workstation just storing the data, or does it run an app or otherwise manipulate the data?

My initial though is a SuperMicro 3u or 4u rackmount in a portable rack. Get one with a SAS expander and use an LSI controller.
 

RazorWind

Supreme [H]ardness
Joined
Feb 11, 2001
Messages
4,106
Semi-portable, as in it will be picked up and moved around often, or shuffled around from site-to-site?

You might want to consider a SAN on a small roller-rack unit. We have a SAN tray in production with 12 2TB SATA disks purely for DB backups and it works flawlessly through iSCSI.

Portable, as in, we'll plug it in in a hotel on day 0 of the survey, fly on days 1 through 5, and then put it in an SUV on day 6 and drive it back to HQ.

I guess what I had in mind looks more like a big meaty workstation PC, and not any sort of rackmount setup.

What's your budget?

I'm hesitant to give any number, because I don't really know. I guess the best way to answer this is "whatever a good solution to this problem costs."

Once the data is off the LIDAR system and onto this workstation does it stay there, or does it then get moved to a NAS/SAN for long term storage?

Is the workstation just storing the data, or does it run an app or otherwise manipulate the data?

I should have mentioned that. Ideally, the machine would run windows 7, and mount the large storage as local volumes. There is some processing that needs to be done after the mission is flown to make sure that we got adequate geographic coverage. The hardware requirements for that are easy - a regular GTX 680, fast Core i7 and 16 to 32GB of memory are more than enough.

This could be done with two machines if necessary, but for logistical reasons, it makes more sense to me personally to do it with one, and then build another for redundancy, if needed.

When we get back to HQ, the backup copy of the data will be copied to a NAS (which we already have), and then the processing is done again on the whole dataset.

My initial though is a SuperMicro 3u or 4u rackmount in a portable rack. Get one with a SAS expander and use an LSI controller.

I addressed this earlier, but I'm not sure rackmount hardware is the answer. The racks themselves would add quite a bit of weight that we have to move around, which would make the logistics of our campaigns more difficult than we really want.
 

CrazyRob

[H]ard|Gawd
Joined
Aug 26, 2004
Messages
1,273
I've built an inexpensive redundant backup array in my office (an offsite backup for backup servers at remote offices) using an lsi 8888elp controller with bbu and a 16 bay sas expander enclosure (similar to this refurbished bundle).

I'm no storage guru, and there's a lot of better informed storage pros on here, but this was an easy and inexpensive way to get a large array up and running without too much hassle. Currently running 11x 2TB drives in RAID6.

*edit* I just read that you're trying to avoid racks/rackmount. Cramming that much storage into a single chassis is gonna be a tight fit. Additionally, to my knowledge, there aren't many internal hotswap bays that offer sas expandability, which would mean each drive would need it's own channel, increasing the cost of your raid controller.
 
Last edited:

hotcrandel

Gawd
Joined
Feb 26, 2010
Messages
781
Have you considered distributing the data through multiple systems? I would be reluctant to keep all my "Eggs in one basket". I would capture a days worth of data, and back it up on to two systems. Use a NAS program like freenas or OI to store the data. Use small form factor cases to make the systems easy to move, and keep the original shipping boxes to provide them some protection.

I've built a few systems for around the office for various backups with the Lian-Li case mentioned below, and I am really happy with it. Each of these systems is fairly light weight and all three could sit in the back seat of a car easily enough, or fit into a larger packing crate that hopefully is nice and padded. Even having a whole system fail, you should be able to piece something working together. Use simple automation programs, and a gigabit switch to back up the data between systems.

Primary Landing System; Estimated Cost - $2200 (Why the GTX 680? gotta game on the road? :))

Lian-Li Q25B Mini File Server Case (Link)
Intel Based mITX board (Link)
i7 processor (Link)
16GB Memory (Link)
SSD Bootable Disk (Link)
2x 4tb Hitachi disks (Link) (RAID1, 1 days worth of data)
GTX 680 (Link)
650W Supply (Link)

Data Storage System; Using OI or solaris running ZFS RAIDZ. Use onboard ports and the CPU to do the parity calculations instead of a big fancy raid card.
Estimated Cost ~$1725, or $3550 for 2 systems.

(2) Lian-Li Q25B Mini File Server Case (Link)
(2) Intel Based mITX board (Link)
(2) 16GB Memory (Link)
(2) i3 Processor (Link)
(2) SSD's for OS (Link)
(8) x 4tb Hitachi disks (Link) (RAIDZ, 5 days worth of storage space)
(2) 380w power supplye (Link)
 

RazorWind

Supreme [H]ardness
Joined
Feb 11, 2001
Messages
4,106
One approach I'm considering is to use two of these in a full-size tower case, with 10 drives, plus an SSD connected to the motherboard with Windows on it for the OS and apps. This appears to offer all the features I need:
http://www.newegg.com/Product/Product.aspx?Item=N82E16816117214

Possibly this specific model of drives. I'm mostly brand agnostic in this regard, but the guy who buys most of our hard drives seems to have a pathological fear of Hitachi for some reason, and the availability of the Western Digital equivalent appears to be a little thin right now.
http://www.newegg.com/Product/Product.aspx?Item=N82E16822148757

The case is not that huge a problem - there are plenty of sufficiently large ATX cases. I asked about the case mostly because I was hoping someone knew about one with wheels like a piece of roll-on luggage, and I didn't want to make a thread in the case mod forum just for that.
Failing the availability of one with wheels, something like this ought to do, I would think:
http://www.newegg.com/Product/Product.aspx?Item=N82E16811119252

I could just get a little luggage rack thing for it, if I needed to, like this.

Anyway, If I can use two of those RAID controllers on one motherboard, I *think* I'm pretty much set.

The Afraid-of-Hitachi guy seems to think that you can never use more than one RAID controller on any given system. I'm pretty sure that's not true - I've found plenty of anecdotal evidence suggesting you can, but none of the RAID controller manuals that I've looked at mention it specifically. Anyone know?
 

RazorWind

Supreme [H]ardness
Joined
Feb 11, 2001
Messages
4,106
Have you considered distributing the data through multiple systems? I would be reluctant to keep all my "Eggs in one basket". I would capture a days worth of data, and back it up on to two systems. Use a NAS program like freenas or OI to store the data. Use small form factor cases to make the systems easy to move, and keep the original shipping boxes to provide them some protection.

I've built a few systems for around the office for various backups with the Lian-Li case mentioned below, and I am really happy with it. Each of these systems is fairly light weight and all three could sit in the back seat of a car easily enough, or fit into a larger packing crate that hopefully is nice and padded. Even having a whole system fail, you should be able to piece something working together. Use simple automation programs, and a gigabit switch to back up the data between systems.

Primary Landing System; Estimated Cost - $2200 (Why the GTX 680? gotta game on the road? :))

Lian-Li Q25B Mini File Server Case (Link)
Intel Based mITX board (Link)
i7 processor (Link)
16GB Memory (Link)
SSD Bootable Disk (Link)
2x 4tb Hitachi disks (Link) (RAID1, 1 days worth of data)
GTX 680 (Link)
650W Supply (Link)

Data Storage System; Using OI or solaris running ZFS RAIDZ. Use onboard ports and the CPU to do the parity calculations instead of a big fancy raid card.
Estimated Cost ~$1725, or $3550 for 2 systems.

(2) Lian-Li Q25B Mini File Server Case (Link)
(2) Intel Based mITX board (Link)
(2) 16GB Memory (Link)
(2) i3 Processor (Link)
(2) SSD's for OS (Link)
(8) x 4tb Hitachi disks (Link) (RAIDZ, 5 days worth of storage space)
(2) 380w power supplye (Link)
I kind of like this idea, although I'm not sure my coworkers will. I certainly haven't ruled out the possibility of using multiple machines, but it would be easier to sell them on one box with redundancy inside it.

It should be noted that while I will likely be on-site at the time, I am not the intended user for this machine in the long run. My primary function is to ride in the aircraft, and operate the LIDAR, with supporting the ground operation being a secondary duty. I'm the only one in the group with anything resembling strong IT skills, though.

The GTX 680 is there to run a piece of custom software for processing the LIDAR data that relies heavily on CUDA, and to run the processing software, which does some 3D visualization. The folks who wrote the software didn't seem to think the extra cost of Quadro cards was worth it for this application, so I took their word for it.
 
Joined
Sep 14, 2008
Messages
1,622
You can have a single controller do this with a SAS expander but even if you don't ARC-1880 or ARC-1882 has some of the best disk compatability and is lighting fast on raid6 (2.5 GB/sec read 2 GB/sec write on a 24x3TB raid6 volume with 11 hour rebuild on my testing on a ARC-1882).

I use a ARC-1280ML + ARC-1880x in my machine and they both work together just fine:

Code:
Copyright (c) 2004-2011 Areca, Inc. All Rights Reserved.
Areca CLI, Version: 1.85, Arclib: 310, Date: Aug 22 2011( Linux )

 S  #   Name       Type             Interface
==================================================
[*] 1   ARC-1280   Raid Controller  PCI
[ ] 2   ARC-1880   Raid Controller  PCI
==================================================

Code:
root@dekabutsu: 02:16 PM :~# cli64 vsf info
CLI>   # Name             Raid Name       Level   Capacity Ch/Id/Lun  State
===============================================================================
  1 WINDOWS VOLUME   48TB RAID SET   Raid6    129.0GB 00/00/00   Normal
  2 MAC VOLUME       48TB RAID SET   Raid6     30.0GB 00/00/01   Normal
  3 LINUX VOLUME     48TB RAID SET   Raid6    129.0GB 00/00/02   Normal
  4 DATA VOLUME      48TB RAID SET   Raid6   43712.0GB 00/00/03   Normal
===============================================================================
GuiErrMsg<0x00>: Success.

CLI> GuiErrMsg<0x00>: Success.

CLI>   # Name             Raid Name       Level   Capacity Ch/Id/Lun  State
===============================================================================
  1 DATA 2 VOLUME    90TB RAID SET   Raid6   84000.0GB 00/01/00   Normal
===============================================================================
GuiErrMsg<0x00>: Success.

CLI>
root@dekabutsu: 02:16 PM :~# df -H
Filesystem      Size  Used Avail Use% Mounted on
rootfs          129G   56G   74G  43% /
/dev/root       129G   56G   74G  43% /
rc-svcdir       1.1M  205k  844k  20% /lib/rc/init.d
udev             11M  246k   11M   3% /dev
tmpfs            26G     0   26G   0% /dev/shm
/dev/sda1       129G   78G   52G  61% /winxp
/dev/sdd1        44T   32T   12T  73% /data
/dev/sde1        84T   56T   29T  66% /data2

I forget if on areca you can use three or four raid controllers in a single machine.
 

Quartz-1

Supreme [H]ardness
Joined
May 20, 2011
Messages
4,257
Here's my situation:

My employer has recently taken delivery of an airborne LIDAR system that, at full spread, produces over a terabyte of data per day of flying.

I've dealt with something very similar. You're just storing the data, right? The data may well be very compressible - of the order of 100:1. The trick is to use 7-zip with an enlarged frame size to compress it. Compressing such a large amount of data is, of course, non-trivial, and best left to run overnight. You might also consider a tape system.

Beyond that, this needs to be robust and is such a large amount of data that I would question using HDDs at all and suggest an all SSD setup. For a mission-critical application like this I would suggest RAID 6 rather than RAID 5. Note that 1 TB SSDs are not cheap.
 

wkearney99

Limp Gawd
Joined
Sep 8, 2010
Messages
367
I would not dismiss using a decent case designed for moving equipment around. Stuff gets abused, accidents happen, etc. Better to have that data in a case designed to handle at least a bit of abuse. It'd be a shame to have all the data get lost due to using a desktop case that got dropped, hit or otherwise manhandled. And you plan on leaving this in the hotel room during the day? Unattended? Might be a bad plan to use anything that looks like a regular computer. I'd much rather have mini rack setup in a travel case. Better durability, easier to lug around without being 'too worried' about durability, and too big for the hotel staff to steal.
 

tschopp

n00b
Joined
Oct 13, 2011
Messages
15
I second the use of a good case. We have used Hardigg rackmount cases in the field. 2U or 3U supermicro w/ Areca raid card and you are all set. The cases have good handles, wheels, and shock protection. If you decide to Fedex that case keep in mind hotswap drives are not intended for shipping, fill the space between the drive bays and the shipping case with foam or the drives might come out.
 

Scotch77

Gawd
Joined
Jul 27, 2005
Messages
769
Here's my situation:

My employer has recently taken delivery of an airborne LIDAR system that, at full spread, produces over a terabyte of data per day of flying. We'll be taking delivery of an add-on to it in a month or two that produces an additional terabyte of data per day. That's ~2TB for every day that we're in the field. A typical field campaign is about 5 days.

The data comes out of the system on a series of ordinary SATA SSDs. I need to build a computer that can copy the data off of them, and onto a set of large spinning disks so that:
1. We need to have at least two copies of the data - one working copy, and one backup.
2. It is enormously preferable that those two copies be two huge ~12 TB volumes, as opposed to say, four 3TB disks. (We need to automate some of the process of moving the data around, and that becomes much more difficult if the target is more than one disk)

My question is, can anyone recommend either a single RAID controller that will handle ten disks (two arrays of 5 3TB disks, RAID 5) or a model that can peacefully coexist with an identical one? (Two controllers, each with a single array of 5 3TB disks, RAID )

Also, if this needed to be semi-portable, can anyone recommend a case?


Just make sure you do have a backup at all times and that the delta between backup and production is an acceptable loss. raid 5 cannot really support 15TB upon a rebuild.
 

Aesma

[H]ard|Gawd
Joined
Mar 24, 2010
Messages
1,854
Since my other passion aside from computers is everything that flies, can you tell what is the plane used ?
 

paret0

Limp Gawd
Joined
Aug 29, 2012
Messages
368
Here's my situation:

My employer has recently taken delivery of an airborne LIDAR system that, at full spread, produces over a terabyte of data per day of flying. ...




PLS here. Passing familiarity w/LIDAR. Call Oracle.

You can do it yourself easycheezy w/ZFS, but this type of data is big$$, and has a very long life cycle. You guys don't need to dick around trying to save $10k/yr on your LIDAR data cycle. You'll charge your clients more than that for 1 day's flying. ;)
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
32,894
People make fun of them around here, but I have had great luck with DROBO devices.

They have a few different 8 bay products that should suit your purposes just fine, and they are very easy to use. They manage themselves. Just pop whatever mismatched drives you want into them and they create the arrays for you.

You'd be looking at either the Drobo Pro: (for professionals)
http://www.drobo.com/products/professionals/drobo-pro/
8 Bays
up to 32TB
iSCSI / FireWire 800 / USB 2.0 connections

Or one of the Business models:
B800i:
http://www.drobo.com/products/business/b800i/index.php
8 Bays
up to 32TB
iSCSI

B1200i:
http://www.drobo.com/products/business/b1200i/index.php
12 Bays
up to 48TB
3x iSCSI

I use their little brother (the 5 bay Drobo S) and I am very happy with it.

The ease of managing these units can not be overstated.

They support RAID5 style drive redundancy (one redundant drive by default, configurable to two drives) with thin provisioning so you can set it up huge from the beginning and then expand by upgrading drives as you go.

After a quick GUI setup they entirely take care of themselves. They warn you if a drive has failed or if you just want to replace it with another you replace it, you just walk up, yank it out (being careful to not exceed the number of redundant drives out at any given time), and pop another one in, and it automatically rebuilds the array.

Any combination of drives can be used. The 12 bay model even accepts two SSD's for automated data aware tiering.

The professional models are fully validated for production server use.

Then simply hook one of these up to your favorite laptop, and viola! great system on the go.
 
Last edited:

Draugauth

Limp Gawd
Joined
Mar 29, 2007
Messages
302
I would recommend the Cooler Master Storm Trooper case. Integrated hand rail at the top. It's a big case with more than enough space to put all the hardware in it. I use it right now as a gaming rig but when I move over to rack mounted cases I'll probably turn it into a portable server for lan games.

Do you really need 8 cores for data processing? If not then go with an i5. They are the same as the i7's but without HT. So only 4 cores but still cheaper. On the other hand you could go with server/workstation quality stuff. I really like my Asus P8BWS Server/Workstation Motherboard with my Xeon E3-1230V2 CPU and 32GB ECC ram. It has 4 PCI-E X16 ports on it so it can handle heavy graphics with the right video cards. Memory, Motherboard, Processor, and Case will run you roughly 800$. Then you just need the raid controller and additional HDDs. If you have the 5 in 3 hot swap bays the case can handle 15 HDDs plus 2 SSD drives internally.

Hope this helps.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
32,894
OP is a narc

Firstly, Narc? What are you, some kind of criminal loser? You mean the upstanding law enforcement officers whose job it is to put criminal pieces of shit like you in jail? :D

Secondly, airborne LIDAR is really cool and used for lots of different things...

Some examples:

1.) Agricultural surveying:

Helps high tech industrial farmers figure a number of things, including where to apply the most costly fertilizers for best effect.


2.) Archaeological Surveying:

Can help archaeologists find previously hidden artifacts concealed by canopy or other things not directly visible to the human eye.


3.) Biology and Conservation:

Surveying the ground for such things as canopy heights, and generating topographic maps


4.) Geology and Soil Science:

Detailed topographical studies help in the fields of geomorphology as well as tectonic studies, looking for fault lines supporting work trying to predict earthquake outbreaks, among other things.


5.) Meteorology and atmospheric environment

Helps in studying storms and weather prediction (usually ground based and called "doppler" radar, but can sometimes be airborne as well.


6.) Military use:

It has been suggested that the military use airborne LIDAR for reconnaissance purposes, but details are classified.


7.) Construction surveying

For some large projects (bridges, hydroelectric dams, photovoltaic and wind power layouts, stuff that requires large and very detailed ground surveys) airborne LIDAR is the best way to get that data.



Long story short.

Law enforcement does use LIDAR guns to measure vehicle speeds, but they are typically not airborne.

Airborne speed enforcement is typically just a normal helicopter or plane with an operator who measures the time it takes a car to travel between two clearly marked lines (or other landmarks) on the ground. In other words, very low tech, and its not entirely clear that Airborne LIDAR would do much to improve on this method. The technology would possibly be overkill, and possibly not work for that purpose at all.
 
Joined
Aug 16, 2012
Messages
769
I won't go into my drobo horror stories here but it really sounds like a small nas is the way to go. You can use 4tb disks these days in a synology/thecus/drobo and 8x4=32tb raw at weight light enough to carry one handed.
 

Zarathustra[H]

Extremely [H]
Joined
Oct 29, 2000
Messages
32,894
I won't go into my drobo horror stories here but it really sounds like a small nas is the way to go. You can use 4tb disks these days in a synology/thecus/drobo and 8x4=32tb raw at weight light enough to carry one handed.

Maybe this isn't the place, but I am curious about your drobo horror stories.

I've had a 5 bay Drobo S hooked up to my Linux server for almost 2 years now, not a single issue. I love it, and how low maintenance it is.
 
Joined
Aug 16, 2012
Messages
769
Zarathustra[H];1039124604 said:
Maybe this isn't the place, but I am curious about your drobo horror stories.

I've had a 5 bay Drobo S hooked up to my Linux server for almost 2 years now, not a single issue. I love it, and how low maintenance it is.

I'm sure most drobo's are fine, they are still around as a company afterall. Here's one of the many complaint threads: http://hardforum.com/showthread.php?t=1700909 but basically, I've had three, all three failed within days of actual use. It's like any other product, if you had 3 fords in a row fail within 100 miles, you'd likely never buy another ford, but plenty of people do and are happy.
 

ND40oz

[H]F Junkie
Joined
Jul 31, 2005
Messages
12,297
The only problem I had with the drobo's is they can't talk across subnets, so as long as you're on the same subnet as it, it works fine. We replaced ours with a qnap because we needed to be able to reach it from multiple vlans.
 

J Macker

[H]F Junkie
Joined
Jan 25, 2001
Messages
10,172
Cramming that much storage into a single chassis is gonna be a tight fit.

8x3TB drives is easily doable in just about any standard mid-ATX case.
NZXT Source 210 elite is $50 case and has:
8x3.5 bays
3x5.25 bays

Get a 4-in-3 bay adapter and you can have 12 drives * 3TB = 36TB.

The controller is a different story. The cheapest route would be an IBM m1015 on ebay around $100. It has 2 sas breakouts for 8 sata ports.
It can be cross-flashed to an LSI9211 HBA 6.0gbps SAS / SATA III adaptor

Drivebender is a Windows7 tool for pooling hard drives. You do not have to have all hard drives connected to the one RAID card.
 

paret0

Limp Gawd
Joined
Aug 29, 2012
Messages
368
People are still suggesting little Drobo boxes for the narc's Big Data problem?...:p

Datalogging/storage is just the tip of the iceberg here....13TB of RAM to manipulate 12TB point clouds is what I wanna see. :D
 
Last edited:

wixter

Weaksauce
Joined
Mar 8, 2012
Messages
88
I would recommend the Cooler Master Storm Trooper case. Integrated hand rail at the top. It's a big case with more than enough space to put all the hardware in it. I use it right now as a gaming rig but when I move over to rack mounted cases I'll probably turn it into a portable server for lan games.

I actually would advise the OP against this particular case. I was initially considering the Storm Trooper too for a new build but decided against it and went with the old reliable HAF932 series. I made sure to stop by Frys to examine a Storm Trooper case before buying and I was sorely disappointed at how much plastic is being used in cases today whereas they may have previously used light steel or aluminum.

Especially in the OP's situation having to haul it around, I think the Storm Trooper case would be doing well to last even 6 months. I don't have a particular case suggestion except to say it better be built with metal of some kind.
 

RazorWind

Supreme [H]ardness
Joined
Feb 11, 2001
Messages
4,106
People are still suggesting little Drobo boxes for the narc's Big Data problem?...:p

Datalogging/storage is just the tip of the iceberg here....13TB of RAM to manipulate 12TB point clouds is what I wanna see. :D

Actually, the processing isn't that big of a deal. I/O is typically the limiting factor, even on relatively slow hardware. Almost all of the processing and interpretation can be done on a robust, but otherwise ordinary desktop machine.

Nevertheless, this is in the building next to us, and we're pretty good buddies with the guys that run it:
http://www.tacc.utexas.edu/resources/hpc;jsessionid=73E9301F33C510D98BF43E0B4A436ACF.jvm2

Also, for the record, we are not a law enforcement agency. I work for the Bureau of Economic Geology, at the University of Texas at Austin. Our primary mission is scientific, usually related to geology in some way, and we turn down requests for surveys that don't offer us any sort of scientific application for the data.

Anyway, an update:
I took the advice some of you guys gave me, and went with rackmount hardware. This increased the size and weight of the system substantially, but I think it will be much better protected, and the logistics won't be that big of a problem.

I'd also reexamined our needs, and realized that the actual amount we were going to need eventually was a lot greater than I'd originally anticipated. Closer to 70 or 80TB than 20.

Here's a couple of photos of the finished system. As you can see, the enclosures are rackmount type, mounted in Pelican-Hardigg Blackbox racks.

lidar1.jpg


lidar2.jpg


Cases:
3 x Pelican-Hardigg "Blackbox" 4U portable racks.

The hardware specs are:
Antec rackmount server case
PSU: Seasonic Platinum series, 860 watt
Asus Sabertooth Z77 motherboard
Intel Core i7 3770S CPU
32GB of Corsair DDR3 memory
MSI GTX680 GPU
3 x 480GB Corsair Force series SSDs. One for the OS and software, two for use as scratch space for data processing

The storage arrays consist of:
2 x LSI MegaRaid 9280-8e RAID controllers
2 x Norco RAID enclosures, with Areca SAS expanders
2 x 12 Western Digital WD4000FYYZ 4TB hard drives, plus a few spares.

I ended up with one 12 bay enclosure and one 24 bay, because the 24 kept selling out, and we needed to go ahead and get the thing built, but the idea is that we could add additional drives with relatively short notice if the need arose.

lidar3.jpg


I give the Blackboxes about a 7 or an 8 out of 10. They seem VERY sturdy, but what sales folks don't mention is that they come with threaded rack holes, which makes mounting IT gear in them a GIANT pain in the neck if adapters aren't available for your rails. I ended up having to modify the rails to get the rackmount things to fit in the boxes.

Each of the boxes weighs about 50 pounds by itself, with another 40 to 60 pounds of hardware inside, making the total weight of the system about 300 pounds. If it were one giant unit, it would be unmanageable for us to move it around, but in three parts, it's not what I'd call fun, but is manageable with two people.

I sort of wish I'd picked a different motherboard, as the Sabertooth seems to only barely have enough PCI express lanes for our needs, but it gets the job done, and seems very well made.

The LSI RAID controllers are also kind of a 7/10 deal. I had a drive fail during the first 24 hours of runtime, and after replacing it, I could not for the life of me convince that controller to not beep incessantly after a reboot. The UI would report both arrays as optimal, with everything working properly, but the alarm would still go off every time it rebooted. According to LSI support, this was the result of a stuck setting in the NVRAM, and there's an elaborate procedure for fixing it. I haven't tried that yet, as I just swapped the spare I bought in.

Other than that one issue, though, the LSI cards seem to be excellent. They're very easy to set up and use, and I was impressed by how easy it was to recover from a drive failure. Just yank the bad one, stick a fresh one in, and away it goes.
 

RazorWind

Supreme [H]ardness
Joined
Feb 11, 2001
Messages
4,106
Benchmarks from the RAID arrays.

array_bench1.png

array_bench2.png


The second one is running out of bandwidth across the PCI-E bus, as it only has four lanes available to it, hence the lower performance. It's not an ideal situation, but it's almost twice as fast as the SSDs from the instrument seem to read, so it's not really a big deal. Plenty fast for our needs.

Each array is 12 4TB WD4000FYYZ enterprise-class disks, set up in RAID 5. The arrays are in physically separated enclosures, in sturdy portable racks, each with its own separate controller. One array is the primary, and the other is a backup.

I've considered going to RAID 6, as I could get away with the loss in volume size for now, but I'm sort of tired of messing with it, and I have at least two layers of redundancy as it is. I've got a couple of weeks to decide if I want to mess with it.
 

Lui

n00b
Joined
Aug 18, 2011
Messages
15
Well in 6 years of LIDAR data acquisition and processing I find out that this solution works best for me:
- I use 3 level of data: raw data (the largest), geolocated nonclassified point cloud (scanlines matched), working data
- as soon as possible convert data from LAS format into LAZ via LASzip (loseless, 10 - 15 times less space used (probably just after georeferencing)
- for processing use SSD (2x512GB) in RAID 0 (you can have backup data on HW RAID6)
And your solution seems quite fine to me.
 

RazorWind

Supreme [H]ardness
Joined
Feb 11, 2001
Messages
4,106
Oh, cool. What type of system have you been using?

Ours is a new design made by AHAB (http://www.airbornehydro.com). The processing steps are a little different from that of an Optech system, and we'll be pairing it with a suite of hyperspectral cameras. The result is that only 25% or so of the data, by volume, will actually be LIDAR points. The rest would be images from a suite of RGB and hyperspectral cameras, and I'm not sure how well those will respond to compression.

In the field, we're generally just concerned with capturing the raw data, and doing some basic processing to make sure all the equipment worked properly, and we didn't make some stupid procedural mistake, like flying too low. I don't think the plan involves producing LAS files until we make it back to the office, as the AHAB software has some other format it uses, and our processing guy has a different approach he likes better for doing the initial post-flight work.

I don't actually handle the processing, though. My part of the work is mostly done when they shut the airplane down.
 

Lui

n00b
Joined
Aug 18, 2011
Messages
15
Who would think that LIDAR people can meet on data storage forum? But I guess that we have a common problem how efficiently store all our data .
Well regarding LIDAR I'm now mostly in processing and data fusion but I was all-in-one: flight planner, operator (heli) with Riegl LMS560, GPS postprocessing, point cloud processing and data extraction. It was hard and demanding but I gathered knowledge like never before. Actually I'm now more in TLS (terrestrial) scanning and modeling business but LIDAR, especially classification tunning is still fun.
Well I also can say that LIDAR isn't such a volume problem as are photos. 1TB per day is rather normal amount of data (LIDAR + photos) so my knowledge of building custom NAS for offline backup seem to be still quite useful.
BTW I did some test flights with green laser (Riegl) and results on clear and still water were amazing. Unfortunately we were working mostly in river hydrology where test results were not good enough.

Well we can continue and share our experience via PM. I'm afraid that LIDAR is far to off-topic.
 
Top