fast disks for distributed computing?

[Thuleman]

Folks, I could use some advice. I am currently in the process of deciding what hardware to buy for a project I am working on. I have a master computer that currently contains roughly 600 GB of source data (and growing), and then I have currenly 8 client machines (soon to be 18) that access the master, grab a chunk of data, process it, and return the result to the master machine.

The data blocks that are read and written are small, it seems to me less than 32k at a time, I could get the actual number from the developers I suppose.

The machines are behind a gigabit switch, and do make use of gigabit network speeds.

My question is, what would be a good setup to facilitate fast read and write access?

I'm considering to just create two four disk RAID0 setups in an 8 disk rackmount enclosure using 200GB or 250GB disks. Thus the source data could be read from one striped array and written to another striped array.
Anyone strongly opposed to that based of efficiency or performance?

Edit: Another question is (and I am not sure about which enclosure I am going to buy), is a 2 channel raid controller sufficient for 2 four disk arrays, or would I be considerably better off to have a controller that has as many channels as I a going to use drives?

Another Edit: Ultimately I will have to think of something to back everything up. Tapes are imho out of the question because they just don't hold enough data per cartridge. Any suggestions for a JBOD that doesn't cost an arm and a leg (already forking that out for the rackmount enclosure)?

 

[Snugglebear]

The easiest question here is the one about RAID controllers. If you plan on using PATA units, the max is two drives per channel, so 4 drives per 2-channel controller. You can get controllers with more channels or more controllers if need be. If you want SATA units, one drive per channel, so look for an 8-channel controller or two 4-channel controllers, etc. With SCSI you'd be able to cram ~14 drives on a channel, though I wouldn't recommend doing that. You'd also get better random I/O performance, but then you're also going to pay 2-3x as much per GB.

You need to find out more about the access patterns in play. That's an awful lot of data, but it's being shoved around in tiny chunks, and I have no clue how the application is going to require that data to be fed to it, what the working set is, etc. A 4-drive RAID0 stripe would certainly give you high contiguous read and write speeds, but it's also going to give you a hell of a lot of latency on operations.

Lastly, I can't say much about backups without more information. Is this a 24/7 application? Is it batch-run for only part of the day? Is there a nice window where the data doesn't change? Do you need offsite copies? Is security of the data a factor?

 

[defakto]

avoid raid 0 if this is any kind of important data. period. Unless you do nightly backups, still i'd recommend against it.

 

[Thuleman]

I will contact the developers to find out just how exactly the data is read and written. Currently I am running a test setup on PATA133 drives, and if I connect more than 2 or 3 clients then there is a noticable slowdown in I/O no matter how much CPU cycles I throw at it.

Meaning I tried to compute one data set with 8 clients connected, and one with 3 clients connected, and it took roughly (give or take a couple hours) the same time, for which I have no other explanation than I/O.

As far as backups go. I could write the source data to removale media in time, will probably buy one of those double sided DVD writers for that. Source data could always be reobtained from the data providers (at some cost and lead time of course). However, there is an intermediate data product made out of the source files and needed for computation, and that intermediate product is just as large as the source data, and while the intermediate could be recreated, it would take a lot of time to do so.

The final output is certainly valuable, valuable enough to invest in some real backup.

Ultimately this is a 24/7 operation, there is no downtime per se, however, there will be a time in the near future when neither source data, nor the intermediate product will change more than once or twice a month, and changes will be on the order of a few hundred MB at best.

As far as using a RAID other than 0, I have read up on this a bit, and basically I don't see how any setup other than RAID1 would provide a fail-safe. It seems to me that in all other setups, if one drive goes, my data will be lost unless I have a backup somewhere.

Offsite backup: Offsite backup is not needed. I have arrainged for offsite backup already, a StorageTek Powderhorn will serve that purpose.

 

[unhappy_mage]

raid 5 will provide a backup of 1 disk. if one dies you can still recover your data, you just add another disk and it recreates the data from parity, which takes the capacity of 1 disk. so, if you have 4 disks of 120gb each, you get 360gb of storage. with a hardware raid controller from 3ware, write speeds can be up to 100mb/s, which should be fast enough for your purposes. the 4way controllers are about $250. it may be better to get an 8way for $400 and create 2 seperate arrays, one for reading one for writing. the "writing" array needs fast disks, the "reading" one not so much,

 

[Cardboard Hammer]

Are you sure you're not saturating the PCI bus on the master machine? If it uses 32 bit 33MHz PCI, that could very well be what's killing you.

Also, perhaps the system architecture could use a bit of tweaking. Instead of sending the results back to the master machine, perhaps results could be sent to another machine? Or perhaps seperate all storage from the master machine and put it into a dedicated storage system on network share(s). Perhaps also get work and return results in bigger chunks (instead of getting / returning one unit at a time, grab a few) and have the client machines return results / request work while they're still working on other units. Or you could have the master push to / pull from the clients, which would mean that it wouldn't have to worry about the demands of the clients at all; it'd just keep track of what it assigned, pick up available results from each in turn and assign more work based on how much work each client has pending.

 

[defakto]

You might want to look into the raid 5 option, or a raid 10/01 seriously, raid 5 buy several extra drives as spares to replace a failed drive, make sure you get a controller with hotswap, most likely sata, for best results, array will rebuild, rma the old drive and still have a backup on hand. Raid 10/01 can lose one drive on eah side of the 0 portion and keep going but you lose a hefty chunk of drive space, so it's a trade off.

 

[Snugglebear]

I will contact the developers to find out just how exactly the data is read and written. Currently I am running a test setup on PATA133 drives, and if I connect more than 2 or 3 clients then there is a noticable slowdown in I/O no matter how much CPU cycles I throw at it.

Monitor the disk usage. All modern OSes have this capability in one form or another. I would not be surprised if you either saturate the bus or have outrun ATA's capability to perform I/Os. CPU cycles are incredibly unimportant when you're serving files and other types of data. I see this every day on my precious database servers, and to some extent corporate fileservers. My gut tells me this master machine is going to need to be very robust, with a larger/faster bus(es) and probably at least Raptors as disks if you want to stay with ATA. Chances are this is going to be SCSI if you really want it to work well. And yes, the programmers can have a sizeable impact depending on how they queue, request, and cache data. However, hardware is usually far cheaper than their time, so don't go asking for some ultra-optimized binary unless they're twiddling their thumbs a good chunk of the day.

 

[Cardboard Hammer]

... And yes, the programmers can have a sizeable impact depending on how they queue, request, and cache data. However, hardware is usually far cheaper than their time, so don't go asking for some ultra-optimized binary unless they're twiddling their thumbs a good chunk of the day.

While throwing hardware at good code is probably a better investment than making great code, lousy code can bury even the best hardware. I've seen performance improvements of over 2 orders of magnitude after cleaning up other people's crap code in real world systems. Of course, if the programmers wrote crap code in the first place, they're not likely to be of much help in getting it right.

 

[Thuleman]

The developers are pretty supportive, but I doubt they will "code to spec" at this point, maybe sometime in the future as they update their stuff it will happen, for now I will have to deal with the way it currently works (still trying to get the intricate details on that).

When I first said that there is only a master and clients it only half the story, stay with me if you are interested, this will get somewhat long.

There is a master file server, which stores the geospatial data we are working with. The data has at least two raster components (digital elevation model, and associated imagery).
Then there is a web server that streams the product to the web. Then there are distributed clients who do nothing but calculate stuff.

There are two operating modes, either the data is precompiled before it gets served to the end-user, or it is compiled on the fly.

PRECOMPILED
In precompiled operation mode the master file server issues a command to the distributed clients that they need to compile the final product. The clients acess the master file server, retrieve a chunk of data, process it, return it, retrieve a new chunk, until they have processed all the data the master file server told them to process.

Then the final product gets moved to the web server, and the web server sequentially streams it to end-users on demand.


ON-THE-FLY
An end-user client connects to the web server. The web server determines what the end-user should see and puts out a request for that data to be compiled to the distributed agents. The agents receive the request, reach for the data on the file server, compile it into the format needed and return it to the web server.
The web server sends it out to the end-user.

For the time being we don't expect more than a dozen concurrent web users, and based of the funding received this is really just a proof-of-concept project. I realize that in order to make this work in a production environment hundreds of thousands of dollars will be needed to guarantee smooth operation.

The current objective is to show that it will work, and not only will it work, but it will work at acceptable delivery speeds. We are connected to Internet2, and the primary audience at this point are Internet2 end-users, so network bandwidth is not a concern at this time. It all hangs on how fast we can generate data and in the case of "on-the-fly" serve it out to the web, and the bottleneck is I/O.

 

[ameoba]

RAID5 on a real RAID controller attached to a 64bit-66MHz PCI bus (you've allready got that, right? 32/33PCI is going to drown in traffic if it has to handle all those Gb clients -and- a drive controller) might help with reliability but you're still going to need -some- sort of backup system. In the end, tho, you need to know where your bottlenecks are before you start buying hardware; splitting the data into a 'read-only' array and a 'write-only' array is only a good idea if you can segment the data in such a way..

Tapes suck, but there's no other real options. You can do that "back-up to HDD" thing, (you can set up a 1TB USB enclosure for like $1/MB) but that's more of a "somebody deleted a file last night and wants it back" than a "the file server caught on fire last night & we need to restore everything"..

 

[unhappy_mage]

(you can set up a 1TB USB enclosure for like $1/MB)

i'm going to hope to god that means a dollar a gigabyte. iirc laCie makes these, and on firewire not usb (or maybe both?) i looked at these, they look alright for data storage/backup but not for regular use. anything that holds a tb without parity (at least) is fairly suspect in my book (short tho it is).

hopefully the "cached" versus "onthefly" modes are done without much user intervention. the server does check to see if it's got packet x done first, right? also, the "clients" can request more work if they're done?

 

[ameoba]

Yeah... $1/GB...

and definately this would be a backup setup. I guess it's what you'd call 'near-line' storage...

 

[Thuleman]

I actually bought one of those LaCie 1TB drives, comes with USB, Firewire 400 and Firewire 800 interface. The enclosure gets fairly warm, even at idle, the powersupply gets what seems to me almost too hot to touch.

I seriously need to read up on just how RAID5 really works. I still don't quite get the "Parity" concept. I mean if you have 4 drives for storage, and one for parity in a RAID5, and one of the storage drives fails. How can it possibly be that you can recover 100% of the data on the failed drive? As I said, foreign concept to me, will try to find a couple good explanations out there.

 

[Cardboard Hammer]

Parity:

A bit is either 1 or 0.
Sum up the bits across a collection of bits and set the parity bit based on whether the sum is even or odd.
If you lose one of the bits, you still can retrieve your data: if it's the parity bit that's lost, no big deal, just recalculate the parity bit for future use; if it's another bit that's lost, sum up the bits that weren't lost and use the parity bit to determine what the missing bit must've been to give the parity that it did, then restore that bit for future use.

EDIT: There isn't a dedicated parity drive in RAID 5, the parity is distributed in blocks over all the drives. IIRC, RAID 4 is like RAID 5, except that it does use a dedicated parity drive (which is worse for performance and why it isn't used).

EDIT 2: http://www.lostcircuits.com/hdd/hdd9/

 

[unhappy_mage]

raid 5 is definitely worth the read up. it is imo the best raid level if you care about data loss and speed. the basis of parity is the xor command. the lostcircuits link above is a good one, if you look around you can find more on this.

hardware raid 5 is your best bet, at least until this gets the funding you think it will. then i'd go with raid 1+0 or 5+0. you will need very very high end hardware to do this all in hardware. i would suggest instead that you get some hardware raid5 cards (2 at least) and do a software raid on top of it. linux is my os of choice for this. i suggest you read through the software raid howto to get a fuller understanding. before you run out and buy hardware, you may want to try a software raid5 with linux. this will give you a feel for how it works, but _not_ performance-wise. software raid 5 has the disadvantage that the processor is fairly 'far' from the disks. thus, the processor has to wait for the disks to do some extra operations for any writes to succeed. this link will explain more about why these extra reads must take place.

as i said before, get a raid5 for the read side, and raid 0+1 or 1+0 for the writes.