Anyone have experience with nbd in Linux?

[LightningCrash]

In my search for an expandable, fault-tolerant network storage solution, I've come across nbd in Linux, which I haven't heard much about.
AFAIK I can use LVM to run SW RAID5 across ndb volumes, and even expand the array.

I've considered building two servers with 8ea 250GB drives (I found a reasonably cheap board with dual onboard gigabit and 8 SATA ports) and connecting them via a gigabit backplane to a controller computer of sorts.
My initial impression is that I should make the controlling box a little more beefy than the storage nodes, as far as processing power and available RAM. I'd planned on putting Pentium D 820s with 1GB of DDR2-667 in the storage nodes, and maybe a Conroe with 2-4GB of RAM in the controller.

Basically, I'd like a storage solution where my roommates and I can chip in for another 2TB node later on, just drop it in, and expand the LVM RAID5 array to include the new nbd devices.

Unfortunately I have ZERO experience with nbd/enbd. I'd like some input on the matter, as unfortunately, I can't find any documentation on google that would educate me sufficiently on the topic. I've found a couple of newsgroups postings discussing LVM/enbd, so I assume it is working in some instances. Input on speed/throughput would also be valuable.

Many thanks.

 

[unhappy_mage]

EVMS is recommended over LVM tools; it's newer and a little neater IMO.

You might want to look at iSCSI rather than nbd; I get the impression that nbd's deprecated these days. iSCSI will also give you cross-platform compatibility, if you might want to throw a Windows 2k box or whatnot in the mix. There's a free initiator from MS out there somewhere.

So are you going to have the frontend box do samba/NFS and use iSCSI over the backend boxes? Sounds like a sufficiently enterprisey solution

 

[LightningCrash]

nice, i hadn't even thought about iSCSI on Linux

i've thought about EVMS, i think i'm just too used to using LVM. i guess i should bring my EVMS knowledge up to speed.

right, i was originally planning on using samba/NFS on the frontend

would there be any caveats to running an iSCSI Target to the aggregate RAID5 EVMS array on the Frontend box, and having all of the machines connect to that array via iSCSI initiators? (by caveats, i mean the usual: sharing conflicts, file locks, etc)
edit: Or is it even possible to have mutliple initiators connected to the same disk?

this is really just going to be well organized mass storage for everything (ripped DVDs, MythTV recordings, drive images, etc etc) and won't see a lot of random I/O.

 

[unhappy_mage]

My problem with this scenario is: what happens if one machine goes down? What if you accidentally unplug the "backend" switch? Linux will willfully and with malice aforethought destroy any semblance of a raid 5 array if more than one of its members disappears.

I think your best bet, unfortunately, is to build several seperate machines with seperate storage containers. You lose a little flexibility, but if one of them goes offline, only the data on that machine is unaccessible, and no raid5 arrays are destroyed.

 

[LightningCrash]

another good point, and i've considered that

if i accidentally kill the switch, then no writes will be made to the storage nodes by the controller.....as long as the controller box doesn't massacre the config files in /etc , i should be able to bring the array back online.
also, i've got a pretty beefy setup in the garage (full height rack, 2KVA APC 2U Rackmount UPS) and all of this will be rackmounted and well maintained.

as far as the backend boxes going down, i guess i only have a couple of options there
1) use 3 storage nodes, locally SW RAID0 the drives on the storage nodes, then export via iSCSI to the controller node, where the 3 iSCSI mounts are then in an array again (RAID5 or RAID6). this approach just sounds like it would be horribly slow
2) your suggestion, individual storage units


i have considered the thought that, by the time we fill up 1.75TB, it would be much easier just to build another single array again.

 

[unhappy_mage]

if i accidentally kill the switch, then no writes will be made to the storage nodes by the controller.....as long as the controller box doesn't massacre the config files in /etc , i should be able to bring the array back online.

Except that the new LVM doesn't use /etc/raidtab anymore - it writes the raid config information to the header of the disk. So when a disk (iscsi or otherwise) drops out, it writes that data to the header.

as far as the backend boxes going down, i guess i only have a couple of options there
1) use 3 storage nodes, locally SW RAID0 the drives on the storage nodes, then export via iSCSI to the controller node, where the 3 iSCSI mounts are then in an array again (RAID5 or RAID6). this approach just sounds like it would be horribly slow
2) your suggestion, individual storage units

You could mount the many machines to a single directory on one, and export that: mount machinea:/data on /mnt/data/TV, for example, and machineb:/data on /mnt/data/DVDs.

i have considered the thought that, by the time we fill up 1.75TB, it would be much easier just to build another single array again.

True enough. And you could sell off the old array once the new one's built, to recoup the costs of building the new one.

 

[LightningCrash]

goddamn you're smart.

/adds you to my buddy list

 

[unhappy_mage]

No, if I were smart I'd be doing my homework instead of answering your question Thanks for the compliment, though

So have you figured out what you're doing with this setup? Multiple raid5s either on one machine mounted locally, or on several machines mounted over NFS? The first might be cheaper, but you need a heckuva power supply to keep that many drives going. The second gives you more compute power, and you can buy several small(er) power supplies for the cost of one huge one. And it's more fault-tolerant to have multiple machines - if your TV archive machine goes down, you can still access the dvd archive machine.

The tradeoff is flexibility of where you put files - if you decide that you get a lot more media to store, you have to pick one array to put them on. And the arrays aren't easily resizable - it takes several hours to resize.

You could use EVMS to make containers for the volumes, and resize those as demand increases. Most recent filesystems get bigger easily; shrinking isn't so easy. So start with 100 gigs each for tv, movies, and music, and grow as necessary. EVMS will let you move containers from array to array. It might not be as easy over network, but it's at least hypothetically possible.

Last hypothetical - have you considered netbooting the storage machines? It seems silly, but then you can fit in more storage disks and fewer OS disks. There's more complexity, and it puts a little more load on the master machine, but it gives you room for more disks if that's a concern. PXE is standard these days, and it's not hard to set up tftp and dhcpd to boot machines. Making the machine image isn't quite so trivial, but it's not rocket science.

Just very, very close

 

[LightningCrash]

I think I'm just going to put together one box with 8 drives (for now, anyway)

I might go ahead and drop in a real RAID controller (like an Areca) during buildout just to solve this whole EVMS headache for the time being

 

[LightningCrash]

i just remembered that my Intel D975XBX has 8 SATA ports on it

4ea 500GB Drive : $760 shipped

2TB storage computer : $1500 shipped

I think the choice is obvious, i'll save the platform costs and just put the drives in my desktop for now

we'll see about a storage cluster later on (when i might already have rigs put up for other purposes, ie F@H)