Somewhat of a complicated question.

[400+whpwrx]

I am looking to sort out what the best option is for storage system.

Current hardware
dual opteron sun box (2gb, scsi)

Requirements
Lots of storage
Online expansion/rebuild capability
Ability to have multiple systems access filesystems (ability to create multiple filesystems, this is going to be done by ZFS on solaris 10)
Hot swap drives if they fail
Raid6 (unless someone else has a better idea)
Ability to do up to 16-20 drives)
Filesystem has to be able to hold lots of files (ZFS again) and have good IO for DB back ends and the like.

Issues in question

Should i build one file server and then use NFS from the multiple systems to it? (comes down to how good is NFS for housing DB's etc)


Or should i build one large box with a massive array and use Solaris 10 zones to partition the box into multiple "instances"? (i would be using zfs on solaris 10) Note this option would mean spending more on MB/CPU/Ram vs building a straight up Fileserve.


Hardware im looking at

CM Stacker for the case
Supermicro drive cages (like the one posted in the terabyte club thread)
areca Raid card (16 or 20 port not sure yet, any input into pci-e or pci-x would be awesome)

Hardware i still need to sort out

Power supply (needs to be able to handle 20 drives or so, if i build a SMP system this will be a large expense as the power req goes up)
Hard drives (best price per gb and performance?)


My thoughts
To keep costs down i was just going to swap the Sun systems mobo into the CM stacker and then purchase more ram (it has 2gb was looking at adding another 2-4gb), and the drives and raid card.

Your thoughts? I am still in the planning stages at this point, but i will most likely pick up the case and drive enclosures soon as i dont see prices on those two items going down to much in the next month or two.

 

[400+whpwrx]

Just for everyones info..

http://sunsolve.sun.com/handbook_pub/Systems/W2100z_shared/spec.html

Its the w2100z

So it does have a pci-x slot (which i am leaning towards using)

 

[unhappy_mage]

How many machines will be accessing this? How much data is there to deal with? How many databases do you want to store?

I want to hear the answers to those questions before I get too far, but I'd tend to build this to use with a software iScsi stack. That way the other machines deal with doing the filesystem stuff, and the database stuff, and this box just has to read and write blocks to disk.

If you're doing databases and speed is an issue, raid 6 is not the way to go. It's slow even in sequential writes - and database access patterns tend to be a) smaller than (numdatadisks * blocksize) (which for writes causes a read (to every disk!) and then a write) and b) involve a fairly high portion of writes, which means slow is a nice way of putting it. Raid 1+0 is the only way to go for databases.

 

[400+whpwrx]

How many machines will be accessing this? How much data is there to deal with? How many databases do you want to store?

I want to hear the answers to those questions before I get too far, but I'd tend to build this to use with a software iScsi stack. That way the other machines deal with doing the filesystem stuff, and the database stuff, and this box just has to read and write blocks to disk.

If you're doing databases and speed is an issue, raid 6 is not the way to go. It's slow even in sequential writes - and database access patterns tend to be a) smaller than (numdatadisks * blocksize) (which for writes causes a read (to every disk!) and then a write) and b) involve a fairly high portion of writes, which means slow is a nice way of putting it. Raid 1+0 is the only way to go for databases.

At first go it would be around 2-3 systems.. Could ballon up into 10+ depending. (all gig e of course) Data would be small at first but could ballon into much larger set of files and dbs. (i would of course like to use the array for system backups or partial backups of systems if possible, i think this would be the single largest use of space)

Biggest concern at this point is redundancy and data integrity. dont get me wrong performance is key but at this point i dont have a large tape library to back up the data.. so it will have to live on the array. So that is my major concern.

At first i am looking at one database could ballon into 3-5 databases. The array would also be used to store files (both small and large, but mainly small files in the range of 10k-4mb). Again my main concern is data integrity and ability to avert the entire array going down and losing everything.


My view of things is basicly this. Amass a massive amount of disk space and carve it up in ZFS or the actual card its self into nice bite size chunks (maybe a R1+0 set for the DB's with a R6 set for the files?). Just my half educated look at things at this point..

 

[unhappy_mage]

Have you considered using the database over network? That is, rather than storing the data on NFS or iScsi and having a *SQL machine write to that filesystem or block device, store the files locally and let machines perform database operations over network? I know you can do this with MySQL and PGSQL, and would imagine many DBs support this. That seems like it'd be simpler to me, but the CPU load would probably be pretty high. I'd give it a try that way; the idea of running the DB on one machine and storing DB files on another strikes me as a bad one.

 

[400+whpwrx]

Have you considered using the database over network? That is, rather than storing the data on NFS or iScsi and having a *SQL machine write to that filesystem or block device, store the files locally and let machines perform database operations over network? I know you can do this with MySQL and PGSQL, and would imagine many DBs support this. That seems like it'd be simpler to me, but the CPU load would probably be pretty high. I'd give it a try that way; the idea of running the DB on one machine and storing DB files on another strikes me as a bad one.

thinking of doing somehting like this.. need to do some research on ZFS with NFSv4..

That was the setup i was thinking of when posting the first post (using a NFS server for the FS and haveing the DB instance on another server that was using the NFS server).

I may just move over the sun box to the stacker case if it works out well nuff.. Then use my old desktop as a front end machine for time being.