The [H]ard Forum Storage Showoff Thread - Post your 10TB+ systems

Status
Not open for further replies.
Ill need a new case first, the case i have only allows 4 hard drives, then i have to figure out what card i need to run more sata!

it all takes time!
I hear you! Too much time researching, especially for what controllers are actually available in Portugal... lol

I already have a big enough case (this one), with 3x 4-in-1 CM drive bays (got them all at once to keep the look clean).

Now the real problem is the controller I need to get. I either have the option of 4-port SiI-based PCI dumb controller, which I need two of (total cost of about €160) to get the extra 8 ports I can handle (not considering the extra space below the uATX motherboard, of course), which kind of sucks because they're not that cheap and 8 drives over the PCI bus is asking for trouble, even if we are talking about a simple WHS machine with rather low bandwidth requirements; or I try and find the elusive Supermicro 8-port PCIe 4x SAS controller, which is €100+ easy (IF I can get it in Portugal, of course), all the while crossing my fingers that it will work on the single PCIe 16x slot on my motherboard (shared with the GMA3100 IGP, btw).

HDDs are the easy pick. The controller is what sucks...

@Nitro: I have already read about the potential issues with the EARS drives. While I'm mostly a Samsung guy when it comes to HDDs (about 30% to 50% cheaper for the same size, at least in Portugal, is hard to beat...), it seems to me that an EARS drive should be just fine as a pool drive, either with or without the offset jumper, right? As long as you don't accidentally plug or unplug the jumper on an already pool-added drive, the most you'll get should be lower performance on a non-jumpered drive, because of the sector misalignment, right?

Oh, and since we're at it, how are the new 2TB F4EGs working for pool drives? Are they 4K-cluster, too?
 
Theoretically yes, but its a little concerning because DE does do weird things. It also appears odd that MS would issue a KB article on Advanced Format Drives specifically stating to not use them on WHS, when for 2003/XP they said to align/use jumpers.

Just wait a few months and there will be a thread, "I lost all of my data on my WHS" Sure enough it will be on advanced format drives.
 
I hear you! Too much time researching, especially for what controllers are actually available in Portugal... lol

I already have a big enough case (this one), with 3x 4-in-1 CM drive bays (got them all at once to keep the look clean).

Now the real problem is the controller I need to get. I either have the option of 4-port SiI-based PCI dumb controller, which I need two of (total cost of about €160) to get the extra 8 ports I can handle (not considering the extra space below the uATX motherboard, of course), which kind of sucks because they're not that cheap and 8 drives over the PCI bus is asking for trouble, even if we are talking about a simple WHS machine with rather low bandwidth requirements; or I try and find the elusive Supermicro 8-port PCIe 4x SAS controller, which is €100+ easy (IF I can get it in Portugal, of course), all the while crossing my fingers that it will work on the single PCIe 16x slot on my motherboard (shared with the GMA3100 IGP, btw).

HDDs are the easy pick. The controller is what sucks...

Yeah hard drives are a easy pick for me, however case and controller are hardest. I really want a rack mount case. It will go with my rack mounted astaro unit, my rack mounted switches and other stuff.

I have never done any thing with larger raid setups or hotswap. But i do know i want a rack case just so it can be mounted into my rack that im going to keep looking for.

Then when its time to move into my house i don't have to do much other than put the rack in the closet and get it up and wired.

j'
 
Just a quick question with your cpu heatsink. Do you have air coming in from the rear of the computer or fresh from the front and exhaust to the back? I ask because I think you mounted your cooler backwards but I wasn't sure if it was planned

I have not even got to turn it on yet. When I fired it up I found that I had been shipped a PSU that was DOA. Now I am waiting on the RMA. The front HD fans are blowing in and I will have the back fans blowing out. I will flip the cooler if it is backwards. It is killing me waiting on this RMA. Thanks for the heads up.
 
I really want a rack mount case.
Oh, you're on the "I've got rack mountable hardware" crew. That makes choosing a case for WHS a breeze.

In those cases, my instant thought is "Norco it and be done with it". And don't you guys DARE to use that marketing phrase without me saying it's OK... lol

I will flip the cooler if it is backwards.
As the owner of a very similar cooler (probably the same model), I can confirm it's mounted backwards. That fan pushes air to the cooler, not away from it. Unfortunately, you'll need to pull it out, re-apply TIM and put it back in, since the fan only really fits one way. :(

Theoretically yes, but its a little concerning because DE does do weird things. It also appears odd that MS would issue a KB article on Advanced Format Drives specifically stating to not use them on WHS, when for 2003/XP they said to align/use jumpers.
Yes, that is somewhat disturbing. Luckily Vail won't have any problems. I'm actually itching for it to launch, it seems a MAJOR overhaul and a great OS to boot, so I'm dying to migrate my whole server to Vail. Best to do it now, while I'm still sub-10TB, right? lol
 
I think ill grab this case for my server, sucks that it takes a special powersupply not even listed on the new egg site.

NORCO RPC-2008 2U
11-219-024-S02
 
I suppose that posting pictures of my company's NetApp FAS2050 and corresponding disk shelves would be cheating? :D
 
I think ill grab this case for my server, sucks that it takes a special powersupply not even listed on the new egg site.

NORCO RPC-2008 2U

Call Norco and ask if they have a backplate to mount a standard ATX PSU. ATX fits in 2U and Norco sells other 2U chassis with ATX backplates. Its likely one of them might fit this case.
 
Ive posted before but this is a completely new setup.

Specs:
P4 3.02 Ghz
1 G Ram
Server 2008 (32-bit)
Norco RPC-470
Supermicro 5-in-3 hotswap
OCZ 600 Watt PSU
2x250 G IDE drives
2x1.5 Tb Seageates
3x2 Tb WD Green
1x750 G WD Green

Now lives in my room at school, the fan noise doesn't bother me at all when i sleep.



New Case
JYY9Q.jpg

bddSb.jpg


Motherboard and PSU in. The fans are moved to the other side because of the depth of the Supermicro 5-in-3.
fpyIA.jpg

Drives in, Cabling started
ED54K.jpg

E7owD.jpg

All Done
X41Yc.jpg

FL0PN.jpg


I7 + Router + Switch + Server
oShnF.jpg
 
im curious
why you kept midfanwall?
5x hotswap has its own fan and 5x bays both have been equiped with 120s
you realy needed the extra airflow?
 
im curious
why you kept midfanwall?
5x hotswap has its own fan and 5x bays both have been equiped with 120s
you realy needed the extra airflow?

I wanted the extra airflow because that Supermicro gets pretty hot even with its own fan. The Supermicro controller i'm using doesn't support spinning the drives down so they are always spinning. Also im living on the 4th floor of a building with no AC in Michigan. its fine now but It did overheat once during August so i had to borrow a window unit for a while.

I guess i could move 2 of the wall fans to the back and take out the other 3 but whatevs
 
I think ill grab this case for my server, sucks that it takes a special powersupply not even listed on the new egg site.

NORCO RPC-2008 2U
11-219-024-S02

Quick question...Unless you are tight for space, why not drop alittle more money on a 4020 case which can hold tons more drives. Odds are you will outgrow that case pretty quickly, otherwise wouldn't it just be worth picking up a 8-12 bay tower case for $100 or cheaper?
 
Which RAID / SATA controller supports multiple 2TB (3TB) and Western Digital drives do spin down and up right?
 
If you're thinking of the green series' speed change, that's a totally internal function of the HDD. The type of SATA controller should have no bearing on it.
As for a total halt of drives, I would really not recommend it for a RAID setup.
 
What case is that in the top left?

It is Thermaltake Tai-Chi case, for my gamming PC. Actually i am building my own case for file server, it will very small with a miniATX mainboard plus 14 x 2TB harddriver that why i am putting all of my ... file servre in the chair now. :D
 
Amount of total storage: 60TB (the older system will be published soon)
Titanic: 40TB

Case: Supermicro, SuperChassis 846A-R900B
PSU: 900w include in the case
Motherboard: X8DTi-LN4F
CPU: 2xE5620
RAM 12x4GB DDR3 ECC Registered 1333MHz Kingstom
GPU no
Controller Cards ARC-1680i-24
Optical Drives no
Hard Drives 23xHD204UI (Samsung)
Battery Backup Units: Areca ARC-BBM
Operating System: Fedora13

short description for now, this thread is still under construction

My first post :) , after all the questions about the reliability of samsung F4eg in a raid system , maybe this post could help (i will check carefully over the CLI log )



This data server is for data storage of scientific calculations ( dynamical trajectories ) usually directories of 300 files of 0.5 to 3Go.

the raid array size is 22 disks + 1 for spare and 1 disk for system (WDEAR 1TB )

all perfomance parameters for the raid card are on, and for the volume we chose a raid6 configuration with 128k stripe.

the actual file system used for test is ext4 , we adjust the format parameter as follow

partition table GPT created with gparted and partition adjusted to cylinder

mkfs.ext4 -E stride=32,stripe-width=640 -i 65536 -m 0 -O sparse_super -T largefile4 -b 4096 -L ice /dev/sdb1

tune2fs -c 0 -i 0 -r 102400 /dev/sdb1

i haven't touch the readahead parameter for now

i am doing some benchmark for now over mounting parameters

here are some results

iozone (test0 is default option)
http://global.phoronix-test-suite.com/index.php?k=profile&u=baobab-32746-21016-28006

disk (test2 is default option)
http://global.phoronix-test-suite.com/index.php?k=profile&u=baobab-3343-4532-23013

some dd results

nice -20 dd oflag=direct bs=1M count=50000 if=/dev/zero of=./50gb.bin
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 83.0157 s, 632 MB/s
nice -20 dd bs=1M if=./50gb.bin of=/dev/null
50000+0 records in
50000+0 records out
52428800000 bytes (52 GB) copied, 69.9714 s, 749 MB/s

front:


back:


Update1:
After two week of usage no error but at the first volume chek (with areca raid web interface) i got 17 warning "Time-out error", it's not fun ;(

2010-10-23 14:03:25 Enc#2 SLOT 02 Time Out Error

And it's not for just one disk, i guess volume check is more stressing for the system than normal usage.
 
Last edited:
Here is my little update:

It's not very fast, but it's secure and fits well into my Mac environment :)

4x Onnto DataTales (FW800@Mac Mini)
"Storage A": 4x1TB=3TB@RAID5
"Storage B": 4x1TB=3TB@RAID5
"Company Data": 2x 640GB=640GB@RAID1
"Backup": 4x1.5TB=4.5TB@RAID5

Complete: 15.28TB
Usable: 11.15TB

Some photos:

The Rack
darkshot.jpg


Opened Rack door - at the bottom my IPCop (SuperMicro Celeron 2.4GHz, 1.2GB RAM, 160GB IDE) & DELL PowerEdge 1650, 4GB RAM, 3x9GB SCSI/SCA
completeset.jpg


4th DataTale on top (the small one) and the UPS (APC RS800)
lildatatale.jpg


Patchpanel
Cisco/Linksys 16 Port GBit/s Switch
Mac Mini & Elgato EyeTV Sat DVB-S2 Box
racktop.jpg


- DSL Modem of 2nd DSL line
- the 3 big DataTales
rackmid.jpg


The already mentioned servers
rackbottom.jpg


Impressive less power usage for all this stuff running the same time :)
204w.jpg


Ciao
Dennis
 
Here is my little update:

It's not very fast, but it's secure and fits well into my Mac environment :)

4x Onnto DataTales (FW800@Mac Mini)
"Storage A": 4x1TB=3TB@RAID5
"Storage B": 4x1TB=3TB@RAID5
"Company Data": 2x 640GB=640GB@RAID1
"Backup": 4x1.5TB=4.5TB@RAID5

Complete: 15.28TB
Usable: 11.15TB

Some photos:

The Rack
darkshot.jpg


Opened Rack door - at the bottom my IPCop (SuperMicro Celeron 2.4GHz, 1.2GB RAM, 160GB IDE) & DELL PowerEdge 1650, 4GB RAM, 3x9GB SCSI/SCA
completeset.jpg


4th DataTale on top (the small one) and the UPS (APC RS800)
lildatatale.jpg


Patchpanel
Cisco/Linksys 16 Port GBit/s Switch
Mac Mini & Elgato EyeTV Sat DVB-S2 Box
racktop.jpg


- DSL Modem of 2nd DSL line
- the 3 big DataTales
rackmid.jpg


The already mentioned servers
rackbottom.jpg


Impressive less power usage for all this stuff running the same time :)
204w.jpg


Ciao
Dennis

nice setup so does the mac mini act as a dvr of some sort?
 
nice setup so does the mac mini act as a dvr of some sort?

Also :) Download machine, Fileserver (central), VM-Server (just a few very small Debian VMs) and Recording System. We can schedule TV shows via iPad for example.
 
Also :) Download machine, Fileserver (central), VM-Server (just a few very small Debian VMs) and Recording System. We can schedule TV shows via iPad for example.

sweet setup, i suppose it uses some kind of IR blaster for the cable box, i've got a similar setup (pc instead) - I honestly can't wait for a better system where I don't need a cable box.......and no - please don't say cablecard tuner - those things are WAY too expensive!!!!!!
 
sweet setup, i suppose it uses some kind of IR blaster for the cable box, i've got a similar setup (pc instead) - I honestly can't wait for a better system where I don't need a cable box.......and no - please don't say cablecard tuner - those things are WAY too expensive!!!!!!

Honestly, no IR Booster :) We just use it to record shows (even in HD) - not to view it live. EyeTV (recording software) allows remote programming.
 
Honestly, no IR Booster :) We just use it to record shows (even in HD) - not to view it live. EyeTV (recording software) allows remote programming.

even better :) i'm working on my 6tb media server.........still a few drives away from being able to boast on this forum!
 
the actual file system used for test is ext4 , we adjust the format parameter as follow

partition table GPT created with gparted and partition adjusted to cylinder

mkfs.ext4 -E stride=32,stripe-width=640 -i 65536 -m 0 -O sparse_super -T largefile4 -b 4096 -L ice /dev/sdb1

tune2fs -c 0 -i 0 -r 102400 /dev/sdb1

I wouldn't recommend ext4 if you are using all the storage on a single file system (over 16 TiB). Using > 16 TiB is still not mature on ext4 and when I emailed their mailing list asking about it a few months ago I was told that it isn't considered stable and I shouldn't use ext4 if I wanted stability.

The only file-systems that I know of that seem to handle >16 TiB correctly is JFS/XFS. I am not a big fan of XFS do to corruption I Have seen in the past and JFS has a bit of annoying bug where an fsck is required on an unclean shutdown if you have more than 12 TB of data on the array do to the log not being able to re replayed.

I chose JFS as an fsck on my 36 TB file-system only takes around 15 minutes and i almost never unmount uncleanly. It took around 8 minutes on my 18 TB file-system.
 
DieTa,

Nice little sexy rack setup you have there, nice and clean!

I wonder if i could build something like that!
 
Status
Not open for further replies.
Back
Top