• A Great friend to the HardForum with a great kid that he is trying to get a scholorship to continue his schooling. Please give hime a vote! Only 24 hours left! Thanks.
    If you have an VOTE FOR KEENAN!

defragmenting a 5TB RAID5 array

C

cyr0n_k0r

Supreme [H]ardness
Joined
Mar 30, 2001
Messages
5,360
I am about to put into production a 5TB RAID5 array running on a dell Perc 5/e / MD1000 combo.

I have used diskkeeper in the past on several smaller arrays to try and manage to keep the defragmenting down to a low level. I would just leave it run 24x7 when the array wasn't under very much load. And it would just sit there and constantly defragment.

I am about to bump up the number of read writes to the hundreds per minute range , and will be stepping up the files stores into the millions. (very small files)

I was wondering if anyone would reccommend diskkeeper to continue my trend in this new kind of enviroment, or just not even worry about it.

Or if anyone has a better solution/program, please let me know.
 
If the files are so small, it won't matter whether the disk is fragmented or not. Every I/O will be a random seek.

Raid 5 is really hurting your performance, by the way. Small random writes are an anathema to raid 5. Switching to raid 10 will take more disks, but improve performance a great deal. Raid 5 does a read-modify-write cycle each time a small chunk of disk is modified, which takes two reads and two writes to do. With raid 10, you skip the read-modify stages and skip directly to writing the blocks to both halves of the array.

What disks are you using? Do you need all 5 TB? What's the read/write ratio? How much cache do you have?
 
So if you use something other than FAT or NTFS, what does that leave? Doesn't that format also need to be defragged as well?
 
no, this is not linux and no, I will not be using anything except NTFS.

The disk is currently a GPT disk running 12 (soon to be 15) 500GB SATA drives.
Yes, we need ALL the storage, and will be adding more here in the next 4 months. Probably daisey chain another MD1000 and continue building the array.

RAID10 might be a better solution I agree, but the cost is prohibitive when dealing with the amount of storage we are looking at. Were a small business and just had to drop almost $4,000 on the array plus adding more hard drives.

unhappy_mage, because everything will be a random read write, is defragmenting even really necessary?
 
cyr0n_k0r said:
RAID10 might be a better solution I agree, but the cost is prohibitive when dealing with the amount of storage we are looking at. Were a small business and just had to drop almost $4,000 on the array plus adding more hard drives.
Click to expand...
The question to ask yourself is, can you afford not to? Is your load likely to increase? $4000 is a necessary expense, and if spending more will let you serve your customer better, it might be better to spend the more.
cyr0n_k0r said:
unhappy_mage, because everything will be a random read write, is defragmenting even really necessary?
Click to expand...
Necessary, probably not. Will it help? Maybe, probably not. Is there any pattern to the accesses? Have you considered storing the files in a database as BLOBs (or otherwise) instead of on the filesystem? What are these files, and is there a hierarchy to them?
 
they are hundreds of thousands of small files that is the output of encrypted file traffic. The folders are constantly being updated, deleted, and added to.

Currently were on about ~500,000 folders, with maybe 2-3 million files. (The output of the encryption, nothing can be changed in how it spits out files) The data can not be put into a database or otherwise altered.
 
Im far from expert on the area but I have some suggestions:

Make several partitions, perhaps for every 50gb or so, and fill them with data. Once full you go to the next one. Each partition should once full be defragged in alphabetic or most accessed order. Idea here is to fill up each partition as much as possible (instead of leaving a few gb for performance) so once you delete one file the replacement will be on the same place in one piece.

Reason for this is to keep the data as much locked in one place as possible. Perhaps you could switch between the backup and the real one and do defrag over night or something on the backup while the live version is working.

Just some loose thoughts.
 
cyr0n_k0r said:
Were a small business and just had to drop almost $4,000 on the array plus adding more hard drives.
Click to expand...

Raise your hand if you think a 5TB raid array is not something a 'small' business would be using.

You, yes, you, in the back,, you only need to raise on hand.

unhappy_mage, because everything will be a random read write, is defragmenting even really necessary?
Click to expand...

Listen to unhappy mage. :) Although the defragmenting isn't going to be that necessary. How small are these files? 1 cluster sized? What you consider a small file, some might consider huge. Small files for me are anything I can fit more than two copies of on my external 750GB drive. But I work with video. Small files for you might be icons that are 76 bytes..

If your files are smaller than a cluster, you'd not need to defragment at all.
 
I cannot imagine a person responsible for 5T of data needing to ask this question.

----

5T if hard drives cost under $2000 (close to $1100). How you spent $4000 would be instructive.
 
GeorgeHR said:
5T if hard drives cost under $2000 (close to $1100). How you spent $4000 would be instructive.
Click to expand...

I can only imagine a person with "HR" in their nick asking this question :p There are power supplies, hotswap chassis, a rackmount case, and a SAS expander to pay for, too. Spending $1 or $2 per GB is not unusual for storage of this caliber. The MD1000 mentioned is $8k with 15 500GB disks in it.
 
There are certainly expensive ways to build the described system. I go for the simple and cheap.
 
GeorgeHR said:
There are certainly expensive ways to build the described system. I go for the simple and cheap.
Click to expand...

I don't mean to derail the thread, but I think cyr0n_k0r's question has been answered adequately. What parts would you build a 5TB array from simply and cheaply?
 
unhappy_mage said:
I don't mean to derail the thread, but I think cyr0n_k0r's question has been answered adequately. What parts would you build a 5TB array from simply and cheaply?
Click to expand...

here's my cheap build

8 x 750GB drives
stacker
2 x 4 in 3 drive cages
pcp&c 1kw
low end intel mb and a e2140
2gb ddr2
highpoint 2320

should be well under $3k
 
GeorgeHR said:
I cannot imagine a person responsible for 5T of data needing to ask this question.

----

5T if hard drives cost under $2000 (close to $1100). How you spent $4000 would be instructive.
Click to expand...

Look at some SUN Store Edge servers. We just got one in from a client that's 1TB. It cost them close to $20,000.
 
750s: $220 each, total $1760
Stacker: $170
5-in-3s (might as well have the expansion space): $120
PCP&C 1KW - $520
e2140: $80, cheap Intel board: $77
2gb ddr2: $80
2320: $240

$1407 at Newegg shipped to where I live, plus $1760 from ZZF. But my point isn't that you spent more than $3k, my point is that this build is inadequate. You've got 6TB of capacity, and that's good. Let me list the problems with it, though.
  • There's no boot disk for the array. Putting the OS on a raid 5 array is generally a problem. Add two more disks in raid 1.
  • You have no cold spares (well, you might use one disk as a spare... but you didn't specify that). One hot and one cold would be a good start.
  • It's not very expandable. I know, you can get 20 disks in it if you push real hard. What do you do when you realize you need 200 disks?
  • You didn't specify additional cooling. I know the case comes with a few 120mm fans, but fans, power supplies and disks are the things that go out most (followed by memory - see next item). Having on-site replacements for all of those would be best.
  • There's no ECC. This is not a server. Period.
  • It doesn't rack mount. That won't float, or at least not for long. When you've got three storage machines it's okay, but when you get number four you'll start thinking about racks in a different way.
  • There's only one power supply. PCP&C is supposed to be reliable, but every manufacturer makes lemons.
  • There's only one disk controller, and it isn't hardware based. The machine is doing other things, don't make it do software raid too.
  • The disks are 7200 RPM, and you've only got one parity disk.
But worst of all - *you* put it together. It's warranted by you. If the board goes tomorrow, guess who has to replace it. If the memory goes in three years, after you've moved on, either they'll call you or they'll be stuck with a broken machine. If Dell builds it, Dell gets to deal with that.

I don't mean to pick on your build - it'd be a fine machine for home use. But I'd be a lot more likely to recommend something from Sun or HP for a business. Cheap, fast, good.
 
unhappy_mage said:
750s: $220 each, total $1760
Stacker: $170
5-in-3s (might as well have the expansion space): $120
PCP&C 1KW - $520
e2140: $80, cheap Intel board: $77
2gb ddr2: $80
2320: $240

$1407 at Newegg shipped to where I live, plus $1760 from ZZF. But my point isn't that you spent more than $3k, my point is that this build is inadequate. You've got 6TB of capacity, and that's good. Let me list the problems with it, though.
  • There's no boot disk for the array. Putting the OS on a raid 5 array is generally a problem. Add two more disks in raid 1.
  • You have no cold spares (well, you might use one disk as a spare... but you didn't specify that). One hot and one cold would be a good start.
  • It's not very expandable. I know, you can get 20 disks in it if you push real hard. What do you do when you realize you need 200 disks?
  • You didn't specify additional cooling. I know the case comes with a few 120mm fans, but fans, power supplies and disks are the things that go out most (followed by memory - see next item). Having on-site replacements for all of those would be best.
  • There's no ECC. This is not a server. Period.
  • It doesn't rack mount. That won't float, or at least not for long. When you've got three storage machines it's okay, but when you get number four you'll start thinking about racks in a different way.
  • There's only one power supply. PCP&C is supposed to be reliable, but every manufacturer makes lemons.
  • There's only one disk controller, and it isn't hardware based. The machine is doing other things, don't make it do software raid too.
  • The disks are 7200 RPM, and you've only got one parity disk.
But worst of all - *you* put it together. It's warranted by you. If the board goes tomorrow, guess who has to replace it. If the memory goes in three years, after you've moved on, either they'll call you or they'll be stuck with a broken machine. If Dell builds it, Dell gets to deal with that.

I don't mean to pick on your build - it'd be a fine machine for home use. But I'd be a lot more likely to recommend something from Sun or HP for a business. Cheap, fast, good.
Click to expand...

yeah I know there are a lot of negatives, but you said a cheap 5TB system, of course you're going to make some sacrifices
if I had to build it with that budget, I'd get a cheaper case, throw a boot drive in somewhere, and probably settle for a good ol' pcp&c 510w
 
CyberDeus-RagDoll said:
Raise your hand if you think a 5TB raid array is not something a 'small' business would be using.

You, yes, you, in the back,, you only need to raise on hand.



Listen to unhappy mage. :) Although the defragmenting isn't going to be that necessary. How small are these files? 1 cluster sized? What you consider a small file, some might consider huge. Small files for me are anything I can fit more than two copies of on my external 750GB drive. But I work with video. Small files for you might be icons that are 76 bytes..

If your files are smaller than a cluster, you'd not need to defragment at all.
Click to expand...

The files are mostly under 2KB. However 10% of the files account for almost 95% of the storage. The way the software spits out the encrypted data is thousands and thousands of small files, then maybe a dozen or so files that are like 600MB, 1.5GB, 500MB, etc.

Also, the way our customers are setup, we cannot have multiple 50GB or even 500GB partitions. Each customer has their own folder that they write to. Sometimes they can change over 100GB in one day, either through updating the files, or outright deleting them. So the customers must have LOTS of room to play.
 
cyr0n_k0r said:
The files are mostly under 2KB. However 10% of the files account for almost 95% of the storage. The way the software spits out the encrypted data is thousands and thousands of small files, then maybe a dozen or so files that are like 600MB, 1.5GB, 500MB, etc.

Also, the way our customers are setup, we cannot have multiple 50GB or even 500GB partitions. Each customer has their own folder that they write to. Sometimes they can change over 100GB in one day, either through updating the files, or outright deleting them. So the customers must have LOTS of room to play.
Click to expand...

And that is why no one uses NTFS.

Expandable partitions is no big deal then. Yahoo (for example) uses it every day.

You really think every yahoo account created automatically partitions you out a gig when next to nobody would actually use that much space?
 
smizack said:
And that is why no one uses NTFS.

Expandable partitions is no big deal then. Yahoo (for example) uses it every day.

You really think every yahoo account created automatically partitions you out a gig when next to nobody would actually use that much space?
Click to expand...
Can you explain this. This is a very good topic and would like to know more about this. Any knowledge I can gain will be helpful down the line when I become a systems admin one day.
 
Vengance_01 said:
Can you explain this. This is a very good topic and would like to know more about this. Any knowledge I can gain will be helpful down the line when I become a systems admin one day.
Click to expand...

I'll have to do some digging. It's one a NetApp 12TB Cabinet we have.
Basically, you set a maximum partition size, and it grows the partition as needed up to the max size you set, so there's not a lot of wasted space doing nothing.
 
In the same way most Server/Workstation Visualisation software, in which a disk size is set, will only use the amount it needs.

No point in having a full 10gb allocated all the time for a specific virtual machine if it sits using 512mb all day long, nice to know though if it needs too then it can expand its virual HDD to 10gb.

Also I have to agree with the mage above, no one should build their own servers for business use unless you have a vast array of spare parts available, will just bite you in the ass otherwise.
 
Eickst said:
Can I ask why? Is there a valid business or technical reason for this or do you just love NTFS for some reason?
Click to expand...

Any other alternatives for small/large business running 2003/x64 flavors? I'm of course asking more for humor.

I'm a Windows system admin for 300+ server (about 2000 total). NTFS is the only format type used and support by us.

As far as the OPs question. Please keep in mind, you're not defragging an array; You're defgragging a partition. How you organize your logical volume into partions is critical in maintaining them.

If you make one huge F: drive, I'm am so sorry for you. You are going to have a management nightmare, plus all sorts of backup issues. BTW, how the heck ARE you going to back this stuff up? The reason I ask is because I currently have a share with around 2.8 million files, and we can't currently back it up using conventional Veritas Netbackup/Windows methods. We do RAW backups/restores only.

I also don't agree with your RAID type. If in fact your storage will be doing lots of small writes, RAID 5 sucks ass. Use RAID 0+1 or RAID 10 for that.
 
Rocco123 said:
If you make one huge F: drive, I'm am so sorry for you. You are going to have a management nightmare, plus all sorts of backup issues. BTW, how the heck ARE you going to back this stuff up? The reason I ask is because I currently have a share with around 2.8 million files, and we can't currently back it up using conventional Veritas Netbackup/Windows methods. We do RAW backups/restores only.

I also don't agree with your RAID type. If in fact your storage will be doing lots of small writes, RAID 5 sucks ass. Use RAID 0+1 or RAID 10 for that.
Click to expand...

Why would one large partition be a nightmare? Each customer has their own folder inside the partition.

We backup using raw data also. We do monthly's and send them to california.

RAID10 might be something we explore when we expand our array at the end of the year, but right now were sticking with RAID5 just to get as much storage online as possible right now. Once we've made the money back that we had to drop on all this equipment, we can look at expansion, and alternate RAID levels.
 
You must log in or register to reply here.
Back
Top