RAID 10 for Oracle database?

hardware_failure

hardware_failure

[H]ard|Gawd
Joined
Mar 21, 2008
Messages
1,370
Ive been given the task of suggesting a build for an oracle database. Its about 200GB (and growing) with about 30-60 users in it at any given time.

I was looking at this barebones system SUPERMICRO SYS-6026T-NTR+ 2U with a 5520 xeon.

The big question is: What controller, what drives? Any reason not to go with the newer 600GB WD velociraptors? What about RAID 10 vs RAID 5?

Any feedback would be greatly appreciated. Thanks!
 
Any reason not to go with the newer 600GB WD velociraptors?
Click to expand...

A single SSD would be faster then any hard drive array. Cheaper if you count the cost of a high end raid controller which will not be needed for a single SSD.

What about RAID 10 vs RAID 5?
Click to expand...
Raid 10 would be better at random reads/writes that are typical in database.

What controller, what drives?
Click to expand...
For raid 10, the Intel motherboard ports are very good. However they do not have a large battery backed cache like expensive raid cards have.
 
I suppose I should have provided some more specifics.

Im thinking this would be the order of needs:
  • The storage volume needs to be at least ~500GB
  • RELIABILITY - massive I/O reads/writes to a very active Oracle 10.0.5.1g database.
  • Redundancy
  • Performance
  • Price is a concern but not overly tight.
 
RAID 10 for performance and reliability, we have over 5 informix servers running and only ever use RAID10 , RAID5 is a recipe for disaster and performance for DB applications will suck.
and some sage advice from an experienced DBA:

RAID 5 vs RAID 10

From Art S. Kagel research findings:

If a drive costs $1000US (and most are far less expensive than that) then switching from a 4 pair RAID10 array to a 5 drive RAID5 array will save 3 drives or $3000US. What is the cost of overtime, wear and tear on the technicians, DBAs, managers, and customers of even a recovery scare? What is the cost of reduced performance and possibly reduced customer satisfaction? Finally what is the cost of lost business if data is unrecoverable? I maintain that the drives are FAR cheaper! Hence my mantra:
NO RAID5! NO RAID5! NO RAID5! NO RAID5! NO RAID5! NO RAID5! NO RAID5!

Is RAID 5 Really a Bargain?

Cary Millsap, manager of Hotsos LLC and the editor of Hotsos Journal found the following facts - Is RAID 5 Really a Bargain?":

* RAID 5 costs more for write-intensive applications than RAID 1.
* RAID 5 is less outage resilient than RAID 1.
* RAID 5 suffers massive performance degradation during partial outage.
* RAID 5 is less architecturally flexible than RAID 1.
* Correcting RAID 5 performance problems can be very expensive.

:)
 
drescherjm said:
For redundancy, I assume you will be doing a nightly dump of the database to tape or some disk array that is not part of this machine.

http://www.orafaq.com/wiki/Oracle_d...Q#Why_and_when_should_I_backup_my_database.3F
Click to expand...
I was hoping that the RAID would just never fail and leave it at that ;)

The network has a dedicated backup media server running symantec backup exec.
Nightly hot backups via backup exec oracle agent.
Weekend cold backups with pre and post job batch files to down and start the database.
 
KaptainKhaos said:
RAID 10 for performance and reliability, we have over 5 informix servers running and only ever use RAID10 , RAID5 is a recipe for disaster and performance for DB applications will suck.
and some sage advice from an experienced DBA:
.....
Click to expand...
The last few servers I have been responsible for have all been RAID 10 as well. I really like them. Was mostly looking for other opinions. RAID 10 is becoming more and more of an easy decision to make.

Next is the controller and drives decision. Im worried about using the onboard controller incase of a failure.. my budget usually allows to keep a spare identical controller in a cabinet near by. MB's are often cheaper than raid cards but quite the PITA to replace during a recovery.

Thanks for all of the great feedback thus far...
 
onboard controller incase of a failure.
Click to expand...
That actually gives you the best failure option. I mean Intel is very good about backward compatibility. If your motherboard dies move the array to any other board with intel raid and you should be fine.
 
RAID 10 for databases. No question. SSD's are a bad idea unless you go with high-end SLC, and even that may not be reliable enough.
 
drescherjm said:
That actually gives you the best failure option. I mean Intel is very good about backward compatibility. If your motherboard dies move the array to any other board with intel raid and you should be fine.
Click to expand...
Okay, thats something to consider. It does look like the intels raid support has come along way from a few years ago...

Anyone have any particularly good or bad exp with recent drives in RAID 10? I see alot of mixed opinions about different drives firmware effecting raid performance..
 
my budget usually allows to keep a spare identical controller in a cabinet near by. MB's are often cheaper than raid cards but quite the PITA to replace during a recovery.
Click to expand...

In that case I recommend LSI based SAS HBAs. An Intel branded SASUC8I card runs for around $155 US.

http://www.newegg.com/Product/Produ...117157&cm_re=INTEL_sas-_-16-117-157-_-Product

However you may want to get a card with a battery backed cache for better database performance. That will cost at least 4 times as much though.
 
@OP

RAID 10 is usually the array topology of choice for databases, however RAID 1 is worth looking at also, as large performance gains can be had, at least with SQL Server. Have a look here.

Taken from page two of the article:

I replaced the massive 20-drive 10K RAID10 array with 8 pairs of RAID1 consisting of 16 15K RPM drives. The new 15K RPM drives had roughly a 10% I/O performance advantage over the 10K RPM drives they were replacing, but there were 20% fewer of the newer drives — which meant that drive speed was more or less a wash. The biggest difference would be the RAID configuration.Microsoft SQL Server fortunately permits seamless distribution of its data tables among any number of volumes you can throw at it. The use of row-level extent-level striping in MS SQL Server allows even distribution of data and workload across any number of volumes, and I used that simple mechanism to distribute the database over the 8 pairs of RAID1.

...

As a result of this change in RAID configuration, the queue depth (the number of outstanding I/O transactions backed up due to storage congestion) dropped a whopping 97%! This in turn resulted in a massive reduction in SQL transactions from a painfully slow 600ms per transaction to 200ms per transaction. The result was so dramatic that it was hard for everyone to believe it during the first week. They had thought that perhaps there was some mistake or anomaly and that this might be due to a drop in usage, but doubts subsided as time went on and the performance kept up. Even the engineer from the storage vendor who initially doubted the effectiveness of this solution became a believer after he ran some tests to verify that the load evenly distributed across the 8 RAID1 pairs.
Click to expand...

Certainly worth looking into. :)


EDIT:

Keep away from RAID 5, unless you intend using ZFS' RAID-Z. If you can, go for 15K SAS drives and an LSI controller.
 
Last edited:
Certainly worth looking into.
Click to expand...

For that I would first see if Oracle will allow you to split a database into 8 parts so you can gain the advantage of short stroking each array. I guess 1 could create a software raid 0 of the individual raid1s at the OS level if oracle does not let you do that. I am not sure what hardware raid cards would let you stack raid levels like this.
 
What OS will you be using? Will you be using ASM?

Raid 1 might be a better option for the various ways you can split out I/O. Also with ASM you can simulate the striping of Raid 10, with automated balancing through the database engine.
 
drescherjm said:
For that I would first see if Oracle will allow you to split a database into 8 parts so you can gain the advantage of short stroking each array. I guess 1 could create a software raid 0 of the individual raid1s at the OS level if oracle does not let you do that. I am not sure what hardware raid cards would let you stack raid levels like this.
Click to expand...

Very good point and in fact the article does go on to say that Oracle doesn't allow you to use multiple volumes in quite the same way. However, I assumed that things would have changed since that article was written. I can't believe Oracle have not thoroughly investigated SQL Server.

I agree with you in that I also do not think that a hardware RAID card will allow you to stack RAID levels on-card like that. Additionally, which do you think would be more desirable: a software RAID 0 of the individual RAID 1s or an LVM volume of the RAID 1s, since LVM can also stripe (and grow)?
 
parityboy said:
Very good point and in fact the article does go on to say that Oracle doesn't allow you to use multiple volumes in quite the same way. However, I assumed that things would have changed since that article was written. I can't believe Oracle have not thoroughly investigated SQL Server.

I agree with you in that I also do not think that a hardware RAID card will allow you to stack RAID levels on-card like that. Additionally, which do you think would be more desirable: a software RAID 0 of the individual RAID 1s or an LVM volume of the RAID 1s, since LVM can also stripe (and grow)?
Click to expand...

This article is a bit dated and the author is not completely informed. This is also coming from the view of someone well versed in SQL Server, not Oracle.

Oracle has had the ability to spread tables across spindles for years. This is built into the ability to assign datafiles to tablespaces and balance object extents across the datafiles. Before ASM this was a manual process requiring table moves between tablespaces to re-distribute extents across the disks. ASM was released with 10g in 2005, so yes, they had investigated and implemented that option.

With ASM you add volumes to a pool and it dynamically loads the extents across the volumes. There is also a background process that re-balances the extents in the background.
 
Additionally, which do you think would be more desirable: a software RAID 0 of the individual RAID 1s or an LVM volume of the RAID 1s, since LVM can also stripe (and grow)?
Click to expand...

I do not have enough experience with LVM stripping to answer that question. I use LVM a lot but only rarely do I have more than 1 pv (physical volume) per vg (volume group).
 
Wow some really good stuff here, thank you all for such good info!

The OS will simply be server 2003 64 bit. This is how the product is typically sold and run. I believe there is a linux install supported as well but there is a mixed staff that will support this server and windows is much more commonly known, so it is preferred. AFAIK the database will not be using ASM.

I am nothing close to a DBA but I have some basic exp with oracle... Spanning datafiles on different volumes sounds like a great idea for performance, but Im not feeling that adventurous. Creating something simple. with little room for failure and being easy to support is very important.

I am currently looking at LSI cards and 15k SAS drives. Again thank you all for the great feedback.
 
parityboy said:
EDIT:

Keep away from RAID 5, unless you intend using ZFS' RAID-Z. If you can, go for 15K SAS drives and an LSI controller.
Click to expand...

So this thread almost had me convinced to do a few disks in a raid 10 on my upcoming NAS build for mysql, until I read this post. Are you implying RAID-Z (or possibly 2 or 3z) could be a viable option when looking for good db performance?

I originaly planned to do one or more volumes most likely 2z, but this thread had me considering if I needed to get a proper RAID card for some RAID 10 or at least do some software RAID 10. If it will still give significant improvement over RAID-Z, I will do it. But I want RAID-Z for most of my VMDKs so I can try out dedupe. As usual, I am picking up a ton of tips on these threads.
 
Define Massive IOs.

Generally 15k SAS drives RAID10 is standard for Databases, but depending on your IO requirements, that could change.
 
hardware_failure said:
What is the difference between RAID 1E/10E and RAID 10 when using an even number of disks?
Click to expand...

Level 1E: RAID level 1E is also referred to as RAID level 1 "Enhanced". In terms of RAID, it can be thought of as enhanced mirroring or striped mirroring. The name sounds like RAID 10, but RAID 1E uses an odd number of disks to achieve data protection while RAID 10 requires the use of an even number of disks. RAID 1E strips data across all disks in the array like RAID 0. However, RAID 1E takes a copy of the data and then strips it across all of the disks as well. RAID 1E shifts the second copy of the data over on each physical disk to move the data stripe copies to other disks. Minimum number of drives is three.

Level 10E: Level 1E is enhanced mirroring, which offers the same functionality as RAID 10 on an odd number of drives, starting with three. RAID 10E is the same as RAID 1E, but on an even number of drives. Minimum number of drives is four.
http://www.intel.com/support/motherboards/server/sb/CS-029839.htm
 
I read elsewhere that level 10E was like 10 but with an odd number of drives, but that sounds like exactly what 1E is. Hard to argue with Intels official page.

So then.. whats the difference between 10E and "regular" 10?
 
Problem is 1E and 10E as well(IIRC) were proprietary IBM raid levels that have since been adopted (modified?) by different vendors and implemented on current HW RAID cards.

So really it depends on the controller you are using and you should consult their documentation for an exact answer as it applies to you, but generally the *e levels are the same as their counterparts for an odd # of disks.
 
Ahhh, that makes sense. So if one establishes a 10E array, they will be even more "married" to always using the specific used RAID card vs something that supports "normal" RAID 10? (ie for replacement in the event of failure, etc)
 
hardware_failure said:
Wow some really good stuff here, thank you all for such good info!

The OS will simply be server 2003 64 bit. This is how the product is typically sold and run. I believe there is a linux install supported as well but there is a mixed staff that will support this server and windows is much more commonly known, so it is preferred. AFAIK the database will not be using ASM.

I am nothing close to a DBA but I have some basic exp with oracle... Spanning datafiles on different volumes sounds like a great idea for performance, but Im not feeling that adventurous. Creating something simple. with little room for failure and being easy to support is very important.

I am currently looking at LSI cards and 15k SAS drives. Again thank you all for the great feedback.
Click to expand...

Is 2003 x64 a vendor requirement? 2008 or 2008 R2 would be greatly suggested, if you can't do something like Oracle Unbreakable(RHAS with a oracle maintainted yum repo) as the OS.
 
geiger said:
Is 2003 x64 a vendor requirement? 2008 or 2008 R2 would be greatly suggested, if you can't do something like Oracle Unbreakable(RHAS with a oracle maintainted yum repo) as the OS.
Click to expand...
At this time, yes it is :mad:

"future upgrades will support server 2008"
 
hardware_failure said:
Ahhh, that makes sense. So if one establishes a 10E array, they will be even more "married" to always using the specific used RAID card vs something that supports "normal" RAID 10? (ie for replacement in the event of failure, etc)
Click to expand...

TBH i dont think swapping from an Adaptec RAID10 to LSI/Areca/insert other here RAID10 card is going to work because of the way that different vendors have written their metadata to the drives. (ie. the other card will not recognize the array, even though the data is striped/mirrored the same way).

That said if you have to recover data using a raid reconstructor such as R-Studio, it will make recovery very manual with a proprietary format such as 10e and you will have to evaluate each drives data in a hex editor to reconstruct the array.
With a Standard R10 level that is prob not required as the software should automatically do it for you.
 
nitrobass24 said:
...That said if you have to recover data using a raid reconstructor such as R-Studio, it will make recovery very manual with a proprietary format such as 10e and you will have to evaluate each drives data in a hex editor to reconstruct the array... for you.
Click to expand...
Thats it? Sounds like peanuts. I evaluate drive data in hex editors all day long :)

Yikes.
 
edlin303 said:
So this thread almost had me convinced to do a few disks in a raid 10 on my upcoming NAS build for mysql, until I read this post. Are you implying RAID-Z (or possibly 2 or 3z) could be a viable option when looking for good db performance?

I originaly planned to do one or more volumes most likely 2z, but this thread had me considering if I needed to get a proper RAID card for some RAID 10 or at least do some software RAID 10. If it will still give significant improvement over RAID-Z, I will do it. But I want RAID-Z for most of my VMDKs so I can try out dedupe. As usual, I am picking up a ton of tips on these threads.
Click to expand...

I haven't tried it myself, but from what I have learned, ZFS' RAID-Z engines avoid the small file write penalty of traditional RAID 5 by using variable stripe sizes, thereby ensuring every write is a full-stripe write. How this performs compared to the no-parity-needed RAID 10 I don't know, but modern CPUs are much faster than a RAID-on-Chip processor.

Since RAID-Z is a very advanced form of parity RAID, a ZFS fileserver needs to be treated like a giant RAID controller, i.e. lots of ECC RAM, fast CPU for the checksumming and a decent UPS. You can also use an SSD as a write-cache, if I'm not wrong. :)

Look here for a big ZFS build.
 
parityboy said:
I haven't tried it myself, but from what I have learned, ZFS' RAID-Z engines avoid the small file write penalty of traditional RAID 5 by using variable stripe sizes, thereby ensuring every write is a full-stripe write. How this performs compared to the no-parity-needed RAID 10 I don't know, but modern CPUs are much faster than a RAID-on-Chip processor.

Since RAID-Z is a very advanced form of parity RAID, a ZFS fileserver needs to be treated like a giant RAID controller, i.e. lots of ECC RAM, fast CPU for the checksumming and a decent UPS. You can also use an SSD as a write-cache, if I'm not wrong. :)

Look here for a big ZFS build.
Click to expand...

Thanks. I already have a pretty good idea about the actual ZFS build. I was just reading more into your statement than I should have RE: raid 10 vs. RAID-Z. I'll do my own basic benchmarks to see how the difference looks when the hardware gets here I think.
 
You must log in or register to reply here.
Back
Top