2TB+ on RHEL?

[dreamnid]

Hi,

I ordered a server from SoftLayer that has 4x750gig in RAID 5 running on a 3ware 9500S-8 controller. They hit a snag trying to create an ext3 partition bigger than 2 TB so they created two partition to temporarily solve the problem.

I've been looking around and found a couple solutions:

1 - Increase the block size for ext3. RHES does not seem to allow you to specify the block size during setup, so I'm assumming you have to create the partitions manually before running setup.

2 - Use another filesystem (XFS, JFS). Again, RHES only allows you to use ext2, or ext3.

Another thing is this Errata note from RHES Update 1:
http://mirror.centos.org/centos/4/docs/html/release-notes/as-x86/RELEASE-NOTES-U1-en.html

Section "Changes to Drivers and Hardware Support"
It mentions that you need to install Update 1 in order to use 2TB + devices. Since it was very late, I couldn't understand what needed to be done (as it is pretty late).

So I was wondering if anybody has successfully used a 2TB+ system in RHES4 and the steps to over come it. We're mainly using the server to serve jpg files (say less than 100k).

Also, is it beneficial to setup LVM for this setup?

Finally, is EVMS just a management tool to handle RAID and LVM or it is a separate thing? I'm kinda confused why you have to rebuild the kernel in the former case. Again, since I was researching about this issue very late at night, I might have missed something.

We do want to get this server setup as soon as possible - so I appreciate any help that I can get!

 

[unhappy_mage]

Well, my first suggestion would be to use a seperate boot disk. That'd make management much simpler.

Using another filesystem for data storage might be a good thing. XFS was built by SGI to run on high-end stuff, and with suitably high-end hardware it performs rather nicely.

Using LVM won't help a whole lot, since it sounds like you're dealing with fairly simple volume layouts, but I'd tend to set up EVMS anyways. EVMS (I think) deals with the 2TB+ problem in such a way that it's transparent to you.

Finally, another distro might be a good choice. Debian FTW.

 

[dreamnid]

Well, my first suggestion would be to use a seperate boot disk. That'd make management much simpler.

You mean a separate hard drive or a separate partition? I believe there is a 100MB or so partition for the /boot mount

Finally, another distro might be a good choice. Debian FTW.

Well, I would love to but the only choices from SL are RHES 4, CentOS, FC5, FreeBSD, and Win2003.

I guess we could look into FC5 or FreeBSD if they have built-in support XFS.

By the way, is XFS really that bad at recovering from a crash? We are still looking into ways to back the images up so it would suck to lose the partition due to a power failure (which shouldn't happen anyways).

Thanks mage!

 

[unhappy_mage]

You mean a separate hard drive or a separate partition? I believe there is a 100MB or so partition for the /boot mount

A seperate disk would be good. It also helps to isolate "stuff that needs daily backup" from "stuff that needs monthly backup".

Well, I would love to but the only choices from SL are RHES 4, CentOS, FC5, FreeBSD, and Win2003.

I guess we could look into FC5 or FreeBSD if they have built-in support XFS.

Hm. Why not install your own OS? It's not that hard - are they providing any support with those OS installs, or would your own install be on an even footing? And if they're installing it, why not let them deal with the problems?

By the way, is XFS really that bad at recovering from a crash? We are still looking into ways to back the images up so it would suck to lose the partition due to a power failure (which shouldn't happen anyways).

I hear AMANDA is a good way to go for medium to huge installations. xfs_repair works fine for me, btw.

 

[dreamnid]

Hm. Why not install your own OS? It's not that hard - are they providing any support with those OS installs, or would your own install be on an even footing?

Well, I would if it is possible to get the installation files on there (which RHES requires a login to download the OS). Again, maybe it is better to choose another OS? I never really liked RedHat but I believe most dedicated servers defaults with RHES.

And if they're installing it, why not let them deal with the problems?

You would think if they offer the option to do 4x750gigs, they would know how to get around the problem .

I'm simply trying to find information that will help them get this problem solved as soon as possible. Besides, I know we need a RAID driver for the 3ware card, so I rather have them install the OS.

The tech I was speaking with was going to try talking to their Linux guru although that will probably happen on Monday.

I hear AMANDA is a good way to go for medium to huge installations. xfs_repair works fine for me, btw.

According to the RHES errata I posted above, AMANDA doesn't let you specify GFS for non-Itanium build. Personally, I'm not sure how GFS fits into all of this since it first says GFS is a requirement for 2TB builds but it mentions later on that the the maximum size for ext3 is 8 TB.

I appreciate your help in the matter, even though you're more of a Debian guy.

 

[eth00]

Have you considered using XFS? Right now I manage a fair number of servers with 2TB+ partitions and we have gone with XFS over ext3. It so far has been treating us pretty good and we have not had any trouble with filesystem corruption. With ext3 you run the risk of having to run a fsck...don't even being to think about what happens with it then! If you store large numbers of images in a single directory and have to undergo a fsck it is very possible you could "loose" a lot of the data.

For that matter I am just burning in a 14x320 Gb machine right now that uses 2x80gb for the OS I did a normal OS install on the 2 80's then will do XFS with RAID5 + 1 hotswap drive on the main array.

If you decide to go with XFS I can post some more information about getting it working - beware you will need to recompile your kernel so if you are not familiar with source kernels you may not want to do it yourself.

 

[unhappy_mage]

Well, I would if it is possible to get the installation files on there (which RHES requires a login to download the OS). Again, maybe it is better to choose another OS? I never really liked RedHat but I believe most dedicated servers defaults with RHES.

I would expect the reseller to include a hard copy of the disks or a login. I would personally use a different distro, but if you've got experience doing things the RH way stick with that.

I'm simply trying to find information that will help them get this problem solved as soon as possible. Besides, I know we need a RAID driver for the 3ware card, so I rather have them install the OS.

The tech I was speaking with was going to try talking to their Linux guru although that will probably happen on Monday.

The 3ware drivers built into the kernel are pretty good, from what I hear. And you can build the array in BIOS. Unless you need management tools for the OS (which you do, true) everything should work stock.

According to the RHES errata I posted above, AMANDA doesn't let you specify GFS for non-Itanium build. Personally, I'm not sure how GFS fits into all of this since it first says GFS is a requirement for 2TB builds but it mentions later on that the the maximum size for ext3 is 8 TB.

That's Anaconda (the install system) not AMANDA (the backup system). Anaconda requires that your / and /boot partitions be on a >2TB device (not a >2TB partition, notice!). Also, GFS is a filesystem. GPT is what you'll be using, or evms devices, which work slightly differently IIRC. Note that you can't boot from a GPT disk with most bioses IIRC. This means that you're pretty much forced to have a seperate boot disk.

 

[dreamnid]

Have you considered using XFS? Right now I manage a fair number of servers with 2TB+ partitions and we have gone with XFS over ext3. It so far has been treating us pretty good and we have not had any trouble with filesystem corruption.

Well, I would consider it if RHES let us format a partition with it.

If you decide to go with XFS I can post some more information about getting it working - beware you will need to recompile your kernel so if you are not familiar with source kernels you may not want to do it yourself.

I have compiled kernels before (in particular FreeBSD and an old version of RedHat) so I'm not too bothered by it. But out of curiousity, if the RHES installer does not offer to use XFS, I'm assumming you'll need to re-format "/" as XFS and reinstall it (although now that I'm thinking about it - since the setup program doesn't allow you to choose XFS, it probably won't recognize those partitions). Thus, I would think that you can safely convert the partition to XFS without a format.

 

[dreamnid]

I would expect the reseller to include a hard copy of the disks or a login.

Err, sorry. I'm not understanding what you mean by "hard copy of the disks." Keep in mind SoftLayer is a webhosting company that offers remote dedicated servers (ie, I don't get physical access to the machine).

The 3ware drivers built into the kernel are pretty good, from what I hear. And you can build the array in BIOS.

Well, they have a page that specifies that servers that have the 3ware card needs to use their kernel configuration scripts so I'm assumming that this is not hte case.

RAID Card (3Ware 9550SX) Drivers
If you have a RedHat/CentOS operating system and a 3ware RAID card, it has been provisioned with a kernel and RAID drivers that have been thoroughly tested with your SoftLayer hardware. If at some time in the future, you elect to upgrade your kernel, please refer to the following driver section for pertinent driver updates.

Your server has a 3Ware 9550SX RAID card. The driver for this RAID card is not included in default RedHat/Centos kernels. SoftLayer provides tested drivers for the following kernels:

That's Anaconda (the install system) not AMANDA (the backup system). Anaconda requires that your / and /boot partitions be on a >2TB device (not a >2TB partition, notice!)...This means that you're pretty much forced to have a seperate boot disk.

Whoops, big mis-read on my part. Could you read this for me:

Since I did not have time to meet with the engineers, I was forced to use the auto carve feature on the controller which allows for the breaking down of multiple volumes of 2TB max.

Your remarks and his remarks seems contradictory. I really have not worked with RAID so my understanding of how it works is probably wrong. I think he created a 2TB raid volume which is within the 2TB limit even though the "device" total space is 2.1TB leaving another volume which is around 50 gigs unused. This setup does work.

Here is what df reports:

[root@www2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda2             2.0T  1.5G  1.9T   1% /
/dev/sda1              99M   12M   83M  12% /boot
none                  2.0G     0  2.0G   0% /dev/shm

Thus, if this works, can't we do something like create a 20gig raid volume for the OS and boot partition and the rest will be on another volume? Or do we really need to install another hard drive just for the boot partition? I know I'm probably missing something here so I again apologize if I am misunderstanding something here.

Thanks!

 

[unhappy_mage]

Err, sorry. I'm not understanding what you mean by "hard copy of the disks." Keep in mind SoftLayer is a webhosting company that offers remote dedicated servers (ie, I don't get physical access to the machine).

Ah. My bad, I assumed you were getting a box delivered.

Well, they have a page that specifies that servers that have the 3ware card needs to use their kernel configuration scripts so I'm assumming that this is not hte case.

Well, I guess they might be using other drivers, or the script might be starting the web administration thing, or something like that. I dunno.

Whoops, big mis-read on my part. Could you read this for me:

What this means is there's a mode on the controller which creates a 2TB array and a seperate .1TB array. If that works for you, fine, go with it.

Thus, if this works, can't we do something like create a 20gig raid volume for the OS and boot partition and the rest will be on another volume? Or do we really need to install another hard drive just for the boot partition? I know I'm probably missing something here so I again apologize if I am misunderstanding something here.

A seperate hard drive is always my suggestion for this type of thing, but if budget doesn't allow that, having a seperate array for the OS is a good start. Maybe you can do a raid 1 for the OS? I don't know how complicated Areca lets you get with arrays. This'd be something like Intel's matrix raid thingy - part of each disk goes to the raid 1 array for OS, the rest to the raid 5 for storage.

 

[dreamnid]

A seperate hard drive is always my suggestion for this type of thing, but if budget doesn't allow that, having a seperate array for the OS is a good start. Maybe you can do a raid 1 for the OS? I don't know how complicated Areca lets you get with arrays. This'd be something like Intel's matrix raid thingy - part of each disk goes to the raid 1 array for OS, the rest to the raid 5 for storage.

Hmm.. do you mean 3ware instead of Areca unless the companies are now one of the same?
Your idea on having the OS on a RAID 1 array sounds really cool. However, since 4 hard drives is the largest number of hard drives they offer, I don't think they can fit any more (I could ask though).

I appreciate your help again, mage! You answered my questions very well and hopefully I can get this worked out with them.

Thanks again!

 

[eth00]

I have compiled kernels before (in particular FreeBSD and an old version of RedHat) so I'm not too bothered by it. But out of curiousity, if the RHES installer does not offer to use XFS, I'm assumming you'll need to re-format "/" as XFS and reinstall it (although now that I'm thinking about it - since the setup program doesn't allow you to choose XFS, it probably won't recognize those partitions). Thus, I would think that you can safely convert the partition to XFS without a format.

Sorry no what I was suggesting is a seperate /boot and / for the actual OS then put the rest in say a /data partition which is only the data. RHES itself does not support it you have to do a custom kernel as well as load the XFS tools from non-RH sources.

 

[unhappy_mage]

Hmm.. do you mean 3ware instead of Areca unless the companies are now one of the same?

Nope, my bad. I thought you were on an Areca and forgot to check my facts.

Your idea on having the OS on a RAID 1 array sounds really cool. However, since 4 hard drives is the largest number of hard drives they offer, I don't think they can fit any more (I could ask though).

What I meant was that the controller might allow splitting up the disks into partitions, and then doing raid on those partitions:

disk 1 +-----+-------------------+
disk 2 | OS  |        Data       |
disk 3 |raid1|      raid 5       |
disk 4 +-----+-------------------+

but I don't know if the 3ware (or the Areca, for that matter) will let you do that. But it might be worth a try.

 

[dreamnid]

Sorry no what I was suggesting is a seperate /boot and / for the actual OS then put the rest in say a /data partition which is only the data. RHES itself does not support it you have to do a custom kernel as well as load the XFS tools from non-RH sources.

Oh, that makes total sense. I am interested for my own personal curiosity, but if you don't want to write a quick guide, that's okay.

What I meant was that the controller might allow splitting up the disks into partitions, and then doing raid on those partitions:

Heh, sounds pretty advanced. Anyway, I figured that since only the OS will be in that raid volume, it can go bad and all I have to do is request to reload the OS on the go. Thus, it would be a waste to create a RAID 1 array for the OS when I could be using that space. However, if both volumes are RAID 5, then I would probably lose both volumes if more than 1 hard drives died? Sigh, the sticky mess I get myself into :-(

Well, they are going to try what I said before - 20gig raid volume for the os and the rest for the primary storage. Hopefully, everything will work out this time. :crosses fingers:

 

[dreamnid]

Hi gang,

An update on my issue: SL finally reloaded the server with my request configuration. The thing is I can't figure out if "/home2" mount is using all the available space in the array.

Here's a snippet of the ticket that I sent back to SL:

Hi,

I just want to make sure I'm intrepreting this right. I apologize if the following command outputs are not formatted correctly when I post this response.

[root@www2 ~]# df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda2 ext3 19G 1.5G 17G 9% /
/dev/sda1 ext3 99M 12M 83M 12% /boot
none tmpfs 2.0G 0 2.0G 0% /dev/shm
/dev/sdb1 ext3 2.0T 113M 1.9T 1% /home2

So the above output shows available size as 2.0T, but 1.9T free, as if the partition size is really 2TB. I wasn't sure either way since the available column was missing another decimal point.

I then tried playing using block sizes of both G and GB:

[root@www2 ~]# df -H --block-size G
Filesystem 1G-blocks Used Available Use% Mounted on
/dev/sda2 19G 2G 17G 9% /
/dev/sda1 1G 1G 1G 12% /boot
none 2G 0G 2G 0% /dev/shm
/dev/sdb1 2043G 1G 1939G 1% /home2
[code]

[code]
[root@www2 ~]# df -T --block-size GB
Filesystem Type 1GB-blocks Used Available Use% Mounted on
/dev/sda2 ext3 20GB 2GB 18GB 9% /
/dev/sda1 ext3 1GB 1GB 1GB 12% /boot
none tmpfs 3GB 0GB 3GB 0% /dev/shm
/dev/sdb1 ext3 2194GB 1GB 2082GB 1% /home2

The difference between the two output is whether a GB is 1024 MB or 1000MB. As you can see with the G option, the available size is slightly smaller the 2TB while the GB option has an available size slightly bigger than 2TB.

I just trying to figure out some way to confirm that the /home2 mount is indeed using all the available disk space left and is hopefully slightly bigger than the first configuration.

Unfortunately, I should have ran the same reports in the first disk configuration so I don't have anything to compare this to.

So as of now, I think I'm just going to let the issue go since I'm sure the way the numbers are reported are different that what I calculated (1024 vs 1000), but just wanted to hear what you guys thought about it.

Thanks!

 

[dreamnid]

Ok, I got this back from them. Just for completeness, I'll post it here just in case somebody runs through the same issue as us or wanted to verify their disk space.

Hardware:
Dual AMD Opterons 248
4x750gig Hard Drive in RAID 5
3ware 9500S-8 controller

//www2> info c0

Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC
------------------------------------------------------------------------------
u0 RAID-5 INITIALIZING 6 64K 2095.44 OFF OFF OFF

Port Status Unit Size Blocks Serial
---------------------------------------------------------------
p0 OK u0 698.63 GB 1465149168 3QD00XCP
p1 OK u0 698.63 GB 1465149168 3QD028SK
p2 OK u0 698.63 GB 1465149168 3QD01LHX
p3 OK u0 698.63 GB 1465149168 3QD029KV

As you probably know, you will never see the full size of the drive, so the 750G drives present themselves as 698.63GB.

You can see that works out to over close to 2T of space, digging further you see:

//www2> /c0/u0 show all
/c0/u0 status = INITIALIZING
/c0/u0 is not rebuilding, its current state is INITIALIZING
/c0/u0 is not verifying, its current state is INITIALIZING
/c0/u0 is initializing with Percent Completion = 7
/c0/u0 volume(s) = 2
/c0/u0 name =
/c0/u0 serial number = 3QD00XCPC56DA20024C3
/c0/u0 Storsave Policy = protect
/c0/u0 Command Queuing Policy = off

Unit UnitType Status %Cmpl Port Stripe Size(GB) Blocks
-----------------------------------------------------------------------
u0 RAID-5 INITIALIZING 7 - 64K 2095.44 4394465280
u0-0 DISK OK - p3 - 698.481 1464821760
u0-1 DISK OK - p2 - 698.481 1464821760
u0-2 DISK OK - p1 - 698.481 1464821760
u0-3 DISK OK - p0 - 698.481 1464821760

So we now see there are 2 volumes. Now that we have that information we need to quit and go into the OS.

Using parted and printing out the info for /dev/sda you see:

Using /dev/sda
(parted) p
Disk geometry for /dev/sda: 0.000-20479.999 megabytes
Disk label type: msdos
Minor Start End Type Filesystem Flags
1 0.031 101.975 primary ext3 boot
2 101.975 19445.866 primary ext3
3 19445.867 20473.461 primary linux-swap

Now for /dev/sdb :

Using /dev/sdb
(parted) p
Disk geometry for /dev/sdb: 0.000-2125255.000 megabytes
Disk label type: gpt
Minor Start End Filesystem Name Flags
1 0.017 2125254.984 ext3

And finally you get a full report with fdisk as well:

[root@www2 3ware]# fdisk -l

Disk /dev/sda: 21.4 GB, 21474835968 bytes
255 heads, 63 sectors/track, 2610 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 2479 19808145 83 Linux
/dev/sda3 xxxxxxxxxxxxxxxx - CC_FILTER+ 82 Linux swap

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.


Disk /dev/sdb: 2228.4 GB, 2228491387392 bytes
255 heads, 63 sectors/track, 270931 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sdb1 1 267350 2147483647+ ee EFI GPT

Thanks again guys! I appreciated everybody's help in this!

 

[dreamnid]

Well - update on the current situation

When we put the server into production, the server didn't handle the load very well that the load was always around 5-10+. I have since switched back to the old server and trying to see what is the problem with the new one.

Quick comparison between old and new server
(New)
CPU: Dual Opteron 148
Mem: 4 gigs
HD: 4 x 750 gigs on 3ware 9500 in RAID 5
OS: RHES 4 U4 (kernel 2.6.9-42)

(Old)
CPU: Dual Xeons 3GHz
Mem: 4 gigs
HD: 4 x 250 gigs in RAID 10
OS: RHES 3

I tried using the following command to see if I can reproduce the load on the new server (Yes, I realize ab is probably not the best way to benchmark the hard drives and it will theoretically cause the IOwait to sky rocket):

ab -n 1000 -c 10 (path to script)

The PHP script simply gets an image file and displays it. However, with this relatively simple script, the new server would be unresponsive for some time. I believe the IOWait pegs at 100% but I can't tell for sure. The old server is much more responsive and the iowait is usually around 0.1%. The weird thing is that the new server is usually 10-20 seconds faster than the old server.... so I don't know if I should be worried about the system responsiveness?

The newest kernel (2.6.9-42) was supposed to fix the IOwait issue for many people - but I don't see any visible changes with the above script so I'm not even sure if the IOwait issue is a problem?

I realize there is a huge difference between RAID 5 and 10 but I don't think RAID 5 should have any problem with our situation?

So does anybody have any suggestions on things to test to make sure the server can indeed handle the load fine before I put it back into production?

Thanks!

 

[unhappy_mage]

What happens if you run "top" and request an image? How about attaching strace, requesting an image, and viewing the log?

 

[longblock454]

I realize there is a huge difference between RAID 5 and 10 but I don't think RAID 5 should have any problem with our situation?

So does anybody have any suggestions on things to test to make sure the server can indeed handle the load fine before I put it back into production?

Thanks!

Before doing anymore IO tests, how about a bandwidth test?

Unhappy's test is very simple:

date; dd if=/dev/zero of=test.file bs=1k count=1M; sync; date

Or do a Bonnie++ run, with 4gigs of memory you'd have to do an 8gig file (not even sure that is possible!) or remove 3gigs of memory then do a 2gig test.

Can you expand a bit on how the new server was being taxed?

 

[dreamnid]

mage... what flags do you want used for strace? I'm assuming I should attach it to the apache process.

Before doing anymore IO tests, how about a bandwidth test?

Unhappy's test is very simple:

date; dd if=/dev/zero of=test.file bs=1k count=1M; sync; date

Or do a Bonnie++ run, with 4gigs of memory you'd have to do an 8gig file (not even sure that is possible!) or remove 3gigs of memory then do a 2gig test.

Can you expand a bit on how the new server was being taxed?

I tried Bonnie before, but there is something that stops it when it runs out of RAM (I guess I could ask them to stop our server management company to stop it for a few).

And unfortuantely, this server is a dedicated server at SoftLayer which means I don't have any physical access to it.

Here's the result of the dd command:

[root@svr3 ggo]# date; dd if=/dev/zero of=test.file bs=1k count=1M; sync; date
Mon Aug 14 20:31:27 CDT 2006
1048576+0 records in
1048576+0 records out
Mon Aug 14 20:34:43 CDT 2006

btw, any reason for the two date command? Why not just the time command?

The problem like I said before is that the server will be unresponsive. When we put hte server in production, the images won't load or takes a very long time to load. I've mentioned before that we're using the same httpd settings as the old server, so I don't think it is an Apache problem. During this, we examined that the IOwait will be spiking all the time to 70-80%.

When I did the ab test, the ssh console doesn't respond very well in that you can't enter commands and top won't refresh when it normally does.

 

[dreamnid]

Top during me accessing the image script through a browser:

top - 20:44:06 up 4 days, 10:31,  3 users,  load average: 0.00, 0.65, 2.21
Tasks: 103 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3% us,  0.0% sy,  0.0% ni, 99.0% id,  0.7% wa,  0.0% hi,  0.0% si
Mem:   4139144k total,  2293328k used,  1845816k free,   307228k buffers
Swap:  1052248k total,        0k used,  1052248k free,  1420212k cached

strace

17659 20:53:47.782156 select(0, NULL, NULL, NULL, {0, 787000}) = 0 (Timeout)
17659 20:53:48.569347 waitpid(-1, 0xbffb7578, WNOHANG|WSTOPPED) = 0
17659 20:53:48.569396 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
17659 20:53:49.570283 waitpid(-1, 0xbffb7578, WNOHANG|WSTOPPED) = 0
17659 20:53:49.570326 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
17659 20:53:50.570230 waitpid(-1, 0xbffb7578, WNOHANG|WSTOPPED) = 0
17659 20:53:50.570274 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
17659 20:53:51.570175 waitpid(-1, 0xbffb7578, WNOHANG|WSTOPPED) = 0
17659 20:53:51.570218 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
17659 20:53:52.570162 waitpid(-1, 0xbffb7578, WNOHANG|WSTOPPED) = 0
17659 20:53:52.570221 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
17659 20:53:53.570105 waitpid(-1, 0xbffb7578, WNOHANG|WSTOPPED) = 0
17659 20:53:53.570164 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
17659 20:53:54.570048 waitpid(-1, 0xbffb7578, WNOHANG|WSTOPPED) = 0
17659 20:53:54.570104 select(0, NULL, NULL, NULL, {1, 0} <unfinished ...>

=============================================
Top during the ab bench

top - 20:55:52 up 4 days, 10:42,  3 users,  load average: 1.06, 10.72, 9.46
Tasks: 114 total,   2 running, 112 sleeping,   0 stopped,   0 zombie
Cpu(s):  7.0% us,  4.3% sy,  0.0% ni, 43.1% id, 45.6% wa,  0.0% hi,  0.0% si
Mem:   4139144k total,  2322184k used,  1816960k free,   315472k buffers
Swap:  1052248k total,        0k used,  1052248k free,  1422368k cached

The strace for the ab bench is humongous...
I'll do the first couple lines

(lines similar to the first strace were snipped)
17659 20:55:45.573161 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
17659 20:55:46.577846 waitpid(-1, 0xbffb7578, WNOHANG|WSTOPPED) = 0
17659 20:55:46.577907 select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
17659 20:55:47.603465 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f9e708) = 17931
17931 20:55:47.604767 rt_sigaction(SIGHUP, {0x3fe372, [], SA_RESTORER|SA_INTERRUPT, 0x827898},  <unfinished ...>
17659 20:55:47.604850 waitpid(-1,  <unfinished ...>
17931 20:55:47.604879 <... rt_sigaction resumed> {0x3fe3a4, [HUP USR1], SA_RESTORER, 0x827898}, 8) = 0
17659 20:55:47.604929 <... waitpid resumed> 0xbffb7578, WNOHANG|WSTOPPED) = 0
17931 20:55:47.604964 rt_sigaction(SIGTERM, {0x3fe372, [], SA_RESTORER|SA_INTERRUPT, 0x827898},  <unfinished ...>
17659 20:55:47.605010 select(0, NULL, NULL, NULL, {1, 0} <unfinished ...>
17931 20:55:47.605044 <... rt_sigaction resumed> {0x3fe37f, [], SA_RESTORER, 0x827898}, 8) = 0
17931 20:55:47.605090 rt_sigaction(SIGUSR1, {SIG_IGN}, {0x3fe3a4, [HUP USR1], SA_RESTORER, 0x827898}, 8) = 0
17931 20:55:47.605284 geteuid32()       = 0
17931 20:55:47.605350 setgid32(48)      = 0
17931 20:55:47.605405 open("/proc/sys/kernel/ngroups_max", O_RDONLY) = 15
17931 20:55:47.605498 read(15, "65536\n", 31) = 6
17931 20:55:47.605558 close(15)         = 0
...

Anything else, please let me know

 

[unhappy_mage]

mage... what flags do you want used for strace? I'm assuming I should attach it to the apache process.

I think so; what does the php script run under? That's what I'd trace.

btw, any reason for the two date command? Why not just the time command?

Because 'time' only times a single command, and with several GB of ram the 'dd' write could be completely done into ram. Thus the 'sync' forces all buffer to be written to disk. Really, one should run 'sync' before starting the timed run - I think that might be skewing your times a wee bit. 1024 MB in 196 seconds is only 5 MB/s - if that's actually what you get, there's a Problem.

Those strace logs aren't very helpful, unfortunately - all it shows is that a child is started, and then waited for, and it takes a long time to finish. Try "strace -f" and then whatever other options you used. Write the output to a file, and email it to me if it's too long to post.

 

[dreamnid]

Because 'time' only times a single command, and with several GB of ram the 'dd' write could be completely done into ram. Thus the 'sync' forces all buffer to be written to disk. Really, one should run 'sync' before starting the timed run - I think that might be skewing your times a wee bit. 1024 MB in 196 seconds is only 5 MB/s - if that's actually what you get, there's a Problem.

After I sent the e-mail to you, I was going to run the same command with time to prove that it is the same.... until I realized that there was indeed a sync command after 'dd'. I'm pretty sure I did see another thread that had the same command without the sync (or maybe my eyes missed it then too), but your explanation about 'time' only works with one command makes sense.

I suppose you could make a batch script if you do want to use the time command however.

Anyways, hopefully you'll find something in the strace log. If not, I'll see if I can get Bonnie++ to work.

 

[dreamnid]

Possible more info:

root@svr3 httpd]# /root/3ware/tw_cli
//svr3> /c0/u0 show all
/c0/u0 status = OK
/c0/u0 is not rebuilding, its current state is OK
/c0/u0 is not verifying, its current state is OK
/c0/u0 is initialized.
/c0/u0 volume(s) = 2
/c0/u0 name =
/c0/u0 serial number = 3QD00XCPC56DA20024C3
/c0/u0 Storsave Policy = protect
/c0/u0 Command Queuing Policy = on

Unit     UnitType  Status         %Cmpl  Port  Stripe  Size(GB)  Blocks
-----------------------------------------------------------------------
u0       RAID-5    OK             -      -     64K     2095.44   4394465280
u0-0     DISK      OK             -      p3    -       698.481   1464821760
u0-1     DISK      OK             -      p2    -       698.481   1464821760
u0-2     DISK      OK             -      p1    -       698.481   1464821760
u0-3     DISK      OK             -      p0    -       698.481   1464821760