![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
I am using: FreeBSD 9.1-RELEASE-p7 amd64.
System hardware:
Output from camcontrol devlist:
I'm seeing very stable but slow performance from drives in the Rackable SE3016 on the BR10i/LSI SAS3081e. Using dd to do large sequential reads from one drive does not saturate the bandwidth, but two or more seem to hit a limit at about 200 MB/sec. I'm using gstat to determine this. The transfer rates on drives on scbus12 never add up to more than about 200 MB/sec. I'm using, for example:
In the example given, I see about 50 MB/sec from all four drives. I see very similar behavior after constructing a ZFS pool and scrubbing. Scrubs of terabytes of data do not show any errors.
The Rackable enclosure provides expander functionality but the OEM expander manufacturer is undetermined (camcontrol identifies as RACKABLE SE3016-SAS 0227). I'm attaching it with a SFF-8087 to SFF-8088 cable and have tried two different cables with the same result.
I confirmed four PHY links appear to be up and active on the onboard LEDs, so I think I've ruled out my initial suspicion that it wasn't linking up a wide (4 PHY) port.
I've verified that an Adaptec 1045 card on a similar X7SBE running a Linux live CD gets me a much higher ceiling, closer to 800-900 MB/sec, when running similar dd testing.
I attempted to use camcontrol smpphylist /dev/da8 to examine the SAS topology but I get:
Internet searches don't seem to indicate that this is a common problem. I am planning on booting the main FreeBSD server using a Linux live CD to try to isolate whether this is a hardware or FreeBSD/mpt driver problem but that requires a maintenance window. Are there any other tools that I can use to further determine the cause?
System hardware:
- Supermicro X7SBE
- Intel Q9300 (Core 2 Quad)
- Onboard Intel ICH-9 SATA-II ports
- Onboard Intel 82573E/L 1 Gbps Ethernet ports
- Onboard ATI ES1000 video
- IBM BR10i (crossflashed to LSI SAS3081e, 1068e controller) as PCI-Express 8x card (Fusion-MPT SAS)
- Supermicro AOC-SAT2-MV8 (MV88SX6081 8-port SATA II controller) on PCI-X card
- Adaptec ASC-39320 SCSI controller on PCI-X card
- Silicon Image PCI0680 PCI ATA-133 host controller on PCI card
- Dual 82546EB gigabit Ethernet controller on PCI-X card
- 2x Supermicro CSE-M35S enclosure attached to ASC-39320
- 3x HTS721010G9SA00 (Hitachi 2.5" 100 GB SATAI) attached to ICH-9
- 2x SSDSA2CW120G3 (Intel 120 GB SATAII SSD) attached to ICH-9
- 1x Samsung UM410 (Samsung 16 GB SATAII SSD) attached to ICH-9
- 8x Fujitsu MBA3300NC, four in each CSE-M35S attached to ASC-39320
- 8x Hitachi HDS721010CLA332 attached to AOC-SAT2-MV8[/file]
- 4x BenQ DW1640/DW1650 (DVD+RW drive) attached to PCI0680
- 1x Rackable (SGI) SE3016 attached to BR10i
- Assortment of drives in the SE3016 on the BR10i, including Fujitsu MBA3300RC, Toshiba DT01ACA200, Hitachi HTS721010G9SA00, etc.
Output from camcontrol devlist:
Code:
root@cadence:/tmp # camcontrol devlist
<BENQ DVD DD DW1640 BSLB> at scbus0 target 0 lun 0 (cd0,pass0)
<BENQ DVD DD DW1640 BSLB> at scbus0 target 1 lun 0 (cd1,pass1)
<BENQ DVD DD DW1640 BSLB> at scbus1 target 0 lun 0 (cd2,pass2)
<BENQ DVD DD DW1650 BCIC> at scbus1 target 1 lun 0 (cd3,pass3)
<FUJITSU MBA3300NC 0102> at scbus2 target 0 lun 0 (da0,pass4)
<FUJITSU MBA3300NC 0102> at scbus2 target 1 lun 0 (da1,pass5)
<FUJITSU MBA3300NC 4801> at scbus2 target 2 lun 0 (da2,pass6)
<FUJITSU MBA3300NC 0102> at scbus2 target 4 lun 0 (da3,pass7)
<SUPER GEM318 0> at scbus2 target 6 lun 0 (ses0,pass8)
<FUJITSU MBA3300NC 5E03> at scbus3 target 0 lun 0 (da4,pass9)
<FUJITSU MBA3300NC 5E03> at scbus3 target 1 lun 0 (da5,pass10)
<FUJITSU MBA3300NC 5E03> at scbus3 target 2 lun 0 (da6,pass11)
<FUJITSU MBA3300NC 4801> at scbus3 target 4 lun 0 (da7,pass12)
<SUPER GEM318 0> at scbus3 target 6 lun 0 (ses1,pass13)
<Hitachi HDS721010CLA332 JP4OA25C> at scbus4 target 0 lun 0 (ada0,pass14)
<Hitachi HDS721010CLA332 JP4OA25C> at scbus5 target 0 lun 0 (ada1,pass15)
<Hitachi HDS721010CLA332 JP4OA39C> at scbus6 target 0 lun 0 (ada2,pass16)
<Hitachi HDS721010CLA332 JP4OA39C> at scbus7 target 0 lun 0 (ada3,pass17)
<Hitachi HDS721010CLA332 JP4OA39C> at scbus8 target 0 lun 0 (ada4,pass18)
<Hitachi HDS721010CLA332 JP4OA39C> at scbus9 target 0 lun 0 (ada5,pass19)
<Hitachi HDS721010CLA332 JP4OA39C> at scbus10 target 0 lun 0 (ada6,pass20)
<Hitachi HDS721010CLA332 JP4OA39C> at scbus11 target 0 lun 0 (ada7,pass21)
<ATA TOSHIBA DT01ACA2 ABB0> at scbus12 target 7 lun 0 (pass40,da19)
<ATA ST3250824AS H> at scbus12 target 8 lun 0 (pass34,da13)
<RACKABLE SE3016-SAS 0227> at scbus12 target 9 lun 0 (pass29,ses2)
<ATA ST3160815AS A> at scbus12 target 10 lun 0 (pass38,da17)
<ATA ST3250824AS H> at scbus12 target 12 lun 0 (pass39,da18)
<ATA Hitachi HDS72101 A3MA> at scbus12 target 13 lun 0 (pass32,da11)
<ATA Hitachi HDS72101 A3MA> at scbus12 target 14 lun 0 (pass33,da12)
<ATA ST3250824AS H> at scbus12 target 15 lun 0 (pass35,da14)
<ATA ST3160815AS A> at scbus12 target 16 lun 0 (pass37,da16)
<ATA ST3160812AS J> at scbus12 target 17 lun 0 (pass30,da9)
<ATA ST3250824AS H> at scbus12 target 18 lun 0 (pass36,da15)
<ATA Hitachi HDS72101 A39C> at scbus12 target 19 lun 0 (pass31,da10)
<FUJITSU MBA3300RC D305> at scbus12 target 20 lun 0 (pass28,da8)
<Hitachi HTS721010G9SA00 MCZOC10H> at scbus13 target 0 lun 0 (ada8,pass22)
<Hitachi HTS721010G9SA00 MCZOC10H> at scbus14 target 0 lun 0 (ada9,pass23)
<HTS721010G9SA00 MCZOC10V> at scbus15 target 0 lun 0 (ada10,pass24)
<SAMSUNG SSD UM410 Series 2.5" 16GB VAM12D1Q> at scbus16 target 0 lun 0 (ada11,pass25)
<INTEL SSDSA2CW120G3 4PC10362> at scbus17 target 0 lun 0 (ada12,pass26)
<INTEL SSDSA2CW120G3 4PC10362> at scbus18 target 0 lun 0 (ada13,pass27)I'm seeing very stable but slow performance from drives in the Rackable SE3016 on the BR10i/LSI SAS3081e. Using dd to do large sequential reads from one drive does not saturate the bandwidth, but two or more seem to hit a limit at about 200 MB/sec. I'm using gstat to determine this. The transfer rates on drives on scbus12 never add up to more than about 200 MB/sec. I'm using, for example:
Code:
dd bs=256k if=/dev/da8 of=/dev/null
dd bs=256k if=/dev/da9 of=/dev/null
dd bs=256k if=/dev/da10 of=/dev/null
dd bs=256k if=/dev/da11 of=/dev/nullIn the example given, I see about 50 MB/sec from all four drives. I see very similar behavior after constructing a ZFS pool and scrubbing. Scrubs of terabytes of data do not show any errors.
The Rackable enclosure provides expander functionality but the OEM expander manufacturer is undetermined (camcontrol identifies as RACKABLE SE3016-SAS 0227). I'm attaching it with a SFF-8087 to SFF-8088 cable and have tried two different cables with the same result.
I confirmed four PHY links appear to be up and active on the onboard LEDs, so I think I've ruled out my initial suspicion that it wasn't linking up a wide (4 PHY) port.
I've verified that an Adaptec 1045 card on a similar X7SBE running a Linux live CD gets me a much higher ceiling, closer to 800-900 MB/sec, when running similar dd testing.
I attempted to use camcontrol smpphylist /dev/da8 to examine the SAS topology but I get:
Code:
camcontrol: error sending command
Segmentation fault (core dumped)Internet searches don't seem to indicate that this is a common problem. I am planning on booting the main FreeBSD server using a Linux live CD to try to isolate whether this is a hardware or FreeBSD/mpt driver problem but that requires a maintenance window. Are there any other tools that I can use to further determine the cause?