New Server (slow speed?)

[emieldegroot]

Since 2 months I have the following server. All works but I have doubts about the speed of my file server.

Specs:
Case: Lian Li PC-Q25B
PSU: FSP Aurum Gold 400
Motherboard: Intel Desktop Board DQ67EPB3
CPU: Intel Core i5 2500
RAM: 16 gb (2x8) GeiL Black Dragon GB316GB1333C9DC
Controller Cards: 1x IBM ServeRAID M1015 (flashed)
Hard Pool: 21TB (7x Hitachi Deskstar 5K3000, 3TB)
Hard Drives (VM’s): 128gb SSD (Crucial m4 CT128M4SSD2)
Software: Esxi 5 (ZFS pool: openindiana 151a5 with napp-it 0.8h)

On a daily basis I run these 6 Virtual machines:

1. OpenIndiana/Napp-it (ZFS Pool, SABnzbd, Sickbeard: 8192mb, 4 cpu’s, VMXNET3, 30Gb SSD, direct acces LSI2008 )
2. . WinXP (Squeezebox / Subsonic: 1024mb memory, 1cpu, VMXNET3, 10GB VM on SSD)
3. . WinXP (1024mb memory, 1cpu, VMXNET3, 10GB VM on SSD)
4. . WinXP (1024mb memory, 1cpu, VMXNET3, 10GB VM on SSD)
5. WinXP (1024mb memory, 1cpu, VMXNET3, 10GB VM on SSD)
6. WinXP (1024mb memory, 1cpu, VMXNET3, 10GB VM on SSD)

Screen esxi

Screen napp-it

Now when I acces the ZFS pool from a external win7 machine within my apartment I get the following results in crystalmark

When I copy a file from my server to this machine (SSD hdd) It is not so fast: 25-32mb sec.

The “funny” thing is when I am using DD bench from napp-it I get the following results

These results look fine….

My question: How can I optimize my server and do you have some tips or tricks to get a better performance?

 

[_Gea]

Your main problem is that ESXi requests sync writes on NFS for best data security.
If you disable sync on your NFS shared folder, you may see massive improvements in ESXi (asume you use NFS shared storage)

To keep this performance and the security, you need a Log device.
Only RAM based ones can keep the whole performance but a SSD (4-8GB is enough) can help a little.

Next is your pool layout. While a Raid Z1 of seven disks can give a good sequential performance, the I/O performance (small random read and writes, important for ESXi) is like a single disk. If you create a pool from 3 x mirrors with a hotspare, ESXi will be also faster.

 

[ST3F]

The same to me, in Samba / CIFS on

• Xeon x3450 4c/8t 2,66 / 3,1
• Intel S3420GPV
• 8 GB RAM : 4x 2 Go ECC HP 1066
• Operating System on SSD OCZ Vertex Series Plus 60 Go (last JMicron Firmware) partitioned @ 30 Go (.... for 30 Go provisionning ^^)
• IBM M1015 flashed in IT 9211-8i
• vDev #1 : 5x 1 TB WD Black WD1001FALS 7200 Trs in Icy Dock MB455SPF-B 5 in 3
• vDev #2 : 5x 2 TB Hitachi HDS72302
• 2x SSD STEC Mach 16 50 Go(ZIL write cache mirrored)
• 1x SSD OCZ Vertex 3 60 Go (LOG) / last firmware
• Nic : Dual port 10 GbE SFP+ SR
• Sharkoon Rebel 12
• Seasonic M2II 650w

write 10.24 GB via dd, please wait...
time dd if=/dev/zero of=/black/dd.tst bs=1024000 count=10000

10000+0 records in
10000+0 records out

real       12.3
user        0.0
sys         5.0

10.24 GB in 12.3s = 832.52 MB/s Write

read 10.24 GB via dd, please wait...
time dd if=/black/dd.tst of=/dev/null bs=1024000

10000+0 records in
10000+0 records out

real       14.4
user        0.0
sys         3.5

10.24 GB in 14.4s = 711.11 MB/s Read

Still through Samba, folder "video" mounted on disk drive (Y:\) in Windows 7 Pro 64b (NFS client protocol doesn't exist for the Pro version ; only on Enterprise & Ultimate)

This is my test box, before the "Faster, Higher, Stronger : powerful storage".

Why CIFS / Samba protocol is so slow on read ?

Cheers.

St3F.

 

[D-EJ915]

Also dump the vCPUs down on that VM down.

 

[emieldegroot]

I access the server with the CIFS / Samba protocol. In Napp-it the option NFS is turned off on each folder.

Your main problem is that ESXi requests sync writes on NFS for best data security.
If you disable sync on your NFS shared folder, you may see massive improvements in ESXi (asume you use NFS shared storage)

My VM”s are stored on the SSD (controlled by the motherboard SATA controller). My ZFS Pool is directly connect to OpenIndiana (all on the IBM M1015). How come ESXi is using the ZFS pool?

When disabling the sync writes on a folder I do get slight improvement

Old (sync standard):

New (disable sync):

Copying file from server is around 68 mb/sec (do not know why there is a difference with crystalmark) with syn disable. It is around 60mb/sec when enabled.

To keep this performance and the security, you need a Log device.
Only RAM based ones can keep the whole performance but a SSD (4-8GB is enough) can help a little.

Thanks for the suggestion. I do have some additional questions:
a) When using a LOG device, is it still safe to disable the sync? (what if power shuts down, I do not own a UPS).

b) Is 1x LOG device enough (no mirror). Is it best to connect to last port on my flashed IBM1015? The IBM has a direct pass-through in esxi to openindiana.

c) Like a 8Gb Transcend SSD (Half SLIM)? The only problem is that this is a MLC. The SLC version (MSA500) is not available where i live (Netherlands)

d) I guess I will need a SLC SSD but those are really hard to get… any suggestions?

Also dump the vCPUs down on that VM down

What do you mean exactly? Do you mean on the VM: Edit settings -> Resources -> CPU -> Shares “Low”?

My resources in esxi

Next is your pool layout. While a Raid Z1 of seven disks can give a good sequential performance, the I/O performance (small random read and writes, important for ESXi) is like a single disk. If you create a pool from 3 x mirrors with a hotspare, ESXi will be also faster.

Do you mean: 3x normal HDD, 3x Mirror, 1x Hotspare? Then I gave 9 tb left right?


Other questions about possible speed improvement:

Does my (read) speed if I use L2arc? If so then:
- What size SSD is needed? I cannot find a guideline
- Because the IBM is out of free Sata Port I guess I need to use my free ports on the motherboard. Only I cannot pass through the Sata controller to OpenIndiana (otherwise the SSD with all the VM’s on it is unusable for esxi). Does this effect speed?


Will my speed improve with other ZFS alignment (ashift=12 instead of ashift=9)
- Can I change the ashift without losing my current data in the pool?