Very sluggish performance testing Nexentastor 3.1.2

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
Hi there,

I'm testing Nexentastor community version on a HP ML350 G5, but so far using iSCSI or NFS the performance is being very sluggish. The server has a 2x dual core intel CPU, 6GB of ram and Gbit network, 6x 73GB 15k SAS drives for the dataset test and 2x 73GB for nexentastor.

I'm having problems with write speeds not going further than 40MB/sec on sequencial writtes and reads about 700KB second, really weird... I've tested the network, everything at the physical level is fine. I've tried a array of "raid0" sets or mirrors, with and without sync but so far no luck.

Could you guys give me some directions of what I can do to really get what ZFS can offer performance wise please?
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
You might want to try the OI+NappIT solution well documented in this forum to see if you get the same perf bottleneck. Might rule out a few things. Also, consider posting here as they would probably have more insight as to troubleshooting nexenta issues:

http://nexentastor.org/projects/site/boards

I'll try NappIT tomorrow and check how it goes, what the OI stands for? never heard about it

I'm posting on nexenta forums, but so far no luck with it :/
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
OI = http://openindiana.org/
There is how-to's in the Napp-IT thread: http://hardforum.com/showthread.php?t=1573272

As for nexentastor.org forums, sometimes you need to bump the thread and nag a bit to get a response. annoying unfortunately, but reality. What is the thread on there? post a link here as well if you dont mind.


Hi mate,

Thanks for this,

here is the link http://www.nexentastor.org/boards/2/topics/5662

I started with a VM then went to baremetal server to ensure it was not the VM.

I'm evaluating ZFS as I would like to implement commercial solutions in the future, but so far I'm not having much luck with it. Would the commercial version of nexenta any different?
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
I'm evaluating ZFS as I would like to implement commercial solutions in the future, but so far I'm not having much luck with it. Would the commercial version of nexenta any different?

The Enterprise Edition NexentaStor EE is the same like the Community Edition NexentaStor CE.
EE edition adds support and free or nonfree plugins.

I would try:
- If you use ZFS/ iSCSI with ESXi, all writes are sync write requests and ZFS will follow this demands.
You can disable this strict sync write behaviour to check if this is your problem. Performance may be much better.
You have then the option to use a UPS or buy a ZIL device to combine highest security with performance.

- Current NexentaStor 3 is based on a quite old code base. They currently move NexentaStor to a Illumian base
(it is then something like OpenIndiana with Debian like package managing + support + plugins).
You may try Illumian/ OpenIndiana now as well. You can expect the same performance like with next NexentaStor 4.
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
The Enterprise Edition NexentaStor EE is the same like the Community Edition NexentaStor CE.
EE edition adds support and free or nonfree plugins.

I would try:
- If you use ZFS/ iSCSI with ESXi, all writes are sync write requests and ZFS will follow this demands.
You can disable this strict sync write behaviour to check if this is your problem. Performance may be much better.
You have then the option to use a UPS or buy a ZIL device to combine highest security with performance.

- Current NexentaStor 3 is based on a quite old code base. They currently move NexentaStor to a Illumian base
(it is then something like OpenIndiana with Debian like package managing + support + plugins).
You may try Illumian/ OpenIndiana now as well. You can expect the same performance like with next NexentaStor 4.

Hi mate,

I've tried with sync disabled but it didn't make any difference.

I'll give OI a go now, test and then I come back with some metrics :)

thanks!
 

ajm83

n00b
Joined
Jan 25, 2012
Messages
32
Are the drives connected via a smartarray?

I had to use set zfs:zfs_vdev_max_pending=1 and set zfs:zfs_nocacheflush=1 to get mine working.
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
Are the drives connected via a smartarray?

I had to use set zfs:zfs_vdev_max_pending=1 and set zfs:zfs_nocacheflush=1 to get mine working.

I'm using a LSI adapter (PERC H200, son't remember what LSI model exactly it is) with the IT firmware presenting the RAW drives to the OS
 

brutalizer

[H]ard|Gawd
Joined
Oct 23, 2010
Messages
1,600
You can also try the LiveCD Solaris 11 from Oracle. The license allows testing purposes. If it does not work in Solaris 11, then you might have a problem.
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
You can also try the LiveCD Solaris 11 from Oracle. The license allows testing purposes. If it does not work in Solaris 11, then you might have a problem.

It has a trial time limit isn't it?

I want to test for a couple oh months putting exchange and sql databases on it and check if I can really see the benefits of using ZFS and all its features
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
The installation of napp-it is not running for some reason

20120419_093225.jpg
 

spankit

Limp Gawd
Joined
Oct 18, 2010
Messages
262
You can elevate the prompt by type su (ENTER) and type your root password. Type the same command in and it will work like a charm.
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
Hi Guys,

Thanks, I used to type sudo and then the command but it looks a bit different on openindiana.

Anyway, I installed it and on openindiana and on nappit I can see the disks as bellow:

napp1.JPG


then when I try to create a volume I get this:

napp2.JPG


I used a partition manager to ensure that all disks were clean, but even after restarting the server, only 3 disks show up
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
This happens when smartmontools (smartctrl) is not installed
.
With OI this happens with an earlier napp-it wget installer due a bug in OI 151
prior developer release a3 where you must install extra tools one by one followed by a reboot.

workaround:
update to OI 151a3 and/or
rerun the napp-it wget installer and look at the setup messages
you may need to run the installer twice

about your disks:
do not use any partition tool, you need raw disks with OI
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
I managed to create the array with those other disks manually using zpool byt becuase the names are than long it is a bit boring, if there any way to shortcut the name of the drives?

I'm using the update manager on OI and reinstalling nappit, hope things get better then :)

I just ised gparted to delete the partitions and make sure the disks were blank.

I get back to you guys when I finish testing it, thanks for your help again, much appreciated
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
I managed to create the array with those other disks manually using zpool byt becuase the names are than long it is a bit boring, if there any way to shortcut the name of the drives?

I'm using the update manager on OI and reinstalling nappit, hope things get better then :)

I just ised gparted to delete the partitions and make sure the disks were blank.

I get back to you guys when I finish testing it, thanks for your help again, much appreciated


These long numbers are not controller id's but unique disk numbers (WWN).
They keep the same when you move a disk to another controller port.

They are always used with LSI SAS2 controller in IT mode.
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
The update on OI didn't make any difference. I'll get a solaris ISO and try it now as it seems to be the best option.

What are your thoughts between solaris and OI ?
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
Installed OI a3 from scratch now instead of 3 is showing only 2 drives to create pools on nappit. Loading solaris 11 now, if nappit doesn't work, is there any other good alternative? I would like to use a GUI for now until I learn better how to deal with ZFS
 

hotalot

n00b
Joined
Dec 24, 2011
Messages
14
Hi there,

I'm testing Nexentastor community version on a HP ML350 G5, but so far using iSCSI or NFS the performance is being very sluggish. The server has a 2x dual core intel CPU, 6GB of ram and Gbit network, 6x 73GB 15k SAS drives for the dataset test and 2x 73GB for nexentastor.

I'm having problems with write speeds not going further than 40MB/sec on sequencial writtes and reads about 700KB second, really weird... I've tested the network, everything at the physical level is fine. I've tried a array of "raid0" sets or mirrors, with and without sync but so far no luck.

Could you guys give me some directions of what I can do to really get what ZFS can offer performance wise please?

If you are using linux remember that linux defaults to nfs version 3, OI defaults to version 4. You can try mounting your volume using either one and see which works best.

To make things easier you can create a folder in you linux machine where you'll always mount your server volume, a good place for it is the /media folder created by the system when you installed linux.

Let's make a folder caller Firstest in /media. In a terminal type:

sudo mkdir Firstest /media/Firstest

Let's say your pool is called mypool and your datastore is called data.To mount it with version 3 type the following (in a terminal):

sudo mount -t nfs -o vers=3 <server IP>:/mypool/data /media/Firstest

To mount the volume using version 4 type:

sudo mount -t nfs4 <server IP>:/mypool/data /media/Firstest

In my linux machine (MInt 12) it takes almost 2 minutes to mount nfs version 4 but it's faster than version 3 on transfers. Version 3 mounts in 2 or 3 seconds.

You can also try disabling sync on your server:

zfs set sync=disabled mypool
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
Installed OI a3 from scratch now instead of 3 is showing only 2 drives to create pools on nappit. Loading solaris 11 now, if nappit doesn't work, is there any other good alternative? I would like to use a GUI for now until I learn better how to deal with ZFS

napp-it use default Solaris tools to detect disks like iostat -En or format
You can try these tools to find the problem.

I would expect disk problems (partition), controller problems or cabling problems
 

cantalup

Gawd
Joined
Feb 8, 2012
Messages
758
If you are using linux remember that linux defaults to nfs version 3, OI defaults to version 4.
........

not exactly true ...
it depends on linux distro
some distros are using nfs4 by default when try to mount nfs.
Centos, debian and Ubuntu as I remember; I had to define to mount nfs3 on mount option.

you do not need to define nfs4 on most current release distro where nfs4 is pretty stable and common as today.
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
Hi guys,

Sorry I've been away for several weeks and broke my left foot. Back to business, the server detects all disks, but nappit for some reason still doesn't when I go to create a pool. Otherwise nappit display the disks if I select to view the physical disks. Such strange thing!

Could you guys give me a hand please?
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
I'm guessing the other drives are already in a pool?

there is no any pool as you can see on the screen shots, also all drives don't have any data, partition, nothing, they are plain blank.

Not sure what is really going on with nappit :/
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
napp-it disk detection is based on the Solaris iostat -Ensr command (in /var/web-gui/data/napp-it/zfsos/_lib/get-disk.pl script)
It expects an unformatted or not-Solaris formatted disk that is part of an active/ not destroyed pool.

Not detected ot not usable discs are mostly due to
- unsupported controller (not the case)
- disks have unsupported partitions
- disks are part of active/not destroyed pools
- a napp-it bug with your config

Try disks from stock or repartition to ntfs or hfs on Windows/Mac or use OI format.
Check napp-it menu disk - details for more infos
 

madrebel

Gawd
Joined
Sep 23, 2011
Messages
724
or just run the format command from the cli. if format doesn't see the drives, they aren't visible.
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
or just run the format command from the cli. if format doesn't see the drives, they aren't visible.

Searching for disks...
Mode sense page(3) reports nsect value as 640, adjusting it to 935
Mode sense page(3) reports nsect value as 640, adjusting it to 935
Mode sense page(3) reports nsect value as 640, adjusting it to 935
Mode sense page(3) reports nsect value as 640, adjusting it to 935
Mode sense page(3) reports nsect value as 640, adjusting it to 935
Mode sense page(3) reports nsect value as 640, adjusting it to 935
done

c8t5000C50000E5FC3Dd0: configured with capacity of 68.30GB
c9t5000C50000E96485d0: configured with capacity of 68.30GB
c10t5000C500014292E1d0: configured with capacity of 68.30GB
c11t5000C5000144A03Dd0: configured with capacity of 68.30GB
c12t5000C5000144F205d0: configured with capacity of 68.30GB
c13t5000C50001450ECDd0: configured with capacity of 68.30GB
c14t500000E01FB35062d0: configured with capacity of 68.36GB


AVAILABLE DISK SELECTIONS:
0. c3t500000E01E724EF2d0 <HP-DH072BAAKN-HPD3 cyl 8921 alt 2 hd 255 sec 63>
/pci@0,0/pci8086,25f7@2/pci8086,3500@0/pci8086,3514@1/pci1028,1f1e@0/iport@80/disk@w500000e01e724ef2,0
1. c8t5000C50000E5FC3Dd0 <HP-DH072ABAA6-HPDB cyl 38296 alt 2 hd 4 sec 935>
/pci@0,0/pci8086,25f7@2/pci8086,3500@0/pci8086,3514@1/pci1028,1f1e@0/iport@1/disk@w5000c50000e5fc3d,0
2. c9t5000C50000E96485d0 <HP-DH072ABAA6-HPDB cyl 38296 alt 2 hd 4 sec 935>
/pci@0,0/pci8086,25f7@2/pci8086,3500@0/pci8086,3514@1/pci1028,1f1e@0/iport@2/disk@w5000c50000e96485,0
3. c10t5000C500014292E1d0 <HP-DH072ABAA6-HPDB cyl 38296 alt 2 hd 4 sec 935>
/pci@0,0/pci8086,25f7@2/pci8086,3500@0/pci8086,3514@1/pci1028,1f1e@0/iport@4/disk@w5000c500014292e1,0
4. c11t5000C5000144A03Dd0 <HP-DH072ABAA6-HPDB cyl 38296 alt 2 hd 4 sec 935>
/pci@0,0/pci8086,25f7@2/pci8086,3500@0/pci8086,3514@1/pci1028,1f1e@0/iport@8/disk@w5000c5000144a03d,0
5. c12t5000C5000144F205d0 <HP-DH072ABAA6-HPDB cyl 38296 alt 2 hd 4 sec 935>
/pci@0,0/pci8086,25f7@2/pci8086,3500@0/pci8086,3514@1/pci1028,1f1e@0/iport@10/disk@w5000c5000144f205,0
6. c13t5000C50001450ECDd0 <HP-DH072ABAA6-HPDB cyl 38296 alt 2 hd 4 sec 935>
/pci@0,0/pci8086,25f7@2/pci8086,3500@0/pci8086,3514@1/pci1028,1f1e@0/iport@20/disk@w5000c50001450ecd,0
7. c14t500000E01FB35062d0 <HP-DH072BAAKN-HPD3 cyl 41676 alt 2 hd 4 sec 860>
/pci@0,0/pci8086,25f7@2/pci8086,3500@0/pci8086,3514@1/pci1028,1f1e@0/i
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
napp-it disk detection is based on the Solaris iostat -Ensr command (in /var/web-gui/data/napp-it/zfsos/_lib/get-disk.pl script)
It expects an unformatted or not-Solaris formatted disk that is part of an active/ not destroyed pool.

Not detected ot not usable discs are mostly due to
- unsupported controller (not the case)
- disks have unsupported partitions
- disks are part of active/not destroyed pools
- a napp-it bug with your config

Try disks from stock or repartition to ntfs or hfs on Windows/Mac or use OI format.
Check napp-it menu disk - details for more infos

All disks formated at NTFS, still the same, no more than 2 disks available at the nappit GUI
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
All disks formated at NTFS, still the same, no more than 2 disks available at the nappit GUI

please check:
1. napp-it menu disk details: look if disks are configured under "busy"
2. enable edit (top right menu), klick on %disk to display all collected disk values,
look under "unused" - these are the disks used in pool - create

optionally
- check if you see a difference between the two disks that are displayed and others.
- can you try another disk beside your HP's to check against
- do you have the option to use another controller (like onboard AHCI Sata)
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
That is all the devices listed there:

all

->

'c3t500000E01E724EF2d0\tc8t5000C50000E5FC3Dd0\tc9t5000C50000E96485d0\tc10t5000C500014292E1d0\tc11t5000C5000144A03Dd0\tc12t5000C5000144F205d0\tc5t0d0\tc13t5000C50001450ECDd0\tc14t500000E01FB35062d0'

The strangely I have several disks like this:
c10t5000C500014292E1_busy -> 'configured'
c11t5000C5000144A03D_busy -> 'configured'
c12t5000C5000144F205_busy -> 'configured'
c13t5000C50001450ECD_busy -> 'configured'

and the list goes with the rest of the disks, then the last bit:




configured ->'c3t500000E01E724EF2d0\tc8t5000C50000E5FC3Dd0\tc9t5000C50000E96485d0\tc10t5000C500014292E1\tc11t5000C5000144A03D\tc12t5000C5000144F205\tc13t5000C50001450ECD\tc14t500000E01FB35062'

connected ->
'c3t500000E01E724EF2d0\tc8t5000C50000E5FC3Dd0\tc9t5000C50000E96485d0\tc10t5000C500014292E1\tc11t5000C5000144A03D\tc12t5000C5000144F205\tc13t5000C50001450ECD\tc14t500000E01FB35062'

controller2 ->
'c10\tc11\tc12\tc13\tc14\tc3t\tc5t\tc8t\tc9t'



controller ->
'c3\tc6\tc7\tc8\tc9\tc10\tc11\tc12\tc13\tc14\tusb4/2\tusb7/1\tc1\tc5'

disks_basic -> 'c3t500000E01E724EF2d0'

disks_cache ->

disks_logs ->

disks_mirror ->

disks_raidz ->

disks_spares ->

disks_unavail ->

notconfigured ->

replaceable -> 'c8t5000C50000E5FC3Dd0\tc9t5000C50000E96485d0'

unused -> 'c8t5000C50000E5FC3Dd0\tc9t5000C50000E96485d0'

used -> 'c3t500000E01E724EF2d0\t'
 

_Gea

2[H]4U
Joined
Dec 5, 2010
Messages
4,094
Code:
configured ->'c3t500000E01E724EF2d0\tc8t5000C50000E5FC3Dd0\tc9t5000C50000E96485d0\tc10t5000C500014292E1\tc11t5000C5000144A03D\tc12t5000C5000144F205\tc13t5000C50001450ECD\tc14t500000E01FB35062'

connected -> 
'c3t500000E01E724EF2d0\tc8t5000C50000E5FC3Dd0\tc9t5000C50000E96485d0\tc10t5000C500014292E1\tc11t5000C5000144A03D\tc12t5000C5000144F205\tc13t5000C50001450ECD\tc14t500000E01FB35062'

unused -> 'c8t5000C50000E5FC3Dd0\tc9t5000C50000E96485d0'

The disk-Ids are different (with and without d0 at the end).
What is your output of iostat -Ensr (this is the relevant command)
 

unhappy_mage

[H]ard|DCer of the Month - October 2005
Joined
Jun 29, 2004
Messages
11,455
I'm having problems with write speeds not going further than 40MB/sec on sequencial writtes and reads about 700KB second, really weird...

Check into Nagle's algorithm, specifically disabling it.
Solaris
Windows 7
Windows in general

If that doesn't make a difference, it may be instructive to capture network traffic from either the server or the client with Wireshark or tcpdump and post it.
 

mephistopoa

Weaksauce
Joined
Dec 11, 2010
Messages
122
I've installed nexenta 3.1.3 to check how things would go. I can see all the disks just fine, create pools and everything else. So far I'm having problems with NFS only, the windows hosts can't map it, and when one of them can I can't even start a file transfer. I've tested FTP and I can get sequential writes of 40MB to whatever amount of disks I have. I tested the machine using windows 2008 r2 and I get 100 mb/sec sequencial write and 110 reads with write cache enabled, with it disabled I get 70MB/sec write speed. Using the disks in software RAID0 escalated the speed proportionaly to the amount of disks I have, so with 6 I managed to get 600MB write speeds.

Definetely is not the controller, definetely is not the host. There must be something else in this equation. I'm so annoyed that so far I could not make any ZFS solution work according to what it can do, speed and large caching. Not even simple file transfers work properly between a workstation and nexenta using NFS, also their support so far is non-existent.
 

Stanza33

Gawd
Joined
Mar 31, 2010
Messages
538
Give me the full spec of your ML350 G5

I have one here I can directly compare with.

Mine is

Dual quad core 5345's
20g ram

These I feel are NOT good Solaris boxes I am afraid ....
You will see massive pauses here and there with them.... Mines been frustrating me for a while.

When it works it works good, still havn't figured ou the pauses .....

Here is what I have found tho.

To get the best out of it, you need to have drives attached to the onboard e200 pig of a smart array controller

Seems the smart array gets confused without drives attached, I am guessing using the Solaris drivers.

Mine is configured as a test machine, and for fiddling with all manner of things....

So, try the following

Attach smartarray port one to top 4 drive bays
Pop in 2x72gb drives in the top two bays....press f8 at boot and configure the drives as a mirror pair...
The boot a smart start cd I used v9.20 and configure controller like this

100% read cache
Then enable drive write cache
The only way to get some performance out of the shit e200 controller.
Top two drives mirrored will be your OS installation.

Then hook up your h200 to the bottom 4 drive bays (so you can only play with 4x72gig drives... But it should be a start for your testing) don't install them yet tho...

If you have a spare intel LAN card pop that in and cable it up. Solaris doesn't seem to correctly play with onboard LAN ports. ILO port doesn't work... 2nd port (Broadcom) I recon identifies wrongly.... And doesn't work correctly

Install OS onto mirrored 72gb drives

Shutdown, install 4 other drives
Reboot

Install napp-it If you like and setup your 4 drives in a pool.

Make a share

Copy stuff from windows to the share and report back the copy speed

Those drives are slow tho.... Eg 50mb/s each.... But the IOPS are quite high

.
 
Top