OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

_Gea · Dec 3, 2014

I would not expect the IBM 1015 to be the problem

I would check:
- disable sync on your NFS datastore filesystem
(ESXi requests slow but safe sync write that can lower performance extremly without a fast dedicated ZIL logdevice)

What I would do
- use/buy any 30 GB+ disk or SSD as bootdisk for ESXi and OmniOS

- create a raid-Z2 from your 4 TB disks (Z1 + hotspare is not good idea)
optionally think of a 6 disk Z2 as it offers the best raw capacity/ usable capacity ratio

- use this pool for general fileserving and backups
If you need to put VMs on it, either disable sync (danger of corrupted ESXi filesystems on a powerloss)
or add a dedicated ZIL like a Intel S3700-100 GB

or better:
- use your 120 GB SSDs as a mirror for a second pool.
Put your NFS datastore on it. Should be large enough for some VMs
Do not fill the SSD over 80%, optionally do a secure erase and create a 20 GB host protected area.
The SSD reports then 100 GB capacity. The hidden capacity helps the firmware to keep write performance high under load.

about RAM
Give OmniOS RAM that is used as storage read cache
With 64 GB RAM and 4 VMs I would assign 16-32 GB

about vnic
e1000 is slower than VMXnet3
I would add a second vmxnet3 vnic and do some performance /stability tests.
If its stable, prefer vmxnet3

Optionally:
There are some tuning options regarding network, ip and NFS settings
that you can try if you want more..

check last threads here or last threads at
https://forums.servethehome.com/index.php?forums/solaris-nexenta-openindiana-and-napp-it.26/

_Gea · Dec 3, 2014

J-san said:
Looks like we should test with at least 1000MB (1GB) testsize in CrystalDiskMark to avoid SSD
disk controller possibly write caching all of the testdata and giving inflated throughput results..

You have always the problem that any sort of read or write caching affects benchmarks.
You can increase the filesizes to 2 x RAM to limit cache effects on reads.

When you set sync=always, you need to commit every block on your ZIL (without a dedicated ZIL the ZIL is on pool) .
This is done additionally to your regular writes via RAM cache but for writes the RAM is not that relevant compared to reads.

Additional problem with many SSDs
When new, they start with very high iops values but after some time of load this can go down to a fraction.
You can manually overprovision a new SSD to lower this problem (just like enterprise SSDs do per default,
example a 100 GB Intel is in reality a 128GB disk with 28GB overprovisioning)

Biggest problem: Benchmarks with large filesizes last forever. If you do not need absolute values but only a hint of
what is better or a estimation about the effect of a setting, you can use smaller filesizes as you look at the differences
nd do not the perfect absolute values.

OS or network tuning can help like any sort of tunings in some cases.
https://docs.oracle.com/cd/E26505_01/html/E37386/chapterzfs-6.html

I mostly avoid OS or ZFS tunings as long as a default config is fast enough and prefer the next level of "bigger" hardware
as long as I do not need to reach a extraordinary high perfomance level where this is not enough.

J-san · Dec 3, 2014

docjay said:
Hi Gea,

I was hoping you could give me a suggestion(s) for the best way to setup my disks?
<snip>
Controller: IBM 1015 (flashed to IT mode with P20 firmware) - thinking of getting a 9211-8i instead
<snip>
Currently, the raid is super slow as I was using it as a NFS datastore to install my first VM. It took over two hours to finish setting up Win7. <snip>

Hi Docjay, one thing I would suggest is to downgrade to IT mode P19 firmware on your IBM 1015.
I had an extremely high number of transport controller errors with my LSI 9211-8i cards with IT mode P20 firmware.

You can check if you have errors relating to this in your OmniOS now before attempting to downgrade the firmware:

Code:

# iostat -En | grep Transport

c10t50014EE05926E121d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
c14t50014EE25FB90889d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
<snip>
c16t55CD2E404B65494Ed0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

If you have many Transport Errors or other hard errors it is quite likely to be the LSI 8211-8i P20 firmware causing issues.

If you don't have Transport errors then I would check some of Geas suggestions.
But if you do have many Transport errors, I would try downgrading to P19 from P20 first to rule that out.

Another cause for Transport errors could be a SAS data cable not plugged in all the way or damaged, so you could try re-seating your connections or changing the cable. In my case downgrading the firmware to P19 fixed the problem.

jmk396 · Dec 3, 2014

This isn't directly related to napp-it or ZFS, but hopefully somebody can help me.

I'm trying to share some NFS exports from Solaris (OmniOS) to Linux and I'm confused how to setup the UIDs and GIDs.

Linux creates new groups for every user but Solaris seems to have one main "staff" (gid 10) group. What's the best practice to resolve this so all my UIDs and GIDs map correctly? (is there an NFS client option to squash all GIDs to 10 or something?)

_Gea · Dec 3, 2014

Every user that you create within napp-it is in groups staff per default:
That is correct but that is a napp-it default not a Solaris default

When you create a user on Solaris (either CLI or napp-it)
you can override this with a dedicated uid/gid

But the main problem remains
- every client OS handles uid/gid over NFS different
- there is no real user or client authentication with NFS 3
all is done on a "good will base" based on client ip - you can always fake any ip/uid/gid

I use NFS3 with ESXi but with the following restrictions
- only allowed on a secure network
- open everything to everyone (everyone@=modify, you cannot really restrict)
- optionally set aclmode to restricted or discard to avoid unwanted chmods via NFS

jmk396 · Dec 3, 2014

Thanks ... My network is only a small home network so I'm not too concerned about security and everything being opened up. I'm only using several different user accounts so I can keep track of which application(s) are creating or modifying files.

Is this a good situation for NFS?

I've synchronized UIDs across the ZFS/NFS server and my Linux servers, but I'm not sure how to setup the GIDs. I'm wondering if I should create a new group on the ZFS/NFS server called "Local Users" or something, and then assign that as the GID to each user on all my Linux boxes to keep everything synchronized?

My other option is to use all_squash on ZFS/NFS and then just use a single UID/GID for everything ...

_Gea · Dec 3, 2014

my solution is
- accept that NFS3 is extremely fast but completely unsecure by design
- use it as simple as possible (open everything)

Audio-Catalyst · Dec 3, 2014

Found the issue, but no way of fixing it.
The remark of another member got me thinking, what did i install that had 64 B....

went back to a snap and started step by step repeating what i did untill i got my error.

Installation of deludge from the blackdot repo did the trick...

skipping that for now

docjay · Dec 3, 2014

J-san,

thanks, I think I'll be trying P19. Here is my output:

Code:

root@OMNIOS:/root# iostat -En | grep Transport
c2t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
c1t0d0           Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
c3t5000CCA24CC7FDF8d0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 0
c3t5000CCA23DF648C3d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
c3t5000CCA24CC9612Ad0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 0
c3t5000CCA24CCC15DAd0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

J-san · Dec 3, 2014

docjay said:

J-san,

thanks, I think I'll be trying P19. Here is my output:

Code:

root@OMNIOS:/root# iostat -En | grep Transport
c3t5000CCA24CC7FDF8d0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 0
c3t5000CCA23DF648C3d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0
c3t5000CCA24CC9612Ad0 Soft Errors: 0 Hard Errors: 1 Transport Errors: 0
c3t5000CCA24CCC15DAd0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

Docjay, when I built my server and I flashed to the latest available firmware for the 9211-8i cards (P20) I always encountered those errors upon benchmarking in Napp-it.

Try running the bonnie++ benchmarks, and fileserver.s and see if your errors keep increasing.

It was much more severe on my SSD pool versus a pool of slower 4 WD RE4 7200 sata drives. After local benchmarking via napp-it benchmarks there were about 3 or 4 transport or hard errors on the sata drives. If you aren't stressing the disks in the pool (ie with slow NFS performance due to lack of separate ZIL) then you may not see any or only 1 like you saw.

Since I downgraded to the P19 Firmware I don't think I've seen any. Good luck!

docjay · Dec 4, 2014

_Gea said:
I would not expect the IBM 1015 to be the problem

I would check:
- disable sync on your NFS datastore filesystem
(ESXi requests slow but safe sync write that can lower performance extremly without a fast dedicated ZIL logdevice)

What I would do
- use/buy any 30 GB+ disk or SSD as bootdisk for ESXi and OmniOS

- create a raid-Z2 from your 4 TB disks (Z1 + hotspare is not good idea)
optionally think of a 6 disk Z2 as it offers the best raw capacity/ usable capacity ratio

- use this pool for general fileserving and backups
If you need to put VMs on it, either disable sync (danger of corrupted ESXi filesystems on a powerloss)
or add a dedicated ZIL like a Intel S3700-100 GB

or better:
- use your 120 GB SSDs as a mirror for a second pool.
Put your NFS datastore on it. Should be large enough for some VMs
Do not fill the SSD over 80%, optionally do a secure erase and create a 20 GB host protected area.
The SSD reports then 100 GB capacity. The hidden capacity helps the firmware to keep write performance high under load.

about RAM
Give OmniOS RAM that is used as storage read cache
With 64 GB RAM and 4 VMs I would assign 16-32 GB

about vnic
e1000 is slower than VMXnet3
I would add a second vmxnet3 vnic and do some performance /stability tests.
If its stable, prefer vmxnet3

Optionally:
There are some tuning options regarding network, ip and NFS settings
that you can try if you want more..

check last threads here or last threads at
https://forums.servethehome.com/index.php?forums/solaris-nexenta-openindiana-and-napp-it.26/

Gea,

Thank you very much for your input. I think I'll be picking up a couple of 500GB samsung 840 Pro SSDs and mirror them for my VM datastore and use one of my 120GB SSD disks for ESXI/OMNIOS install. If I'm going to run 6 disks for a raid-z2 then I'll have to rethink my 'case' situation as I only have a chenbro SR209, might need to get the SR10769 instead.

Then I'll up the RAM I'm giving to OMNIOS from 8GB to 16GB and go from there. Also try out the VMXnet3 nic again as that didn't help me out initially. Probably due to the P20 FW on my IBM 1015 controller. I'll flash down to P19 as I did have hard errors that iostat reported. ( thanks J-san!)

docjay · Dec 4, 2014

J-san said:
Docjay, when I built my server and I flashed to the latest available firmware for the 9211-8i cards (P20) I always encountered those errors upon benchmarking in Napp-it.

Try running the bonnie++ benchmarks, and fileserver.s and see if your errors keep increasing.

It was much more severe on my SSD pool versus a pool of slower 4 WD RE4 7200 sata drives. After local benchmarking via napp-it benchmarks there were about 3 or 4 transport or hard errors on the sata drives. If you aren't stressing the disks in the pool (ie with slow NFS performance due to lack of separate ZIL) then you may not see any or only 1 like you saw.

Since I downgraded to the P19 Firmware I don't think I've seen any. Good luck!

J-san,

Did you encounter slow performance on your pools that were connected to your LSI before you went back to P19?

Do you mind posting your disk/pool setup that you are currently running? I'm betting you might have already done this, but this thread is over 300 pages.

J-san · Dec 4, 2014

docjay said:
J-san,
Did you encounter slow performance on your pools that were connected to your LSI before you went back to P19?

I couldn't say if the performance was that slow on the SATA disk portion as I had only started benchmarking at that point and wasn't using it intensively.

Here's a benchmark of Striped Mirrors - 4 X 2TB WD Re SATA disks with Bonnie with the different firmware:

Code:

NAME  	 SIZE  	 Bonnie  	 Date(y.m.d)  	 File  	 Seq-Wr-Chr  	 %CPU  	 Seq-Write  	 %CPU  	 Seq-Rewr  	 %CPU  	 Seq-Rd-Chr  	 %CPU  	 Seq-Read  	 %CPU  	 Rnd Seeks  	 %CPU  	 Files  	 Seq-Create  	 Rnd-Create 

Striped Mirrors - 4 X 2TB WD Re
Phase 20 firmware:   	    	    	    	    	    	    	    	    	    	    	    	    	    	    	    	    	    	    	   
 tank  	 3.62T   start  	 2014.11.03  	 32G  	 311 MB/s  	 80  	 309 MB/s  	 27  	 113 MB/s  	 11  	 299 MB/s  	 85  	 434 MB/s  	 15  	 1621.9/s  	 3  	 16  	 +++++/s  	 +++++/s
 tank  	 3.62T 	 start  	 2014.11.03  	 32G  	 308 MB/s  	 81  	 309 MB/s  	 27  	 109 MB/s  	 11  	 292 MB/s  	 85  	 423 MB/s  	 14  	 1552.9/s  	 3  	 16  	 +++++/s  	 +++++/s

Phase 19 firmware: 
 tank  	 3.62T 	 start  	 2014.11.04  	 32G  	 309 MB/s  	 78  	 309 MB/s  	 27  	 110 MB/s  	 11  	 298 MB/s  	 86  	 423 MB/s  	 15  	 1577.3/s  	 2  	 16  	 +++++/s  	 +++++/s  
 tank/wlog 3.62T start  	 2014.11.04  	 32G  	 308 MB/s  	 80  	 308 MB/s  	 27  	 110 MB/s  	 11  	 290 MB/s  	 82  	 420 MB/s  	 14  	 1604.0/s  	 3  	 16  	 +++++/s  	 +++++/s

Code:

Single 480GB Intel S3500 SSD

NAME  	 SIZE  	 Bonnie  	 Date(y.m.d)  	 File  	 Seq-Wr-Chr  	 %CPU  	 Seq-Write  	 %CPU  	 Seq-Rewr  	 %CPU  	 Seq-Rd-Chr  	 %CPU  	 Seq-Read  	 %CPU  	 Rnd Seeks  	 %CPU  	 Files  	 Seq-Create  	 Rnd-Create 
 Phase 20 firmware:
 vm-7d 	 444G  	 start  	 2014.11.03  	 32G  	 428 MB/s  	 98  	 455 MB/s  	 37  	 156 MB/s  	 15  	 293 MB/s  	 84  	 219 MB/s  	 7  	 5006.9/s  	 7  	 16  	 +++++/s  	 +++++/s
 vm-7d 	 444G  	 start  	 2014.11.03  	 32G  	 399 MB/s  	 99  	 402 MB/s  	 30  	 148 MB/s  	 14  	 249 MB/s  	 73  	 166 MB/s  	 5  	 3692.8/s  	 5  	 16  	 +++++/s  	 +++++/s

Phase 19 firmware:
 vm-7d 	 444G  	 start  	 2014.11.04  	 32G  	 396 MB/s  	 99  	 418 MB/s  	 33  	 161 MB/s  	 17  	 323 MB/s  	 94  	 355 MB/s  	 12  	 9714.9/s  	 14  	 16  	 +++++/s  	 +++++/s

docjay said:
Do you mind posting your disk/pool setup that you are currently running? I'm betting you might have already done this, but this thread is over 300 pages.

Here you go..

Tank striped mirror drives are WD RE4 2TB 7200 rpm SATA disks,
with an 100GB Intel S3700 SSD for zil slog.
Speed: with SSD slog: 200MB/s sequential write speed, 47.42 MB/s write @ 4k QD32 with sync=standard.

vm1 mirror drives are 480GB Intel S3500 SSD.
255MB/s sequential write speed, 25.79 MB/s write @ 4k QD32 with sync=always.

Both speeds were from CrystalDiskMark /w 1000MB test size over NFS mounted VMDK in Win2K8R2 VM.
OmniOS has sd.conf modified for Intel SSDs, as the S3500 and S3700 have end-to-end power protection, so we can set cache-nonvolatile:true to enable sync writes go faster for SLOG and SSD pool.
(reasoning behind sd.conf settings: http://www.c0t0d0s0.org/archives/7679-Configuration-hint-when-using-F40F80-cards.html)

Code:

 pool: tank
 state: ONLINE
  scan: resilvered 104K in 0h0m with 0 errors on Fri Nov 28 23:11:46 2014
config:

        NAME                        STATE     READ WRITE CKSUM
        tank                        ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            c10t50014EE05926E121d0  ONLINE       0     0     0
            c9t50014EE25F104CBEd0   ONLINE       0     0     0
          mirror-1                  ONLINE       0     0     0
            c13t50014EE2B50E9766d0  ONLINE       0     0     0
            c14t50014EE25FB90889d0  ONLINE       0     0     0
        logs
          c16t55CD2E404B65494Ed0    ONLINE       0     0     0

errors: No known data errors

  pool: vm1
 state: ONLINE
  scan: resilvered 44.4G in 0h2m with 0 errors on Tue Nov 25 00:22:56 2014
config:

        NAME                        STATE     READ WRITE CKSUM
        vm1                         ONLINE       0     0     0
          mirror-0                  ONLINE       0     0     0
            c12t55CD2E404B58E103d0  ONLINE       0     0     0
            c7t55CD2E404B58B0A7d0   ONLINE       0     0     0

docjay · Dec 5, 2014

Great, thanks for sharing. I was wondering if you were running a disk for the SLOG or not. I need to design mine correctly as I don't want my system to suffer.

omegatotal · Dec 6, 2014

Konkonex said:
Wonder if anyone can assist with an issue I'm experiencing.

I have a HP N54L with 6x 3TB drives and 1 x usb storage drive as the boot.

There are two drives sitting on top of each other at the top of the case where the optical disk drive should go - one connected to on board sata, one connected to ESATA on the rear.

The problem it seems to have is during heavy tasks - the two connected to sata and esata appear to often hit 100% busy/wait whereas all the other disks will be significantly less.

I've tried this with different disks/different tasks/etc.. but I get the same issue.

My thought is that the ESATA and the SATA port somehow share the same path somehow resulting in half speeds on both drives?

Or that it's because I'm using a molex to 2x sata splitter to power both these drives.

Any thoughts on this?

edit - I'm using openindiana - raidz2 - but have used raidz1 and get the same issue with both configurations.

The n54l has the ODD port and esata port artificially limited via a hard coded bios setting to sata1 standard I believe

your best bet is to purchase a decent sata/sas card that is supported by your O.S and stick it in one of the 2 slots.

I would get a higher end 8xSAS card as the bays in the device actually support SAS drives, and you can get break out cables for the other 4 devices.

a popular card in the freenas arena is the LSI based, IBM m1015 using the IT firmware (no hardware raid support)

HP also has cards based on LSI but probably not as cheap as the IBM

jim82 · Dec 7, 2014

Thanks for the hint about the P20 firmware, I had a ton of hard errors and transport errors on my 6x3TB mirror pool on a new build.

Reverted to using P19 and all is well again

jim82 · Dec 7, 2014

Not sure if this is the right forum to post in, but I'll try anyway.

Running my ESXi 5,5 U2(latest build) All-in-one, Gea's concept basically.
All my VM's use VMXNET3 NICs.
One OmniOS VM w/ 32GB RAM, 4vCPU, LSI9211-8i(P19 firmware)

Napp-it runs MTU9000 against the vSwitch, presenting datastores via NFS.

Zpool config:

Code:

  pool: HP_SAS_1TB
 state: ONLINE
  scan: scrub repaired 0 in 1h23m with 0 errors on Sun Dec  7 04:23:53 2014
config:

	NAME                       STATE     READ WRITE CKSUM      CAP            Product /napp-it   IOstat mess
	HP_SAS_1TB                 ONLINE       0     0     0
	  mirror-0                 ONLINE       0     0     0
	    c7t5000C50041A77369d0  ONLINE       0     0     0      1 TB           MM1000FBFVR   S:0 H:0 T:0
	    c8t5000C500415FE571d0  ONLINE       0     0     0      1 TB           MM1000FBFVR   S:0 H:0 T:0

errors: No known data errors

  pool: RedDataPool
 state: ONLINE
  scan: none requested
config:

	NAME                       STATE     READ WRITE CKSUM      CAP            Product /napp-it   IOstat mess
	RedDataPool                ONLINE       0     0     0
	  mirror-0                 ONLINE       0     0     0
	    c6t50014EE2B5781C3Ed0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68E   S:0 H:0 T:0
	    c6t50014EE2B578224Ed0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68E   S:0 H:0 T:0
	    c6t50014EE603195DFAd0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0
	  mirror-1                 ONLINE       0     0     0
	    c6t50014EE603EC0E6Bd0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68E   S:0 H:0 T:0
	    c6t50014EE604FC0106d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68E   S:0 H:0 T:0
	    c6t50014EE658712391d0  ONLINE       0     0     0      3 TB           WDC WD30EFRX-68A   S:0 H:0 T:0

errors: No known data errors

Bonnie test from Napp-IT console:

I'm testing my ZFS pools/disks from the Windows VM and everything seems fine:

BUT, when I try to copy a file from the zfs disk to a direct attached SSD disk, I see very low transfer speeds:

So I tested the SSD, also this seems fine:

I have no idea, why it's behaving like this

My VM's does NOT run Jumbo frames, but my VMKernel and Napp-it VM does run MTU9000.

Can anyone shed some light on this or point me in the correct direction?

Much appreciated
Jim

J-san · Dec 7, 2014

jim82 said:
BUT, when I try to copy a file from the zfs disk to a direct attached SSD disk, I see very low transfer speeds:

I have no idea, why it's behaving like this

My VM's does NOT run Jumbo frames, but my VMKernel and Napp-it VM does run MTU9000.

Can anyone shed some light on this or point me in the correct direction?

Much appreciated
Jim

How is the direct attached SSD disk connected to the VM?
Are you presenting a VMDK disk to the VM?
(Eg the ssd is formatted as a VMFS datastore?)

Is the VMDK passed in as a SCSI disk on the same virtual scsi controller?
(Maybe its attached as an IDE disk by accident?)

When you benchmarked the SSD is it connected via the same cable as when you are having the slow copy?
Eg SATA cable from MB, or SATA connection via LSI hba card?

One last thing, is that SSD direct attached to the ESXi all-in-one server?

Or are you talking about copying a file across a physical network switch from the ESXi server to a separate Windows box (eg over 1 GB ethernet)?
(if this is the setup, maybe you have a slower realtek network card in the Windows box limiting the network speed.
I've seen about 45MB/s 1GB network copy over a slower NIC.)

Pyrran · Dec 8, 2014

J-san said:
RE: Which version of LSI 9211-i8 IT-mode firmware should I flash? Should I flash the latest P20?

<snip>

So if you have a LSI 9211-8i hba I would stick with P19.
There seems to be some error with P20 and these cards.

(Also, someone on IRC also mentioned 9201-16e having problems with P20)

Hope this saves someone else some time.

Note: I also had a striped mirror of 2TB WD Re drives that also displayed some hard + transport errors with P20, but much less than the SSDs. (only 2 or 4 per WDRe drive after a benchmark run)
So it seems the faster speed of the SSD was triggering the problem more frequently on P20.

Thought I'd post to say thanks for your efforts finding the solution to this problem. Spent two days pulling my hair out, after building a new storage system and taking the opportunity of "upgrading/updating" firmwares etc, and then moving from FreeNAS to nas4free to OmniOS/napp-it - all giving variations in reporting on the same error(s). Just wish I'd found your life-saving advice before I'd dropped the cash for another raid/hba card - still it's good to have a spare I guess

Downgraded from P20 to P19 and no a single error since. Thanks again.

jim82 · Dec 8, 2014

J-san said:
How is the direct attached SSD disk connected to the VM?
Are you presenting a VMDK disk to the VM?
(Eg the ssd is formatted as a VMFS datastore?)

Is the VMDK passed in as a SCSI disk on the same virtual scsi controller?
(Maybe its attached as an IDE disk by accident?)

When you benchmarked the SSD is it connected via the same cable as when you are having the slow copy?
Eg SATA cable from MB, or SATA connection via LSI hba card?

One last thing, is that SSD direct attached to the ESXi all-in-one server?

Or are you talking about copying a file across a physical network switch from the ESXi server to a separate Windows box (eg over 1 GB ethernet)?
(if this is the setup, maybe you have a slower realtek network card in the Windows box limiting the network speed.
I've seen about 45MB/s 1GB network copy over a slower NIC.)

Hi J-San,

Thanks for taking time to reply

Direct attached SSD is a VM datastore attached to the VM as SCSI 0:2.

SSD is connected directly to the motherboard, with the same cable. Never touched it.

I'm talking about internal VM to VM copy, from the ZFS datastore to the SSD datastore using a Win2012 R2 client.

BR Jim

J-san · Dec 8, 2014

Jim: As your VM -> datastore connection is good speed and you haven't changed the physical connection I would suspect it's something about the network setup.

You didn't say what type of copy you are doing, probably a SMB share from ZFS pool to the Window's client's local SSD datastore vmdk disk?

Have you got "atime=on" (enabled) on the ZFS datastore that you are copying the file from? If so you might want to disable it.

Check (mine is off for VM storage):

Code:

# zfs get atime tank/dstore/vm
NAME            PROPERTY  VALUE  SOURCE
tank/dstore/vm  atime     off    inherited from tank

If it's enabled then on each access the file's last "accessed" time has to be updated to the current date/time stamp and flushed to disk.

Hopefully disabling atime fixes it for you.

jim82 · Dec 8, 2014

J-san said:
Jim: As your VM -> datastore connection is good speed and you haven't changed the physical connection I would suspect it's something about the network setup.

You didn't say what type of copy you are doing, probably a SMB share from ZFS pool to the Window's client's local SSD datastore vmdk disk?

Have you got "atime=on" (enabled) on the ZFS datastore that you are copying the file from? If so you might want to disable it.

Check (mine is off for VM storage):

Code:

# zfs get atime tank/dstore/vm NAME PROPERTY VALUE SOURCE tank/dstore/vm atime off inherited from tank

If it's enabled then on each access the file's last "accessed" time has to be updated to the current date/time stamp and flushed to disk.

Hopefully disabling atime fixes it for you.

Hi J-San,

Thanks for your reply.

Unfortunatly disabling ATIME didn't help anything. I was actually enabled on my pool, so thanks for the tip.

I'm copying from a volume that resides on the NFS datastore being presented by OmniOS, to the SSD. I'm copying a 5GB file. See below:

Copy from->to:

Copy status:

Datastores:

Storage Network 1:

Storage Network 2:

Storage Network 3:

Thanks for your assistance.
BR Jim

J-san · Dec 8, 2014

Hi Jim,

How is the D: drive that you are copying FROM mounted in Windows?
(Is it a SMB share mounted to a drive letter?)

What file (or filetype) are you copying?
Is the file in use? (eg a VMDK and the VM it is attached to is still powered on)

jim82 · Dec 8, 2014

J-san said:
Hi Jim,

How is the D: drive that you are copying FROM mounted in Windows?
(Is it a SMB share mounted to a drive letter?)

What file (or filetype) are you copying?
Is the file in use? (eg a VMDK and the VM it is attached to is still powered on)

The D-drive is a harddrive(VMDK) file mounted in ESXi, just like regular ESXi drives.

I'm trying to copy an MKV file about 5GB in size.

See below, green arrow is SSD and red arrow is ZFS datastore via NFS:

BR Jim

jim82 · Dec 8, 2014

.........performing the EXACT same operation in Ubuntu, yielded a much better result

J-san · Dec 8, 2014

Jim - It looks like you're using the same virtual SCSI controller 0 for all your disks:

LSI Logic SAS

I would try powering off the VM then:

- Removing the SSD hard disk (green arrow VMDK) and clicking OK to save.
- Re-adding the Harddisk on the SSD VMDK.
- Choosing SCSI 2:0 as the address (which will add an additional Scsi controller)
- Before saving the change to the VM change the new controller type to "Paravirtual".

Not sure if that will do it, but it's worth a try.
(you have VMware tools installed in the Windows VM right?)

Actually is that green arrow your boot VMDK disk?
(C: and D: are different partitions on the same VMDK disk?)

I think before Paravirtual scsi controllers had a problem booting but I think it's many versions of ESXi ago.

jim82 · Dec 9, 2014

J-san said:
Jim - It looks like you're using the same virtual SCSI controller 0 for all your disks:

LSI Logic SAS

I would try powering off the VM then:

- Removing the SSD hard disk (green arrow VMDK) and clicking OK to save.
- Re-adding the Harddisk on the SSD VMDK.
- Choosing SCSI 2:0 as the address (which will add an additional Scsi controller)
- Before saving the change to the VM change the new controller type to "Paravirtual".

Not sure if that will do it, but it's worth a try.
(you have VMware tools installed in the Windows VM right?)

Actually is that green arrow your boot VMDK disk?
(C: and D: are different partitions on the same VMDK disk?)

I think before Paravirtual scsi controllers had a problem booting but I think it's many versions of ESXi ago.

Hi J-san,

Thanks for your help.

I tried adding the disk as SCSI 2:0 - paravirtual. Same results, unfortunately.

Each disk is it's own VMDK. I have not partitioned the drives added from ESXi. VMWare tools is installed and updated to latest version.

So weird that the Ubuntu runs @ ~200MiB/s, when Windows is about a quarter of that speed

Do you see any issues with running MTU 9000 on the Storage network only? In my optics, this is a seperated network, where the actual VM traffic is not running.

BR Jim

J-san · Dec 9, 2014

Hi Jim,

I'm running with 9000 mtu on my separate virtual storage network. And left the 1500 MTU for the regular VM network traffic. Network performance should be fine as the VM <-> VMDK disk traffic over NFS should only be on the storage network @ 9000 mtu. Physical direct attached SSD wouldn't even go over any network..

The only last thing I would check is:
- Is the VM hardware level at 8 (created in ESXi 5.5) or is it lower. (doubt it would be this)
- Check the Write cache settting in Windows for the disk.
eg. Is the Windows disk driver setting set to write cache disabled? (forcing sync)

1. Right-click on Hard Disk (SSD) you are copying to
2. Properties -> Hardware
3. The right hard disk should be selected, click properties for it.
4. Click "Policies" tab

What is the Write caching setting set for?
Quick Removal (no write caching), or Optimize for Performance, there's also "Enable write caching on the disk".

Failing that.. if it's an old VM, maybe re-install a new version of Windows and benchmark on that?

jim82 · Dec 9, 2014

J-san said:
Hi Jim,

I'm running with 9000 mtu on my separate virtual storage network. And left the 1500 MTU for the regular VM network traffic. Network performance should be fine as the VM <-> VMDK disk traffic over NFS should only be on the storage network @ 9000 mtu. Physical direct attached SSD wouldn't even go over any network..

The only last thing I would check is:
- Is the VM hardware level at 8 (created in ESXi 5.5) or is it lower. (doubt it would be this)
- Check the Write cache settting in Windows for the disk.
eg. Is the Windows disk driver setting set to write cache disabled? (forcing sync)

1. Right-click on Hard Disk (SSD) you are copying to
2. Properties -> Hardware
3. The right hard disk should be selected, click properties for it.
4. Click "Policies" tab

What is the Write caching setting set for?
Quick Removal (no write caching), or Optimize for Performance, there's also "Enable write caching on the disk".

Failing that.. if it's an old VM, maybe re-install a new version of Windows and benchmark on that?

Hi J-san,

Thanks for your help. I think I've narrowed it down to actual disk/zfs performance.

Even though CrystalDiskMark displays wonderful results, I came to think about that it creates a 1GB file which ZFS then hold in ARC. This effectively means I'm running disk test from the ARC cache.

I tested it like this:

On VM-1 I copied a file from a disk(vmdk located on vsan) - speeds were as you see in my earlier posts(40-50MiB/s). I let the file copy complete.

On VM-2 I copied the same file, which resulted in speeds of around 1GiB/s

On my normal desktop PC I copied that same file, which maxed out my 1Gbit connection.

Running the HD Tune benchmarks shows 40-60MiB/s on the VMDK connected to vsan.

It makes sense, besides the Bonnie++ benchmarks done from the Napp-it interface. They show speeds of around 1GiB/s also.

Now I'm confused even more

Something tells me, that I only get ultra fast speeds when the item I'm copying or working with is located in ARC cache(RAM).

I also noticed that Napp-it network monitor never goes higher than 6-700Mbit/s, when I'm working with random data(searching for a file or performing any data READ related tasks) This matches the throughput I'm seeing to much, to be just a coincidence.

Any opinions?

BR Jim

J-san · Dec 9, 2014

Hi Jim,

There's some good information here on speed of vdevs in mirrored mode and raidz mode:
http://constantin.glez.de/blog/2010/06/closer-look-zfs-vdevs-and-performance

It also depends upon the disks chosen (and their performance).

But mirrored vdevs instead of raid-z should according to the article give you better performance especially if it's not stored sequentially in a vmdk.

If you don't have a lot on your system yet, you could backup to your single SSD datastore and then destroy your data pool and re-create as a mirrored zpool instead of raidz.

You will lose a good chunk of disk space going this route.

Eg 4 x 2 TB disks in mirrored vdev will only give you roughly 3.7TB storage space.

But each drive in the mirror can supply read bandwidth, and position disk heads independently for seeking..

But if as you said you don't need more performance you will keep a lot more disk space in your Raid-z zpool setup.

J-san · Dec 11, 2014

J-san said:
Comparing iscsi 10G pool w/ ZeusRAM slog:
(from Napp-it benchmarks pdf)

Code:

sync, ZeusRAM, DRAM 8 GB, iSCSI CrystalDiskMark 3.0.2 x64 ------------------------------------------------------- Seq 406.6 [R] 169.2 [W] MB/s 512k 347.8 [R] 131.6 [W] MB/s 4k 12.26 [R] 12.99 [W] MB/s 4k QD32 178.1 [R] 113.8 [W] MB/s -------------------------------------------------------

To 100GB S3700 $225:

Code:

OmniOS VM - [B]16GB Ram[/B] sync=always, 100 GB S3700, NFS thin prov vmdk NTFS 4k sectors Win2008R2 - LSI SCSI driver to VMDK. sd.conf values: "physical-block-size:4096, cache-nonvolatile:true, throttle-max:32, disksort:false" CrystalDiskMark 3.0.3 x64 - 100 MB test size ------------------------------------------------------- Seq 981.6 [R] 184.9 [W] MB/s 512k 347.8 [R] 186.7 [W] MB/s 4k 44.78 [R] 18.52 [W] MB/s 4k QD32 230.1 [R] 143.0 [W] MB/s -------------------------------------------------------

(pool backed by 4 WD RE4 2TB 7200RPM Drives, in Raid 10 configuration - striped mirrors)
100 MB test size

It looks like the 200 GB version of the S3700 can do around 365 MB/s sequential writes.
The 400GB version can do 460 MB/s sequential writes.

If those drives could do 77% of the sequential speeds (as mine did above) in 4k QD32 those would be getting seriously fast numbers!.

Code:

sync=always, 100 GB S3700, NFS thin prov vmdk NTFS 4k sectors, Win2008R2 - Paravirtual SCSI driver to VMDK. sd.conf values: "physical-block-size:4096, cache-nonvolatile:true, throttle-max:32, disksort:false" CrystalDiskMark 3.0.3 x64 - 1 GB test size ------------------------------------------------------- Seq 906.3 [R] 182.1 [W] MB/s 512k 598.3 [R] 183.7 [W] MB/s 4k 31.27 [R] 16.86 [W] MB/s 4k QD32 256.5 [R] 47.43 [W] MB/s -------------------------------------------------------

(pool backed by 4 WD RE4 2TB 7200RPM Drives, in Raid 10 configuration - striped mirrors)
1 GB test size

I just discovered today, that this much slower 4k@QD32 benchmark with 1GB was caused by the ZFS write throttle engaging when the underlying 7200rpm vdevs RE4 sata disks couldn't keep up to the SSD.

Benchmarking with a larger zfs_dirty_data_max doesn't trip the write throttle, or using a 500MB test.

Looks like I need another WD RE4 sata vdev to keep up with the sustained async transaction group writes coming in (after they are written sync by the SSD zil slog), as the data is too fast for the 2 RE4 sata vdevs and so they are building up in ram and causing the write throttle to engage.

Benchmarking with 500MB size, or a larger zfs_dirty_data_max gave me:

Code:

OmniOS VM memory at 14GB
CrystalDiskMark 3.0.3 x64
Test size: 1000MB
setting zfs_dirty_data_max=2495MB

[B]127 MB/s 4K@QD32 write speed[/B]

(yes 127MB/s that's not a typo, with sync=standard)

See the following post I created about the cause:
https://forums.servethehome.com/ind...hrottle-tuning-nfs-benchmarking-4k-qd32.4331/

So even the slowest 100G S37000 is a very solid performer for zfs slog.

Audio-Catalyst · Dec 11, 2014

Guys,
been having a odd issue with my Omnios box and c3750
the omnios box has 6 interfaces ( 2 e1000g and 4 igb)
somehow the LACP bonding between the igb's and my cisco's will not stay in sync, while the same lacp config runs fine over the e1000's :

port configs :

interface Port-channel6
!
interface Port-channel7
!
interface GigabitEthernet1/0/10
description PSD01-IGB0
switchport mode access
duplex full
channel-group 6 mode active
spanning-tree link-type point-to-point
!
interface GigabitEthernet1/0/12
description PSD01-IGB1
switchport mode access
duplex full
channel-group 6 mode active
spanning-tree link-type point-to-point
!
interface GigabitEthernet2/0/10
description PSD01-IGB2
switchport mode access
duplex full
channel-group 7 mode active
spanning-tree link-type point-to-point
!
interface GigabitEthernet2/0/11
description PSD02-IGB3
switchport mode access
duplex full
channel-group 7 mode active
spanning-tree portfast
spanning-tree link-type point-to-point
!

channel 5 = e1000g's channel 6 and 7 are the igb's :

Channel group 5 neighbors
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi1/0/14 FA 4096 0030.48d5.ec94 23s 0x0 0x3E8 0x2 0x3F
Gi2/0/14 FA 4096 0030.48d5.ec94 23s 0x0 0x3E8 0x1 0x3F

Channel group 6 neighbors
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi1/0/10 FA 4096 a036.9f02.c13a 0s 0x0 0x3EA 0x6 0x47
Gi1/0/12 FA 4096 a036.9f02.c13a 0s 0x0 0x3EA 0x5 0x47

Channel group 7 neighbors
Partner's information:
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Gi2/0/10 FA 4096 a036.9f02.c138 0s 0x0 0x3E9 0x4 0x47
Gi2/0/11 FA 4096 a036.9f02.c138 0s 0x0 0x3E9 0x3 0x47

ipadm :

LINK POLICY ADDRPOLICY LACPACTIVITY LACPTIMER FLAGS
aggr0 L4 auto active short -----
aggr1 L4 auto active short -----
aggr2 L4 auto active short -----
aggr0 -- up 1000Mb full static x.x.x.x 255.255.255.0 x:x:x:x:x:x 1500
e1000g0 attached up 1000Mb full - - - x:x:x:x:x:x 1500
e1000g1 attached up 1000Mb full - - - x:x:x:x:x:x 1500
aggr1 -- up 1000Mb full static x.x.x.x 255.255.255.0 x:x:x:x:x:x 1500
igb0 attached up 1000Mb full - - - x:x:x:x:x:x 1500
igb1 attached up 1000Mb full - - - x:x:x:x:x:x 1500
aggr2 -- up 1000Mb full static x.x.x.x 255.255.255.0 x:x:x:x:x:x 1500
igb2 attached up 1000Mb full - - - x:x:x:x:x:x 1500
igb3 attached up 1000Mb full - - - x:x:x:x:x:x 1500

Log from the Cisco :

058673: Dec 8 22:15:51.593: lacp_mux Gi2/0/10 - mux: during state WAITING, got event 1(selected)
058674: Dec 8 22:15:51.593: @@@ lacp_mux Gi2/0/10 - mux: WAITING -> WAITING
058675: Dec 8 22:15:51.593: LACP: Gi2/0/10 lacp_action_mx_waiting entered
058676: Dec 8 22:15:51.593: LACP: timer lacp_w(Gi2/0/10) started with interval 2000.
058677: Dec 8 22:15:51.895: LACP :lacp_bugpak: Receive LACP-PDU packet via Gi2/0/10
058678: Dec 8 22:15:51.895: LACP : packet size: 124
058679: Dec 8 22:15:51.895: LACP: pdu: subtype: 1, version: 1
058680: Dec 8 22:15:51.895: LACP: Act: tlv:1, tlv-len:20, key:0x3E9, p-pri:0x1000, p:0x4, p-state:0x47,
s-pri:0x1000, s-mac:a036.9f02.c138
058681: Dec 8 22:15:51.895: LACP: Part: tlv:2, tlv-len:20, key:0x0, p-pri:0x0, p:0x0, p-state:0xA,
s-pri:0x0, s-mac:0000.0000.0000
058682: Dec 8 22:15:51.895: LACP: col-tlv:3, col-tlv-len:16, col-max-d:0xA
058683: Dec 8 22:15:51.895: LACP: term-tlv:0 termr-tlv-len:0
058684: Dec 8 22:15:51.895: LACP: Gi2/0/10 LACP packet received, processing
058685: Dec 8 22:15:51.895: lacp_rx Gi2/0/10 - rx: during state CURRENT, got event 5(recv_lacpdu)
058686: Dec 8 22:15:51.895: @@@ lacp_rx Gi2/0/10 - rx: CURRENT -> CURRENT
058687: Dec 8 22:15:51.895: LACP: Gi2/0/10 lacp_action_rx_current entered
058688: Dec 8 22:15:51.895: LACP: Gi2/0/10 set to UNSELECTED
058689: Dec 8 22:15:51.895: lacp_mux Gi2/0/10 - mux: during state WAITING, got event 3(unselected)
058690: Dec 8 22:15:51.895: @@@ lacp_mux Gi2/0/10 - mux: WAITING -> DETACHED
058691: Dec 8 22:15:51.895: LACP: Gi2/0/10 lacp_action_mx_detached entered
058692: Dec 8 22:15:51.895: LACP: Gi2/0/10 Resetting hw_sw constraints
058693: Dec 8 22:15:51.895: LACP: lacp_insert_partner_cd_inhibitor: didn't change sync flag.
058694: Dec 8 22:15:51.895: LACP: lacp_send_lacpdu: (Gi2/0/10) About to send the 110 LACPDU
058695: Dec 8 22:15:51.895: LACP :lacp_bugpak: Send LACP-PDU packet via Gi2/0/10
058696: Dec 8 22:15:51.895: LACP : packet size: 124
058697: Dec 8 22:15:51.895: LACP: pdu: subtype: 1, version: 1
058698: Dec 8 22:15:51.895: LACP: Act: tlv:1, tlv-len:20, key:0x7, p-pri:0x8000, p:0x20B, p-state:0x5,
s-pri:0x8000, s-mac:0019.aa89.b580
058699: Dec 8 22:15:51.895: LACP: Part: tlv:2, tlv-len:20, key:0x3E9, p-pri:0x1000, p:0x4, p-state:0x47,
s-pri:0x1000, s-mac:a036.9f02.c138
058700: Dec 8 22:15:51.895: LACP: col-tlv:3, col-tlv-len:16, col-max-d:0x8000
058701: Dec 8 22:15:51.895: LACP: term-tlv:0 termr-tlv-len:0
058702: Dec 8 22:15:51.895: LACP: lacp_write: LACP 124 bytes out Gi2/0/10
058703: Dec 8 22:15:51.895: lacp_handle_standby_port_internal called, depth = 1
058704: Dec 8 22:15:51.895: LACP: recordPDU Gi2/0/10 LACP PDU Rcvd. Partners oper state is hex 47
058705: Dec 8 22:15:51.895: LACP: recordPDU Gi2/0/10 Partner out of sync
058706: Dec 8 22:15:51.895: LACP: Gi2/0/10 Partners oper state is hex 47
058707: Dec 8 22:15:51.895: LACP: timer lacp_c_l(Gi2/0/10) started with interval 90000.
058708: Dec 8 22:15:51.895: LACP: Gi2/0/10 LAG_PARTNER_UP.
058709: Dec 8 22:15:51.895: LACP: Gi2/0/10 LAG unchanged
058710: Dec 8 22:15:51.903: LACP: Gi2/0/10 STANDBY aggregator hex address is 5FB71A4
058711: Dec 8 22:15:51.903: LACP: Gi2/0/10 set to STANDBY
058712: Dec 8 22:15:51.903: lacp_mux Gi2/0/10 - mux: during state DETACHED, got event 2(standby)
058713: Dec 8 22:15:51.903: @@@ lacp_mux Gi2/0/10 - mux: DETACHED -> WAITING
058714: Dec 8 22:15:51.903: LACP: Gi2/0/10 lacp_action_mx_waiting entered
058715: Dec 8 22:15:51.903: @@@ lacp_mux Gi2/0/10 - mux: WAITING -> WAITING
058716: Dec 8 22:15:51.903: lacp_mux Gi2/0/10 - mux: during state WAITING, got event 6(outof_sync) (ignored)
058717: Dec 8 22:15:51.903: LACP :lacp_bugpak: Receive LACP-PDU packet via Gi2/0/11
058718: Dec 8 22:15:51.903: LACP : packet size: 124
058719: Dec 8 22:15:51.903: LACP: pdu: subtype: 1, version: 1
058720: Dec 8 22:15:51.903: LACP: Act: tlv:1, tlv-len:20, key:0x3E9, p-pri:0x1000, p:0x3, p-state:0x47,
s-pri:0x1000, s-mac:a036.9f02.c138
058721: Dec 8 22:15:51.903: LACP: Part: tlv:2, tlv-len:20, key:0x0, p-pri:0x0, p:0x0, p-state:0xA,
s-pri:0x0, s-mac:0000.0000.0000
058722: Dec 8 22:15:51.903: LACP: col-tlv:3, col-tlv-len:16, col-max-d:0xA
058723: Dec 8 22:15:51.903: LACP: term-tlv:0 termr-tlv-len:0
058724: Dec 8 22:15:51.903: LACP: Gi2/0/11 LACP packet received, processing
058725: Dec 8 22:15:51.903: lacp_rx Gi2/0/11 - rx: during state CURRENT, got event 5(recv_lacpdu)
058726: Dec 8 22:15:51.903: @@@ lacp_rx Gi2/0/11 - rx: CURRENT -> CURRENT
058727: Dec 8 22:15:51.903: LACP: Gi2/0/11 lacp_action_rx_current entered
058728: Dec 8 22:15:51.903: LACP: Gi2/0/11 set to UNSELECTED
058729: Dec 8 22:15:51.903: lacp_mux Gi2/0/11 - mux: during state WAITING, got event 3(unselected)
058730: Dec 8 22:15:51.903: @@@ lacp_mux Gi2/0/11 - mux: WAITING -> DETACHED
058731: Dec 8 22:15:51.903: LACP: Gi2/0/11 lacp_action_mx_detached entered
058732: Dec 8 22:15:51.903: LACP: Gi2/0/11 Resetting hw_sw constraints
058733: Dec 8 22:15:51.903: LACP: lacp_insert_partner_cd_inhibitor: didn't change sync flag.
058734: Dec 8 22:15:51.903: LACP: lacp_send_lacpdu: (Gi2/0/11) About to send the 110 LACPDU
058735: Dec 8 22:15:51.903: LACP :lacp_bugpak: Send LACP-PDU packet via Gi2/0/11
058736: Dec 8 22:15:51.903: LACP : packet size: 124
058737: Dec 8 22:15:51.903: LACP: pdu: subtype: 1, version: 1
058738: Dec 8 22:15:51.903: LACP: Act: tlv:1, tlv-len:20, key:0x7, p-pri:0x8000, p:0x20C, p-state:0x5,
s-pri:0x8000, s-mac:0019.aa89.b580
058739: Dec 8 22:15:51.903: LACP: Part: tlv:2, tlv-len:20, key:0x3E9, p-pri:0x1000, p:0x3, p-state:0x47,
s-pri:0x1000, s-mac:a036.9f02.c138

so in short, i see it comming up, shows out of sync, and down it goes...
have tried short times, long times active , passive... nothing will work

J-san · Dec 11, 2014

Just saw a note in the changelog for VMware 5.5:
VMware ESXi 5.5, Patch ESXi-5.5.0-20141004001-standard (2087364)

This patch resolves:
...

- When you run Solaris 10 or 11 virtual machines using the e1000 adapter, the network throughput within the Solaris guest OS is very poor.

..

Audio-Catalyst · Dec 12, 2014

this just keeps getting stranger by the minute.
after removing the aggr's and setting a static ip on each interface,
i can ping from the host itself, but not through a wire, direct or through a switch.
plugged in a new intel i350, same deal...
when looking at the driver, i see that the is a update date of november 6....

zeroARMY · Dec 14, 2014

Does anyone know why line 16 has a different description from the rest of the drives? There are many more of that same model in the list. Just firmware weirdness or something bad?

moose517 · Dec 14, 2014

Having some issues with napp-it, for some reason when the computer boots all the hard drives show up when the SAS card and expander are booting, but when it gets into solaris suddently none of them are there. I've got a M1015 i believe it is with the intel SAS expander if that is of any help. Whats more odd is if i look at the disks individually through the web GUI it shows them all as connected and online, just the pool doesn't seem to recognize anything, but when i try and do anything with the pool it says that pool I/O is suspended.

_Gea · Dec 15, 2014

I would suppose, one disk is blocking the bus or controller.
What you can do on a hotpluggable server is (to find a single disk problem)
- power off and remove all disks
- power on, boot Solaris
- plug in a disk and wait some seconds until it comes up in menu disks
- do this with all disks. With one disk you may have problems and this is the disk that you need to replace

Unlike hardwareraid, such a procedure is not a problem with ZFS
The pool simply stays offline until enough disks come back, if a disk is missing within the redundancy the poolstate is degraded.
When the last disk is back the poolstate goes to online. If you have modified data in the meantime, it initiates a resilver.

If this does not help, you may need another setup where you can test
if the pool itself is the problem or
- power
- the SAS controller
- the expander
- the backplane

toelie · Dec 15, 2014

I have multiple all-in-one napp-it appliances running omnios. Updating vmware tools seems to break the napp-it webinterface. After logging in it is stuck at "processing please wait". Reverting to the previous BE fixes it.
This happens with ESXi 5.1 vmware tools 2000251 and 2323236.
Installation is done by running the vmware perl script.

Does anybody know how to solve this?

J-san · Dec 15, 2014

toelie said:
I have multiple all-in-one napp-it appliances running omnios. Updating vmware tools seems to break the napp-it webinterface. After logging in it is stuck at "processing please wait". Reverting to the previous BE fixes it.
This happens with ESXi 5.1 vmware tools 2000251 and 2323236.
Installation is done by running the vmware perl script.

Does anybody know how to solve this?

This might be related to enabling 9000MTU and jumbo frames, or using Solaris 10 vs 11 vmware drivers. Did you modify the vmware tools script to try and select Solaris 11 drivers for OmniOS?

I've stuck with the regular Solaris 10 driver version that is installed if you do not modify the tools.

If you modify MTU to 9000 and then reboot your network on vmxnet3s will be verrryyy slow/flakey until you enable jumbo-frames on each of your vmxnet3s interfaces:

Code:

ndd -set /dev/vmxnet3s0 accept-jumbo 1
ndd -set /dev/vmxnet3s1 accept-jumbo 1

This must be done after each boot, as I have not found an option to make it persistent via a driver setting.

I just hacked a quick script to apply it automatically upon boot in my post on servethehome:
https://forums.servethehome.com/ind...-it-vmxnet3-and-jumbo-frames.2853/#post-36083

I thought napp-it had broken at one point as well as I couldn't log in, but it turned out to be the 9000MTU without the accept-jumbo 1 settings. You'll notice your datastores dropping connections from ESXi if you're doing the all-in-one thing with ESXi until you correct it as above.

_Gea · Dec 15, 2014

hello J-San
first, thanks about all the valuable infos

You may insert this startup command in menu
Services >> Bonjour and Autostart

While this is intended for Bonjour startup commands it should work for you as well

I would suggest to keep the napp-it management interface on a default setting and use another interface for tunings so it stays online on a problematic setting.

OpenSolaris derived ZFS NAS/ SAN (OmniOS, OpenIndiana, Solaris and napp-it)

Supreme [H]ardness

Supreme [H]ardness

n00b

Gawd

Supreme [H]ardness

Gawd

Supreme [H]ardness

n00b

n00b

n00b

n00b

n00b

n00b

n00b

Gawd

Weaksauce

Weaksauce

n00b

n00b

Weaksauce

n00b

Weaksauce

n00b

Weaksauce

Weaksauce

n00b

Weaksauce

n00b

Weaksauce

n00b

n00b

n00b

n00b

n00b

[H]ard|Gawd

Gawd

Supreme [H]ardness

n00b

n00b

Supreme [H]ardness