Slow NAS speed - one way only?

[TeeJayHoward]

I've got two boxes, labeled "nas" and "backup". "nas" is a physical Solaris 11.1 box - E3, 8GB, 7x3TB RAIDZ2. "backup" is a Solaris 11.1 VM w/ 4GB, 2 cores, 6x3TB RAIDZ passed through via LSI2308. The VM's stored on an SSD on the NAS. I need to take the data on my backup server and put it on my NAS. I initiated an rsync last night before bed. I woke up this morning to discover that almost nothing had copied over. Any SSH connections to my backup server ran ridiculously slowly - You could type, wait about 10 seconds, and then see what you typed on screen. The console worked fine. Canceling the rsync allowed SSH to work normally again. I did a few quick DD tests before I left for work:

From backup to nas: 0.5MB/s
From backup to backup: 175.4MB/s
From nas to nas: 195MB/s write
From nas to backup: 71MB/s

What's going on?

 

[FnordMan]

could be a lot of things.
the one thing that stands out right now is "VM" for backup.

what's the VM software? how is the thing set up? namely, how is the network card set up? (virtio or emulating a card?)

 

[TeeJayHoward]

could be a lot of things.
the one thing that stands out right now is "VM" for backup.

what's the VM software? how is the thing set up? namely, how is the network card set up? (virtio or emulating a card?)

ESXi 5.5. There's only a single NIC in use for the whole system right now. If it's network related, I've got another NIC I could enable and perhaps even passthrough to the VM. The VM has whatever the default is set up for networking. Physically, both machines are connected to a HP 1810-24G. Nothing special is set up on the switch. No other VMs are running on the backup VM's host.

 

[dandragonrage]

What kind of NIC are you using on the VM? E1000 or VMXNet? Try the opposite of what you're using currently. IIRC VMXNet currently has issues with some OSes, and I think Solaris might be one of them. I think E1000 has had issues with some VMWare versions in the past as well.

Passing through a second NIC is also a viable solution.

 

[TeeJayHoward]

I decided to remove the VM from the equation. I installed Solaris 11.1 to a USB thumbdrive, and booted off that. It was then that I discovered that the i210-AT included in the Supermicro X10-series motherboards is not supported by Oracle unless you pay out the wazoo for a support contract, at which point in time they will hand you the appropriate driver in the form of an SRU. After hours of trying to get igb or e1000g to work, I gave up and installed a NIC which I know was supported. That's when I discovered that "ipadm create-if" was turned into "ipadm create-ip" for some unknown reason.

Oh, did you know that Solaris disables all network access if no NIC is detected at install? My new NIC had an IP assigned. It had a route set. It was physically connected. And yet, any time I tried to ping the gateway, I'd get a "Network is unreachable". I had to disable ipfilter. (svcadm disable svc:/network/ipfilter) to get outside of the box. ipfstat. Thank god for that tool.

Anyway, after another hour or two trying to get the networking working right, I finally managed to connect to the nas from my backup server. I initiated an rsync, and watched as files poured out over the network. I SSH'd into my nas, fired up iostat, and my heart sank. Things seemed to be transferring at the right speed, but my pool was only getting about 6 Mw/s. The folder structure was there, but the files were all missing! That's what I get for using a GUI to enable NFS instead of doing it manually, I suppose. After some screwing around with napp-it, I found a fix. I needed to change the permissions via ZFS Filesystems -> acl extension -> ACL on folders -> Reset ACL's (Use the 777 option)

That's right. Triple-seven. I'm as (un)secure as you can get, baby! *snicker*

I have no idea why the default setting for NFS shares is set that way, but after changing it, rsync started working great. I figure it should take about 2 days to copy the 12-13TB of data over.

Then comes step 2. Oracle's Solaris has irritated me enough. I'm switching over to OmniOS. That means downgrading my zpool... Which means, after I finish this copy, I get to turn around and delete my backup pool, turn my backup server back into a VM, and create a new v28 zpool. Then comes another two days of copying data.

Oh, and for what it's worth, the VM was using E1000 for the NIC.

TL;DR: Solaris hates me. Napp-It's configured weird. I turned my VM into a physical box and started the rsync. It works.

 

[TeeJayHoward]

Oh, and I ran some benchmarks. Here's a 7-disk RAIDZ2 pool of assorted 3TB Seagate 7200RPM drives, and a single 840 Pro 256GB SSD for comparison.

 NAME 	 SIZE 	 File 	 Seq-Wr-Chr 	 %CPU 	 Seq-Write 	 %CPU 	 Seq-Rewr 	 %CPU 	 Seq-Rd-Chr 	 %CPU 	 Seq-Read 	 %CPU 	 Rnd Seeks 	 %CPU 	 Files 	 Seq-Create 	 Rnd-Create 
   	   	   	   	   	   	   	   	   	   	   	   	   	   	   	   	   	   	   	   
 pool 	 19T 	 16G 	 446 MB/s 	 96 	 501 MB/s 	 33 	 248 MB/s 	 22 	 385 MB/s 	 98 	 633 MB/s 	 21 	 1093.1/s 	 4 	 16 	 +++++/s 	 +++++/s  
 vms 	 238G 	 16G 	 454 MB/s 	 95 	 473 MB/s 	 29 	 102 MB/s 	 8 	 359 MB/s 	 92 	 529 MB/s 	 13 	 13192.2/s 	 12 	 16 	 +++++/s 	 +++++/s

 

[_Gea]

I suppose, you know that
- NFS3 is 100% unsecure by design and should be used in secure networks only
- access is based on client IP without verification
- there is no user authentication
- different client use different ways, some use nobody as fileowner other the client UID

If you have access to the network of your NFS server and if you know an allowed client IP or client UID, you have access to NFS. In doubt this is no more than a try different IPs and UIDs.
If you move clients or configs with the result of access denied, this is quite normal. You can fix either by a 777 permission or by a change owner to that uid that is used by your NFS client .

This is not a Solaris problem but related to NFS

 

[TeeJayHoward]

This is not a Solaris problem but related to NFS

Yup yup. The weird part was that a user could create a folder, but not a file. That's what lead me to look into ACLs, since by default, UNIX permissions don't cover that level of granularity.

As of right now, I've copied ~750GB of data over in 14 hours. Let's see... 53.5GB/hour, 54784MB/hour, 913MB/minute, 15.2MB/s average.

Damnit. That's the opposite of fast. I mean, it's a freaking ton of itty-bitty files, so I know it's not going to fly across at line speed, but I was expecting better than that. When I went to bed last night, it was copying an .ISO over at 98MB/s. Maybe it'll pick up once it hits my video folder.

 

[TeeJayHoward]

4.14TB copied so far (in 45 hours) for a speed of 26.8MB/s. Looks like it picked up quite a bit once it hit the 300+MB files.

So, I've got 2 NICs in both my backup server and my NAS. I've not bonded them or done anything special. Both NICs have an IP on both systems. Is it using both NICs to copy data over? Because that's the only reason I can think of to be getting iostats like this on my NAS:

My Mw/s has been consistantly over 100, and occasionally over 200 ever since I've gotten home from work today. I've never seen a transfer over this network faster than 118MB/s (except when I was playing with SMB3 on Windows 2012).

Wait, come to think of it... That's a RAIDZ2... If I pull two disks out of the equation, it's 133MB/s, and it always alternates high-low-high-low on 1s intervals. That'd be about gigabit line speed. Running iostat on the backup server and pulling a disk out of the equation comes up with 95MB/s read speeds...

Grr. Is there a nice, simple network performance monitor I can run on Solaris-y boxen to find out (in human readable format) what kind of transfer speed is actually occurring?

edit: Well, if this script is to be trusted, I'm only getting a transfer rate of under 20MB/s. Gah!

edit2: Disabling sync added another 2.5MB/s to the results presented by the script. I don't trust the results. DD from backup to NAS via NFS now creates a 1,000,000KB file in 9.214s (106MB/s) The script showed 23MB/s.