FreeNAS to VMware - NFS or iSCSI

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
Just wondering what makes the most sense to setup. I used FreeNAS awhile back with NFS, but haven't touched it in awhile. Here is the specs of the SAN.

2x 10 Gb NICs (LACP)
38x 2 TB SATA Drives
16x 1 TB SATA Drives
12x 600 GB SAS Drives

I don't currently have a use for the SAS drives, but they were already in the SAN so I created a volume with them. The 16x 1 TB drive may end up being iSCSI mounted in Windows for use with Backup Exec, but I am undecided. And then the primary volume is the 38x 2 TB drives that I wanted mounted to several ESXi hosts. It will be used for Veeam backups, Oracle RMAN backups, and Zerto replication.

Right now all other storage on the ESXi hosts is mounted using iSCSI, so I was thinking about going that route unless there is a good reason to use NFS.
 
iSCSI... The latest target is better than the old ones from stability perspective, but at least works. FreeBSD with NFS these days has a new way of interpreting FILE_SYNC writes, which means unless you want to spend a lot of time tuning SLOG for the ZIL, you're going to, well, get utter shit for performance. iSCSI bypasses that. :(

This is assuming ZFS, of course.

Or you could apply the kernel patch to enable async NFS override for FILE_SYNC, but then you're in danger of data loss. Choice is yours, as they say.
 
Seems to be something wrong with this server. It used to run Openfiler just fine. I swapped the RAID card for a brand new HBA and added a 10 Gb NIC. Installed FreeNAS on the existing SSD that Openfiler was installed on.

So far twice has the large volume gone into an unknown state, and now it is degraded. But what I don't understand is when this happens, the OS seems to stop responding too. If I start clicking around the web interface it just hangs. I go to SSH into it and it hangs after typing in the password. The first time this happened, after rebooting the 38 disk volume was only showing 31 disks. The second time it looked like 20 disks all in a row disappeared. I would say maybe it is the backplane, but drives from another volume weren't affected. Seems to just be the 38x 2 TB volume.

What would cause the OS to just stop responding?
 
Here is what I am getting after rebooting. When I view the disks, 21 of the 2 TB disks are missing.

 
I did notice the following warning.

WARNING: Firmware version 20 does not match driver version 16 for /dev/mps0

Could it be a driver issue? And how the hell do you update the driver on FreeBSD? I barely know Linux, let alone BSD.
 
i prefer freebsd over freenas

nfs is much easier to manage compared with block storage like fc o iscsi

KapsZ28 said:
Just wondering what makes the most sense to setup. I used FreeNAS awhile back with NFS, but haven't touched it in awhile. Here is the specs of the SAN.

2x 10 Gb NICs (LACP)
38x 2 TB SATA Drives
16x 1 TB SATA Drives
12x 600 GB SAS Drives

I don't currently have a use for the SAS drives, but they were already in the SAN so I created a volume with them. The 16x 1 TB drive may end up being iSCSI mounted in Windows for use with Backup Exec, but I am undecided. And then the primary volume is the 38x 2 TB drives that I wanted mounted to several ESXi hosts. It will be used for Veeam backups, Oracle RMAN backups, and Zerto replication.

Right now all other storage on the ESXi hosts is mounted using iSCSI, so I was thinking about going that route unless there is a good reason to use NFS.
Click to expand...
 
FreeBSD still needs the same kernel patch - the FILE_SYNC change was made in the mainline kernel, IIRC. With that patch, absolutely.

I'm not familiar with that error, Kaps - anything on google on it? :(
 
KapsZ28 said:
I did notice the following warning.

WARNING: Firmware version 20 does not match driver version 16 for /dev/mps0

Could it be a driver issue? And how the hell do you update the driver on FreeBSD? I barely know Linux, let alone BSD.
Click to expand...
There's your problem straight away, downgrade the card to at most v19. There are known corruption issues with v20, there's a few posts on the freenas forum and a bug report about it.

"Officially" you should be at 16 to match the driver until they update it, but i'm running 19 (and 17 before that) on 2x 2008 without issues.

Also, not one to argue with lopoetve but they recommend zvols for iscsi now afaik.
 
bmh.01 said:
There's your problem straight away, downgrade the card to at most v19. There are known corruption issues with v20, there's a few posts on the freenas forum and a bug report about it.

"Officially" you should be at 16 to match the driver until they update it, but i'm running 19 (and 17 before that) on 2x 2008 without issues.

Also, not one to argue with lopoetve but they recommend zvols for iscsi now afaik.
Click to expand...

Cool. I will look into downgrading the firmware instead of upgrading the device driver.

Not sure about the zvols either. I was watching a video that recommended file instead of device but I don't recall what version FreeNAS.
 
bmh.01 said:
There's your problem straight away, downgrade the card to at most v19. There are known corruption issues with v20, there's a few posts on the freenas forum and a bug report about it.

"Officially" you should be at 16 to match the driver until they update it, but i'm running 19 (and 17 before that) on 2x 2008 without issues.

Also, not one to argue with lopoetve but they recommend zvols for iscsi now afaik.
Click to expand...

They used to, but I've seen recently everyone switching to file :confused: A few new articles on it working "better" apparently, but who knows?

I honestly don't know which side to believe on it - try one, then the other?
 
lopoetve said:
They used to, but I've seen recently everyone switching to file :confused: A few new articles on it working "better" apparently, but who knows?

I honestly don't know which side to believe on it - try one, then the other?
Click to expand...

I tried file before the system crashed and performance was pretty damn good. :D

Now hopefully when I get that firmware rolled back, I won't have to worry about it crashing.
 
bmh.01 said:
There's your problem straight away, downgrade the card to at most v19. There are known corruption issues with v20, there's a few posts on the freenas forum and a bug report about it.

"Officially" you should be at 16 to match the driver until they update it, but i'm running 19 (and 17 before that) on 2x 2008 without issues.

Also, not one to argue with lopoetve but they recommend zvols for iscsi now afaik.
Click to expand...

So how do you successfully downgrade? I keep having issues and haven't seen a solution yet. I read to do the following.

sas2flsh -o -e 6
sas2flsh -f XXXXXX.bin -b mptsas2.rom

This is what I am getting.

Code: 
[root@SAN01] /tmp# sas2flsh -o -e 6

CORRECT>sas2flash -o -e 6 (y|n|e|a)? yes
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

        Advanced Mode Set

        Adapter Selected is a LSI SAS: SAS2308_2(D1)

        Executing Operation: Erase Flash

        Erasing Flash Region...

                Erase Flash Command not Supported on this platform.

        Resetting Adapter...
        Reset Successful!

        Due to Exception Command not Executed. IOCStatus=0x1, IOCLogInfo=0x0
        Finished Processing Commands Successfully.
        Exiting SAS2Flash.
[root@SAN01] /tmp# sas2flash -f 9207-4i4e.bin -b x64sas2.rom
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

        Adapter Selected is a LSI SAS: SAS2308_2(D1)

        Executing Operation: Flash Firmware Image

                Firmware Image has a Valid Checksum.
                Firmware Version 19.00.00.00
                Firmware Image compatible with Controller.

                Valid NVDATA Image found.
                NVDATA Version 11.00.00.00
                Checking for a compatible NVData image...

                NVDATA Device ID and Chip Revision match verified.
                ERROR: Cannot downgrade NVDATA version 0x14000006 to 0x11000006.

                ERROR: Failed to get valid NVDATA image from File!

                Firmware Image Validation Failed!

        Due to error remaining commands will not be executed.
        Unable to Process Commands.
        Exiting SAS2Flash.
 
Unless it must be done in DOS... Which would suck being that the virtual media on the IPMI isn't working and I will have to drive an hour just to do this.
 
That's the process I use, although I've only ever done it in dos i've got a freedos boot stick with every firmware I usually come across for this. Although i've sucessfully managed it in an efi shell as well.

They used to recommend file extents with the older istgt daemon as it game much better performance, when they switched to ctld with 9.2.1.8 I believe the performance was better with zvols and the recommedation changed.

http://doc.freenas.org/9.3/freenas_sharing.html#extents

Although looking at that these days it seems it's up to your use case, i'd try both and see what you find. If you get chance report back as it'd be something i'd be interested in seeing/knowing. I'm currently using zvols and get pretty decent performance once the l2arc is hot.
 
Unfortunately this motherboard seems to have issues. It is a Supermicro X8DTI-F and it won't boot up the DOS or FreeDOS off of USB. I've used the same USB stick on many other Supermicro servers to update BIOS, but this one just doesn't work.
 
Well, I went with firmware 16 since that is the driver version. Hope it doesn't change anytime soon since I can't update from this motherboard. I had to pull the card out and stick it in another motherboard just to flash the firmware. Good to go now. Just have to start testing.


 
I am not looking for a ton of performance as this is mostly going to be used for backups, but kind of curious as I was reading about the Intel 750 series today. Can you pop in a 1.2 TB PCIe SSD card and use that for SSD caching? Would be pretty cool since it is only about $1,000.
 
Sure, ish. I don't remember how the SLOG works for ZVOL devices, but you could use it to try and accelerate the NFS side for certain. Then you get into the transaction queue commit rates, RAID levels, and SLOG size though, which may be more trouble than it's worth (and that SSD may be massive overkill size wise, depending on a lot of things).
 
danswartz said:
SLOG is no different for zvol than files. If ZIL is in play, so is the SLOG.
Click to expand...

ZIL is always in play - it's how ZFS does journaling - ZIL is often used to refer to the Separate Log (SLOG) device to accelerate ZIL commits based on which write pipeline you're using, which is what I suspect you're referring to. The question is, are you doing FILE_SYNC, or just SYNC for writes. FILE_SYNC requires one write pipeline, SYNC requires a slightly different one, and ASYNC (not VMs) uses a third. And then you have to size against the transaction queue size and rate (dependent, normally, on system ram, time (ZFS version), and speed of the spinning disks). Without a separate device, the ZIL lives on the same disks as the volume itself.

Given that most ZVOL devices don't do FILE_SYNC (since iSCSI doesn't have the same concept, but NFS does, and VMware at least sends FILE_SYNC for NFS), you tend to get better performance out of them without a SLOG device of some kind for the ZIL, but it's very possible to mis-size one of the many parts of the pipeline and either waste resources (SSD SLOG device is too large, never gets fully utilized based on transaction queue settings/time), or get lousy performance (spinning disks can't handle the transaction writes to stable storage in time and you timeout the queue).

ZFS is awesome, but it's really easy to do it 'wrong', as it were, or spend more than you need to. :)

edit: gack, did you ninja edit? I could have sworn you said something slightly different when I hit quote at first. Either way - the NFS to iSCSI sync differences make a huge difference in performance based on how ZFS has to handle "stable" storage for FILE_SYNC.
edit2: FILE_SYNC vs SYNC will also differ if you're on BSD, Linux, or Solaris based ZFS implementations, as it also relies on how the kernel NFS server(s) do business, and that changes things.
 
Last edited:
Well, good thing is the SAN has been stable since I downgraded the LSI firmware. But I am not sure if I should laugh or cry about the performance.

SATA - 36 disks in RAIDZ2



SAS - 11 disks in RAIDZ2 - 15k



Granted there is only one VM running on the SAN at the moment, but I believe those numbers are slightly better than when I tested our NetApp when it was new. Now our NetApp is much lower with higher latency and sadly the NetApp has flash pool while the FreeNAS is all spinning disks.
 
Welcome to low-end NetApp!

Good IOPS :) Whatcha running the test on - VM or native box?
 
Don't know what the CPU/RAM specs are but if you can get your working set (and keep it there) into arc/l2arc then it flies along for the cost.
 
bmh.01 said:
Don't know what the CPU/RAM specs are but if you can get your working set (and keep it there) into arc/l2arc then it flies along for the cost.
Click to expand...

Nothing exciting. Intel E5506 which is an older quad core and 48 GB of RAM. This is really just for backups, so I don't intend on putting SSD into it.
 
Back to the topic of which extent type to use. Currently I am using the file type extent for the iSCSI mounts. I noticed that Delete Status is Unsupported. I was doing some reading and it seems you need a ZVOL for Dead Space Reclamation (UNMAP).

So, I guess that UNMAP is not available to the file type extent and my only option to have UNMAP available is to create a ZVOL and use the "Device" Extent Type? Unless there is some other way to reclaim space on a file extent.

Also, what Block size do you usually go with for ZVOL and logical block size for the file type extent?
 
Pretty sure UNMAP is zvol only as you've worked out, I'd have to check but I think I settled on 8k zvols in the end.
 
I switched to ZVOLs and UNMAP is now available. Performance seems slightly slower, but it could also be due to block size. I went with the 8k block for the ZVOL and 512 logical block on the extent.
 
Keep in mind if you have an SLOG, small block zvols are going to pound on the SLOG hard. e.g. hundreds/thousands of small writes coming into the SLOG, whereas the writes to the pool disks will be much larger chunks. You could try redoing the zvol with 16K, 32K, etc and see if writes improve...
 
No SLOG. I am not really looking for a lot of performance, but like to see numbers. This is only being used for backups and replication. Nothing that requires a lot of performance.
 
I'm guessing your write performance will probably suck (unless you have writeback cache mode set on the iSCSI target...)
 
This is the SATA disks. First using the file extend and then changing to a ZVOL with device extent.





Pretty big difference on this one. Not sure if UNMAP is worth the performance drop or if it has to do with block size on the ZVOL.
 
I already fixed the firmware issue. Had to put the HBA into a different server. The initial issues have been resolved. Now it is more about using the best settings to meet our needs.
 
You must log in or register to reply here.
Back
Top