Poor write performance using all-in-one server (ESXi + Solaris + napp-it)

TheLastBoyscout

Limp Gawd
Joined
Feb 13, 2011
Messages
142
Hi,

I've implemented an all-in-one sever following _gea's (popular) guide:
OpenSolaris derived ZFS NAS/ SAN (Nexenta*, OpenIndiana, Solaris Express
However, despite using 5x 600GB VelociRaptor drives in a "RAID10" + spare configuration, the write performance from within the Windows (XP and 7) VMs is simply unacceptable (benchmarks and perceived performance are roughly on par with a 10 year old HDD :eek:), while the performance within the Solaris OS (OI 148) seems to be OK.

There are a few parameters to tweak and I looked at the following:
1. e1000 vs. vmxnet3 network interface (within the Solaris SAN):
While the vmxnet3 network interface much yields faster (peak) throughput, I noticed that the e1000 is much more consistent and provides a better overall average (running HD tune read tests from within the Windows VMs – write requires an empty HDD or in the case virtual HDD).

2. Disabling the write synchronization:
This gives write performance quite a boost, but if I understand it correctly, it is as risky as running a HW RAID5 without a battery backup and defeats the purpose of a reliable system.
_gea also posted here that this is not recommended (see post # 401).
Now what puzzles me is the following:
When I copy a 7 GB .iso file directly to the VMstorage on Solaris (shared via NFS and SMB) from outside the all-in-one system, it takes the same amount of time regardless of whether write sync is disabled or not. However when I copy the same file to a share on a Windows XP VM running on the all-in-one, I see about 4x improvement when the sync is disabled.
Conclusion: The write sync has a huge impact on NFS write performance.

What I have not tried is the use of a ZIL, simply because I don't have an SSD at hand and I don't want to buy one to find out that it doesn't improve the situation. Looking for one, I have not found much information on usable (SSD) drives. The ZIL can be small (with the above listed ~1TB storage a 2 GB ZIL should be sufficient), but it needs a "supercapacitor" or similar device to allow its SRAM or DRAM cache to be written to the flash in case of a power loss. Without the supercapacitor we're having the same problem in case of an abrupt power-loss as already state above.
Now finding an SSD with a supercapacitor isn't that easy. I found the Wintec FileMate 8GB SLC and the similarly priced 40 GB Intel 320.

Is a ZIL the way to go to avoid this (NFS?) write bottleneck with this type of application?
Which SSDs do you use for a ZIL and how much did it improve the write performance?

Thanks for helping!
-TLB
 
You're misunderstanding gea's post #401. He is referring to the general recommendation. If you are running an all in one, there is no real risk, since if the host crashes, everything, including the ZFS SAN is down too.
 
Thanks Dan!

You are correct in that when the power is down, the Solaris as well as the VMs are off, but there is no mechanism that saves the data of transaction that are in progress?

Well, thinking more about it it really seems that it wouldn't matter much: The difference is really just a matter of whether the power for the VM is killed e few transactions earlier (data in progress to be written doesn't completely make it to the disk) or not. Either way, I may end up with a corrupt or inconsistent set of files (e.g. if the power goes out while a Windows update runs that replaces multiple system files). While a UPS can protect my system from "outside issues", it's still vulnerable to inside issues (power supply or MB dies). I guess that's really nothing I can prevent and only counter that with regular storage backups and a replacement server.

I have not played that much with apcupsd yet, right now it's only configured to shut down the Solaris SAN OS when the battery goes down, but I have not looked into options of how to shut down the VMs automatically before the Solaris shuts down...
But that's for another thread ;)

-TLB
 
I am virtualizing on my Solaris 11 Express PC, via VirtualBox. Works great, and it is easy to do. What are your reasons to not try VirtualBox?
 
brutalizer, after over a year persisting with VBox on a solaris host...I have given up and moved to esxi.

I had issues with stability and performance issues across a range of hosts, windiws, ubuntu and centos. LAN transfers were slow in vm's and the vm performace was slow too.

I just moved to esxi, with passthrough to the soaris guest and I get 600MBs reads and writes on a 10 disk raidz2 as well as 250MBs plus between Windows vm's. memory use is great too. read/writes over the LAN maxes out Ethernet.

why go esxi, because it works, fast.

to the op, you should not disable sync IMHO, ever.
 
@davros123
I see your point with VB. Yes, it is quite unstable in complex configurations. But for my basic needs, it works great. Is ESXi easy to setup if you compare to VB? Which takes most time to administer?

Have you considered the new StormOS? Minimal Solaris kernel (ZFS, DTrace, etc) with KVM. KVM has been ported in the StormOS.
 
I have to say, esxi has been an absolute breeze to move to...the free VMWare converter does a real time migration of the VBox machines (windows and linux)...in under 5 in's I had perted my machines across to esxi and the were up and running! Amazing. I know this sounds like an add...but it really was too easy.

I have seen stormOS...and was going to check it out but the ease of passthrough on esxi with a vt-d mobo (ie 3 clicks) was too good to pass up...so I have not tried it but would be keen to hear your thoughts after giving it a go.

I even moved my Solaris Nas over...not as easy, but only took about 30 mins...

I posted this over at another forum which outlines some of the steps and the experience of moving.
----------------------------------------------
E3-1230
Supermicro X9SCM-F Mobo, 4x pcie, C204, IPMI and iKVM (http://www.supermicro.com/products/m...04/X9SCM-F.cfm)
16GB ECC Kingston KVR1333D3E9S/4G
Intel dual onboard nic's
IBM M1015 – flash with P10 firmware (~100 on ebay).
Crucial M4 128GB for the VM's
ESXi on a 1 GB usb (already have an slc stick)
Passthrough of 10 3TB drives to Solaris 11 Express (nas/san)

Well after a couple of weeks awaiting all components have arrive, so the cast of this weekend was to put together the new server and see how easily I could migrate from my physical Solaris box and virtual box machines.

Well am pleased to say that it was an absolute delight to transition across to ESXi. I'm not a great fan of VMware, but I must say it is so very easy. I installed ESXi onto a one gig SLC thumb drive which mounted directly onto the motherboard. I am currently using a 128 gig M4 SSD as a data store for the main VM's with a 500GB hdd as a temp storage while I get the data pool copied across to the new 3TB drive pool.

I used this guide to help me get started, , http://www.napp-it.org/doc/downloads/all-in-one.pdf. Passthrough of the drives to Solaris is just a matter of clicking a few boxes. Sooooo, easy with VT-d compat. hardware...honestly, I thought there would be more to it...but nup, click, done.

But as most of my machines were existing VM's under virtual box and also my "host" physical Solaris box I wanted a simple way to just migrate these existing physical and virtual machines.

As it turned out, migrating Windows and Linux vm's (and physical machines) was an absolute piece of cake! Just download VMWare Converter from the VMware site. Install it to the running machine that you want a copy (you can even do this remotely), enter the details of your ESXi server and in about 5 min it will clone and deploy the machine to ESXi. How simple was that. All done via a GUI, no commandline is required, no messing with anything. Your machine is now available to run in ESXi. I would usually tweak a few things like change the ethernet adapter to a vmxnet3 (10GB) adapter... But that's about it. Shut down your virtual box machine and start it up over in ESXi. Nice.

I use this to migrate my for all five Windows machines as well is centre OS and Ubuntu Linux. At about 5 minutes a pop. I was very happy.

Now, the big one Solaris 11 express... I first installed a fresh copy of Solaris just confirm that everything would work okay under ESXi, it did so I started Googling to find out how I could migrate S11E across to ESXi. After a bit of hunting, I found a fantastic guide here, http://blogs.oracle.com/mrj/entry/solaris_10_opensolaris_p2p_p2v. This talks about migrating to virtual box, however the process is exactly the same to go to ESXi. Thanks to the magic of ZFS snapshots, (they really are magic), and zfs send. This is actually an extremely simple process involving only a handful of commands, all with no down time on your Solaris machine! Freaking cool. I was able to move from physical to virtual in about 30 minutes, and most of that was waiting for the zfs send/receive to complete :)

There is one mistake I found in the instructions when installing the grub boot loader, and it should read...
On the VirtualBox guest, install grub
installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c4t0d0s0

Also, note that once I had migrated the rpool, I booted the system into single user mode (basically add "-s" to a line when you boot and get the grub menu)... I logged in as root, and then from the command prompt, ran "touch /reconfigure"...which causes the system to reconfigure itself on the next boot. I then rebooted and presto! I have migrated Solaris 11 Express from physical to virtual.


Note : a trap for players, make sure that before you start the Solaris P2v migration process, on your S11E box, go into preferences, appearance, visual effects and click on none...ie. no visual effects. If you do not ESXi will not be able to handle the additional graphics requirements, and while it will boot it will present as a white screen. That took a while to work out!

So, I now have all my vm's running on the ESXi box and I am just in the process of doing a final copy of the data from the old data pool (1.5TB hard drives) to the new pool of 3TB drives.

A couple of additional comments, I recommend that you download the evaluation version of VMware is the centre server and install at two Windows machine. There are a few nice things that this provides above the free version, including the ability to clone machines and create templates as well as do easy migration across data stores. I think the evaluation licence last 60 days, so more than enough time to get everything set up.

Make sure you only assigned 2 cpu's to solaris, I understand it can become unreliable when booting with more than two CPUs.

So far performance on the new machine is simply outstanding. So much better than running virtual box. Even with just a single CPU assigned to a Windows VM, it literally flys when compared to virtual box. Certainly some of this is attributable to being on a new generation processor, however it is clear that the virtualisation ESXi has it hands down over virtual box.

I accidentally created my 10 disk pool of the new 3 TB drives as a raid0 array... Hence the need to recopy the data over.... Bummer. But I did take the opportunity to test the speed in this configuration. Writes were over 600 MB/s and read was in excess of 1 GB/s. Once you get the new array installed, I'll run a few bench tests and see how it performs. At this stage, however it's looking like the performance using pass through is very close if not the same as running directly on the host.

I really like the direct passthrough...It does not touch the drives at all, just presents them straight to the vm. So handy knowing that the data pool is just like any other zfs pool...ie. I can export it and take it another PC and import the pool. Great to know I am not tied to esxi should the server die. Infact that's what I did to get the pool over, I pugged it into my other Solaris Server, copied from th old pool to the new pool (using zfs send/receive)...and then shutdown, unplugged the cards/drives and plugged them into the esxi server...zpool import, presto, done.

The speed of this new setup compared to my prev. server is just amazing...truely amazing. I wonder how much of it is down to the new components (Sandy xeon vs. 3370 xeon, 16GB vs 8GB...) but I am sure that the ESXi is certainly a huge factor with VM's feeling REALLY snappy and transfers between the machines @ 300+MB/s vs. limping along @5-10MB/s on VBox.

Should have gone a proper hypervisor long ago..like this or KVM.

Still I have learned alot and I'm not sure passthrough was as mature 2 years ago...

So, do I recomend it, well based upon a few hours running...HELL YEAH!
 
to the op, you should not disable sync IMHO, ever.
Hi davros123,

Could you elaborate on this a bit? Since I am getting horrible write performance with this NFS setup, I need to find a way to make this work. Since I don't use it as a NAS, but use it to run multiple VMs off it, I was really shooting for good IOPS, but I have not been able to do so without disabling the write sync.
I know that CrystalMark has its "challenges", but I since the lack of performance is so noticable, I needed something that visualizes this quickly. I posted some test results here (does this look like a RAID10 of 600 GB Raptors?) and here.

Thanks,
-TLB
 
Again, unless someone can provide a credible reason why sync=disabled is bad in an all-in-one scenario, I will repeat the recommendation.
 
As another comparison: The guys at LegitReviews.com tested a single 600 GB Raptor and got the following results for the File Benchmark (HD Tune) on Windows 7, 64-bit:
wd_raptor_hdtune5.jpg


When I run HD tune from a Windows 7 (64-bit) VM, I get this with sync=standard:
win764hdtunefilesyncstd.png


And I get the following with sync=disabled
win764hdtunefilesyncdis.png


While one could get hung up on the high numbers for large files, it's the small files (<32k) that count for the responsiveness of the OS. And in this case, it's not even close - even though I should get much better numbers just from the fact that I run 4 of those drives in as striped mirrors, right?
Again, it's not that I'm much into benchmarks, the only reason I started looking into these, is that I noticed that the performance was sub-par.

Am I the only one that gets "throttled" performance inside the VMs?

-TLB
 
What kind of vnics are you using? I had to stop using vmxnet3 due to performance anomalies (burstiness, etc...) e1000 on all guests works very well for me - about 3gb/sec sustained thruput between guests. Note: this assume you have vmci device set to "Unrestricted" for all of the guests? If not, try that...
 
I'm on the e1000 "NIC". While the vmxnet3 provides higher peak throughput, it is very flaky in performance and the resulting average of data transferred is much lower compared with the e1000.

I'm not familiar with the settings for the vmci device. Where do I look for those (in ESXi or inside the Windows VM)?

-TLB
 
Go to the settings for the VM. It should show you the devices. You want the VMCI device. I believe the default is "Restricted". Try changing to "Unrestricted" (for all affected VMs...)
 
Dan, the sync=disabled question has been covered before, and i do not want to derail this thread.

I suggest that sync=disabled is not this guys issue and he should find the root cause, not seek to mask it by disabling a key feature of zfs, the zil. his issues lies elsewhere. I have not disabled the zil, and am running a slower disk setup with 10 drives in raidz2, but get better performance...so I do noth ink sync is his key issue.

Removing it means zfs can not guarantee the writes as the writes are stored in volatile ram. IMHO, this is important as I like to know my data is integral, one of the key reasons I use ecc, a xeon and zfs.

as for a scenario on an all in one, well, what about the SAN vm faulting and the others staying up?
 
I was not derailing the thread. I agreed already that sync=disabled is bad in the usual case, but this is NOT the usual case; it's an all in one. In that case, sync=disabled is a quick and easy fix for the write performance sucking. As for this not being his problem, how do you figure that? His performance went up by a large factor when he disabled it. Yes, it is theoretically possible for the SAN VM to fault and die, everyone must evaluate what risk they can afford - I have never had a VM (not windows anyway) fault and die. Your rig is not his - so when he changes one setting (whose default state is known to impact write performance from ESXi NFS), and it works much better, what else would it be? You are making vague assertions what his problem is or isn't. I made a specific suggestion that resolved his issue, which made sense given that this is a well known issue (with ESXi NFS writes.) Not everyone is using ECC and such, they mainly want the other nice features of zfs.
 
Last edited:
Sorry, I was crabby. My point is still a valid one. This is absolutely a known issue, and the only two options I've ever seen documented are a high-speed (SSD) ZIL or disabling sync entirely.
 
no prob's mate...all good.

yes, your point is a valid one...but I still wouldn't want to disable the zil :)

I just moved a few more vm's over to nfs and ran zilstat.ksh and therre is definitely alot of zil activity...and the vm's were slowing down...so I can see the issue manifesting on my server with nfs.

I created an iscsi lun and connected that to esxi, migrated the vm's and it's obviously not doing sync writes..so seems to scale better...pitty about the loss of easy (vm level) snapshot restores...but it may be a viable alternative.

give it a shot and see how it performs.
 
Different strokes, I guess. I wouldn't make such a fundamental change (moving from nfs to iscsi) unless I had absolutely no choice. So you don't have a ZIL?
 
Not at the moment. I have some ssd's that I could use, but they are MLC nor supcap.

the move to iscsi was pretty easy, in VCentre, click vm, click migrate...wait 5 minutes, done.
 
I have a similar all-in-one setup. Mine has a SuperMicro X9SCI-LN4F-O, Xeon E3-1230 Sandy Bridge and 16 GB RAM. There are 6x2TB Hitachi 7k3000 drives in a striped mirror (ZFS "RAID10"?), a 120GB SSD as a boot drive and a 64GB SSD as read cache. Three of the the 4 GigE ports are load balanced and dedicated to file services via NFS and iSCSI. The physical switch uses LACP for link aggregation.

My setup has similar symptoms as the ones reported by the OP. I also had to switch the Ethernet drivers from vmxnet3 to e1000 to get consistent NFS read performance. With vmxnet3 there were bursts of very high performance but also periodic bursts of slow performance. With e1000 read performance is slower than the vmxnet3 peaks, but consistently very good. With sync=standard both had slow NFS write performance. With sync=disabled, NFS write performance is very good, about as fast as read performance in most cases.

So, I am also looking for any hints to improve NFS write performance in Open Indiana. Operating with sync=disabled may not be safe for a production server that handles requests from dozens of VMs. Is there a setting to enable striping in a zpool, or is it automatic when you have two or more disk pairs? Would RAIDZ perform better on writes than striped mirrors?

Also, how can I restrict the NFS and iSCSI services in OpenIndiana to only listen to requests to the IP address dedicated to SAN traffic? Those 3 GigE ports are on a separate VLAN in a physical switch that is accessible to other NFS or iSCSI clients. Yet those services are also available in the IP address of the remaining GigE interface used for external traffic. That interface is on a separate subnet and VLAN.
 
Raidz can perform better for writes, AFAIK. I think an SSD for the boot drive on an all in one is gross overkill - the OI VM is going to do very little read/write from/to it. If you are worried about the OI VM crashing (which is the only reason not to disable sync), I'd rather see you add a small SSD as a ZIL. From what I remember, the recommended ZIL size is 1/2 the size of RAM assigned to the OI. The reason raidz writes can outperform: consider a 3x2 raid10 vs 6-drive raidz. For the former, you have 3 spindles participating in writes (logically), in the latter, you have 5 (not counting parity drive.) On the other hand, for reads, raid10 will generally outperform raidz, since reads can be spread across both sides of mirrors (there are those here who report conflicting results, but I am going by my results, as well as everything I've seen on official sun blogs and such...)
 
Last edited:
As far as restricting access, I can't speak to iSCSI, but for NFS, you can do your sharenfs like this:

NAME PROPERTY VALUE SOURCE
tank/esxi-datastore sharenfs rw=@10.0.0.8 local
 
Thanks for your suggestions.

Your argument about RAIDZ being faster than RAID10 on writes because of the spindle activity sounds logical. I'll check on the web for ZFS data to determine if I should recreate the pool with RAIDZ.

I agree with you that using an SSD for boot was probably an overkill. I'll get a small SATA drive for boot and then use that SSD for ZIL.
 
TechTrend...very similar setup to mine.

Have you tried iscsi? I am seeing good performance and none of the sync slow write issues of nfs. give it a shot and let us know how you go.

also, you might consider running your high io vm's off the ssd...that's what I am doing and it's very fast. you'll need a backup strategy for the vm's though...take a look at backup exec.

edit: found this...http://blog.laspina.ca/ubiquitous/running-zfs-over-nfs-as-a-vmware-store

In this blog entry we will explore how to implement an NFS share for VMware ESX using OpenSolaris and ZFS. We will also explore a new way of accelerating the servers I/O performance with a new product called the DDRdrive X1.
With an architecture set in place let now explore some of the pros and cons of running VMware on Opensolaris NFS.

Some of the obvious pros are:

VMware uses NFS in a thin provisioned format.
VMDKs are stored as files and are mountable over a variety of hosts.
Simple backup and recovery.
Simple cloning and migration.
Scalable storage volumes.
And some of the less obvious pros:

IP based transports can be virtualized and encapsulated for disaster recovery.
No vendor lock-in
ZFS retains NFS share properties within the ZFS filesystem.
ZFS will dedup VMDKs files at the block level.
And there are the cons:

Every write I/O from VMware is an O_SYNC write.
Firewall setups are complex.
Limited in its application. Only NFS clients can consume NFS file systems.
General+ protocol security challenges. (RPC)
VMware kernel constraints
High CPU overhead.
Bursty data flow.
Before we break out into the configuration detail level lets examine some of the VMware and NFS behaviors so as to gain some insight into the reason I primarily use iSCSI for most VMware implementations.
 
Last edited:
as i understand the problem

sync write is a method for better data consistency in case of a crash
ex. if you write a chain of transaktions, the next write is only done if the prior
is confirmed and really stored on disk.

if you compare it with a method without sync writes, i expect the same level
of data security like NFS and sync write disabled. In case of a all-in-one where
Esxi, Solaris and all VM's crashes at the same time in case of a power failure,
so no better data security is reachable with sync writes enabled.

Additional problem
Sync writes are done on block level. In case of a multi-GB VM a crash could always
be a problem also with sync enabled. You must always check if the VM is running afterwards
correctly or you must go back to a functional snap (best a ESXi hot-snap)

The main advantage i see with NFS is its simplicity. You can move, copy, import, clone
or access/restore snaps easily even with free ESXi. If you share it also via smb
or you can use Nautilus time slider or Win previous version to access snaps,
even if you have more than one VM on a share. With iscsi you can only restore the whole target.
Performance should be the same if you disable sync or add a ssd write cache.
 
Last edited:
Have you tried iscsi? I am seeing good performance and none of the sync slow write issues of nfs. give it a shot and let us know how you go.
I was leaning towards NFS because of the advantages of having file level access to the VM directories. It is also simpler to setup. Yet this write performance issue is making me reconsider. Some NFS clients are VMs on physical servers external to the all-in-one. If the SAN crashes those external VMs may still be running, making sync=disabled more dangerous.

Allocating a separate LUN for each VM makes the logistics more complicated when using iSCSI. Since all access will be from VMware VMs, one alternative may be to allocate a large LUN and then allow VMware to control space usage via its VMDK files. Are there problems or disadvantages using this scheme with the Comstar iSCSI in OI? If this is viable, would thick provisioning of the large LUN be advisable? In some SANs (non-OI) using thin provisioning with SATA arrays gradually degrades performance over time as VMs get created /deleted /resized.
 
Since all access will be from VMware VMs, one alternative may be to allocate a large LUN and then allow VMware to control space usage via its VMDK files. Are there problems or disadvantages using this scheme with the Comstar iSCSI in OI?
That's what I have done..Setup a single 200GB thick provisioned iscsi LUN. Exposed this to esxi and let it manage the files/filesystem.

The only issue I have is that as my setup is an all in one, the SAN vm needs to start first, to get the iscsi up, but by then esxi has decided it can not find the target, so set's the other vm's to unknown and willnot boot them. Last night I created a perl script that runs on the Solaris machine to ssh into esxi and rescan for iscsi datastores. This works. I might add it as a cron job to the esxi machine too. I also updated my nagios (also in a vm) scripts to control the vm's and do a rescan.

You will not have this issue with iscsi as you are not an all in one.

If you can afford it, you may want to look at Backup Exec. It can connect into esxi and backup the (running) vm's...but interestingly can not back itself up!

danswartz will laugh, but I am slowly migrating to his point of view about sync and nfs.
 
Last edited:
No laughter, this is not about ego for me. One other thing, you have hit the same annoying nit I did - namely by the time the SAN VM is up, the esxi iscsi stuff has timed out. NFS seems to be more resilient in that respect...
 
the SAN vm needs to start first, to get the iscsi up, but by then esxi has decided it can not find the target, so set's the other vm's to unknown and willnot boot them.
One workaround for this may be to reduce the startup delay for the SAN VM (to 10 secs?) while leaving the other internal VMs with the default (120 secs). You can override the startup delay using the VSphere client by selecting Configuration, VM Startup/Shutdown, Properties. Of course, that doesn't help with VMs on external servers using the SAN. On those the monitoring scripts are a good alternative.

I have been testing with iSCSI for the past couple of days with good performance. I am leaning towards using iSCSI as the default datastore for VM disks and NFS for file storage and sharing. Thanks to all for your suggestions.
 
I think I tried something like that but it didn't work. If I recall, ESXi still wanted to look up the other VMs, even if it didn't start them yet.
 
Note: this assume you have vmci device set to "Unrestricted" for all of the guests? If not, try that...
I did not have that set originally, but after changing the VMs over, I see only marginal improvements (sync still disabled):
VMCI restricted (default, same as above screenshot):
win764hdtunefilesyncdis.png


VMCI unrestricted:
win764hdtunefilesyncdis.png


I am still "surprised" that the striped mirror (through NFS) is significantly slower than a single drive (directly connected); see post # 10.

-TLB
 
I'm not - in the nfs case, you have nfs protocol overhead, etc... The win is if you have striped drives both of which are directly connected?
 
What do you mean with "directly connected"? The five HDDs are connected to the LSI 1068E which is passed-through to OpenIndiana and then set up as follows:
Code:
config:

                NAME        STATE     READ WRITE CKSUM
                VMstorage   ONLINE       0     0     0
                  mirror-0  ONLINE       0     0     0
                    c3t5d0  ONLINE       0     0     0
                    c3t6d0  ONLINE       0     0     0
                  mirror-1  ONLINE       0     0     0
                    c3t7d0  ONLINE       0     0     0
                    c3t8d0  ONLINE       0     0     0
                spares
                  c3t9d0    AVAIL
-TLB
 
You tell me what 'directly connected' means - you used the term in your previous post? When you said 'striped mirror thru nfs', I assumed it was on a datastore via nfs, but you were talking like you had a raid10 exported via nfs vs a single passed-thru drive. I guess I am not sure I understand what you are doing...
 
Sorry about the confusion. I basically followed _gea's guide to the letter and set up a striped mirror of 600 GB WD Raptors in OpenIndiana 148 (OI). The drives are all connected to the on-board LSI 1068E which is passed-through to OI. OI then shares that storage pool "back" to ESXi via NFS.

When I mentioned "directly connected", I referred to the test results of a single 600 GB Raptor that is tested when connected to the test OS "directly" - without going through any sorts of network sharing (i.e. the drive is local to your system - like your boot drive in most cases).

I realize that there is *some* overhead of an additional protocol, but with these results, I am wondering why _gea uses a "RAID10" SSD pool? The same protocol overhead that "kills" the performance on my setup would be the same, so SSDs should only be marginally faster, while much more expensive....
Thus, I am wondering whether there is something that can be done to improve the small file performance with my setup?

-TLB
 
Thanks for the clarification. I don't actually agree with your 'marginally faster' analysis. For a virtual disk accessed via NFS, there will be two delay components - disk access time and network delay. In the case of a raid10 SSD pool, the disk access time should be virtually zero, no? I reviewed your other post, and I'm not understanding something. I'm not sure offhand why small file access is so much worse, but I would argue that in the common cases, most of those will be cached by the client OS, so I'm not sure why that is a killer for you (do you have a detectable issue, or is this a concern only due to running a benchmark?)
 
Also, how can I restrict the NFS and iSCSI services in OpenIndiana to only listen to requests to the IP address dedicated to SAN traffic?
Using Target Portal Groups in Comstar turned out to be a good way to restrict iSCSI targets to some interfaces. Using napp-it, this can be done with these menu options:

Comstar, Target Portal Groups, Create
Comstar, Targets, Add Portal
 
Back
Top