New ESXi Build with Questions & Project Log

spectrumbx said:
This has been discussed so many times.
Running your VMs on a RAID volume (other than RAID 1) is just not recommended.
People always get caught up with disk transfer rate throughput and forget about I/O requests scaling, when that's what matters in this scenario.
Click to expand...

I'm confident that 100% of production ESX environments are run on storage with a RAID level of some sort. Fast controllers and fast disks are more than capable of handling the I/O demands of VMs. Just configure according to your needs.
 
I agree with Lizardking I've never seen someone run ESX in enterprise on independent disks without RAID. We run RAID 5, 50, RAID-DP (Netapp RAID 6) but mostly RAID-DP on our ESX clusters at work and don't have a single problem with IO scaling. RAID with parity like 50, 5, and 6 actually makes more sense on enterprise arrays as you are often dealing with large raid sets and need to minimize exposure to data corruption and random drive failure as opposed to RAID 10 which doesn't give you parity checking and during a drive failure you are taxing the exact drive you wouldn't want to fail to rebuild the mirror.

You do take a performance hit with RAID 5 on writes (usually 15-30% depending on your RAID controller and app) which we don't take that on RAID-DP (parity is calculated in NVRAM on Netapp filers rather than adding an extra read/write I/O to every transaction to calculate parity) but 15-30% write penalty is the cost of not having to waste half the raw capacity of the array which means you can afford to throw more spindles in it to compensate for the write hit.
 
Of the esx clusters ive encountered, which is curently around 1000, 85% run RAID5, 5% RAID1, and 10% a custom or array virtualized RAID level. If you're not on RAID in a production environment you're crazy.
 
lopoetve said:
Except ESXi was originally released to enterprise customers, and then made free - it was never designed for small customers in the first place.

And sure they support virtualization of local disk - you just have to use something like LeftHand's VSA appliance - I'm pretty sure there's a demo for it even, although it requires ESX Classic, IIRC. VMware is not a storage vendor, it's a hypervisor vendor.

What do you call large? To me, large is 3 clustered DMX-4s with 2.5 PB of storage on 8gig fibre or a 10gbe front end.... :) No local storage will ever touch that. THATS what ESX was designed for.

Why would you want to do a raw device to a local disk anyway? What's the point? You gain maybe a percentage point of performance, but that's about it... Can't split it up to multiple vms, only a single one has access at a time (you can't MSCS around local disk - the SCSI3 reservation would lock ESX up totally since it's a device level reservation)... It just makes no sense at all for normal use.
Click to expand...

1. Sure, I would agree that it was designed for the enterprise market and then re-marketed. So, fair enough.

2. That VSA applicance will use VMDK files, which is a problem for me. A corrupt VMDK means you lose everything in that file.

I want my virtualized file server to have access to the raw disks.
This way, I can pull a disk and have it readable in another computer.
Again, I don't want to use VMDK files for my data.
Note, I like running the VM OSes in VMDK files, but just not my data (6TB+ worth of it).

Right now, I have a mega server that does everything for me (VMware/File server).
The first partition (about 50GB) of every disk in that system is reserved for VMware use.
The remainder of every disk is then used for data storage, which is protected under FlexRAID.
This setup gives me the best I/O performance and value out of that system.
It might not be "enterprise", but I like it and it works very well for me. :)

Thuleman said:
This may hold true for your home-grown RAID setup, but in an Enterprise environment where VMs run off of SANs, those are all RAIDed.
Click to expand...
True.
I wouldn't have expected less. :)

I guess I would be doing some sort of RAID too on the VM storage if my VMs needed such protection.
 
Openfiler then- quick and easy iscsi for raw disk and performance isn't bad if you load balance.

Study the man page for vmkfstools. I'm not going to say how but you may be able to rdm a local vmhba device
 
lopoetve said:
Openfiler then- quick and easy iscsi for raw disk and performance isn't bad if you load balance.

Study the man page for vmkfstools. I'm not going to say how but you may be able to rdm a local vmhba device
Click to expand...

not possible, I spent a long time researching rdm a local disk
 
spectrumbx said:
If you want the best disk performance, it is recommended that you use independent disks as opposed to a RAID volume.
You are bound to have I/O issues with many VMs on a single RAID volume.
Remember, most disk activities are going to be small reads and writes.
With many VMs, this would lead to a lot of I/O requests.

This has been discussed so many times.
Running your VMs on a RAID volume (other than RAID 1) is just not recommended.
People always get caught up with disk transfer rate throughput and forget about I/O requests scaling, when that's what matters in this scenario.

I had a lot of fun playing with ESXi 3.5 update 3 yesterday on both my T60 laptop and the server in my sig.
I was quite surprised to find the performance to be really good using a NFS datastore.
The NFS server was running in a Ubuntu VM on another laptop (a Dell 9300).

I am a bit disappointed that ESXi does not support using raw local disks.
If RDM is well supported for external storage, I don't see why raw local disks can't be well supported as well.
Having to build a separate storage server for datastore use kinds of defeat the point of consolidation unless you really need shared storage.
Click to expand...

I agree with this, planning on build a nas soon because of it
 
Ndel said:
not possible, I spent a long time researching rdm a local disk
Click to expand...

It's possible if you know what you're doing. Not supported, but possible.

At least, it worked in 3.0.2 - haven't tried it in 3.5, as I haven't had the need to. I'll try it when I get back to work on Monday.
 
Thuleman said:
Just out of curiosity, why not?
The performance hit is negligible in the global scheme of things.
Click to expand...

probably the 2TB VMFS single volume limit. I can understand that one - Extents are bad, and one of the most often things we get called on.
 
lopoetve said:
It's possible if you know what you're doing. Not supported, but possible.

At least, it worked in 3.0.2 - haven't tried it in 3.5, as I haven't had the need to. I'll try it when I get back to work on Monday.
Click to expand...

I will worship the ground you walk on if you can help me with this.

I apologize for my abrupt response, it was meant to entice people to prove me wrong =)
 
Thuleman said:
Just out of curiosity, why not?
The performance hit is negligible in the global scheme of things.
Click to expand...
It is not about performance at all.
As lopoetve pointed out, there this 2TB limit on a single VMFS volume.
But mostly, a corrupted VMDK file means you lose everything in that file.
This is one additional layer of risk I would prefer to do without.

lopoetve said:
It's possible if you know what you're doing. Not supported, but possible.

At least, it worked in 3.0.2 - haven't tried it in 3.5, as I haven't had the need to. I'll try it when I get back to work on Monday.
Click to expand...

I sure would like to hear more on this.
Big thanks in advance for investigating. :)
 
spectrumbx said:
It is not about performance at all.
As lopoetve pointed out, there this 2TB limit on a single VMFS volume.
But mostly, a corrupted VMDK file means you lose everything in that file.
This is one additional layer of risk I would prefer to do without.



I sure would like to hear more on this.
Big thanks in advance for investigating. :)
Click to expand...

I got it to work, pm me for details, figured it out thanks to the virtual file server thread in this sub forum
 
Ndel said:
I got it to work, pm me for details, figured it out thanks to the virtual file server thread in this sub forum
Click to expand...

May as well share it with everyone, odds are more folks will stumble across this thread and would appreciate to read the solution. ;)
 
Ndel said:
I will worship the ground you walk on if you can help me with this.

I apologize for my abrupt response, it was meant to entice people to prove me wrong =)
Click to expand...



I'll see if the old trick still works.

One note though - it's definitely not supported, and VMware support won't be able to help you if it goes south.

And do NOT ever, ever ever ever, use it on a partition on the drives that the ESX install is on. Do it on separate drives only. There is a good reason for this, and it has to do with how ESX handles SCSI reservations.
 
Hello,

Does anyone know if this has been resolved?
I currently am in the same situation, and sent a PM to Ndel, but he didn't answered so far, and I'd like to progress in my install...
So, if someone managed to present a Raid array to ESX from an adaptec 5805 and use it as either raw device or scsi passthrough, I would appreciate to know how they achieved this...

If, for any reason, they don't want to publish the solution, I'll be happy to have it PM'ed to me...
 
I managed in the meantime to do it...
Here's an extract of my mail sent to some friends with the same hardware (and issues) as me. It is not supported by vmware of course, and the commands/parameters may need to be adjusted to specific hardware, but at least it did work for me:

I installed the latest Adaptec driver, without any problem.

I then created a new HD for my OpenFiler (OF) of 1Mb in size.



Logged in the ESX console, I went to: /vmfs/volumes/sfcesx01:storage1/sfcopenfiler01

I passed the command: vmkfstools -i sfcopenfiler01_1.vmdk -d raw:/vmfs/devices/disks/vmhba1\:0\:0\:0 A5805R5.vmdk

Note: sfcopenfiler01_1.vmdk was my temp HD, A5805R5.vmdk is the name of the RAW disk I wanted to create. vmhba1\:0\:0\:0 is equivalent to vmhba1:0:0:0, which is the hba ESX sees my disk on.

I then removed the 1Mb drive from the OF, and added the new disk using the wizard.



The size of the disk should be exactly defined in the vmdk. Strange, but OF saw the whole raw device, even if the incorrect size was specified in the vmdk file.

To get the correct size, I logged in the OF, and did:



parted /dev/sdb

Then type:



print-fdisk



The result is:

Disk /dev/sdb: 10485.1 GB, 10485088911360 bytes

255 heads, 63 sectors/track, 1274739 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes



Device Boot Start End Blocks Id System

/dev/sdb1 1 1274739 3796889033+ 8e Linux LVM

(parted)



My guess is that the value 10485088911360 should be defined in the vmdk, because:

10485088911360/1024=10239344640 Kbytes

10239344640/1024=9999360 Mbytes

9999360/1024=9765Gbytes

9765/1024=9.5361328125 TBytes



And for the fun:

9.5361328125/7=1.3623046875 – The size of one of our 1.5Tb Disks…



I guess you have to do this whole stuff, as some data on the disk are generated by vmfstools – like a UUID, etc…

Just edit the file once the first command is completed, and change the line for the raw data size, which will become something like:



# Extent description

RW 10485088911360 VMFSRAW "/vmfs/devices/disks/vmhba1:0:0:0"

Note that the size I calculated is for my 8 x 1.5 Tb hard disks - having the formula might help people with disks of different size...

As usual, use at your own risk, but this is working quite OK for me so far, uploading to OF through Samba at ~35 Mbytes/sec on Gb Lan, and downloading at about 45 MBytes/sec.
 
Again, I have to highly suggest against doing this. Especially if shared with the ESX install. Good luck.
 
Hello,

Lopoetve, you suggest against doing this, but why exactly?
I wanted to host everything in one single machine, and due to vmfs limitations, there is not so much other choices than that...
I did a different setup first, not creating an array on my 5805, only a RAID10 on the 2405 - which currently hold the ESX and the VM vmdks - by creating 8 vmfs partitions, in which I created one disk per partition of the exact size I needed, then using soft raid in the openfiler to get one single 8.5 Tb volume.

I had a lot of issues getting some disks pulled off of the array, and when everything was working had bad performances compared to what I have now.

If you have another suggestion that will let me have a Raid 5, not loosing more space than I'm doing, and safer for any reason, I'd be glad to hear it...

Outside of getting a second machine for the OF, of course - more than 1 server at home is not "waf", not even talking about the environmental impact - more electricity consumption, more noise, etc...
And of course the price of getting a second case, a second PSU, a second CPU, some RAM, the NIC cards, etc...

So, any suggestion welcom - or an explanation of what the risks of this solution are...

Thanks for your feedback,
Felix
 
I did a quick post that disappeared, it seems...
Lopoetve, may I ask you why you recommend not going that way?

I tried to create 8 volumes from my 5805 in the past, with on each a vmfs and a vmdk of about the size of teh disk, then doing soft raid in OF, but had performance and stability issues - disks kicked off the array, about 50% less transfer speeds than now...

Without going for a second machine (cost, electricity consumption, noise, waf...) what would you suggest to do?
 
felix_clerc said:
Hello,

Lopoetve, you suggest against doing this, but why exactly?
I wanted to host everything in one single machine, and due to vmfs limitations, there is not so much other choices than that...
I did a different setup first, not creating an array on my 5805, only a RAID10 on the 2405 - which currently hold the ESX and the VM vmdks - by creating 8 vmfs partitions, in which I created one disk per partition of the exact size I needed, then using soft raid in the openfiler to get one single 8.5 Tb volume.

I had a lot of issues getting some disks pulled off of the array, and when everything was working had bad performances compared to what I have now.

If you have another suggestion that will let me have a Raid 5, not loosing more space than I'm doing, and safer for any reason, I'd be glad to hear it...

Outside of getting a second machine for the OF, of course - more than 1 server at home is not "waf", not even talking about the environmental impact - more electricity consumption, more noise, etc...
And of course the price of getting a second case, a second PSU, a second CPU, some RAM, the NIC cards, etc...

So, any suggestion welcom - or an explanation of what the risks of this solution are...

Thanks for your feedback,
Felix
Click to expand...

Reservation conflict issues, depending on what you have where, are the main ones I've seen and the main explanation I've seen. The local controllers don't play as well with the reservations, and I've seen a reservation conflict result in the entire local filesystem going RO because they used a partition on the controller that was shared with the boot volume, and it didn't handle the reservation correctly.

I'd combine them all in openfiler as a single, massive LUN, and then use an initiator inside of a virtual machine to access them as an iSCSI lun. This bypasses ESX entirely, and thus the 2TB limit, and the initiator performs better than ESX iSCSI stack anyway. Performance should be pretty decent too.
 
That's what I wanted to do, but ESX doesn't allow to create raw volumes straight on the 5805...

My understanding is that it miss a SCSI Id to allow scsi pass through...

Actually, the point is that it is on a separate adpter. I have a RAID10 which is used by ESX, with VMFS, all the VM's disks, etc.. and this big, Raid 5 array used by Openfiler only...
I admit I didn't filled more than 2 Tb on it yet, but created volumes of that size + a few others without any complaint from OF.
Also note tha fdisk on the ESX doesn't report the volume...
So, am I in a bad shape?

Sorry for the noobie questions - I just try to optimize the usage of my hardware, and am not a storage specialist (yet)

Thanks
 
Not sure if this has been asked yet or not, but do you forsee one share on the fileserver requiring more than 2TB?
 
Well,

I actually have one of 2 TB on the Openfiler...And I plan to make it grow!
So far, the OF's UI doesn't seem to prevent doing it, which doesn't mean at all it is feasible...
I lacked time to do tests these days, but I will try to create a 5 TB share (I still have room for that) and use dd in it to fill it up...
That would confirm (or infirm) the 2 TB barrier, and the 4 TB theorical limit on 32 bits OF.
I've been surpised as OF did see the whole volume and allowed me to do a LVM of 8.5 TB without having to create any partition...
I'll post my findings ;-)
 
Correct em if I'm missing something:

1.You plan to use openfiler as a VM.
2.The openfiler will be used as a file server.

Given the 2TB VMFS volume limit, I'd just carve up the raid volume at the controller level, 1.5 for VMs, then 2 1.5TB volumes for the shares. Present the 2 1.5 TB volumes to OF after the VM is created and create 2 shares totaling 3TB.
 
Geiger,

You are right, OF is a VM.
And yes, it will be used as a file server.

What I did is this: I have 2 adaptec raid controllers in the ESX server. The first one is a 2405, with 4 500Gb HD, in Raid 10. Esx is installed on this, the remaining space is formatted in VMFS and used to host teh boot disks of all my VMS.
The second adapter, a 5805, is having 8 HD of 1.5 TB, which in Raid 5 represent 8.5 Tb.

My OF does have a boot disk on the Raid 10 array, and the RDM to the whole 8.5TB as a second "disk".
And in that big "disk", I created a LVM of the whole size, and then volumes and shares.

Now I have to see wether or not I can use more than 2 TB on this big "disk", which I hope I'll have time to test in the coming days..
 
For thos interested, I've been able to create a 6TB volume in my OF, and fill it with 4 TB without any problem.
The only limitation so far is the max size of the files, limited to 2TB
Also note that I contacted Adaptec about this, and they are fine with the way I did it - I mean, theyr support think it is a workable solution, they do not foresee any issue with this solution...
 
I hate bumping a semi-dead thread, but on the other hand I hate finding threads that leave you without a conclusion.

I've finally fixed the stability issues with the mobo. Turns out that it was due to the USB flash drive I was using to boot off of. I tested 2 other flash drives from OCZ and Sandisk, and still the problems remained.

Landed up booting off a 640GB WD SAS Disk I had laying around (on the 5805) and it's been online for the last 3 weeks with zero problems.

Long story short, X7DCA-L and booting of USB for ESX = fail
 
fwiw, USB sticks for embedded are generally a special design, iirc. You were using installable on one?
 
You must log in or register to reply here.
Back
Top