Deciding how to boot ESXi *pics*

I

idea

Gawd
Joined
Jan 24, 2005
Messages
615
I'm trying to decide how I will boot my ESXi whitebox. I definitely want to boot from a flash drive. I don't want any USB drives sticking out. I want something internal. Maybe a flash drive that plugs directly into the USB headers or a CF-to-SATA adapter with a PCI bracket. Anyone have any tips? There are some cheap parts available on newegg and monoprice that are pretty interesting looking. Such as those below:

Adapter for CF-to-SATA, mounted on a PCI bracket
12-186-061-S02


Adapter for USB drive to plug directly into the mobo headers:
12-200-474-S03
 
are you really against just using a normal internal sata drive to boot it? I like keeping things simple, and I am not seeing a real reason to go to this way, unless I am missing something.
 
AceXsmurF said:
are you really against just using a normal internal sata drive to boot it? I like keeping things simple, and I am not seeing a real reason to go to this way, unless I am missing something.
Click to expand...

Yes, I am against it. I also like to keep things simple. Hear me out: running on an internal drive will require 2 drives in RAID1, since it is an OS we are talking about here. It takes up two slots in my mid-tower case, uses more electricity, and generates more heat. Not to mention it's a waste of money to spend on wasted gigabytes. ESXi only needs a cheap 1GB flash drive.
 
what about those InnoDisk IDE Embedded flash modules or USB pin header embedded flash modules?
 
so youd use 2x sata drives in raid1 for a low useage OS install of ESX, but youd have no problems installing ESX onto a plain, single thumb drive?
Ive had thumb drives fail at random, one while using it as well.

Why not get a small 30/40gb SSD and use the rest of the space to store ISO's to use when installing your VM's?
 
relatively unrelated: IBM sold several machines with an internal USB on the motherboard, and charged people $500 to get the enbedded version of ESXi preloaded on the machine (USB thumbdrive, internal) so that they could just power it on when they go it, and go. The problem? The USB key was serialized to the motherboard, and had embedded code to check one against the other. If the numbers didnt match... no boot. So what happened when the board or the USB key died? BOTH had to be replaced and re-serialized. Can you say 3 day downtime? Nice, eh?

It's ESXi, and if it's for personal use, I'd say that you're better off going with the one that has the faster throughput, which would be the CF solution. Keep in mind that if it's ESXi 4 though, you'll need at least 10GB for the installer to not crash. Better get a 16gb card.
 
dbwillis said:
so youd use 2x sata drives in raid1 for a low useage OS install of ESX, but youd have no problems installing ESX onto a plain, single thumb drive?
Ive had thumb drives fail at random, one while using it as well.

Why not get a small 30/40gb SSD and use the rest of the space to store ISO's to use when installing your VM's?
Click to expand...

Well I wasn't planning on a "plain thumb drive." I was planning on a more industrial/enterprise solution inspired by this thread, but I certainly see your point.

SSD's are still too expensive even at that size.
 
Mr.Pernod said:
http://www.innodisk.com/flashstorage.jsp
have a look at the EDC Series (IDE interface) or the USB EDC
Click to expand...

Nice! 1GB for $30

USB EDC(V)
USB%20EDC%28V%29.jpg

Features
. Small and flexible size
. Support capacity up to 4GB (vertical)
. Reliable data transfer performance
. RoHS compliance
Specifications
Capacity 512MB~4GB
interface Hi-Speed USB2.0
Data Transfer Rate Read : 26MB/sec.
Write : 12MB/sec.
Operation Temp. 0°C~+70°C
Storage Temp. -55°C~+95°C
DC Input Voltage +5V single power supply operation
Power Mode Auto stand-by mode
Power Consumption 120mA (max.)
Dimension 15.2 x 34.1 x 6.4 mm(WxLxH)
Click to expand...
 
idea said:
Yes, I am against it. I also like to keep things simple. Hear me out: running on an internal drive will require 2 drives in RAID1, since it is an OS we are talking about here. It takes up two slots in my mid-tower case, uses more electricity, and generates more heat. Not to mention it's a waste of money to spend on wasted gigabytes. ESXi only needs a cheap 1GB flash drive.
Click to expand...

No, it needs a NICE 1gb flash card, or you'll overload the card's bulk transfer allowance, corrupt the card, and crash ESXi.

Don't expect you'll get away with a $5 special from Wal-mart - you'll have problems eventually with the number of writes that i does. :) Just as a FYI for everyone.
 
lopo, dont you mean a NICE 10GB flash card? ESXi cant install into a 1GB device, as far as I know.
 
We just ordered some flash drives from HP to use on our DL380s. The older ones have an internal usb port and I think the newer ones have SD slots to install on. ESXi works great on a flash drive and you save all of your disk space for local vmfs or local SAN storage.
 
keep in mind though that if your boot device is flash based like this, and the partitions are set to all be on this type of device, you need to keep enough free space for swap, and also to install host updates. failure to plan for these will lead to major headaches down the road.
 
I was going to get some SATA DOM's but opted for some Kingston 30GB SSD's to boot my XenServer nodes. Granted @ $90 it was not as cheap as some of the USB stuff. But still, a cheaper low space SSD outperforms one of those USB devices for read/write. I see an OCZ 32GB model on newegg for $75 @ 125/70MB/s read/write. I'm sure if you look around you can find something and just stick with SATA for the same price and less messing around.

Just a thought :)
 
I'm setting up two ESXi whiteboxes at home. I'm simply using a 80GB 5400RPM 2.5" hard drive in each. The VMs will reside on an iSCSI SAN.

Even if a drive fails the VMs will reboot on the other host and it can handle things for the week or two until a replacement drive arrives.

The laptop hard drives are low power and create little heat. They're also more reliable than a flash drive or CF.
 
So, you're saying a spinning magnetic disk is more reliable then flash? Just want to make sure we're clear.
 
ND40oz said:
So, you're saying a spinning magnetic disk is more reliable then flash? Just want to make sure we're clear.
Click to expand...

For installing an OS, I'd rather use a traditional hard drive. However, I also live with two kids who might be inclined to see a thumb drive and remove it. Also, if there are logs being written to a flash drive you're asking for trouble.

ESXi, however, is different since the files are fairly static and logs are stored in a memory file system. I'd advise having a syslog set up to forward those logs to in the case of a crash. A reboot will erase the logs stored in the memory file system.

So, in the case of ESXi, I shouldn't have said a hard drive would be more reliable. For a traditional OS install, they would be because of flash degradation caused by write activity.

However, I chose hard drives for my setup so I can have a local datastore on each host for storing iso images and templates for rapid deployment.
 
Yup, just wanted to make sure you knew we were talking about ESXi and how it uses the visorfs partition.
 
Do you already have the machine? A lot of Intel/ Supermicro and etc boards that are compatible with ESXi have internal USB ports these days.
 
The problem with embedded is that there's no hardware fault tolerance unless you specifically design an embedded system to have such tolerance. When the choice is between 2 spinning disks and one usb/cf card, I'd go with the disks as well.

None of this applies if you are talking about some non-essential home setup where it doesn't matter what happens to the box.
 
Thuleman said:
The problem with embedded is that there's no hardware fault tolerance unless you specifically design an embedded system to have such tolerance. When the choice is between 2 spinning disks and one usb/cf card, I'd go with the disks as well.

None of this applies if you are talking about some non-essential home setup where it doesn't matter what happens to the box.
Click to expand...

Yeah, but what if you're talking about running with HA and then using Fault Tolerance on your essential boxes?
 
ND40oz said:
Yeah, but what if you're talking about running with HA and then using Fault Tolerance on your essential boxes?
Click to expand...

If a host goes down HA will only boot the VMs on another host. There will still be a period of downtime.

Fault Tolerance has so many restrictions not many want to use it.

For an extra couple hundred bucks per host why not run a mirrored set of hard drives in an Enterprise environment to add another layer of protection against VM downtime?
 
If I have something that has that high of an uptime that it can't go down, I'll use fault tolerance, that's why it's there. Sure you can't thin provision, but hey, it'll always be up.

HA can take care of everything else that can take a blip.

What running off of flash does allow me to do is fully populate a machine with disk and use all of that disk as a virtual SAN, if I went with an ESX install and used 2 of the disks just for the OS, I'd lose 600 gigs of capacity.
 
ND40oz said:
If I have something that has that high of an uptime that it can't go down, I'll use fault tolerance, that's why it's there. Sure you can't thin provision, but hey, it'll always be up.

HA can take care of everything else that can take a blip.

What running off of flash does allow me to do is fully populate a machine with disk and use all of that disk as a virtual SAN, if I went with an ESX install and used 2 of the disks just for the OS, I'd lose 600 gigs of capacity.
Click to expand...

You also cannot use vmxnet3, vSMP, VCB backups, DRS, PVSCSI, and must have a Gb NIC solely dedicated to FT logging.

And who cares about losing local storage? You can't use FT on local storage anyway.

Whether flash or traditional hard drives, having a pair of storage devices mirrored for the OS is ideal.
 
So what if you lose those things, they're not essential. If you have essential boxes that have a 100% uptime requirement, fault tolerance is great. Otherwise, you need to do clustering on the OS side.

As for local storage, I use it on my DMZ VMs, keeps the environment isolated which is what is required. Don't need 100% uptime on them, but with mirrored virtual SANs and HA, a box isn't going to be down long if we do have a failure and I don't have to dedicate a SAN to it.
 
FYI, we were (I stress the word were) booting two ESXi hosts off internal USB flash drives provided by IBM corporate.

Both ultimately exhibited very strange problems and error logs. We ended up having to install local SAS drives after the IBM VMware support team specifically told us they've had nothing but trouble trying to run ESXi off any sort of flash media. We're not booting off 2-drive RAID 1 arrays.
 
ND40oz said:
So what if you lose those things, they're not essential. If you have essential boxes that have a 100% uptime requirement, fault tolerance is great. Otherwise, you need to do clustering on the OS side.

As for local storage, I use it on my DMZ VMs, keeps the environment isolated which is what is required. Don't need 100% uptime on them, but with mirrored virtual SANs and HA, a box isn't going to be down long if we do have a failure and I don't have to dedicate a SAN to it.
Click to expand...

To each his own. I'm of the opinion that I'll take steps to avoid downtime whenever possible and a pair of discs in RAID 1 is one of those steps.

FT is nice but if you're willing to spend the money on more NICs (since you'll want two NICs assigned to the FT port group if it needs 100% uptime) and forgo important features like the ability to do VCB based backups then why not spend the little bit of extra money to use a mirrored disc set for the OS?
 
We run ESXI on 1-2GB external USB flash drives... Nothing fancy, but good for hosting Proof of Concept VMs...
They run great connected to the Dell D830 laptops... Again, not for production use.

Our production box hosting unimportant/not heavily used VMs does have ESXi on a 2GB flash drive... I have a 2nd flash drive connected that has a backup of the boot flash drive. So when it dies, I can reboot to the new flash drive and only be down for a short while.

We've been running this setup for about 4-6 months and have had no issues. But we're not really stressing much and I've yet to sit down and do anything major like HA or VLANs...
I'd probably stick ESXi on a 2.5in sata/sas drive for real production use depending on how I plan for a node failure... It's all about how you handle the box failing. If another node can step in quickly, then you don't need the mirrored drives. If you don't, you'll probably want the mirrored drives. It's all about cost and downtime...

oh, at home I run esxi off micro sd cards, because I have a multi card reader in the floppy bay and it was cheaper than compactflash cards. It's small enough that it's not easy to just break off or yank out, and cheap enough to buy extras to have on hand for when it fails. homebuilt cheap esxi whitebox... finalized the setup on the flash drive, cloned it to another, stored nearby...
 
Last edited:
ND40oz said:
So, you're saying we shouldn't be running embedded then?
Click to expand...

Whitebox? Installable.
Custom system with embedded built in? Embedded.
Standard server? Installable.

Spinning drives are still more mature than flash. It's getting there, but if we're talking enterprise - go with what's most proven unless you've got a special setup that only runs embedded.
 
Child of Wonder said:
...FT is nice but if you're willing to spend the money on more NICs (since you'll want two NICs assigned to the FT port group if it needs 100% uptime) and forgo important features like the ability to do VCB based backups then why not spend the little bit of extra money to use a mirrored disc set for the OS?
Click to expand...

Go big and get 4 10GB NICs and the infrastructure to go along with it. You may get by with just two 10GB NICS.

kumquat said:
Both ultimately exhibited very strange problems and error logs. We ended up having to install local SAS drives after the IBM VMware support team specifically told us they've had nothing but trouble trying to run ESXi off any sort of flash media. We're not booting off 2-drive RAID 1 arrays.
Click to expand...

I wonder if Dell is also having troupble on their Proliant R910 and R810s with the RAID 1 SD Flash embedded ESX models...?


Bookmage said:
oh, at home I run esxi off micro sd cards, because I have a multi card reader in the floppy bay and it was cheaper than compactflash cards. It's small enough that it's not easy to just break off or yank out, and cheap enough to buy extras to have on hand for when it fails. homebuilt cheap esxi whitebox... finalized the setup on the flash drive, cloned it to another, stored nearby...
Click to expand...

Could you explain more about how you are setup with this feature. Is a laptop the host for the embedded microSD card?

I thought ESXi needed more space than 2GB to run.
 
You must log in or register to reply here.
Back
Top