ESXi autodeploy without vCenter?

[TCM]

Hi,

ESXi 5.1 supports stateless/stateless caching(?)/stateful autodeploy, i.e. the host boots via PXE and gets its complete ESXi + configuration from a vCenter server.

Now, I only run 1 ESXi host but I do run an extra DHCP, DNS and NTP server on a http://pcengines.ch/alix2d3.htm using NetBSD.

Now my question is, can ESXi be autodeployed "manually"? I suppose it's all just PXE and TFTP, is it? Also, the files are not big. They could fit on the 4GB CF card in the ALIX just fine.

Ideally, my ESXi would boot via PXE and the config would safely be stored on the embedded server. Is this possible or does autodeploy involve complicated communication with vCenter?

I'm asking because my (internally attached) USB key somehow became unreadable in just 60 days of runtime. Since it vanished from ESXi and I couldn't get it to reattach properly, I suppose all configuration I did since then is lost. I'm in the process right now of upgrading to ESXi 5.1 so I don't know the real damage it took. It's even a SanDisk and it barfed.

PXE boot would be a nice way to get rid of yet another failure-prone component.


Update 2013-01-17: posted working(?) method.

 

[itrsteve]

I'm sure it's possible with some pxelinux trickery, but the only method that I know of bypassing vCenter is pulling down the installer .ISO over PXE and then run kickstart scripts to configure the host. But that's an actual install and I realize that's not what you're asking.

Honestly, with only one host and all of those dependent services it seems like a hair bit of over designing.

If you do end up going with multiple hosts and vCenter down the road you'll find that Auto Deploy is extremely easy to setup and why you'll be hard pressed to find somebody that has taken the time to find some other solution.

 

[TCM]

I'll try to get it working. Maybe it's easy enough. I'll start by configuring autodeploy normally and looking at the resulting files.

 

[itrsteve]

I'll try to get it working. Maybe it's easy enough. I'll start by configuring autodeploy normally and looking at the resulting files.

The file (undionly.kpxe.vmw-hardwired) that you set in your TFTP boot options is not human readable (or at least I can't). This calls back to your Auto Deploy server, checks the active rule set for that server and gives it the correct image profile.

Once again, a ton of work that you're making for yourself.

I understand if you want to see how it works, the first time I set up Auto Deploy it was with virtual ESXi VM's in Fusion, makes it a little less painful because they reboot a little quicker Maybe you could do it on a VM?

 

[Child of Wonder]

Does your host have a network card that supports boot from iSCSI? Maybe that's a better option if you just want to eliminate local disk as a point of failure.

 

[TCM]

Yeah, iSCSI would be an option. I will evaluate which path I'm taking once vCenter runs.

But I was just trying to setup the vCenter appliance and all I get is this:

Failed to execute '/usr/sbin/vpxd_servicecfg db write embedded     CENSORED':
/usr/sbin/vpxd_commonutils: line 732:  4675 Segmentation fault      (core dumped) $VPXD "$@"
/usr/sbin/vpxd_commonutils: line 732:  4682 Segmentation fault      (core dumped) $VPXD "$@"
/usr/sbin/vpxd_commonutils: line 732:  5058 Segmentation fault      (core dumped) $VPXD "$@"
VC_DB_SCHEMA_VERSION=
VC_DB_SCHEMA_INITIALIZED=1
VC_CFG_RESULT=407(Error: Failed to initialize schema.)

when it's setting up the initial embedded database.

I was already panicking thinking my hardware is flawed, but it's reproducably the same error and other people are also getting it. One person suggested changing the password before logging in to the web interface. Didn't work.

Any idea?

 

[Nate7311]

Try a redownload of the appliance?

 

[TCM]

No, its SHA1 sum is 7c256bdbacd3331890f9df2a5975e7dcd98af6d0 on my storage, same as on vmware.com.

I deployed it 3 times already. Each time it's the same error. Another user with the same problem: http://communities.vmware.com/message/2114483

Edit: This is on ESXi 5.1. I already tried it on 5.0 before upgrading the host to 5.1. On 5.0 it ran fine.

Edit: The appliance's Linux kernel is throwing a lot of errors like: kernel: [...] vpxd[...] general protection ip:[...]

 

[TCM]

PXE booting ESXi is really simple as I found out.

First, I installed ESXi onto a USB key and applied ESXi510-201212001.zip using esxcli software profile install to see what ended up on the bootdevice and which files were different from the stock install.

Through testing I found out that the installed files don't differ from the files on the CD-ROM at all. They get repacked with gzip somehow but when you unpack them, they don't differ from their CD-ROM versions. There are some files that are different from the CD-ROM versions or that are created and don't exist on the CD-ROM, respectively.

These files are imgdb.tgz (differs), onetime.tgz (created), state.tgz (created) and useropts.gz (differs).

My TFTP root looks like this:

lrwxr-xr-x   1 root  wheel         22 Jan 17 08:26 esxi-boot -> esxi-boot-5.1.0-914609
drwxr-xr-x   2 root  wheel       2048 Jan 17 08:26 esxi-boot-5.1.0-838463
drwxr-xr-x   2 root  wheel       2048 Jan 17 08:26 esxi-boot-5.1.0-914609
lrwxr-xr-x   1 root  wheel         25 Jan 16 07:10 esxi-install -> esxi-install-5.1.0-838463
drwxr-xr-x   4 root  wheel       2048 Jan 17 08:03 esxi-install-5.1.0-838463
-rw-r--r--   1 root  wheel      16794 Apr  1  2010 pxelinux.bin

esxi-install-5.1.0-838463 contains the contents of VMware-VMvisor-Installer-201210001-838463.x86_64.iso

# ls -l esxi-install-5.1.0-838463/
total 612176
-rw-r--r--  1 root  wheel      11225 Oct 23 00:09 a.b00
-rw-r--r--  1 root  wheel      10682 Oct 23 00:09 ata_pata.v00
-rw-r--r--  1 root  wheel       7823 Oct 23 00:09 ata_pata.v01
-rw-r--r--  1 root  wheel       8077 Oct 23 00:09 ata_pata.v02
-rw-r--r--  1 root  wheel       8846 Oct 23 00:09 ata_pata.v03
-rw-r--r--  1 root  wheel      10188 Oct 23 00:09 ata_pata.v04
-rw-r--r--  1 root  wheel       8748 Oct 23 00:09 ata_pata.v05
-rw-r--r--  1 root  wheel       7968 Oct 23 00:09 ata_pata.v06
-rw-r--r--  1 root  wheel       9941 Oct 23 00:09 ata_pata.v07
-rw-r--r--  1 root  wheel      55284 Oct 23 00:09 b.b00
-rw-r--r--  1 root  wheel      26258 Oct 23 00:09 block_cc.v00
-rw-r--r--  1 root  wheel        512 Oct 23 00:09 boot.cat
-rw-r--r--  1 root  wheel       1366 Oct 23 00:09 boot.cfg
-rw-r--r--  1 root  wheel      21448 Oct 23 00:09 chardevs.b00
drwxr-xr-x  3 root  wheel        512 Jan 17 08:00 efi
-rw-r--r--  1 root  wheel    1048576 Oct 23 00:09 efiboot.img
-rw-r--r--  1 root  wheel      32977 Oct 23 00:09 ehci_ehc.v00
-rw-r--r--  1 root  wheel      80343 Oct 23 00:09 esx_dvfi.v00
-rw-r--r--  1 root  wheel     443421 Oct 23 00:09 ima_qla4.v00
-rw-r--r--  1 root  wheel      29219 Oct 23 00:09 imgdb.tgz
-rw-r--r--  1 root  wheel    3110218 Oct 23 00:09 imgpayld.tgz
-rw-r--r--  1 root  wheel      12351 Oct 23 00:09 ipmi_ipm.v00
-rw-r--r--  1 root  wheel      32007 Oct 23 00:09 ipmi_ipm.v01
-rw-r--r--  1 root  wheel      29819 Oct 23 00:09 ipmi_ipm.v02
-rw-r--r--  1 root  wheel      14336 Oct 23 00:09 isolinux.bin
-rw-r--r--  1 root  wheel        275 Oct 23 00:09 isolinux.cfg
-rw-r--r--  1 root  wheel    2646636 Oct 23 00:09 k.b00
-rw-r--r--  1 root  wheel      80008 Oct 23 00:09 mboot.c32
-rw-r--r--  1 root  wheel      53456 Oct 23 00:09 menu.c32
-rw-r--r--  1 root  wheel       7480 Oct 23 00:09 misc_cni.v00
-rw-r--r--  1 root  wheel    1039116 Oct 23 00:09 misc_dri.v00
-rw-r--r--  1 root  wheel      98548 Oct 23 00:09 net_be2n.v00
-rw-r--r--  1 root  wheel     169184 Oct 23 00:09 net_bnx2.v00
-rw-r--r--  1 root  wheel     793806 Oct 23 00:09 net_bnx2.v01
-rw-r--r--  1 root  wheel      37770 Oct 23 00:09 net_cnic.v00
-rw-r--r--  1 root  wheel      91926 Oct 23 00:09 net_e100.v00
-rw-r--r--  1 root  wheel      82884 Oct 23 00:09 net_e100.v01
-rw-r--r--  1 root  wheel      43752 Oct 23 00:09 net_enic.v00
-rw-r--r--  1 root  wheel      40389 Oct 23 00:09 net_forc.v00
-rw-r--r--  1 root  wheel      78829 Oct 23 00:09 net_igb.v00
-rw-r--r--  1 root  wheel     127723 Oct 23 00:09 net_ixgb.v00
-rw-r--r--  1 root  wheel    1944683 Oct 23 00:09 net_nx_n.v00
-rw-r--r--  1 root  wheel      36199 Oct 23 00:09 net_r816.v00
-rw-r--r--  1 root  wheel      25447 Oct 23 00:09 net_r816.v01
-rw-r--r--  1 root  wheel      77738 Oct 23 00:09 net_s2io.v00
-rw-r--r--  1 root  wheel      37079 Oct 23 00:09 net_sky2.v00
-rw-r--r--  1 root  wheel      93942 Oct 23 00:09 net_tg3.v00
-rw-r--r--  1 root  wheel      31761 Oct 23 00:09 net_vmxn.v00
-rw-r--r--  1 root  wheel      19705 Oct 23 00:09 ohci_usb.v00
-rw-r--r--  1 root  wheel  120212819 Oct 23 00:09 s.v00
-rw-r--r--  1 root  wheel      58144 Oct 23 00:09 safeboot.c32
-rw-r--r--  1 root  wheel      26810 Oct 23 00:09 sata_ahc.v00
-rw-r--r--  1 root  wheel      14945 Oct 23 00:09 sata_ata.v00
-rw-r--r--  1 root  wheel      24614 Oct 23 00:09 sata_sat.v00
-rw-r--r--  1 root  wheel      12298 Oct 23 00:09 sata_sat.v01
-rw-r--r--  1 root  wheel      12887 Oct 23 00:09 sata_sat.v02
-rw-r--r--  1 root  wheel      11509 Oct 23 00:09 sata_sat.v03
-rw-r--r--  1 root  wheel       8920 Oct 23 00:09 sata_sat.v04
-rw-r--r--  1 root  wheel      52647 Oct 23 00:09 scsi_aac.v00
-rw-r--r--  1 root  wheel     141176 Oct 23 00:09 scsi_adp.v00
-rw-r--r--  1 root  wheel      98729 Oct 23 00:09 scsi_aic.v00
-rw-r--r--  1 root  wheel      60000 Oct 23 00:09 scsi_bnx.v00
-rw-r--r--  1 root  wheel      46962 Oct 23 00:09 scsi_fni.v00
-rw-r--r--  1 root  wheel      55054 Oct 23 00:09 scsi_hps.v00
-rw-r--r--  1 root  wheel      39032 Oct 23 00:09 scsi_ips.v00
-rw-r--r--  1 root  wheel     414225 Oct 23 00:09 scsi_lpf.v00
-rw-r--r--  1 root  wheel      30451 Oct 23 00:09 scsi_meg.v00
-rw-r--r--  1 root  wheel      51146 Oct 23 00:09 scsi_meg.v01
-rw-r--r--  1 root  wheel      29204 Oct 23 00:09 scsi_meg.v02
-rw-r--r--  1 root  wheel     113461 Oct 23 00:09 scsi_mpt.v00
-rw-r--r--  1 root  wheel     156830 Oct 23 00:09 scsi_mpt.v01
-rw-r--r--  1 root  wheel     131622 Oct 23 00:09 scsi_mpt.v02
-rw-r--r--  1 root  wheel     670375 Oct 23 00:09 scsi_qla.v00
-rw-r--r--  1 root  wheel      80962 Oct 23 00:09 scsi_qla.v01
-rw-r--r--  1 root  wheel     184765 Oct 23 00:09 scsi_rst.v00
-rw-r--r--  1 root  wheel      42369 Oct 23 00:09 tboot.b00
-rw-r--r--  1 root  wheel  170835784 Oct 23 00:09 tools.t00
-rw-r--r--  1 root  wheel      19962 Oct 23 00:09 uhci_usb.v00
drwxr-xr-x  2 root  wheel        512 Jan 17 08:00 upgrade
-rw-r--r--  1 root  wheel     362626 Oct 23 00:09 user.b00
-rw-r--r--  1 root  wheel         20 Oct 23 00:09 useropts.gz
-rw-r--r--  1 root  wheel     812814 Oct 23 00:09 vmware-esx-base-osl.txt
-rw-r--r--  1 root  wheel       2180 Oct 23 00:09 vmware-esx-base-readme
-rw-r--r--  1 root  wheel    3135099 Oct 23 00:09 weaselin.t00
-rw-r--r--  1 root  wheel     748378 Oct 23 00:09 xlibs.v00
-rw-r--r--  1 root  wheel    1520470 Oct 23 00:09 xorg.v00

I linked all needed files from esxi-install-5.1.0-838463 into esxi-boot-5.1.0-914609. To find out which files are needed, I copied the boot.cfg from the installed ESXi on the USB key and looked at the modules= line. The files that get installed by applying the update I pulled from VMware_bootbank_esx-base_5.1.0-0.9.914609.vib inside ESXi510-201212001.zip.

The esxi-boot-5.1.0-914609 dir thus looks like this:

# ls -l esxi-boot-5.1.0-914609/
total 582900
lrwxr-xr-x  1 root  wheel         34 Jan 17 08:10 a.b00 -> ../esxi-install-5.1.0-838463/a.b00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ata_pata.v00 -> ../esxi-install-5.1.0-838463/ata_pata.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ata_pata.v01 -> ../esxi-install-5.1.0-838463/ata_pata.v01
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ata_pata.v02 -> ../esxi-install-5.1.0-838463/ata_pata.v02
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ata_pata.v03 -> ../esxi-install-5.1.0-838463/ata_pata.v03
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ata_pata.v04 -> ../esxi-install-5.1.0-838463/ata_pata.v04
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ata_pata.v05 -> ../esxi-install-5.1.0-838463/ata_pata.v05
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ata_pata.v06 -> ../esxi-install-5.1.0-838463/ata_pata.v06
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ata_pata.v07 -> ../esxi-install-5.1.0-838463/ata_pata.v07
-rw-r--r--  1 root  wheel      55284 Nov 18 12:44 b.b00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 block_cc.v00 -> ../esxi-install-5.1.0-838463/block_cc.v00
-rw-r--r--  1 root  wheel       2437 Jan 17 08:28 boot.cfg
-rw-r--r--  1 root  wheel       1270 Jan 17 07:32 boot.cfg.orig
-rw-r--r--  1 root  wheel      21449 Nov 18 12:44 chardevs.b00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ehci_ehc.v00 -> ../esxi-install-5.1.0-838463/ehci_ehc.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 esx_dvfi.v00 -> ../esxi-install-5.1.0-838463/esx_dvfi.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ima_qla4.v00 -> ../esxi-install-5.1.0-838463/ima_qla4.v00
lrwxr-xr-x  1 root  wheel         38 Jan 17 08:15 imgdb.tgz -> ../esxi-install-5.1.0-838463/imgdb.tgz
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ipmi_ipm.v00 -> ../esxi-install-5.1.0-838463/ipmi_ipm.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ipmi_ipm.v01 -> ../esxi-install-5.1.0-838463/ipmi_ipm.v01
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ipmi_ipm.v02 -> ../esxi-install-5.1.0-838463/ipmi_ipm.v02
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:26 isolinux.cfg -> ../esxi-install-5.1.0-838463/isolinux.cfg
-rw-r--r--  1 root  wheel    2647912 Nov 18 12:44 k.b00
-rw-r--r--  1 root  wheel      80008 Nov 18 12:44 mboot.c32
-rw-r--r--  1 root  wheel      53456 Nov 18 12:44 menu.c32
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 misc_cni.v00 -> ../esxi-install-5.1.0-838463/misc_cni.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 misc_dri.v00 -> ../esxi-install-5.1.0-838463/misc_dri.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_be2n.v00 -> ../esxi-install-5.1.0-838463/net_be2n.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_bnx2.v00 -> ../esxi-install-5.1.0-838463/net_bnx2.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_bnx2.v01 -> ../esxi-install-5.1.0-838463/net_bnx2.v01
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_cnic.v00 -> ../esxi-install-5.1.0-838463/net_cnic.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_e100.v00 -> ../esxi-install-5.1.0-838463/net_e100.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_e100.v01 -> ../esxi-install-5.1.0-838463/net_e100.v01
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_enic.v00 -> ../esxi-install-5.1.0-838463/net_enic.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_forc.v00 -> ../esxi-install-5.1.0-838463/net_forc.v00
lrwxr-xr-x  1 root  wheel         40 Jan 17 08:09 net_igb.v00 -> ../esxi-install-5.1.0-838463/net_igb.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_ixgb.v00 -> ../esxi-install-5.1.0-838463/net_ixgb.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_nx_n.v00 -> ../esxi-install-5.1.0-838463/net_nx_n.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_r816.v00 -> ../esxi-install-5.1.0-838463/net_r816.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_r816.v01 -> ../esxi-install-5.1.0-838463/net_r816.v01
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_s2io.v00 -> ../esxi-install-5.1.0-838463/net_s2io.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_sky2.v00 -> ../esxi-install-5.1.0-838463/net_sky2.v00
lrwxr-xr-x  1 root  wheel         40 Jan 17 08:09 net_tg3.v00 -> ../esxi-install-5.1.0-838463/net_tg3.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 net_vmxn.v00 -> ../esxi-install-5.1.0-838463/net_vmxn.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 ohci_usb.v00 -> ../esxi-install-5.1.0-838463/ohci_usb.v00
-rw-r--r--  1 root  wheel  120167603 Nov 18 12:44 s.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 sata_ahc.v00 -> ../esxi-install-5.1.0-838463/sata_ahc.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 sata_ata.v00 -> ../esxi-install-5.1.0-838463/sata_ata.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 sata_sat.v00 -> ../esxi-install-5.1.0-838463/sata_sat.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 sata_sat.v01 -> ../esxi-install-5.1.0-838463/sata_sat.v01
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 sata_sat.v02 -> ../esxi-install-5.1.0-838463/sata_sat.v02
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 sata_sat.v03 -> ../esxi-install-5.1.0-838463/sata_sat.v03
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 sata_sat.v04 -> ../esxi-install-5.1.0-838463/sata_sat.v04
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_aac.v00 -> ../esxi-install-5.1.0-838463/scsi_aac.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_adp.v00 -> ../esxi-install-5.1.0-838463/scsi_adp.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_aic.v00 -> ../esxi-install-5.1.0-838463/scsi_aic.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_bnx.v00 -> ../esxi-install-5.1.0-838463/scsi_bnx.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_fni.v00 -> ../esxi-install-5.1.0-838463/scsi_fni.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_hps.v00 -> ../esxi-install-5.1.0-838463/scsi_hps.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_ips.v00 -> ../esxi-install-5.1.0-838463/scsi_ips.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_lpf.v00 -> ../esxi-install-5.1.0-838463/scsi_lpf.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_meg.v00 -> ../esxi-install-5.1.0-838463/scsi_meg.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_meg.v01 -> ../esxi-install-5.1.0-838463/scsi_meg.v01
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_meg.v02 -> ../esxi-install-5.1.0-838463/scsi_meg.v02
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_mpt.v00 -> ../esxi-install-5.1.0-838463/scsi_mpt.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_mpt.v01 -> ../esxi-install-5.1.0-838463/scsi_mpt.v01
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_mpt.v02 -> ../esxi-install-5.1.0-838463/scsi_mpt.v02
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_qla.v00 -> ../esxi-install-5.1.0-838463/scsi_qla.v00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_qla.v01 -> ../esxi-install-5.1.0-838463/scsi_qla.v01
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 scsi_rst.v00 -> ../esxi-install-5.1.0-838463/scsi_rst.v00
lrwxr-xr-x  1 root  wheel         38 Jan 17 08:10 tboot.b00 -> ../esxi-install-5.1.0-838463/tboot.b00
-rw-r--r--  1 root  wheel  171699644 Nov 18 12:40 tools.t00
lrwxr-xr-x  1 root  wheel         41 Jan 17 08:09 uhci_usb.v00 -> ../esxi-install-5.1.0-838463/uhci_usb.v00
-rw-r--r--  1 root  wheel     362636 Nov 18 12:44 user.b00
-rw-r--r--  1 root  wheel         20 Nov 18 12:44 useropts.gz
-rw-r--r--  1 root  wheel    3137956 Nov 18 12:44 weaselin.t00
lrwxr-xr-x  1 root  wheel         38 Jan 17 08:09 xlibs.v00 -> ../esxi-install-5.1.0-838463/xlibs.v00
lrwxr-xr-x  1 root  wheel         37 Jan 17 08:09 xorg.v00 -> ../esxi-install-5.1.0-838463/xorg.v00

The files that are not links are the ones that changed through the update and which I pulled from VMware_bootbank_esx-base_5.1.0-0.9.914609.vib inside ESXi510-201212001.zip.

The boot.cfg looks like this:

# cat boot.cfg
bootstate=0
kernel=tboot.b00
kernelopt=no-auto-partition
#modules=b.b00 --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- s.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- weaselin.t00 --- esx_dvfi.v00 --- xlibs.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- misc_cni.v00 --- misc_dri.v00 --- net_be2n.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_nx_n.v00 --- net_r816.v00 --- net_r816.v01 --- net_s2io.v00 --- net_sky2.v00 --- net_tg3.v00 --- net_vmxn.v00 --- ohci_usb.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_lpf.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- scsi_qla.v01 --- scsi_rst.v00 --- uhci_usb.v00 --- xorg.v00 --- imgdb.tgz --- state.tgz
modules=b.b00 --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- s.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- weaselin.t00 --- esx_dvfi.v00 --- xlibs.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- misc_cni.v00 --- misc_dri.v00 --- net_be2n.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_nx_n.v00 --- net_r816.v00 --- net_r816.v01 --- net_s2io.v00 --- net_sky2.v00 --- net_tg3.v00 --- net_vmxn.v00 --- ohci_usb.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_lpf.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- scsi_qla.v01 --- scsi_rst.v00 --- uhci_usb.v00 --- xorg.v00 --- imgdb.tgz
build=5.1.0-0.9.914609
updated=2

Relevant entries in ISC DHCPD config:

# PXELINUX options, taken from http://syslinux.zytor.com/wiki/index.php/PXELINUX
option space pxelinux;
option pxelinux.magic                   code 208 = string;
option pxelinux.configfile              code 209 = text;
option pxelinux.pathprefix              code 210 = text;
option pxelinux.reboottime              code 211 = unsigned integer 32;

host esxi {
        vendor-option-space pxelinux;
        option pxelinux.magic = "F1:00:74:7E";

        filename "pxelinux.bin";
        option pxelinux.pathprefix "/esxi-boot/";
        option pxelinux.configfile = "isolinux.cfg";
}

It's not a complete config. I assume you know your way around in ISC DHCPD. PXELINUX pulls isolinux.cfg which in turn loads boot.cfg. All paths including isolinux.cfg are relative to /esxi-boot/.

Now, this allows for booting into a working (but unconfigured!) ESXi host from stock files without doing any install. Note how I had to remove the state.tgz from the modules= line because that doesn't exist unless you go through an install.

Once the host is running, you can configure it and pull the state.tgz using vicfg-cfgbackup.pl which you can then put on the TFTP server and include in the modules= line.

Also note that no root password is set and that the 4 files I mentioned (imgdb.tgz, onetime.tgz, state.tgz and useropts.gz) are in their stock version (imgdb.tgz, useropts.gz) or don't exist (onetime.tgz, state.tgz). I'm not sure yet what their exact effect on the config is. onetime.tgz is some config that's applied once during boot and then removed. It's not included in the boot.cfg that I pulled from the USB key.

imgdb.tgz contains some .xml files related to the installed profile.

state.tgz is obviously the main config incl. root password so once you changed the config, pulled state.tgz using vicfg-cfgbackup.pl and put it on the TFTP server, you should have a semi-stateful config, i.e. you can choose when you want the config saved and all changes after that are stateless. This allows for tinkering with unknown settings without messing up the host because all you have to do is reboot to get your old state.

Edit: It looks like imgdb.tgz contains the install date in every .xml file after an install. I'm not sure what happens when that info isn't there. So far I haven't noticed any oddities running an "uninstalled" PXE boot.