ESXI 5.1 crashs

C

coldice

n00b
Joined
Oct 28, 2013
Messages
3
Hello,
my whitebox ESXi 5.1 keeps crashing, not daily but with a week between each crash.

Here are the log files I collected each day:
https://dl.dropboxusercontent.com/u/2088720/ESXi_Crash_23-10-2013.zip - Crashed 23-10-2013, not sure why...

https://dl.dropboxusercontent.com/u/2088720/ESXi_Crash_28-10-2013.zip - Crashed 28-10-2013, not sure why...

Anyone that could help me out?
If there is need of more logs or anyhthing else, please specific..

Guest OS's running daily basis:
Win 2008r2 server
CentOS server
Open Indiana (Napp-IT) ZFS

(I only had these VM's powered on up on crashing each time)
 
What is the hardware in your whitebox?

I had an issue with my old whitebox machine pink-screening. It was a ASRock Z77, 3770 (non-k) and 32GB ram. Each of the pieces worked fine separately but when running ESXi 5.1 it would randomly pink-screen.

I ended up upgrading to a E3-1230V2/Supermicro Mobo/32GB ECC and it has been flawless since.
 
AMD FX-8320
Gigabyte GA-970A-D3
Kingston DDR3 HyperX 1600 MHz 16GB Black x 2
LSI 8-PORT INT 6GB/S SATA+SAS PCIE 2.0 HBA (ZFS, should be VMware HCL OK!)

Some Intel NIC (it's checked with VMware HCL)

EDIT:
I've had this server since may, it's been crashing lately. I'm going to upgrade to 5.5 this weekend maybe, I'll see if still crashes then :)
 
Last edited:
Is it the case that this server used to be stable and is now crashing? Or was it never stable?
 
Hey Lop its just syslog files no cores from the DLs...

Do you happen to have the coredumps? pictures of the PSOD?
 
Last edited:
yeah, I noticed. We need the cores or the PSOD at the very least (prefer both)
 
Hey uh, not to piggy back here, but I think I have a CPU on the fritz in one of my nodes. The console just stops responding - the ESX default screen (F2/F12 screen) stays up, but nothing happens when I try to access it - just stays gray scale. vCenter reports that the host is unresponsive, so I know it's hanging itself (host presently is removed from cluster, put "in the corner" in it's own DC with 0 guests).
I've ZIPed up all of the logs - I think - and dropped them here:
http://resting-blade.com/esx4.zip

Can I ask a kind soul to take a look at these and lend me a hand?
 
Eulogy said:
Hey uh, not to piggy back here, but I think I have a CPU on the fritz in one of my nodes. The console just stops responding - the ESX default screen (F2/F12 screen) stays up, but nothing happens when I try to access it - just stays gray scale. vCenter reports that the host is unresponsive, so I know it's hanging itself (host presently is removed from cluster, put "in the corner" in it's own DC with 0 guests).
I've ZIPed up all of the logs - I think - and dropped them here:
http://resting-blade.com/esx4.zip

Can I ask a kind soul to take a look at these and lend me a hand?
Click to expand...

2013-10-28T17:45:24.582Z cpu9:33353)ALERT: NMI: 709: NMI IPI received. Was eip(base):ebp:cs [0x9be8c(0x418019a00000):0x41238925daf0:0x4010](Src 0x1, CPU9)
2013-10-28T17:45:24.582Z cpu6:34927)ALERT: NMI: 709: NMI IPI received. Was eip(base):ebp:cs [0x148b7(0x418019a00000):0x4123a1bdd840:0x4010](Src 0x1, CPU6)
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925daf0:[0x418019a9be8c]SP_WaitLock@vmkernel#nover+0xec stack: 0x41238925db30
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925db50:[0x418019ff05ff]UserTeletypeWriteBuffer@<None>#<None>+0x393 stack: 0x41238925dc70
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dc10:[0x418019ff083c]UserTeletypeWriteInt@<None>#<None>+0x1b4 stack: 0x41238925dc3c
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dc50:[0x418019f6f047]TTYInputCallback@chardevs#1+0x5b stack: 0x3
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdd840:[0x418019a148b7]Interrupts_Enable@vmkernel#nover+0xb stack: 0x5
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dcd0:[0x418019aa2453]TermReceiveInput@vmkernel#nover+0x243 stack: 0x41238925dd50
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdd8f0:[0x418019dad654]SVGAConsoleDraw32@vmkernel#nover+0x274 stack: 0x417fd9c20c01
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dce0:[0x418019d98f1f]KeyboardBH@vmkernel#nover+0x73 stack: 0x23d27c93cb7
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdd930:[0x418019daec4d]SVGAConsolePuts@vmkernel#nover+0x151 stack: 0x4
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dd50:[0x418019a2e5b9]BH_Check@vmkernel#nover+0xc9 stack: 0x410977d65cc0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdd9e0:[0x418019aa1820]Term_Putb@vmkernel#nover+0x4c stack: 0x4108efd202b8
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925deb0:[0x418019c53eb7]CpuSchedDispatch@vmkernel#nover+0xbb7 stack: 0x41238925df00
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdda00:[0x418019f6f528]TTYWriteToTerm@chardevs#1+0x24 stack: 0x417fd9d2b2e0
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925df20:[0x418019c55c65]CpuSchedWait@vmkernel#nover+0x245 stack: 0x0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdda60:[0x418019ff032e]UserTeletypeWriteBuffer@<None>#<None>+0xc2 stack: 0x4123a1bddac0
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dfd0:[0x418019a27d26]VmkTimerQueueWorldFunc@vmkernel#nover+0x542 stack: 0x0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddb20:[0x418019ff083c]UserTeletypeWriteInt@<None>#<None>+0x1b4 stack: 0x412e800dc2c0
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dff0:[0x418019c53242]CpuSched_StartWorld@vmkernel#nover+0xfa stack: 0x0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddb40:[0x418019f6ed5d]TTY_Write@chardevs#1+0x3d stack: 0x41097f827080
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddb70:[0x418019cfbb2d]CharDriverAsyncIO@vmkernel#nover+0x91 stack: 0x4109747a1a30
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddc60:[0x418019cf61ae]FDS_AsyncIO@vmkernel#nover+0x3a2 stack: 0x190
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddcb0:[0x418019cf6810]FDS_DoSyncIO@vmkernel#nover+0xd4 stack: 0x4108da99d8d0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddd20:[0x418019cefaa6]DevFSFileIO@vmkernel#nover+0x24e stack: 0x4123a1bdde7c
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddd90:[0x418019fe8def]UserChardevIO@<None>#<None>+0xe7 stack: 0x4123a1bdde7c
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdddb0:[0x418019fe8e7a]UserChardevWrite@<None>#<None>+0x26 stack: 0x41085f334860
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdde20:[0x418019fe726b]UserVmfs_Writev@<None>#<None>+0x123 stack: 0x4123a1bdde60
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddea0:[0x418019fa1390]LinuxFileDesc_Write@<None>#<None>+0xd4 stack: 0x4123a1bddf00
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddeb0:[0x418019f7b632]Linux32_Write@<None>#<None>+0xe stack: 0x4123a1bddf00
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddf00:[0x418019f7a316]User_LinuxSyscallHandler@<None>#<None>+0x3f6 stack: 0x4123a1bddf20
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddf10:[0x418019aa9b4d]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0xfffd9374
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddf20:[0x418019af1064]gate_entry@vmkernel#nover+0x64 stack: 0x0
2013-10-28T17:48:19.949Z cpu4:33023)usb storage warning (0 throttled) on vmhba32 (SCSI cmd READ_10): command_abort called
usb storage message on vmhba32: scsi cmd done, result=0x0
usb storage message on vmhba32: Bulk command transfer result=0
usb storage message on vmhba32: Bulk data transfer result 0x0
2013-10-28T17:48:19.949Z cpu4:33023)usb storage warning (0 throttled) on vmhba32 (SCSI cmd READ_10): usb_stor_stop_transport called

Talk to me about OS i think i saw ESXi 5.5 are u running on the C6100 Hardware? vSAN? Network Adapters?

There are options on what the issue could be but thats my starting foundation!
 
Last edited:
dasaint said:
2013-10-28T17:48:57.860Z cpu1:34232)WARNING: Heartbeat: 785: PCPU 7 didn't have a heartbeat for 7 seconds; *may* be locked up.
2013-10-28T17:48:57.860Z cpu7:34927)ALERT: NMI: 709: NMI IPI received. Was eip(base):ebp:cs [0xa1811(0x418019a00000):0x4123a1bdd9e0:0x4010](Src 0x1, CPU7)

Talk to me about OS i think i saw ESXi 5.5 are u running on the C6100 Hardware?

There are options on what the issue could be but thats my starting foundation!
Click to expand...
Yeah, this is on a Dell C6100. Three nodes function perfectly fine, this 4th one is having issues. This is after a recent 5.1 to 5.5 upgrade. When on 5.1 it ran perfectly fine, even with a full load. All other nodes are running OK on 5.5. I tried a new thumb drive as well just to be sure it wasn't that or the install itself, but no dice. Generally locks up anywhere from 2 to 40 hours after booting, with or without load. The "PCPU didn't have a heartbeat for 7 seconds" is what made me think PCPU is going south, but I'm not certain. I don't need this node, just would like to have it... running on 3 will be fine for my home lab.
ESX is installed to a USB thumb drive, plugged into the back. VM storage is all NAS to a whitebox NAS (NAS is responsive - it serves quite a few functions without interruptions). All NICs are Intel NICs - most are dual port cards on the HCL.
 
Last edited:
Sorry updated my previous info...

vSan? Antivirus? What are the boxes being used for, just trying to understand more about the configs setup on it since i dont have that file to work with... usually a support bundle helps out
 
dasaint said:
Sorry updated my previous info...

vSan? Antivirus? What are the boxes being used for, just trying to understand more about the configs setup on it since i dont have that file to work with... usually a support bundle helps out
Click to expand...
No vSAN at all. No AV on the VMs - they're isolated from the outside world. Most VMs are Server 2012, some 2012R2, and I think I have one 2008R2 still left. Each VM node is in a cluster, sans this 4th one. DRS, HA are enabled. I have one VM setup with FT as well. I think in total I have somewhere around 30 VMs powered on, 40-45 in total.

I can get more logs or whatever, I'm just not sure what people need or how to get it for them. That zip is just the log files I can easily pull.
 
lopoetve said:
yeah, I noticed. We need the cores or the PSOD at the very least (prefer both)
Click to expand...

Thank you for reply, where do I get "core"-logs og PSOD-logs?


Also, I don't appreciate people hijacking my thread even tho it's same purpose. :p
 
You can both generate logs with the KBs below, if using ESXi you might not have enough space for it to dump the kernel to file (we have a software vSphere esxi dump collector thats free (on vCenter disk) and would need configured if it cannot dump the core... try running the dumps below and setting those files up for us...

In most cases also adding syslog files is a plus in troubleshooting as the host wont grab the syslog files (just for your reference)

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=653

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1010705
 
dasaint said:
You can both generate logs with the KBs below, if using ESXi you might not have enough space for it to dump the kernel to file (we have a software vSphere esxi dump collector thats free (on vCenter disk) and would need configured if it cannot dump the core... try running the dumps below and setting those files up for us...

In most cases also adding syslog files is a plus in troubleshooting as the host wont grab the syslog files (just for your reference)

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=653

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=1010705
Click to expand...
I'll do that and begin my own thread afterward since OP doesn't like piggyback rides :D.
 
dasaint said:
2013-10-28T17:45:24.582Z cpu9:33353)ALERT: NMI: 709: NMI IPI received. Was eip(base):ebp:cs [0x9be8c(0x418019a00000):0x41238925daf0:0x4010](Src 0x1, CPU9)
2013-10-28T17:45:24.582Z cpu6:34927)ALERT: NMI: 709: NMI IPI received. Was eip(base):ebp:cs [0x148b7(0x418019a00000):0x4123a1bdd840:0x4010](Src 0x1, CPU6)
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925daf0:[0x418019a9be8c]SP_WaitLock@vmkernel#nover+0xec stack: 0x41238925db30
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925db50:[0x418019ff05ff]UserTeletypeWriteBuffer@<None>#<None>+0x393 stack: 0x41238925dc70
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dc10:[0x418019ff083c]UserTeletypeWriteInt@<None>#<None>+0x1b4 stack: 0x41238925dc3c
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dc50:[0x418019f6f047]TTYInputCallback@chardevs#1+0x5b stack: 0x3
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdd840:[0x418019a148b7]Interrupts_Enable@vmkernel#nover+0xb stack: 0x5
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dcd0:[0x418019aa2453]TermReceiveInput@vmkernel#nover+0x243 stack: 0x41238925dd50
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdd8f0:[0x418019dad654]SVGAConsoleDraw32@vmkernel#nover+0x274 stack: 0x417fd9c20c01
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dce0:[0x418019d98f1f]KeyboardBH@vmkernel#nover+0x73 stack: 0x23d27c93cb7
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdd930:[0x418019daec4d]SVGAConsolePuts@vmkernel#nover+0x151 stack: 0x4
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dd50:[0x418019a2e5b9]BH_Check@vmkernel#nover+0xc9 stack: 0x410977d65cc0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdd9e0:[0x418019aa1820]Term_Putb@vmkernel#nover+0x4c stack: 0x4108efd202b8
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925deb0:[0x418019c53eb7]CpuSchedDispatch@vmkernel#nover+0xbb7 stack: 0x41238925df00
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdda00:[0x418019f6f528]TTYWriteToTerm@chardevs#1+0x24 stack: 0x417fd9d2b2e0
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925df20:[0x418019c55c65]CpuSchedWait@vmkernel#nover+0x245 stack: 0x0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdda60:[0x418019ff032e]UserTeletypeWriteBuffer@<None>#<None>+0xc2 stack: 0x4123a1bddac0
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dfd0:[0x418019a27d26]VmkTimerQueueWorldFunc@vmkernel#nover+0x542 stack: 0x0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddb20:[0x418019ff083c]UserTeletypeWriteInt@<None>#<None>+0x1b4 stack: 0x412e800dc2c0
2013-10-28T17:45:24.582Z cpu9:33353)0x41238925dff0:[0x418019c53242]CpuSched_StartWorld@vmkernel#nover+0xfa stack: 0x0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddb40:[0x418019f6ed5d]TTY_Write@chardevs#1+0x3d stack: 0x41097f827080
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddb70:[0x418019cfbb2d]CharDriverAsyncIO@vmkernel#nover+0x91 stack: 0x4109747a1a30
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddc60:[0x418019cf61ae]FDS_AsyncIO@vmkernel#nover+0x3a2 stack: 0x190
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddcb0:[0x418019cf6810]FDS_DoSyncIO@vmkernel#nover+0xd4 stack: 0x4108da99d8d0
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddd20:[0x418019cefaa6]DevFSFileIO@vmkernel#nover+0x24e stack: 0x4123a1bdde7c
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddd90:[0x418019fe8def]UserChardevIO@<None>#<None>+0xe7 stack: 0x4123a1bdde7c
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdddb0:[0x418019fe8e7a]UserChardevWrite@<None>#<None>+0x26 stack: 0x41085f334860
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bdde20:[0x418019fe726b]UserVmfs_Writev@<None>#<None>+0x123 stack: 0x4123a1bdde60
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddea0:[0x418019fa1390]LinuxFileDesc_Write@<None>#<None>+0xd4 stack: 0x4123a1bddf00
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddeb0:[0x418019f7b632]Linux32_Write@<None>#<None>+0xe stack: 0x4123a1bddf00
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddf00:[0x418019f7a316]User_LinuxSyscallHandler@<None>#<None>+0x3f6 stack: 0x4123a1bddf20
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddf10:[0x418019aa9b4d]User_LinuxSyscallHandler@vmkernel#nover+0x1d stack: 0xfffd9374
2013-10-28T17:45:24.582Z cpu6:34927)0x4123a1bddf20:[0x418019af1064]gate_entry@vmkernel#nover+0x64 stack: 0x0
2013-10-28T17:48:19.949Z cpu4:33023)usb storage warning (0 throttled) on vmhba32 (SCSI cmd READ_10): command_abort called
usb storage message on vmhba32: scsi cmd done, result=0x0
usb storage message on vmhba32: Bulk command transfer result=0
usb storage message on vmhba32: Bulk data transfer result 0x0
2013-10-28T17:48:19.949Z cpu4:33023)usb storage warning (0 throttled) on vmhba32 (SCSI cmd READ_10): usb_stor_stop_transport called

Talk to me about OS i think i saw ESXi 5.5 are u running on the C6100 Hardware? vSAN? Network Adapters?

There are options on what the issue could be but thats my starting foundation!
Click to expand...

That looks like a bios issue that seems familiar. Any more recent bios updates? What CPUs are those? You booting from USB?
 
lopoetve said:
That looks like a bios issue that seems familiar. Any more recent bios updates? What CPUs are those? You booting from USB?
Click to expand...
No, no recent BIOS updates. To be honest, I'm not even sure HOW to update the BIOS on these blades - I haven't looked into it as of yet.
Dual L5520 CPUs in each blade.
Yes, booting from USB (reinstalled to a brand new USB drive and switched USB ports as well to rule those out as much as possible on this system).
 
i doubt any of this helps but I will post anyway.

I had a GA-M790FXT-UD5P on 5.1. It PSOD like once ever two weeks or so for no reason it seemed. I eventually upgraded the mobo to a GA-990FXT-UD3 and 5.5. Havent had an issue since.
 
I had this issue on a few mb's. I turned out to be an actual hardware problems. use a boot cd and run some hardware tests on your mb, cpu , mem. Probably just need to adjust mem timings, cpu voltage, etc...
 
You must log in or register to reply here.
Back
Top