![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Have an odd dilemma, and I'm trying to find the root cause of this instead of band-aid'ing it.
Recently, my trusty C6100 died. After much troubleshooting, I decided to just buy a new, similar system off eBay. I found one, bought it, and received it yesterday.
Of note, the C6005 I bought was damaged in transit. I notified the seller, and he's giving me the weekend to play/test it to try to make sure it's OK. Old system was dual Intel L5560s, 48GB DDR3 ECCs / node. New system is dual AMD 2419 EEs, 48 GB DDR3 ECC. So definitely some architecture change.
Regardless, I setup the system in a near identical way as my old one was. My evaluation licenses only had a couple of weeks left, so I just re-installed new everything. Each node has it's own 4GB USB drive, running 5.5 1331820.
I spun up a vCetner VM and got it configured. Within vCenter, I created a new data center, and a cluster underneath that containing all three of the "new" nodes. Initially, I had HA and DRS enabled on the cluster. In troubleshooting, I disabled HA. Outside of the cluster, I have one standalone host that's running 5.1.0 1157734.
Within the cluster I have 15 VMs, 12 of which run 24x7. The majority of these are Server 2012 R2, but there is also a CentOS machine, plus various VMWare appliances.
Of the 12 that are running, it seems that any one of them will periodically (random interval from an hour to 3 hours it seems) get "stuck". Meaning, I cannot RDP to it, and when I try to connect to the console, sending Ctrl+Alt+Del has no effect (I do not get taken to the windows login prompt). On the login screen, the time is visibility "stuck" as well - no longer incrementing as it should (right now on the console of the ones that are "stuck" they show 1:20pm, it is 1:32pm). Interestingly enough, the machines still reply to PINGs.
The vmware.log file doesn't show a thing out of the ordinary for these stuck VMs. Here's the log entries around one of the VMs that got "stuck" at 1:20:
I've left the stuck VM just sit for a couple of hours to see if it ever comes back. They do not seem to. Also interesting, it seems that sometimes, performing a vmotion brings the VM back. But not always.
I SSH'd into one of the hosts (note: VMs on all three hosts on the C6005 seem to do this) and ran esxtop. I flipped over to the disk screen, and GAVG is 0.00. Everything else on the memory and cpu screens is normal - higher than I'd ever like to see in a production type environment, but suitable for a home lab.
Anyway, I'm racking my brain here looking for something else to look after. I haven't ruled out network, storage or the hosts at this point - I'm hesitant to point to network or storage, as both have proven fine for the years I've had this lab.
Full setup:
Dell C6005 nodes' two NICS (one is for management/vMotion, other is for VM traffic)-> DLink 24 port managed switch -> Synology DS1513+
ESX Whitebox two NICS (one for management/vMotion, other for VM traffic)-> DLink 24 port managed switch -> Synolog DS1513+
The Synology is setup is an LACP bond, all 4 ports. I'm seeing no issues from the synology dashboard. I'm also seeing no errors on the switch side of things. I'm debating taking out the LACP setup, running just one NIC.
I also have a big spool of cat6 sitting here, so I was thinking of changing my cat5e cables to cat6 - I want to get that done anyway, so maybe at the same time I can rule out any potential cabling issues.
Sorry for the long post. I tried to put all the information I can think of into it. I can grab and post logs if wanted! Thank you for any ideas or help! I'm mostly interested in finding out if the host is actually damaged and I need to return it, or if I'm simply having a network/storage/configuration issue somewhere that I messed up with the new host.
Recently, my trusty C6100 died. After much troubleshooting, I decided to just buy a new, similar system off eBay. I found one, bought it, and received it yesterday.
Of note, the C6005 I bought was damaged in transit. I notified the seller, and he's giving me the weekend to play/test it to try to make sure it's OK. Old system was dual Intel L5560s, 48GB DDR3 ECCs / node. New system is dual AMD 2419 EEs, 48 GB DDR3 ECC. So definitely some architecture change.
Regardless, I setup the system in a near identical way as my old one was. My evaluation licenses only had a couple of weeks left, so I just re-installed new everything. Each node has it's own 4GB USB drive, running 5.5 1331820.
I spun up a vCetner VM and got it configured. Within vCenter, I created a new data center, and a cluster underneath that containing all three of the "new" nodes. Initially, I had HA and DRS enabled on the cluster. In troubleshooting, I disabled HA. Outside of the cluster, I have one standalone host that's running 5.1.0 1157734.
Within the cluster I have 15 VMs, 12 of which run 24x7. The majority of these are Server 2012 R2, but there is also a CentOS machine, plus various VMWare appliances.
Of the 12 that are running, it seems that any one of them will periodically (random interval from an hour to 3 hours it seems) get "stuck". Meaning, I cannot RDP to it, and when I try to connect to the console, sending Ctrl+Alt+Del has no effect (I do not get taken to the windows login prompt). On the login screen, the time is visibility "stuck" as well - no longer incrementing as it should (right now on the console of the ones that are "stuck" they show 1:20pm, it is 1:32pm). Interestingly enough, the machines still reply to PINGs.
The vmware.log file doesn't show a thing out of the ordinary for these stuck VMs. Here's the log entries around one of the VMs that got "stuck" at 1:20:
Code:
2014-07-19T18:59:20.399Z| vmx| I120: TOOLS received request in VMX to set option 'synctime' -> '0'
2014-07-19T18:59:20.465Z| vmx| I120: VMXVmdb_SetCfgState: cfgReqPath=/vm/#_VMX/vmx/cfgState/req/#4/, remDevPath=/vm/#_VMX/vmx/vigor/setCfgStateReq/#82/in/
2014-07-19T20:16:25.450Z| mks| I120: SOCKET 3 (122) Creating VNC remote connection.
2014-07-19T20:27:19.348Z| mks| I120: SSL: syscall error 104: Connection reset by peer
2014-07-19T20:27:19.349Z| mks| I120: SOCKET 3 (122) recv error 104: Connection reset by peer
2014-07-19T20:27:19.349Z| mks| I120: SOCKET 3 (122) VNC Remote Disconnect.
2014-07-19T20:27:28.982Z| mks| I120: SOCKET 4 (122) Creating VNC remote connection.
2014-07-19T20:29:56.127Z| mks| I120: SSL: syscall error 104: Connection reset by peer
2014-07-19T20:29:56.127Z| mks| I120: SOCKET 4 (122) recv error 104: Connection reset by peer
2014-07-19T20:29:56.127Z| mks| I120: SOCKET 4 (122) VNC Remote Disconnect.I SSH'd into one of the hosts (note: VMs on all three hosts on the C6005 seem to do this) and ran esxtop. I flipped over to the disk screen, and GAVG is 0.00. Everything else on the memory and cpu screens is normal - higher than I'd ever like to see in a production type environment, but suitable for a home lab.
Anyway, I'm racking my brain here looking for something else to look after. I haven't ruled out network, storage or the hosts at this point - I'm hesitant to point to network or storage, as both have proven fine for the years I've had this lab.
Full setup:
Dell C6005 nodes' two NICS (one is for management/vMotion, other is for VM traffic)-> DLink 24 port managed switch -> Synology DS1513+
ESX Whitebox two NICS (one for management/vMotion, other for VM traffic)-> DLink 24 port managed switch -> Synolog DS1513+
The Synology is setup is an LACP bond, all 4 ports. I'm seeing no issues from the synology dashboard. I'm also seeing no errors on the switch side of things. I'm debating taking out the LACP setup, running just one NIC.
I also have a big spool of cat6 sitting here, so I was thinking of changing my cat5e cables to cat6 - I want to get that done anyway, so maybe at the same time I can rule out any potential cabling issues.
Sorry for the long post. I tried to put all the information I can think of into it. I can grab and post logs if wanted! Thank you for any ideas or help! I'm mostly interested in finding out if the host is actually damaged and I need to return it, or if I'm simply having a network/storage/configuration issue somewhere that I messed up with the new host.
