![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
Riccochet
Fully [H]
- Joined
- Apr 11, 2007
- Messages
- 29,900
I don't know if anyone else here uses FVP, but I thought I'd post about a possible bug that could compromise your VM's that are being accelerated.
We have what appears to be either a flaky SSD card or the driver for the card is not functioning properly. Mind you this has been in place for 6 months now and has not exhibited any signs of failure prior to this. Anyways, either the card was flaking out causing the driver to crash or the driver was crashing causing the card to flake out. It's not clear looking at the logs, just that one or the other is happening. The card is a Micron P420 700GB card. Essentially FVP failed to remove the card from the flash cluster when errors started appearing. It eventually got to the point where the errors were so frequent that the driver completely crashed. This in turn caused a massive spike in datastore read/write latency since no writes were getting through and no reads were being accelerated or being passed through either. The VM this card was accelerating was completely unresponsive. As this was going on Windows ate itself. There was no option but to power off the VM. Once we were able to remove the card from the flash cluster we powered the VM on and Windows went in to a chkdsk. Windows was so corrupted that it removed itself. LOL I had to spin up another SQL server from template and attach the vmdk as a disk. Luckily the data partition was untouched, mostly, just transaction log corruption I was able to fix. Ultimately this was a lot faster than restoring 25 DB's from backup and we didn't lose any data.
Pernix was sent a metric ton of log files and is investigating the issue. To them it should have not happened the way it did. Another issue we are seeing now is that card keeps re-adding itself back in to the flash cluster. Throughout the day I have to remove it. I could just migrate all those VM's to another host and bring that host down to physically remove the card and/or remove the driver for it. Just haven't had time to do that.
I just thought I'd give yall a heads up in case you run in to a similar situation. Check those log files!
We have what appears to be either a flaky SSD card or the driver for the card is not functioning properly. Mind you this has been in place for 6 months now and has not exhibited any signs of failure prior to this. Anyways, either the card was flaking out causing the driver to crash or the driver was crashing causing the card to flake out. It's not clear looking at the logs, just that one or the other is happening. The card is a Micron P420 700GB card. Essentially FVP failed to remove the card from the flash cluster when errors started appearing. It eventually got to the point where the errors were so frequent that the driver completely crashed. This in turn caused a massive spike in datastore read/write latency since no writes were getting through and no reads were being accelerated or being passed through either. The VM this card was accelerating was completely unresponsive. As this was going on Windows ate itself. There was no option but to power off the VM. Once we were able to remove the card from the flash cluster we powered the VM on and Windows went in to a chkdsk. Windows was so corrupted that it removed itself. LOL I had to spin up another SQL server from template and attach the vmdk as a disk. Luckily the data partition was untouched, mostly, just transaction log corruption I was able to fix. Ultimately this was a lot faster than restoring 25 DB's from backup and we didn't lose any data.
Pernix was sent a metric ton of log files and is investigating the issue. To them it should have not happened the way it did. Another issue we are seeing now is that card keeps re-adding itself back in to the flash cluster. Throughout the day I have to remove it. I could just migrate all those VM's to another host and bring that host down to physically remove the card and/or remove the driver for it. Just haven't had time to do that.
I just thought I'd give yall a heads up in case you run in to a similar situation. Check those log files!