![[H]ard|Forum](/styles/hardforum/xenforo/logo_dark.png){kind=logo}
I recently built an ESXI host server to handle a couple different roles, and allow me to retire some old hardware and move some disks into a more centralized location.
ESXI 5.1.0 Update 1
Motherboard
ASRock 970 Extreme3 R2.0 AM3+ AMD 970 SATA 6Gb/s USB 3.0 ATX AMD Motherboard
One port Realtek 8111E 10/100/1000 Onboard Ethernet
CPU
AMD FX-8320 Vishera 3.5GHz Socket AM3+ 125W Eight-Core Desktop Processor FD8320FRHKBOX
RAM
G.SKILL Ripjaws X Series 16GB (2 x 8GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) Desktop Memory Model F3-12800CL10D-16GBXL
Non-NAS Disk
Kingston SSDNow V300 Series SV300S37A/120G 2.5" 120GB SATA III Internal Solid State Drive (SSD)
Video (mostly only needed for initial configuration)
EVGA 512-P3-1301-KR GeForce 8400 GS 512MB 32-bit DDR3 PCI Express 2.0 x16 HDCP Ready Low Profile Ready Video Card
HDDs
4x SAMSUNG EcoGreen F4 ST2000DL004 2TB 32MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive Bare Drive
I had these in my gaming machine for shared storage utilizing that motherboard's onboard RAID. I decided I wanted to use IOMMU passthrough to send these to a NAS VM.
Installed ESXI to the local SSD. I see a lot of folks online using USB Flash drives for this, but I figured if I have the space there, I might as well use it. That may or may not have been a mistake.
Additionally, the Motherboard I used handles SATA IOMMU Passthrough a little differently. When you pass the controller through, it only passes 4 of the 5 SATA ports. This means that I can pass through all 4 of my HDDs while still allowing the Host to access the 1 SSD with all the VM files and disk images.
I then configured three VMs:
1. NAS-Server
OmniOS and Nappit
Passed the SATA controller through
2 VCPU
4 GB RAM
5 GB drive Stored on the SSD
2. Media-Server
Ubuntu Server 13.10
2 VCPU
4 GB RAM
70 GB drive stored on the SSD
3. MineCraft Server
Windows 2008 R2 SP1
2 VCPU
4 GB RAM
24 GB sored on the SSD
It's worth noting that I don't really have any experience with ESX, Linux, OmniOS, or NAS solutions. A lot of this setup is brand new to me, which is part of the reason why I selected the software that I did.
It was at this point that I started noticing issues. I have a feeling that they're all realted to the same cause, so I'm going to focus on one server and its symptoms to see if anyone has any suggestions.
The primary use for the Media-Server is SABNZBD and NZBDrone. SABNZBD is a usenet downloader, and NZBDrone manages the searching and downloading automatically. On my old hardware downloader, I was averaging a steady 5 MB/s down, and unpacking after download didn't take a whole lot of time. On the VM machine, the download speed fluctuates between 4 KB/S and, at best, 2 MB/s. Additionally, while it's downloading or unpacking, I will see other issues. The interfaces will stop loading properly, certain automatic functions on the server will start to fail, and I even received some SAB errors saying that it timed out while trying to read a file on the local filesystem. Unpacking of files takes ages, I haven't sat down and timed it, but it is a significant increase. When nothing is happening, everything seems to run fine.
I suspect that what I'm seeing is some kind of I/O issue with the SSD. When the server is downloading (writing), unpacking (read/writing), or moving (read/writing) files, I see a performance hit. To the point that the console doesn't refresh when I enter a command while this is going on. I have to close the console window and re-open if I do something like an ls.
I thought it might be some sort of networking issue, with one router port and 1 physical NIC handling 3 VMs, maybe every call is going out to my router and back instead of being handled within ESXI. I'm not sure how to check that, but at this point I'm leaning much more heavily towards some kind of disk problem.
The problem I have is that...I don't really know where to start. I'm going to borrow another SSD from a coworker to rule out a faulty drive. Other than that, I don't have any ideas on how to troubleshoot. Is there some setting in vSphere Client that I'm missing which is causing this? Is it an issue with having everything (ESX, all my VMs) on the same source drive? Is it something else entirely?
I guess that's what I'm looking for some input on. Where do I go next? What can I check, what are some best practices that I may have missed? Are there any tools out there that I can run to gain more insight into what's happening here? Has anyone seen anything like this before? I also see what appear to be buffering problems while running video files from the NAS, I suspect it's just another symptom of whatever this problem is.
ESXI 5.1.0 Update 1
Motherboard
ASRock 970 Extreme3 R2.0 AM3+ AMD 970 SATA 6Gb/s USB 3.0 ATX AMD Motherboard
One port Realtek 8111E 10/100/1000 Onboard Ethernet
CPU
AMD FX-8320 Vishera 3.5GHz Socket AM3+ 125W Eight-Core Desktop Processor FD8320FRHKBOX
RAM
G.SKILL Ripjaws X Series 16GB (2 x 8GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) Desktop Memory Model F3-12800CL10D-16GBXL
Non-NAS Disk
Kingston SSDNow V300 Series SV300S37A/120G 2.5" 120GB SATA III Internal Solid State Drive (SSD)
Video (mostly only needed for initial configuration)
EVGA 512-P3-1301-KR GeForce 8400 GS 512MB 32-bit DDR3 PCI Express 2.0 x16 HDCP Ready Low Profile Ready Video Card
HDDs
4x SAMSUNG EcoGreen F4 ST2000DL004 2TB 32MB Cache SATA 3.0Gb/s 3.5" Internal Hard Drive Bare Drive
I had these in my gaming machine for shared storage utilizing that motherboard's onboard RAID. I decided I wanted to use IOMMU passthrough to send these to a NAS VM.
Installed ESXI to the local SSD. I see a lot of folks online using USB Flash drives for this, but I figured if I have the space there, I might as well use it. That may or may not have been a mistake.
Additionally, the Motherboard I used handles SATA IOMMU Passthrough a little differently. When you pass the controller through, it only passes 4 of the 5 SATA ports. This means that I can pass through all 4 of my HDDs while still allowing the Host to access the 1 SSD with all the VM files and disk images.
I then configured three VMs:
1. NAS-Server
OmniOS and Nappit
Passed the SATA controller through
2 VCPU
4 GB RAM
5 GB drive Stored on the SSD
2. Media-Server
Ubuntu Server 13.10
2 VCPU
4 GB RAM
70 GB drive stored on the SSD
3. MineCraft Server
Windows 2008 R2 SP1
2 VCPU
4 GB RAM
24 GB sored on the SSD
It's worth noting that I don't really have any experience with ESX, Linux, OmniOS, or NAS solutions. A lot of this setup is brand new to me, which is part of the reason why I selected the software that I did.
It was at this point that I started noticing issues. I have a feeling that they're all realted to the same cause, so I'm going to focus on one server and its symptoms to see if anyone has any suggestions.
The primary use for the Media-Server is SABNZBD and NZBDrone. SABNZBD is a usenet downloader, and NZBDrone manages the searching and downloading automatically. On my old hardware downloader, I was averaging a steady 5 MB/s down, and unpacking after download didn't take a whole lot of time. On the VM machine, the download speed fluctuates between 4 KB/S and, at best, 2 MB/s. Additionally, while it's downloading or unpacking, I will see other issues. The interfaces will stop loading properly, certain automatic functions on the server will start to fail, and I even received some SAB errors saying that it timed out while trying to read a file on the local filesystem. Unpacking of files takes ages, I haven't sat down and timed it, but it is a significant increase. When nothing is happening, everything seems to run fine.
I suspect that what I'm seeing is some kind of I/O issue with the SSD. When the server is downloading (writing), unpacking (read/writing), or moving (read/writing) files, I see a performance hit. To the point that the console doesn't refresh when I enter a command while this is going on. I have to close the console window and re-open if I do something like an ls.
I thought it might be some sort of networking issue, with one router port and 1 physical NIC handling 3 VMs, maybe every call is going out to my router and back instead of being handled within ESXI. I'm not sure how to check that, but at this point I'm leaning much more heavily towards some kind of disk problem.
The problem I have is that...I don't really know where to start. I'm going to borrow another SSD from a coworker to rule out a faulty drive. Other than that, I don't have any ideas on how to troubleshoot. Is there some setting in vSphere Client that I'm missing which is causing this? Is it an issue with having everything (ESX, all my VMs) on the same source drive? Is it something else entirely?
I guess that's what I'm looking for some input on. Where do I go next? What can I check, what are some best practices that I may have missed? Are there any tools out there that I can run to gain more insight into what's happening here? Has anyone seen anything like this before? I also see what appear to be buffering problems while running video files from the NAS, I suspect it's just another symptom of whatever this problem is.
Last edited: