Please help with building home ESXi + NAS.

R

Rivera

Limp Gawd
Joined
May 17, 2011
Messages
154
Hello. I'm thinking about rebuilding my primary home server and need some help. So...
Currently i have some Arima motherboard with crappy opteron 2210 CPU and 1G ram.
And... Areca ARC-1160 (PCI-X SATA RAID) with 16x750GB HDD.
Current setup:
ArchLinux on 2xHDD (RAID1), Other 14 HDDs is storage (RAID6).
I'm going to upgrade that with:
Intel S5000PAL motherboard (btw, does that thing have IP-KVM?) (i do not want to buy new 16-port RAID, so this motherboard looks good - with PCI-X port)
2xXeon E5471 (old, but good)
16GB (8X2GB) PC2-5300 FBDIMM (ECC) RAM. (nuff said)
Probably USB flash / DOM flash for ESXi installation.
Also it will be seedbox and NFS server for home media (XBMC client installed on nettop + Samsung TV with ethernet, so only NFS needed from virtual machine, not XBMC)
Transmission daemon
I also need netatalk daemon (timemachine - 2 macs), SAMBA (or other SMB implementation)

Moar about VMs:
1. FreeNAS for storage?
2. ArchLinux or Ubuntu for TimeMachine (netatalk), Transmission (BT), MediaTomb (DLNA), SubSonic (web music player)
3. Ubuntu + GNS3 for learning networking
4. Win2008 server for vCenter
All other will be temporary VMs for testing/learning etc.

I think i will use VMWare ESXi 5 (vSphere) for it, i even have a license for it.

But i still have some questions unaswered, and main is... How i can manage storage? I can buy two SATA drives for system or use two from array (if ESXi can boot from Areca raid). I'm thinkin about installing freenas/openfiler/etc for it, but AFAIK i need to allocate 90% of storage array to virtual machine with FreeNAS? Or is there any other choices?
Also, i need suggestions about FreeNAS-like distros. But it should have web interface.

Sorry, never worked with ESXi.

UPD: i heard i can install ESXi on USB flash (connected to internal USB pins on motherboard). So i can install ESXi on flash and use all my array for VMs, storages and etc. But i still do not know how i should _manage_ my storage.
 
Last edited:
Most people here recommend something solaris based like an openindiana with napp-it. I've tried it but since I'm not very experienced with solaris it wasn't the easiest solution. FreeNAS 8.0.2 on the other hand was very easy to work with. Everything I've tried with it just worked. I'd also like to hear from others what is the best practice when setting up shared storage between VMs.
 
One thing: if you can go with NFS for shared storage, even with an emulated nic like the e1000, I can get 4gb/sec easily, since the traffic never leaves the host. Remember to set the high-speed VMCI option for both guests if you do...
 
danswartz said:
One thing: if you can go with NFS for shared storage, even with an emulated nic like the e1000, I can get 4gb/sec easily, since the traffic never leaves the host. Remember to set the high-speed VMCI option for both guests if you do...
Click to expand...

NFS shares seem like a way to go. What about iSCSI? Is it used mostly to connect external NAS device to ESXi?
 
It all depends on if you want to manage storage from the array (NFS), or from the host (iSCSI/FC/FCoE) and which you're most comfortable with.

I prefer block storage, but I'm a storage guy.
 
What you guys think would be a good solution for me? I have a single ESXi box with few VMs like Win Server 08, pfSense and (as of right now) FreeNAS. I also got Perc 5i for RAID5 for speed and redundancy. Should I set up FreeNAS (alternatively openindiana) to share storage between VMs and other computers in my network and if so, which method would yield the best overall performance?
 
Same here, my working solution right now is FreeNAS VM that is on the same network as the rest of VMs and physical machines. That way I can share files across the whole network plus FreeNAS allows me to create target drive for Time Machine.
 
danswartz said:
Maybe not a lot of people using freenas so no comments?
Click to expand...

I'm not asking about freenas, i'm asking about variants how i should manage storage so it can be accessible from several VMs. And about hardware..
 
danswartz said:
Maybe not a lot of people using freenas so no comments?
Click to expand...

Yeah I think that's seems to be the case. I went with it since it worked for me without any issues. I'll have to give openindiana another try. I just have to figure out whether I should go with hardware raid (perc 5i) or do a ZFS raid.
 
Unfortunately, a lot of the suggestions for how to share/organize is based on the distro you are using for the SAN/NAS, so whether it is freenas is a valid point.
 
Rivera said:
Hello. I'm thinking about rebuilding my primary home server and need some help. So.....

Moar about VMs:
1. FreeNAS for storage?
2. ArchLinux or Ubuntu for TimeMachine (netatalk), Transmission (BT), MediaTomb (DLNA), SubSonic (web music player)
3. Ubuntu + GNS3 for learning networking
4. Win2008 server for vCenter
All other will be temporary VMs for testing/learning etc.

I think i will use VMWare ESXi 5 (vSphere) for it, i even have a license for it.
Click to expand...

You want to build a all.in-one, that is a barebone ESXi server with a virtualized
ZFS NAS/SAN storage server on top with a lot of other VM's.

1.
Especially for the storage part, your selected mainboard is not suitable.
A storage server especially a ZFS one needs direct disk access for performance
and reliability. This can only achieved if you can pass-through a controller/ disks
via hardware virtualization vt-d.

The Intel S5000 is not vt-d capable, do not buy!

2.
Your PCI-X controller is a hardware Raid-controller.
It is slower than modern software RAID, has a write-hole problem
and you loose most of advanced ZFS feataures with it.
Forget it.

I would buy a
- more modern board with Intel server chipsets (3420, 5520 or 202 chipset)
my favourites are from SuperMicro

- buy one/two SAS HBA controller (without Raid, cheapest onboard, best is LSI 2008 SAS2, ok is LSI 1068 SAS1)
optionally use a Expander (SAS2)

- use a cheap sata disk for ESXi and your storage VM
pass-through disks from separate SAS to your storage VM

- share your storage( Storage VM)
via SMB (file services), NFS (ESXi SAN storage) and AFP (Time machine)

-Install other VM's on your NFS SAN storage
(High speed internal link ESXi -> SAN) in Software up to 10 Gb/s

I wrote a miniHowto about (OpenIndiana)
but most of it may work with FreeNAS

http://www.napp-it.org/doc/downloads/all-in-one.pdf

and some hardware suggestions:
http://www.napp-it.org/hardware/index_en.html
 
Last edited:
_Gea said:
You want to build a all.in-one, that is a barebone ESXi server with a virtualized
ZFS NAS/SAN storage server on top with a lot of other VM's.

1.
Especially for the storage part, your selected mainboard is not suitable.
A storage server especially a ZFS one needs direct disk access for performance
and reliability. This can only achieved if you can pass-through a controller/ disks
via hardware virtualization vt-d.

The Intel S5000 is not vt-d capable, do not buy!

2.
Your PCI-X controller is a hardware Raid-controller.
It is slower than modern software RAID, has a write-hole problem
and you loose most of advanced ZFS feataures with it.
Forget it.

I would buy a
- more modern board with Intel server chipsets (3420, 5520 or 202 chipset)
my favourites are from SuperMicro

- buy one/two SAS HBA controller (without Raid, cheapest onboard, best is LSI 2008 SAS2, ok is LSI 1068 SAS1)
optionally use a Expander (SAS2)

- use a cheap sata disk for ESXi and your storage VM
pass-through disks from separate SAS to your storage VM

- share your storage( Storage VM)
via SMB (file services), NFS (ESXi SAN storage) and AFP (Time machine)

-Install other VM's on your NFS SAN storage
(High speed internal link ESXi -> SAN) in Software up to 10 Gb/s

I wrote a miniHowto about (OpenIndiana)
but most of it may work with FreeNAS

http://www.napp-it.org/doc/downloads/all-in-one.pdf
Click to expand...

This right here is some really good information. It answered a lot of my questions. Thanks for that. Seems like hardware raid is unnecessary, I wonder if I should even keep the perc 5i card. Do you recommend the server version of OpenIndiana for older low end system? I've had some troubles installing VM tools in OI, partially why I've been using FreeNAS.
 
i always use OpenIndiana live version.
Easier to handle and TimeSlider to access Snaps by selecting a folder and go back in time is a must have
You need a few GB disk space more and some more RAM but thats not essential (min 2 GB RAM and 8-10 GB disk)

If you select Solaris10-64 bit, you can install the ESXi included tools without problem.
 
@_Gea great post, thanks.

But building such system will cost around 2k+ USD - pretty big sum for me :(
400$ for mobo, 600$ for one cpu, 350$ for non-raid HBA and unknown sum for new PSU and chassis (ARC 1160 uses SATA connectors, not modern SFF connectors). And this is only in USA shops - prices are higher in Russia. As well as customs taxes... It will be easier to build complete new server...
Let's say my budget is around 1k USD.

And about ZFS - i does not want to use it after evil oracle killed OpenSolaris project. AFAIK new ZFS source code will not be available?

About current disk system performance
write:
lex@lex-storage >>> dd if=/dev/zero of=ddfile bs=8k count=2000000
2000000+0 records in
2000000+0 records out
16384000000 bytes (16 GB) copied, 58.9518 s, 278 MB/s
Click to expand...
read:
lex@lex-storage >>> dd if=ddfile of=/dev/null bs=8k
2000000+0 records in
2000000+0 records out
16384000000 bytes (16 GB) copied, 40.0261 s, 409 MB/s
Click to expand...

>>use a cheap sata disk for ESXi and your storage VM
I will buy a SATA DOM for that, thanks. Maybe that model

And this is home lab, not some production system :)

And btw! I can pass-through drives via that Areca card. But i know - PCI-X is old sh*t.
 
_Gea said:
i always use OpenIndiana live version.
Easier to handle and TimeSlider to access Snaps by selecting a folder and go back in time is a must have
You need a few GB disk space more and some more RAM but thats not essential (min 2 GB RAM and 8-10 GB disk)

If you select Solaris10-64 bit, you can install the ESXi included tools without problem.
Click to expand...

Got it, I may have used different option when creating a VM, which led to problems.
 
Rivera said:
@_Gea great post, thanks.

But building such system will cost around 2k+ USD - pretty big sum for me :(
400$ for mobo, 600$ for one cpu, 350$ for non-raid HBA and unknown sum for new PSU and chassis (ARC 1160 uses SATA connectors, not modern SFF connectors). And this is only in USA shops - prices are higher in Russia. As well as customs taxes... It will be easier to build complete new server...
Let's say my budget is around 1k USD.

And about ZFS - i does not want to use it after evil oracle killed OpenSolaris project. AFAIK new ZFS source code will not be available?
.
Click to expand...


You do not need a 8core Xeon board,
one quad + enough RAM is ok ex

SuperMicro X9..-F server class mainboard: about 150 Euro
+ single CPU ex i5: 200 euro
+ hba with LSI 1068: 130 euro
or SuperMicro AOC-USAS2-L8e (same price, SAS2 but wrong side mounted)
sff to sata cable: 20 euro

or
SuperMicro X58 X8ST3-F about 270 Euro with SAS onboard
(max 24 GB cheap DDR3 RAM)

even a highend SuperMicro X8DT6-F and included SAS2 controller
is about 400 Euro in Germany + one (of possible two) Xeon + DDR3 RAM (up to 192GB)
is not too expensive but not needed for a usual server

Dollar price is mostly the same or minimal higher because my prices are with german sales tax
but i do not know about russian tax and price


about OpenSolaris:
The successor (Solaris Express 11) is commercial but
the fork OpenIndiana is free and very alive
 
Last edited:
What about backplane? As i said, it uses SATA2 ports ( image )
It's impossible to find new backplane for that chassis.
 
Rivera said:
Okay, so, for example:
http://www.supermicro.com/products/motherboard/Xeon3000/X58/X8ST3-F.cfm mobo (300$)
Some i7 quad-core (i7-2600k?) - 300$
8-port HBA - 200
And RAM about 100.
But you said about VT-D (directed I/O), and i7 does not support that.
Click to expand...

Actually the non-K chips DO support VT-D, but overclocking K versions do not (don't ask me why). However...the CPU and board you mentioned above do not go together. X8 series is based on X58 chipset with LGA 1366 socket. So mainly Xeon 56xx and i7-9xx series chips. BE aware it seems the 1068e only supports up to 2TB drives.

Perhaps reconsider the newer boards mentioned. IN particular something like the X9SCL+-F. No you don't get the built in HBA, but you can find those 8 port SAS HBA's on ebay for good prices (1068e or SAS2008 based), a lot less than $200 each. You can add up to 3 to this board. A Xeon E3-1220 is the perfect budget CPU at only $210 (US), will easily outperfrom those older Xeon E56xx, and you get VT-D and ECC ram support.
 
Just so you guys know, I am using a 990FXA-UD3 with a AM3 640 quad and it works sweet with Vsphere 4.1/5 with Iommu. The only real drama I have is the controller (I'm using a Dell Perc6i) where I've had to create individual RAID 0 sets per disk to present through to ESX (and then using IOMMU to pass through the controller, and then to the OpenIndiana VM). This is a bit of a problem because if the controller goes bang I'm probably screwed (well, I'd have to restore from backup) so I'd definitely be looking at other solutions for the controller

Runs like a champ though. Set up nappit to present through iscsi to the esx host and have built vms on top of that. Not bad speeds either - was originally getting write over gb lan around 70-100mb/s, but now they've dropped down to around 40-60mb/s. From memory, isn't Gea responsible for this sweet piece of software? If so - kudos to you sir, as Freenas/Openfiler wouldn't work (They seem to have issue seeing the Perc 6i)

Tbh, I'd look at AMD and Iommu over 1366/1155.
 
Silenus said:
Actually the non-K chips DO support VT-D, but overclocking K versions do not (don't ask me why). However...the CPU and board you mentioned above do not go together. X8 series is based on X58 chipset with LGA 1366 socket. So mainly Xeon 56xx and i7-9xx series chips. BE aware it seems the 1068e only supports up to 2TB drives.

Perhaps reconsider the newer boards mentioned. IN particular something like the X9SCL+-F. No you don't get the built in HBA, but you can find those 8 port SAS HBA's on ebay for good prices (1068e or SAS2008 based), a lot less than $200 each. You can add up to 3 to this board. A Xeon E3-1220 is the perfect budget CPU at only $210 (US), will easily outperfrom those older Xeon E56xx, and you get VT-D and ECC ram support.
Click to expand...

Oh. Okay. So probably something like this:
Xeon E3-1260L (3.6 Ghz and only 45W? Impressive)
Supermicro X9SCA? This one has x16 pci-e port - i want to put videocard in it. But no IPMI :(. Maybe x8 is okay for encoding?

Also, does ESXi support PCI passthru? I'd like to do some video encoding in virtual machine
 
Yup - haven't tried passing through a video adapter myself, but everything else seems to work sweet.
 
You must log in or register to reply here.
Back
Top