• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Cheap Whitebox SAN

M

Modder man

[H]ard|Gawd
Joined
May 13, 2009
Messages
1,770
Looking for tips and pointers to build a cheap whitebox SAN to serve as an iSCSI target for my esxi hosts in my home lab. I have a lot to learn and am looking for some tips on places to start.
I have a few 2tb hitachi drives as well as three 128gb ssds that could be used. Or I can get some different drives all together if these are not suited for the task. Not sure at all what is required from a computer perspective, the load seems relatively light but as with anything the more ram the better. I am not trying to mirror a production environment, just understand the concepts. I have also seen how VM's perfrom if they are just running on straight spinning disks and dont really want anything to do with that.
Any tips and help is greatly appreciated.
 
How many hosts do you have?
Are you willing to buy/build another computer for the storage?
What are your speed goals?
What is your capacity goal?
What is your redundancy goal?
How much expandability do you want to be ready for? Future Hosts and extra disks.
What is your budget? Does that include disks?
What are the specs of your current hosts?
What VM's do you run or plan to run?
 
1. I plan to have three hosts
2. Yes the SAN will be on dedicated hardware, may also run a vcenter server if possible
3. I dont think I really know enough to know what my speed goals should be
4. 2TB should be enough for the VM's but I may build more in for overall network storage
5. Not sure I know enough to answer this question intelligently
6. I dont need to be ready for a huge amount of expandability but would like to keep my options open if possible.
7. I dont have a nailed down budget, Just trying top keep it as reasonable as possible. Trying to manage the whole cost to performance ration in my head with reasonable expectations.
8. Current hosts are fx-6300 with 16gb of ram, may expand to 32gb as I find cheap ram kits.
9. This lab is strictly for educational purposes, I am looking at my companies network and trying to build out machines in a similar manner for the experience of how to start from scratch. So things like a domain controller, firewall appliances for testing. Overall just alot of test machines. My main area of interest is in the virtual environment and infrastructure itself.
 
That is a nice looking case but i have a whole stack of norco cases on my desk that I can use. I am trying to determine mainly what I need in terms of cpu/mobo/ram and as far as drives go.
 
16 or 24 bay?

for motherboard I'd go with this sucker:

http://www.newegg.com/Product/Product.aspx?Item=N82E16813132230


Yes I know it's outrageously expensive but the built in CPU, bazillion SATA/SAS ports, and multiple Gig ports really do make it a helluva board.

If you are looking for a dirt cheap alternative, any server grade mobo from ebay will do. Grab a motherboard with dual opteron's and a plethora of DDR2 memory. You'll do just fine.

OS:

https://www.openfiler.com/

built in ISCSI, i've used it, it's not too bad to deal with and it's free.
 
I would
- use a HP Microserver Gen8 or something similar with
Intel serverchipset, i3 or Xeon, Intel nics and ECC RAM

If you are looking for a mainboard, check SuperMicro like a
http://www.supermicro.nl/products/motherboard/Xeon/C220/X10SL7-F.cfm
or a smaller board without the extra LSI HBA (perfect for many disks or all-in-one)

- Prefer a 19" case with a backplane ex from Norco (8-24 bay) or SuperMicro (higher quality)
- Prefer anything with ZFS (use nothing else!)
- Prefer SSDs for ESXi datastores and regular disks for backup and other fileservices

- I use Solaris or the free Solaris fork OmniOS for iSCSI (but prefer ESXi+NFS because of simplicity)
see http://docs.oracle.com/cd/E23824_01/html/821-1459/fncoz.html
and my napp-it ZFS storage server with web-ui http://www.napp-it.org/doc/downloads/napp-it.pdf

With my newest napp-it you can create ZFS snaps (a regular ZFS snap is like a sudden powerloss) with included ESXi hot memory snaps and copy such consistent snaps to a backup machine. This is all free (with OmniOS)
 
Last edited:
_Gea said:
I would
- use a HP Microserver Gen8 or something similar with
Intel serverchipset, i3 or Xeon, Intel nics and ECC RAM

If you are looking for a mainboard, check SuperMicro like a
http://www.supermicro.nl/products/motherboard/Xeon/C220/X10SL7-F.cfm
or a smaller board without the extra LSI HBA (perfect for many disks or all-in-one)

- Prefer a 19" case with a backplane ex from Norco (8-24 bay) or SuperMicro (higher quality)
- Prefer anything with ZFS (use nothing else!)
- Prefer SSDs for ESXi datastores and regular disks for backup and other fileservices

- I use Solaris or the free Solaris fork OmniOS for iSCSI (but prefer ESXi+NFS because of simplicity)
see http://docs.oracle.com/cd/E23824_01/html/821-1459/fncoz.html
and my napp-it ZFS storage server with web-ui http://www.napp-it.org/doc/downloads/napp-it.pdf

With my newest napp-it you can create ZFS snaps (a regular ZFS snap is like a sudden powerloss) with included ESXi hot memory snaps and copy such consistent snaps to a backup machine. This is all free (with OmniOS)
Click to expand...

Any good guides you suggest for setting up prefer ESXi+NFS ?
 
You can google about NFS vs iSCSI
but basically you do not need a special guide

- Share a filesystem over NFS
(avoid access restrictions, with napp-it this is a set NFS to on with menu ZFS filesystems)

- Connect ESXi to the NFS datastore and create VMs on it
You can move/copy/backup all VMs with any NFS client (or a SMB client ex Windows when SMB is also enabled)

Good to know
- MPIO (multipath io) is iSCSI only but I would prefer 10G network if performance is a matter
- NFS and iSCSI have a secure and a fast but insecure setting
this is sync write with NFS and writeback cache with iSCSI. With same settings performance can be similar.
- NFS is less critical regarding timeout and delayed connect on power on
 
I planned on mixing ssds and spinning disks. Gea i have read a bunch of your stuff and planned on possibly using OmniOS with napp-it. Can i somehow use the ssd's for caching the spinning disks to give higher overall throughput?
 
ZFS has an excellent RAM based read cache called ARC.
You can extend that with SSDs and L2ARC.

But on noncached data or on random writes,
nothing can beat SSDs.

As SSDs are quite cheap nowadays, (and you do not need that many space for VMs)
I suggest SSDs for VMs and disks for backup.
 
so i could use my 3 128gb ssds and a few spinning 2TB's and that could be a decent drive setup?
 
Why not.
Pool1 from SSDs as a fast NFS ESXi datastore (mirror or raid Z1) and pool2 from your 2 TB disks.
 
Typically yes, 4x 1Gb connections would be enough.

Another route you could take is:

Supermicro 16 Bay server w/ 16gb memory
2x IBM M1015's
4x SFF-8087 SATA breakout cables
2x Qlogic QLE2462 FC HBA

Run OmniOS+Napp-it, serve your array on COMSTAR over both of the Fiber Channel ports (the other QLE2462 would go in your ESXi box), set up round-robin MPIO and you would have a 8Gb/sec connection between the 2.

If you wanted to expand with more hosts all you need to do is add a FC switch like a Brocade 200e.

Right now the FC setup is much cheaper to expand to more hosts. 10Gb switches with more than 2 ports are excessively priced still. I just picked up a spare 200e w/16 licensed ports and SFP's for $55 shipped.

Here is a post about using a Supermicro FatTwin for the same thing. I think the total setup was like $1400 including drives (including a ZIL write cache and 5x 2TB drives) and a ESXi node with 2x quad core processors and 32GB memory.
 
Last edited:
As an eBay Associate, HardForum may earn from qualifying purchases.
Ok, I can see where 4x1gb connection may not be the best way to go, looks like it would cost almost $70 for the 4x1gb nic in each host. would be cheaper to just go with a fiber channel setup. What is the terminology i would want to search to make sure that the switch has all the gear needed to make it work?
 
If money does not matter, I would go 10 GbE (Intel X520/540 etc)
If you go "cheap Ebay like an older IB or FC device", you must care about compatibilty.
I would not try anything that is not confirmed to be working and stable with your OS of choice.
 
_Gea said:
If money does not matter, I would go 10 GbE (Intel X520/540 etc)
If you go "cheap Ebay like an older IB or FC device", you must care about compatibilty.
I would not try anything that is not confirmed to be working and stable with your OS of choice.
Click to expand...

That happens to be exactly what I like about the FC setup. The qlogic drivers are in everything, right along with emulex drivers too. It just works, with everything.
 
Money definitely matters. I dont want to cheap out but at the same time the purpose of this lab is just to prove concepts and understand things more thoroughly. not even a chance i can afford 10 GbE. I bought two of the FC cards earlier for $4 each, Now i am setting out to find a FC switch. What does it mean when they say liscensed for x# of ports?
 
Modder man said:
Money definitely matters. I dont want to cheap out but at the same time the purpose of this lab is just to prove concepts and understand things more thoroughly. not even a chance i can afford 10 GbE. I bought two of the FC cards earlier for $4 each, Now i am setting out to find a FC switch. What does it mean when they say liscensed for x# of ports?
Click to expand...

Just an fyi, if you only have a 1 host and 2 clients you don't need a fc switch right away. Point to point will work just fine to get you going. You would have a 4gb connection to each host. If you wanted to get to 8gb per host you would need the switch. You did buy the qle2462 right? Only the 4gb & 8gb qlogic and emulex cards will work in target mode. Also, the fiber you will need is "LC-LC duplex 50/125 MultiMode" OM1 will work just fine but if OM3 is cheaper you can get that too. Be sure to order fiber from someone in the USA.

Number of licenced ports refers to how many ports are active on the switch. For example, the brocade 200e is a 16 port switch, by default it came with an 8 port licence. If you wanted to use other ports you purchased an additional 4 or 8 port licence.

If you do want to get a FC switch right now I really recommend the Brocade 200e. They are cheap, low power and you can find them fully licenced for less than $100 typically, including 12-16 modules. If you do need to order modules they are cheap too. Just be sure to order a serial cable (and usb adapter if you don't have a serial port) so no matter what you can configure it when you get the switch. Sometimes people don't factory reset the switch before they sell it.

I actually am running the fc setup I recommended so if you have questions setting it up feel free to contact me. Just remember, to use the fc card in target mode you have to be using a solaris based os for the zfs server. The top one out there is OmniOS, then you install napp-it on top of it.

_Gea is the napp-it man, just so you know.
 
Last edited:
The cheapest one i see on ebay is $160 fully ready to go...was yours fully liscensed and 16 sfp for $50?
 
NAS and SAN are two different things. If you are trying to build a fiber channel SAN, it's going to be out of your budget and so dissimilar to anything that you would ever use in an enterprise that it wouldn't even make sense.

Get a Synology and load it up with your drives, or build a cheapo ZFS or other whitebox NAS. Either will support NFS, iSCSI, and CIFS. That'll allow you to learn block and file-based concepts without breaking the bank.
 
REDYOUCH said:
NAS and SAN are two different things. If you are trying to build a fiber channel SAN, it's going to be out of your budget and so dissimilar to anything that you would ever use in an enterprise that it wouldn't even make sense.

Get a Synology and load it up with your drives, or build a cheapo ZFS or other whitebox NAS. Either will support NFS, iSCSI, and CIFS. That'll allow you to learn block and file-based concepts without breaking the bank.
Click to expand...

All he's trying to do is serve a zfs pool to 2+ servers at higher speed than 1gb for cheap.

As I've said before, you can use omnios+napp-it to serve your zfs pool over fc with comstar. All the fc parts for 1 server and 2 hosts costs $50. That gets you a 4gb link to 2 servers. It works quite well. Add a fc switch into the mix for $110 along with mpio and you have a setup that is as fast as 10gb iSCSI.

Sure, he won't be able to do true ha with it, but he isn't talking about using an external enclosure w/ dual port drives, 2 San heads and ha. Although you can do that, licencing gets expensive.
 
Last edited:
Pointing out that using FC is stupid because it would never be used in an enterprise application and then suggesting I use a Synology instead is comical.

Noticing very few of the brocade 200E's have rack hardware. The rack hardware by itself seems to be quite pricy. Not a deal breaker by any means, did yours have rack hardware?
 
Modder man said:
Pointing out that using FC is stupid because it would never be used in an enterprise application and then suggesting I use a Synology instead is comical.

Noticing very few of the brocade 200E's have rack hardware. The rack hardware by itself seems to be quite pricy. Not a deal breaker by any means, did yours have rack hardware?
Click to expand...

No, but it seems a universal rail kit would work on it. Just the cheap $30-$50 fixed ones on ebay would work. I'm going to have to make a template and drill some holes, but no big deal.
 
Last edited:
I found one that included rails, Working on getting the seller in the $100 range. 16sfps and rails, sounds decent to me.

EDIT: After more negotiation with them they are stopping at $139 shipped. I think that is probably a bit much.
 
Last edited:
Modder man said:
I found one that included rails, Working on getting the seller in the $100 range. 16sfps and rails, sounds decent to me.

EDIT: After more negotiation with them they are stopping at $139 shipped. I think that is probably a bit much.
Click to expand...

Considering the switch usually goes for $100 and the rails go for $80, $139 is good.
 
Another thought I had today, Are you running ECC ram in your SAN? What hardware are you using? I was planning on using an FX6300 but i am not sure if i can get ECC to work on that.
 
I'm using a dl385 g2, 2x opteron 2218, 16gb ecc, lsi sas 9211-8I for the boot drives and ssd's for zil, lsi sas3801e for the sas hba. I've got a rackable se3016 w/ 5z 2tb drIves (raidz1) for media storage, 3x 1.5tb drives (basic pool) for non-critical storage, then a hp msa70 with 25x 73gb sas drives (4x raidz2 + 1 hot spare) with a mirrored zil.

The only 2 things I would like to change is move to a dl380 g5 (with dual quad core's) because it's more power efficient and a sas9200-8e for the external hba.

That supermicro server I linked on the first page of the discussion is a really good server for what you want to do. With a decent pool and a good zil (like a intel s3700 100gb) you'll be able to handle more than a few hosts.
 
wow, that sounds like a power hungry setup. I am trying very hard to avoid a power hungry rack.
 
The 16 bay supermicro would idea around 110w-135w without disks.

Getting a high speed, ultra low power zfs pool can be done, but you'll be spending a bit of money on the processor and motherboard. Adding features like hot swap adds to the cost.

If you do want to go full whitebox look into the c2750 based mobos. They don't use much power and are fast, but the marvel controllers don't work on solaris.
 
Last edited:
I have a stack of norco 8bay 2u cases and PSU's available to use. I just have to pick out the hardware, or go the route you mentioned and pick up a supermicro box. I think the norco diy route will be cheaper but i could be wrong.
 
Then grab a supermicro X8STE, xeon l5520, a 2U 1366 heatsink, 16gb ddr3 ecc (not fb-dimms), a ibm m1015 and you'll be all set.

That should idle at ~80w.
 
Last edited:
Modder man said:
Pointing out that using FC is stupid because it would never be used in an enterprise application and then suggesting I use a Synology instead is comical.

Noticing very few of the brocade 200E's have rack hardware. The rack hardware by itself seems to be quite pricy. Not a deal breaker by any means, did yours have rack hardware?
Click to expand...

It's comical that you are on a tight budget and even considering FC.

I'm simply pointing out that your home grown FC setup will be cost prohibitive and dissimilar to anything you see in an actual production environment.

After re-reading this thread, it's mindboggling that you are considering fiber channel to test a handful of VMs and 3 ESX hosts. Please just try single 1-gbps Ethernet first before wasting so much time and money unnecessarily.
 
REDYOUCH said:
Please just try single 1-gbps Ethernet first before wasting so much time and money unnecessarily.
Click to expand...

are you smoking crack? A single quad port intel nic costs $70. Not to mention he would still need at least an additional intel nic in his hosts because he us using consumer hardware. Even that would leave him with 1gbps. He could go with a qle2464 and serve 2 hosts at 8Gb/sec.

He has to build the fileserver either way.

Cost for intel quad nic and 2x intel nic's is going to be $90-$100

Cost for 4 port qlogic fc hba and 2x 2 port qlogic hba's + fiber is $80-$90.

Same price, 1gbps or 8gbps. What would you choose?

1Gb/sec is Moar better than 8Gb/sec mah! Efernet is aw-ways cheap-arr mah.... duh-huh....
 
bought all the hba's and switch for less than $200.....cant come even close to that with ethernet. I think you misunderstand much of the purpose of this lab. The lab is not so much to learn about the virtual machines and thier OS's, though it can be used for that. My goal is to learn how to setup a virtual infrastructure from start to finish. all of the pieces that make it work. I have 8 hp DL360 G8's at work running my virtual environment. I get to manage that from a machine level every day. What I don't ever get to play with is the hardware so i am giving myself that experience from home.
 
Modder man said:
bought all the hba's and switch for less than $200.....cant come even close to that with ethernet. I think you misunderstand much of the purpose of this lab. The lab is not so much to learn about the virtual machines and thier OS's, though it can be used for that. My goal is to learn how to setup a virtual infrastructure from start to finish. all of the pieces that make it work. I have 8 hp DL360 G8's at work running my virtual environment. I get to manage that from a machine level every day. What I don't ever get to play with is the hardware so i am giving myself that experience from home.
Click to expand...

I agree that it won't be 100% like a San in a business environment but it still forces you to deal with wwn's, mpio, the concept of volumes that some hosts see and others don't. FC also enables you to play with RDM of a FC volume in a VM and teaches you the ups and downs of that setup. FC also enables you to easily use many different OS's with your storage server, providing (what appears to the OS) block level access. Heck, you can even boot from a COMSTAR volume.

There are lots of things to learn from a home-lab with FC.

Here's the guide to enabling target mode with qlogic hba's in solaris.

http://docs.oracle.com/cd/E23824_01/html/821-1459/glddq.html

I would like to point out that he got the hardware to have 1 server and 7 hosts running at 8Gb/sec for $200. Try that with 10 GbE or even 1GbE.
 
Last edited:
You must log in or register to reply here.
Back
Top