• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Cheap Whitebox SAN

Modder man said:
I dont have a regular pci-e slot. The qlogic fiber channel cards are on the other slot
Click to expand...

Forgot that the 1u has a pcie and a uio slot.

The AOC-USAS-L8i states it supports it mode which will make it a plain old hba.

It'll work fine since you are planning on using sas drives. My only concern is that the ssd might be running at 1.5gb/sec, effectively rendering it useless as a zil. It's worth a try but for some reason I remember the 1068e forcing sata drives to 1.5Gb/sec.

If you keep the pool in sync standard you'll still see decent speeds, just make sure you have a ups and automatic shutdown (and good backups)

At least it'll give you a good proof of concept, get you into setting things up and help you decide if you want to build a more capable storage server. If you do build another storage server in the future you can recycle your current one as an esxi node.

Got any pics of your setup so far?
 
Last edited:
My setup is currently a disastrous pile of parts as I dont have a rack yet or any other good way of organizing whats going on. Once i get things a bit organized i will have to take some pics. I think i will have to rule out the AOC-USAS-L8i and figure out a different way. perhaps i need to keep my eyes open for a different piece of hardware altogether.
 
I wonder if the AOC-USAS2-L8i would work. Normally sas2008 controllers treat disks left out of an array as jbod.

Newegg has them for $145 shipped.
 
Last edited:
You want the AOC-USAS2-L8e (= LSI 9211, IT mode)
the L8i is the same board but with IR firmware like a LSI 9211-IR (raid) mode

I would expect that you can
- reflash to IT mode (suggested)
- use it with IR firmware

or
look for the proper L8-e
 
Thanks GEA, I am going to put this project on hold for now. That card cost more than I have spent on all four of these servers put together.
 
So another question. My questions are endless.

buy the AOC-USAS2-L8e GEA mentioned?

Or

Buy an different chassis ready to go? are there any common off lease or otherwise solutions out there? I looked around a little but didnt have a whole lot of luck in what i thought would work well.
 
Just ordered the card you linked from newegg. Time to stop spending money and start playing lol. Found a 24U rack local yesterday so now it is time to get busy and get this environment running. I really appreciate all of the help you all have given me.

Would you suggest the SAN node also have esxi on it and the SAN os be on top of that or give the machine bare hardware?
 
Modder man said:
Just ordered the card you linked from newegg. Time to stop spending money and start playing lol. Found a 24U rack local yesterday so now it is time to get busy and get this environment running. I really appreciate all of the help you all have given me.

Would you suggest the SAN node also have esxi on it and the SAN os be on top of that or give the machine bare hardware?
Click to expand...

I prefer the San node on direct hardware. ESXi passthru tends to me hit or miss from version to version.
 
I have the card now, tried following the instructions you linked. I think I am to green. Missing a few pieces to that puzzle, dos is not something I am familiar with.
 
i logged in as my user via SSH, used the "su" command to become root and literally cut-and-past the commands into the shell, customizing the "devbindings" as necessary. The first command under "How to Display Existing FC Port Bindings" gave me some grief until I realized what symbols I did/didn't need.

Code: 
mdb -k

That will load the interface you need

Code: 
::devbindings -q qlc

That will show you your bindings, you will need those in the second step. In the examples their binding is "pciex1077,2432". You will have to change that to yours.

Code: 
$q

Quits the interface

From there you will need to unbind the "qle" driver:
Code: 
update_drv -d -i 'pciex1077,2432' qlc

Rebind the "qlt" driver:
Code: 
update_drv -a -i 'pciex1077,2432' qlt

Reboot the system

Then to verify the system sees target ports:
Code: 
stmfadm list-target -v

Which will print something close to:
Code: 
Target: wwn.210100E08BA54E60
Operational Status : Offline
Provider Name : qlt(1)
Alias : -
Sessions : 0
Target: wwn.210100E08BA54E60
Operational Status : Offline
Provider Name : qlt(0)
Alias : -
Sessions : 0

After all that is said and done you need to log into napp-it and create a target group for "Port 1" and "Port 2" of the HBA that is now in target mode. After you create the groups use "add members" under target-groups to add the appropriate WWN of the hba to the appropriate group.

Now you have to create 2 host-groups. I would make it "Port 1" and "Port 2" again. Now use the "add-members" under host-groups to add the WWN of one port on the client's HBA to "port 1" and the other WWN to "port 2". Adding all your client's WWN's now is okay.

Create your pool, add the SSD as a write-cache. Use "expand pool".

Create a volume under the "comstar" menu, i prefer thin-provision volumes.

Add a "view" under the comstar menu for your new volume, let "all" see the view.

The ESXi box should see the volume after refreshing the storage. Use ESXi to add storage. View the formatted LUN's properties, manage paths, change it to round robin.

Refresh the storage on the rest of the ESXi boxes and you will see the formatted LUN, then enable round-robin on those too.
 
Last edited:
Woops, I guess I was vague. I will likely need this info later today though so thank you very much for that. I have that raid card now and am trying to flash the firmware and am struggling quite a bit with that.
 
IM thinking its just my unfamiliarity in DOS, they say to just enter the name of the .bat file and it will go. when i do that it says bad command or file name.

Ok now trying to run it in windows 8 i get an error saying cannot downgrade NVDATA version 0x10000012 to 0x030004]

EDIT: Finally got it!!!!! had to use the older versions in MSDOS because the new flash tool does not allow the switch between IR and IT. In short old flasher version with new bios and firmware files = Success
 
Last edited:
These days I have given up on the dos tool, and just use the uefi bios shell tool instead.
Too many issues with new bioses getting in the way and causing it to be unable to flash the card.
 
Do people typically mount their switches in the back of the rack to keep all network runs in the back?
 
Modder man said:
Do people typically mount their switches in the back of the rack to keep all network runs in the back?
Click to expand...

It depends on airflow and cabling requirements. Your FC switch should be mounted to the back because that's how the switch was designed.

Most network switches are designed to be mounted on the front but in environments that you have plenty of airflow, it doesn't matter. In an enclosed cabinet mounting things backwards can be troublesome because of the ventilation currents caused by other devices and vents.

Your switch should go on the front if you have:
  • One switch
  • Inconvenient rear access and/or more distribution drops than server drops
  • Enclosed rack

Your switch should go on the rear if:
  • Your server switch is seperate than your distribution switch
  • You have plenty of access to the rear
  • Open frame rack

In environments with one switch I've seen people put a 48 port switch on the front, a 24 port patch for distribution drops on the front, then a 24 port patch panel on the rear going to another 24 port on the front. That gives you the best of all options. You have ports on the rear of the rack, your airflow pattern is maintained & you have front access to the switch. It works great for sound-controlled enclosures at home.
 
My switches are fanless so i should be fine as far as airflow goes. I just wasnt sure from a wiring perspective what makes more sense. In enterprise i always see them in the front. I have 3 Fanless 12 port junipers for this lab so they could be setup in any order. The other point is there isnt really any outside connectivity to worry about. just one drop to the rest of the house for internet coming in.
 
Here are a few pictures of what it currently looks like. Cableing nightmare right now but its a start.



 
my NAPP-it box is now running and i have all the drives installed. I cannot seem to figure out how to make 2 vdev's of three disks each and strip them. Infact i dont see how to create a vdev at all. I only see how to make a zpool.

Edit: I think i got it figured out. Create the pool with one vdev of 3 disks then extend the pool with the other disks, both vdev's being a raid z. Added my S3700 as zil, and the other 128gb ssd as a read-cache. The machine is running about 550MB read writes in dd-bench. with a 51GB
file.


Next up, time to get the fiber channel configuration started. BDS i followed your comstar configuration setup and when i get to the view all section, it says no LU. EDIT AGAIN: Got it, have to create a LUN on the volume.

NEXT: How does one setup the esxi hosts to connect to that LUN? What sort of setup do i have to do to those FC cards on the hosts?
 
Last edited:
Awesome, the directions I listed about setting up the FC should get you close. I'm interested to see what speeds you will be getting from within a VM. Be sure to do a benchmark with sync standard and write back cache on, then do another one with sync always and write back cache off.
 
The fiber channel is setup on the host/SAN side. What about ESXi? OR were the instructions you gave me for ESXi?
 
Modder man said:
The fiber channel is setup on the host/SAN side. What about ESXi? OR were the instructions you gave me for ESXi?
Click to expand...

Once you configure the san host with the volumes, target groups (should have 2, one for each port on the host), host groups (should have 2 groups with 3 members each) and views the esxi boxes should just see the lun. From there you have to add storage from the "storage configuration" section on esxi and format the lun. After you see it on the first host you can refresh the other hosts storage and they will see the formated volume.
 
Modder man said:
I ended up getting 4 port cards, does that change the setup?
Click to expand...

make 4 target groups, port 1-4.

make 4 host groups, port 1-4

Make hosts 1&2, port 1 WWN's members of the port 1 host group

Make hosts 1&2, port 2 WWN's members of the port 2 host group

Make host 3, port 1 WWN a member of the port 3 host group

Make host 3, port 2 WWN a member of the port 4 host group

That is assuming you are using the fc switch and have 3 hosts.

That'll give you 2 paths (8gb total) to use in MPIO (round robin) back to the San while spreading the load over 4 ports on the San server. You will be using 4 ports on the San server and 2 on each host.


I have found that bonding more than 2 links back to the San can cause latency and lower performance. 8Gb of FC capability is generally the same as 10gb ISCSI performance.
 
Last edited:
As always thank you so much for your help BDS. did you have any trouble with the 200E gui. I think I am having issues with java. The java console says failed to validate certificate, the application will not be executed. I am running java 7.67.

UPDATE: Downgraded to java 7.13 and the gui console works now. In the switch gui I can see my ESXi host connected but the ports the san is hooked into are all showing inactive. The ports the SAN is hooked up to on the 200E show no light.
 
Last edited:
Modder man said:
As always thank you so much for your help BDS. did you have any trouble with the 200E gui. I think I am having issues with java. The java console says failed to validate certificate, the application will not be executed. I am running java 7.67.

UPDATE: Downgraded to java 7.13 and the gui console works now. In the switch gui I can see my ESXi host connected but the ports the san is hooked into are all showing inactive. The ports the SAN is hooked up to on the 200E show no light.
Click to expand...

Be sure the switch was set to factory defaults first. It is was and you still don't have a link you have a physcial problem. The link is up regardless of seeing any luns or anything.
 
I set the switch to factory defualts myself last night, the only thing that has been modified is the management IP. Ok a hardware problem is what I was afraid of. I was hoping there was some change I have the drivers/config wrong on napp-it. When I get home this evening I will try some different ports on the switch and/or try a different HBA. I ended up with like 10 of those 4gb HBA's lol.
 
Do you get the bios prompt at boot for the 2464 and does the OS see the card? If so start by changing patch cords with known good ones, changing sfp's and ports.
 
I will doublecheck, I know one of the machines was flagging an error with the card not having a rom bios on boot. have you ever seen that?

I am thinking most likely the qlogic card in the san is bad.

EDIT: Further testing....either the card or the drivers in omni-os are bad. should be an easy fix though.
 
Last edited:
Modder man said:
I will doublecheck, I know one of the machines was flagging an error with the card not having a rom bios on boot. have you ever seen that?

I am thinking most likely the qlogic card in the san is bad.

EDIT: Further testing....either the card or the drivers in omni-os are bad. should be an easy fix though.
Click to expand...

if you can get into the configuration of the FC card when it prompts the card is at least active. I've seen cards that won't link but still show up too.

Honestly, it seems a little odd that the drivers would be bad in omni-os. The cards have been around for a while and haven't had any issues in opensolaris that i'm aware of.
 
FIXED:

I executed the commands below to get the fiber channel HBA to work on the OMNI-OS SAN.

Install the storage server package.

pkg install storage-server

View the state

svcs stmf

Enable the stmf service

svcadm enable stmf


Always another hick-up though. The HBA's all seem to be working but some of the connections on the brocade 200E have a red X and say invalid connection.

On another note I am having a hard time adding the hosts to the host group. SOLVED: I Added the WWN listed in the brocade to the napp-it and and it worked, the hosts are now listed as members of the host groups. Unfortunately even after the WWN's are added to the host group ESXi is still not seeing the Volume
Sorry for all of the questions, you have explained this a few times and by now I should understand. There is just something small I am overlooking.
 
Last edited:
My esxi hosts are not seeing the LUN's after all of these steps, should i be considering a hardware issue perhaps with the switch or is it more likely I did something wrong with the host and target groups?
 
Have you set a view from your Logical Unit to your target group?
(A view make a LU visible in a target)
 
yeah I setup the view to be view all. I think I am going to try to gut the config and start fresh from a config perspective.
 
If everything on the switch is showing a 4gb link and active then everything should be good. I know I messed up a few WWN's in the host group the first time. If the ESXI hosts WWN's aren't in there correctly nothing show's up.

Don't forget that nothing will show up in the esxi "storage" configuration until you hit add storage and format your new drive. The first place you will see the lun is under storage adapters.

ESXi should look like this (note the appearance of the LUN's and the adapters WWN):
lun.jpg


Comstar host groups should look somewhat like this (note: the ESXi server above uses "port 1" and the WWN appears in "port1" on the host group):
hostgroup.jpg


Could you post pics of your host groups, target groups, views and volumes screens under the comstar menu? Also post pics of the storage adapter menu of one esxi host that has a lit fc connection. That would help a lot.
 
Last edited:
will post some pics in the morning. Today I tried running the config directly from the cli, basically just created a volume then a LUN, I set the LUN to view all. This LUN shows up in one of my three hosts but not the other two. I am starting to speculate an issue with the switch.
 
GOT IT!!! I was having issues with the switch. It is close to plug an play but not quite. You have to run the setup manager, to set ports as storage or HBA. Now that I have that set properly all three hosts see my LUNS. At this point it looks like I may have one bad fiber patch/HBA but all hosts have at least one working port so they work.

Here are drive benchmarks from a win7 vm, seems kinda slow compared to the benchmarks directly on the SAN



 
Looks like you have sync writes enabled. Really good speeds for that. did you enable round robin on esxi?
 
I will have to learn how to do round robin, Is there a way to verify sysc writes are enabled?

EDIT: Round robin Turned on and benchmarks after



 
Last edited:
You must log in or register to reply here.
Back
Top