SANs & Virtual Machines: A Question Or Two

[parityboy]

This is just to clarify things in my own head. I've been doing some reading on SANs, and trying to learn about zoning etc.

1) When people say they have a SAN serving virtual machines, do they simply mean that a given VM's OS "disk" is in fact an iSCSI target, and that every read and write operation performed by the guest OS travels over the iSCSI network to the LUN? Or do they mean something else?

2) RAID over a SAN. Is this worth it, or is the latency and/or overhead too much to make it viable? Is something like ZFS worth deploying on the "front" server, its zpool consisting of the iSCSI targets sitting in a JBOD "behind" the server?

I'll probably have more questions, but this will do for now. Many thanks.

 

[Blue Fox]

A SAN doesn't have to be iSCSI. FC is still widely utilized due to the much higher speed/cost ratio. RAID is something that would generally be handled by a SAN instead of whatever system you have attached to it, so there won't be any extra latencies or overhead. If you want to toy around with a whitebox unit instead of getting a real SAN, download Openfiler and play around with it. A proper SAN (or what most people would consider to be one) isn't going to be running on some x86 hardware though and will be a standalone unit.

 

[packetboy]

This is precisely what I'm doing with ZFS.

vmware ESXI - diskless
| - 1Gbps "SAN"
Storage "Controller" - Solaris/ZFS
| - 24Gbps SAS link
Supermicro SC847 JBOD enclosure (100TB)
|
(future)
JBOD #2 - 100TB
|
JBOD #3 - 100TB
|
etc..

With VMWare ESXI you can create storage pool either via iSCSI *or* NFS.

I prefer NFS as it allows you to see the all the files in the NFS mount from Solaris itself vs. only beeing able to manipulate the files from ESXI.

Then here is where the real power of ZFS comes in.

Let's say you have one VM booting from a ZFS vol here:

/tank/vmware/ubuntu1/...

1) VM snapshot and backup

# zfs snapshot tank/vmware/ubuntu1@20110225

The above literally takes 2 seconds and does not impact the VM at all. Try doing a snapshot from within VMware...it can take minutes and causes the VM to "pause" while the snapshot is in process.

If you need to rollback the VM to a prior state, shutdown the VM, issue zfs rollback, restart VM...and poof you are back to where you were.

2) VM clone

Let's say you want to fire up 5 more Ubuntu VMs:

# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu2
# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu3
# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu4
# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu5
# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu6

Share each of the above with NFS, add them as data stores in ESXI console, then browse each data store and import as new VM. VMware will automatically prompt you to assign a new unique VM ID for each...boot up each VM and you have 5 new VMs all within a few minutes.

Most importantly, because these VMs where cloned from ubuntu1 they all share the same underlying disk blocks (except for delta changes)...combine this with Layer 1 and Layer2 ZFS caching and there's a good chance that the entire disk image for these VMs will be served from high-speed cache, possibly enabling better disk performance than if you have a local physical drive.





[/CODE]

 

[LightningCrash]

Let's say you want to fire up 5 more Ubuntu VMs:

# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu2
# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu3
# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu4
# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu5
# zfs clone tank/vmware/ubuntu1@20110225 tank/vmware/ubuntu6

Share each of the above with NFS, add them as data stores in ESXI console, then browse each data store and import as new VM. VMware will automatically prompt you to assign a new unique VM ID for each...boot up each VM and you have 5 new VMs all within a few minutes

But don't you still have lots of manual processes there? You have to bring them up without active NICs, reconfigure networking, reconfig hostnames, truncate log files, gen new SSH keys, etc.

KS/Preseed could be finished in mere minutes.

 

[parityboy]

Thanks for that packetboy. Is there any value in the VM's OS disk being a direct iSCSI target (a LUN), or is it better to have it as a virtual disk accessed as a file (via NFS for example)?

A SAN doesn't have to be iSCSI. FC is still widely utilized due to the much higher speed/cost ratio. RAID is something that would generally be handled by a SAN instead of whatever system you have attached to it, so there won't be any extra latencies or overhead. If you want to toy around with a whitebox unit instead of getting a real SAN, download Openfiler and play around with it. A proper SAN (or what most people would consider to be one) isn't going to be running on some x86 hardware though and will be a standalone unit.

Yeah, I know a SAN doesn't have to be iSCSI. I'm aware of other SAN technologies such as FibreChannel and ATA-over-Ethernet.

As for what makes a "proper" SAN...from what I have read, the hardware in them is in fact commodity (as opposed to truly custom), it's just that it's been well tested by the vendor and all the parts are known to work cohesively.

There are no doubt many things that make a SAN "proper", but the landscape is changing. For me the stand out items are:

1) Hard drives. As an example, Clariion drives use 520 byte sectors, adding an extra 8 bytes of ECC. This is now negated by 4K sector drives (when they go native). Technologies such as ZFS help to make commodity hardware more attractive.

2) Networking. FibreChannel is the technology of choice, but 10GigE is here and can run over fibre. With FCoE, the next battle looks like it will be iFCP vs FCIP vs iSCSI.

3) Support. This is where proper SANs win IMO. Open solutions have a huge community, but there's no substitute for 4-hour support when something starts to flake out.

For me starting my own business, $40,000 SANs are out of reach. However, I would like to start the business with a scalable infrastructure in place so as to ease the growth process by minimising the inertia, which is why I'm trying to learn as much as I can.



I thought I'd run this past you girls & guys, to garner some opinions.

Router
Dell R210 running vyatta. This will take care of LAN, WLAN, and WAN functions.

Main Server
Dell R510 running Ubuntu Server w/KVM running off 4x SATA in RAID10. This will run a number of virtual machines (all running Ubuntu Server):

- Communications (POP3, IMAP4, SMTP, TeamSpeak, remote access)
- Directory services (LDAP, DNS)
- File serving (NFS, SMB, AFP)
- Source code control (git)

SAN Box(es)
2x Dell R515s running (OS to be decided). Each box exports logical volumes as iSCSI targets, and there is replication between the boxes (like DRBD). The targets are used by the VMs for storage, so for example /home/users/ would be mounted within the file serving VM from an iSCSI target on the SAN. This would allow the VM images themselves to be small (32GB) since they won't actually be holding user-generated data.

Networking
The SAN boxes are connected privately to the main server via a managed GigE switch (perhaps one with iSCSI prioritisation). The main LAN will probably be 100MBit Ethernet. Both switches will be Dell PowerConnect.

Backup
I'm thinking that the main server should run Amanda Enterprise Server on the host, with an client agent on each of the VMs. A client agent on the main server would backup the VM images. A tape library of some sort would hold the physical backups.

I'm looking at a few candidates to run on the SAN boxes. Currently, they are:

1) NexentaStor

2) Nexenta Core & napp-it

3) QuantaStor. I'm not sure how long these guys will be around.

What do you think? Is this viable? Is it overkill? Is there a better way?

 

[LightningCrash]

As for what makes a "proper" SAN...from what I have read, the hardware in them is in fact commodity (as opposed to truly custom), it's just that it's been well tested by the vendor and all the parts are known to work cohesively.

There's no real `commodity` redundant controller, though. I can replace controllers without taking the whole SAN down. I can go from a single controller on an array to a dual controller without killing the SAN. Microcode updates are online and don't interrupt anything, etc.

 

[pjkenned]

A SAN doesn't have to be iSCSI. FC is still widely utilized due to the much higher speed/cost ratio. RAID is something that would generally be handled by a SAN instead of whatever system you have attached to it, so there won't be any extra latencies or overhead. If you want to toy around with a whitebox unit instead of getting a real SAN, download Openfiler and play around with it. A proper SAN (or what most people would consider to be one) isn't going to be running on some x86 hardware though and will be a standalone unit.

NetApp Filers (for example) use x86 CPU's with modified Intel chipsets, custom boards and etc. Not EMC, but on the other hand, still good enough for like $5B in revenue.

BTW @ OP Clariion is out, VNX is in.

 

[Mindflux]

There's no real `commodity` redundant controller, though. I can replace controllers without taking the whole SAN down. I can go from a single controller on an array to a dual controller without killing the SAN. Microcode updates are online and don't interrupt anything, etc.

Yeah, but the days of commodity active/active failover costing less are here. You can have two chassis with failover for less than some single box "enterprise" solutions with those hot pluggable SAS/FC controllers.

 

[sabregen]

NetApp Filers (for example) use x86 CPU's with modified Intel chipsets, custom boards and etc. Not EMC, but on the other hand, still good enough for like $5B in revenue.

BTW @ OP Clariion is out, VNX is in.

NetApp exclusively uses AMD Opteron 2xxx and 8xxx (x64) CPUs, not Intel. It is therefore deducible that they do not have Intel chipsets in them.

OP - SAN is Storage Area Network. The basics are either:

Fibre Channel, iSCSI or SAS connectvity (physical). All use SCSI procotol, just encapsulation methods are different.

FC is the only one where the term Fibre Channel describes not only the physical connectivity from the host to the storage medium, but also the protocol that it rides on. A typical FC SAN involved a FC HBA in the host, a FC switch, and a FC addressable RAID enclosure (but could also be JBOD). FC RAID enclosures, depending on model, can tyipcally support FC, SAS and SATA drives, although the rules of what type can be populated in what storage chassis will vary depending on manufacturer and model. Native FC drives (and in fact, all FC devices, by nature of the FC standard) are dual ported. The HBAs will have a WWN and a WWPN, as will the switches, and the storage controllers and drives. FC drives actually have two data ports on them, and the controller arbitrates traffic to one or the other. By design, it is intended for high availability (and is therefore, per GB of storage), the most expensive to implement (hardware), most of the time.

iSCSI is internet SCSI, and it's (by design) routable (and as the name would imply, over the internet). iSCSI is attractive to many because it can be implemented with regular SATA drives and gigabit network switches (you can also do 10Gb iSCSI, if you have the hardware). many iSCSI RAID controllers can also utilize SAS drives.

SAS SANs are just SAS connected storage devices, and can usually utilitize SATA and SAS drives. Some SAS drives are dual ported, but most are not.

Think of a SAN as a network, with switches, a host, and physical connectivity with a protocol running on top, connecting the host to it's storage, and you've got he basic idea.

 

[parityboy]

NetApp Filers (for example) use x86 CPU's with modified Intel chipsets, custom boards and etc. Not EMC, but on the other hand, still good enough for like $5B in revenue.

BTW @ OP Clariion is out, VNX is in.

Cheers for that. It was the only example that came to mind.

Yeah, but the days of commodity active/active failover costing less are here. You can have two chassis with failover for less than some single box "enterprise" solutions with those hot pluggable SAS/FC controllers.

Exactly this. As these things commoditise and networks get faster and cheaper, you can go from boxes with multiple controllers to multiple boxes with single controllers, connected to each other via trunked GigE or 10GigE.

NetApp exclusively uses AMD Opteron 2xxx and 8xxx (x64) CPUs, not Intel. It is therefore deducible that they do not have Intel chipsets in them.

OP - SAN is Storage Area Network. The basics are either:

...

Cheers for the info. I already had an idea of what a SAN is and what the rationale is for their existence and continued evolution. However, I know I'm missing some information concerning configuration and best practises - hence the questions.

 

[pjkenned]

NetApp exclusively uses AMD Opteron 2xxx and 8xxx (x64) CPUs, not Intel. It is therefore deducible that they do not have Intel chipsets in them.

You are kidding right?

FAS2020/2050 = Mobile Celeron M
FAS32xx = Harpertown http://communities.netapp.com/docs/DOC-9215
FAS62xx = Westmere and Nehalem based http://www.netapp.com/us/communities/tech-ontap/tot-fas-6200-1011-hk.html the FAS6280 is 4x Intel X5670 for example

Now if you were talking last gen 3xxx and 6xxx series that is different, but saying that NetApp exclusively uses AMD, when all of their current-gen hardware use Intel CPUs does seem like a bit of a stretch.