ESXi Disk/LUN size restriction

[Cyberrad]

In ESXi is there a restriction in the size of Disk or LUN that you can use/create?

I have created a RAID 5 of 5 1TB hard drives for 3.64TB of space and I can only get ESXi to create a 1.64TB datastore. Why won't ESXi allow me to use the full array?

 

[NetJunkie]

SCSI2 has a limitation of 2TB. So, what you do is take the RAID Group and make 3 LUNs. Then use extents to make a larger datastore. How many VMs do you plan to have on that?

 

[Cyberrad]

SCSI2 has a limitation of 2TB. So, what you do is take the RAID Group and make 3 LUNs. Then use extents to make a larger datastore. How many VMs do you plan to have on that?

How do I go about doing that? I do not plan to put any VMs on this array. The disks are local so I wasn't sure how to go about just letting one of the VMs see the array.

 

[SpaceHonkey]

I'm going to beat Lopoetve to it -

Extents are BAD - do not use them!!

In the event that you have to rebuild the datastore for some reason and it used extents, good luck.

Now I'll have to let him explain why, but that's one of those Lopoetve's bits-o-knowledge that I filed away in my library.

 

[NetJunkie]

I'm going to beat Lopoetve to it -

Extents are BAD - do not use them!!

In the event that you have to rebuild the datastore for some reason and it used extents, good luck.

Now I'll have to let him explain why, but that's one of those Lopoetve's bits-o-knowledge that I filed away in my library.

Yes. Extents are bad. No doubt. But if he's doing a 4TB datastore I figure this is a test/lab box. No way I'd ever do that in production. Going over 2TB on a datastore should mean you have one VMDK that is over that size...and you need to seriously look at what you're doing.

 

[NetJunkie]

How do I go about doing that? I do not plan to put any VMs on this array. The disks are local so I wasn't sure how to go about just letting one of the VMs see the array.

Partition the drives.

 

[AreEss]

Do not partition the drives. Use smaller LUNs. You do not want to exceed 512GB in a reasonable environment. (This is an older ESX hard limit, and a good rule of thumb.) LUNs should be sized appropriately for the guest. Any application realistically requiring greater than 512GB should be using multiple disks and controllers, or should only ever be run on full hardware.

 

[NetJunkie]

Do not partition the drives. Use smaller LUNs. You do not want to exceed 512GB in a reasonable environment. (This is an older ESX hard limit, and a good rule of thumb.)

Read what he wrote. This isn't a SAN. There is no LUN. It's direct attached disk.

 

[lopoetve]

Yes. Extents are bad. No doubt. But if he's doing a 4TB datastore I figure this is a test/lab box. No way I'd ever do that in production. Going over 2TB on a datastore should mean you have one VMDK that is over that size...and you need to seriously look at what you're doing.

can't even do that

Max filesize = 2TB as well. Greater than that is only really useful for physical mode RDMs (which you can't do on local drives and have their own issues) or windows dynamic disks, and I'll let you figure out the problems with that

 

[lopoetve]

How do I go about doing that? I do not plan to put any VMs on this array. The disks are local so I wasn't sure how to go about just letting one of the VMs see the array.

What exactly are you trying to do?

 

[Thuleman]

I am still waiting on an official VMware statement that says anything about 500GB. I have read a couple three posts on the community forums that call BS on that and say that the LUNs should be sized to fit the need (VM*disk per VM+overhead) rather than some arbitrary number pulled out of someone's hat.

I tend to side with lopoetve on this one since I have no reason not to, but it's strange that best practices doesn't mention the 500GB rule. If I could only find the time it would be awesome to actually set up a 500GB and a 2TB LUN and just run some performance tests on it to see what's what.

 

[AreEss]

Read what he wrote. This isn't a SAN. There is no LUN. It's direct attached disk.

Sigh. So what are you cranky about?
Here's an idea; instead of being cranky, learn about RAID and SCSI before opening your mouth and removing all doubt.
Arrays do not have SCSI IDs. Only LUNs have SCSI IDs. A SCSI ID is a target ID, which is required to be a valid disk. Internal, external, does not matter in the least and never has. That's how RAID functions. You make an array, you make LUNs on the array, the host talks to the LUN, the end.

 

[AreEss]

I am still waiting on an official VMware statement that says anything about 500GB. I have read a couple three posts on the community forums that call BS on that and say that the LUNs should be sized to fit the need (VM*disk per VM+overhead) rather than some arbitrary number pulled out of someone's hat.

512GB, and it's an old limit, as I said. 3.0 and prior. Regardless, sizing that absurdly large indicates that either you're doing things with a guest you really shouldn't in testing or production, or you're taking the lazy way out.
The reasonable sizing is to create the array, then create appropriately sized LUNs for each VM guest from that array. (Yes, I have done this before.) Unless you're using an old 3Ware, Intel Matrix (NONONO) or some oddball not on the HCL, your controller will support not just setting up appropriate LUNs but online resizing. Well, okay, no real online resizing because you need the management software running on hardware (e.g. inside ESXi, which isn't gonna happen,) but you can resize with reboot worst case.

I tend to side with lopoetve on this one since I have no reason not to, but it's strange that best practices doesn't mention the 500GB rule. If I could only find the time it would be awesome to actually set up a 500GB and a 2TB LUN and just run some performance tests on it to see what's what.

I don't disagree with the statement that 512GB is somewhat arbitrary. As I said, it's an old limitation, but generally a good rule of thumb in my book. With no real load, you won't see any performance difference.
However, creating excessively large LUNs reduces the number of spindles in your total pool drastically. The less spindles in the pool, the more load on the spindles you do have, which with large numbers of VMs on limited spindles does create performance problems. (Which Iopoetve will of course, agree with.) Remember that your amount of random seek is inevitably going to be significant, because you have multiple guests asking for multiple blocks. Each time you hit a different VMDK, you have to do a seek. Optimally in your configuration, I would probably RAID1 two drives to serve as the ESXi install and guest OS VDMKs, then add three more drives for a 4+1 RAID5 and shared hotspare. Under testing, 512GB was found to basically be capable of handling enough typical guests that their disk loading exceeded reasonable capability - array loads averaged over 85% load with peak wait of 60ms and average of 26ms on 15k FC2 disks.

My general recommendation to folks on VMWare ESX and ESXi is to create distinct arrays when possible, one on the fastest disks you can afford for the OS images. Size your OS VMDKs appropriately, but small as possible, on as many LUNs as reasonable. (e.g. Server 2k3 might only get a 40-60GB C:.) Spread load across as many fast spindles as you can. You have a paging activity on all guests, which can occur at any time or every time, which hammers the disks. The more seek you do, the slower it gets.
Then, create arrays on disks with appropriate speeds and capacities for application and storage, preventing them from crossing with the OS disks. This allows you to create your 20GB little app that wants fast disk on 15k SAS, and put your file server on seven or eight or a dozen 1TB LUNs on a 7200 SATA array.
But again, this is a general recommendation. Call it part of my "bestest theories of doing it." Not everyone can afford it, and it's not always appropriate for the environment.

 

[Cyberrad]

What exactly are you trying to do?

The end result is to have the hardware RAID 5 from my Host accessible to one of my Guests. The RAID array is 3.64TB (5 1TB hard drives) which in VMware ESXi seems to be limited to 1.64TB when creating a Datastore. This is just for simple storage for a particular guest. Is there a way to map a vmdk file to the physical array? Or rather what do you suggest?

 

[lopoetve]

The end result is to have the hardware RAID 5 from my Host accessible to one of my Guests. The RAID array is 3.64TB (5 1TB hard drives) which in VMware ESXi seems to be limited to 1.64TB when creating a Datastore. This is just for simple storage for a particular guest. Is there a way to map a vmdk file to the physical array? Or rather what do you suggest?

You can't map that much to a guest, and you can't map raw local storage to a guest either. You're going to be limited to 2TB volumes (-512bytes), and you'll have to make VMDK's on those and then use dynamic disks / software raid to put them together.

Moreover though, I expect this is a system that could benefit from ~not~ being virtualized - what is the goal of making it a VM if it needs that much hardware?

 

[lopoetve]

I am still waiting on an official VMware statement that says anything about 500GB. I have read a couple three posts on the community forums that call BS on that and say that the LUNs should be sized to fit the need (VM*disk per VM+overhead) rather than some arbitrary number pulled out of someone's hat.

I tend to side with lopoetve on this one since I have no reason not to, but it's strange that best practices doesn't mention the 500GB rule. If I could only find the time it would be awesome to actually set up a 500GB and a 2TB LUN and just run some performance tests on it to see what's what.

VMFS performance is always happiest in the 400-600gb range - that's a given, but it's not a major difference either way, especially these days. I don't even bother worrying about it anymore, to be honest - the big question is far more the number of VMs per lun / spindle / path - those have a MUCH more significant performance impact than the VMFS filesystem itself.

 

[NetJunkie]

Sigh. So what are you cranky about?
Here's an idea; instead of being cranky, learn about RAID and SCSI before opening your mouth and removing all doubt.
Arrays do not have SCSI IDs. Only LUNs have SCSI IDs. A SCSI ID is a target ID, which is required to be a valid disk. Internal, external, does not matter in the least and never has. That's how RAID functions. You make an array, you make LUNs on the array, the host talks to the LUN, the end.

EDIT: I'll edit so I'm not continuing to be jerky. I know storage..I think a lot of people at EMC would agree I do. I do enough speaking and work for them (I work for a partner). I was being simple as I don't think he has a real deep understanding of SCSI/RAID/LUNs/Targets. A lot of these small RAID cards will do a RAID Group and then just present that to the local host as a single LUN without giving you an option to carve it up. I said to use smaller LUNs in my original reply. If he can locally, awesome.

Going by what he wrote I figured it was a pretty basic setup. Most of these internal RAID boards let you setup one or more RAID Groups and then present that as one default LUN. You don't have the option to carve that RG up in to multiple LUNs and presented locally. If his does, awesome. Do that, which if you notice is what I told him to do in my original reply. But if he can't..... he can't.

 

[lopoetve]

A better thing to do almost is to refer to LUN without the acronym - it's a logical volume, or logical unit - that makes more sense most of the time to local card users, since those are the terms it will normally present.

 

[AreEss]

EDIT: I'll edit so I'm not continuing to be jerky. I know storage..I think a lot of people at EMC would agree I do. I do enough speaking and work for them (I work for a partner). I was being simple as I don't think he has a real deep understanding of SCSI/RAID/LUNs/Targets. A lot of these small RAID cards will do a RAID Group and then just present that to the local host as a single LUN without giving you an option to carve it up. I said to use smaller LUNs in my original reply. If he can locally, awesome.

Going by what he wrote I figured it was a pretty basic setup. Most of these internal RAID boards let you setup one or more RAID Groups and then present that as one default LUN. You don't have the option to carve that RG up in to multiple LUNs and presented locally. If his does, awesome. Do that, which if you notice is what I told him to do in my original reply. But if he can't..... he can't.

My bag isn't just external arrays, but internal arrays. That's why I'm grouchy about it, because I really have to know both sides of the house extensively. (I build high end systems, plus work with high end arrays.)
Almost every single card, with very very few exceptions, on the VMWare HCL will do multiple LUNs off a single physical array. The reason being the same reason we're talking about it here - it's simply not reasonable to present a single 5TB+ LUN to a system. Rather than get fancy and play filesystem-as-RAID (Intel Matrix RAID,) they stuck with a sane solution of LUNs. And before folks crack on m-too late. But either way, I stick to LUNs because it's the standard. Plus Logical Drive gets people thinking Windows software RAID, ugh.

Moreover though, I expect this is a system that could benefit from ~not~ being virtualized - what is the goal of making it a VM if it needs that much hardware?

the big question is far more the number of VMs per lun / spindle / path - those have a MUCH more significant performance impact than the VMFS filesystem itself.

See? What did I tell you? I'm just not the best at explaining things sometime.

 

[Cyberrad]

Well, I appreciate the help guys. I went a head and did what lopoetve suggested. I created individual volumes with data stores. Then I made vmdk of each disk and added it to the guest. Then I created the RAID in the guest.

Thanks for your help again.

 

[NetJunkie]

Moreover though, I expect this is a system that could benefit from ~not~ being virtualized - what is the goal of making it a VM if it needs that much hardware?

Portability. I have customer with a single VM running on a single ESX box. HA/FT/SRM..all benefits.

 

[AreEss]

Portability. I have customer with a single VM running on a single ESX box. HA/FT/SRM..all benefits.

All of which can be performed with Microsoft Cluster, saving thousands of dollars. Generally anything that big on Windows, is a file server. Cluster can handle all of that; I got a crash course on exactly how it's done and how to fix it when it was done wrong. (Software RAID, big no-no.)

 

[NetJunkie]

All of which can be performed with Microsoft Cluster, saving thousands of dollars. Generally anything that big on Windows, is a file server. Cluster can handle all of that; I got a crash course on exactly how it's done and how to fix it when it was done wrong. (Software RAID, big no-no.)

No, I'm sorry but it can't. Microsoft cluster can handle an application or system failover like HA. It can't do a stateful active/active failover like FT. It can't do complete automated environment failover like SRM. By virtualizing systems you gain a lot of flexibility on how you protect and failover your environment. Sure, you can do MS Cluster, NeverFail, WanSync, etc but those are point solutions that greatly increase the complexity by doing it many times in an environment. I can virtualize all the servers and put them under the exact same DR strategy and plan by using something like Site Recovery Manager. That's why we're seeing a lot of our customers finally virtualizing those Tier 1 apps, even if it's 1 App to 1 Server (rare, but it happens).

I have multiple customers moving off MS Cluster right now because they hate it. Complexity. Updates. Just general issues. They are much happier to let VMware HA handle it, or get even better reliability with FT. Then they roll it in to the easily testable DR strategy of SRM. I just gave a great keynote on this at the Carolina VMware Summit.

 

[AreEss]

No, I'm sorry but it can't. Microsoft cluster can handle an application or system failover like HA. It can't do a stateful active/active failover like FT. It can't do complete automated environment failover like SRM. By virtualizing systems you gain a lot of flexibility on how you protect and failover your environment. Sure, you can do MS Cluster, NeverFail, WanSync, etc but those are point solutions that greatly increase the complexity by doing it many times in an environment. I can virtualize all the servers and put them under the exact same DR strategy and plan by using something like Site Recovery Manager. That's why we're seeing a lot of our customers finally virtualizing those Tier 1 apps, even if it's 1 App to 1 Server (rare, but it happens).

Read what I said again; Microsoft Cluster can do all these things in a file/print sharing usage. It does them better, faster, and cheaper than anything else out there. It also does something VMWare et. al. absolutely cannot do, which is Active-Active HA. Users may be connected to one system for one share, then the other system for another share. They won't notice if the first goes down, because the share just migrates to the second system. For that specific application, there simply is nothing better than Microsoft Cluster. Even in VMWare, people don't use ESX's HA to do it. They create two guests, and do an MS Cluster between them.

I have multiple customers moving off MS Cluster right now because they hate it. Complexity. Updates. Just general issues. They are much happier to let VMware HA handle it, or get even better reliability with FT. Then they roll it in to the easily testable DR strategy of SRM. I just gave a great keynote on this at the Carolina VMware Summit.

Hey, never said it was easy, just that it's the best for file/print shares. The stories I could tell you about six MCPs who couldn't set up MS SQL within MS Cluster over a six week period...

 

[NetJunkie]

Read what I said again; Microsoft Cluster can do all these things in a file/print sharing usage. It does them better, faster, and cheaper than anything else out there. It also does something VMWare et. al. absolutely cannot do, which is Active-Active HA. Users may be connected to one system for one share, then the other system for another share. They won't notice if the first goes down, because the share just migrates to the second system. For that specific application, there simply is nothing better than Microsoft Cluster. Even in VMWare, people don't use ESX's HA to do it. They create two guests, and do an MS Cluster between them.



Hey, never said it was easy, just that it's the best for file/print shares. The stories I could tell you about six MCPs who couldn't set up MS SQL within MS Cluster over a six week period...

EDIT To Clarify: What you are talking about isn't Active/Active. Active/Active means two servers are answering requests or processing the same jobs. You have two systems in a MS cluster doing file shares, but it's not Active/Active. It would be A/A if both servers were hosting the same shares, they aren't. When one fails the resources have to be moved to the secondary box, which takes time.

Go look at vSphere FT. Then get back to me on active/active clusters. I can fail a VM to a second box in under second with CPU/memory in the exact same context as before. Microsoft clustering isn't active/active. It's two systems. You can put shares on each but at any one point in time that share only lives on one system (outside of DFS). If one fails you still have to bring up the resources/shares on the second host. That's not instant. vSphere FT isn't true active/active as both VMs aren't answering client requests but it fails in well under a second and logs/ships all CPU/memory writes so it can continue answer the same client requests.

Second, we're talking about a large server with large resource requirements. File servers usually aren't. The only thing they require is disk space, which is easy, and sometimes network throughput, which is easy. Normally a client can withstand a HA failover and a reboot of the VM since the client timeout is so high. But, with something like FT you can do it even faster. Then if your primary DC is a smoking crater you can fail to DR in a few minutes. Portability. Simplicity.'

Microsoft Clustering is a pretty basic active/passive cluster system. It's been around a while..well used and documented, but it's a standard setup. Plus, if you want to keep your apps separate you know have a lot of servers in a clustering arrangement. With something like VMware FT I can right-click, enable FT, on any VM and get instance protection as long as I have resources in my cluster.

Simplicity.

 

[lopoetve]

Read what I said again; Microsoft Cluster can do all these things in a file/print sharing usage. It does them better, faster, and cheaper than anything else out there. It also does something VMWare et. al. absolutely cannot do, which is Active-Active HA. Users may be connected to one system for one share, then the other system for another share. They won't notice if the first goes down, because the share just migrates to the second system. For that specific application, there simply is nothing better than Microsoft Cluster. Even in VMWare, people don't use ESX's HA to do it. They create two guests, and do an MS Cluster between them.



Hey, never said it was easy, just that it's the best for file/print shares. The stories I could tell you about six MCPs who couldn't set up MS SQL within MS Cluster over a six week period...

http://www.vmware.com/products/fault-tolerance/

Give it a read We can do it now. State is maintained between two systems at the same time - much like an ongoing vMotion. Failover is transparent to OS/users, handed entirely at the hypervisor level.

The key benefit that MSCS has at this point is that the 2 nodes are totally separate machines - you can patch one while the other hosts your content. You can't do that with FT, as it's a shadow machine at the hardware level.

 

[AreEss]

http://www.vmware.com/products/fault-tolerance/

Give it a read We can do it now. State is maintained between two systems at the same time - much like an ongoing vMotion. Failover is transparent to OS/users, handed entirely at the hypervisor level.

The key benefit that MSCS has at this point is that the 2 nodes are totally separate machines - you can patch one while the other hosts your content. You can't do that with FT, as it's a shadow machine at the hardware level.

Yeah, I'm fairly familiar with it, Iopoetve. It's the closest POWER6 PowerVM+PowerHA wannabe. The problem is that obviously, it's a lot more resource expensive, to put it mildly.

But yes, that's also it's weak point. You have nowhere to fail to during patching, nowhere to fail to if it's a software problem. Sure, you have hardware protection, but you have absolutely no software protection whatsoever. And the fact is that you are far, far, far more likely to have a software fault or failure than a hardware fault.

 

[lopoetve]

Yeah, I'm fairly familiar with it, Iopoetve. It's the closest POWER6 PowerVM+PowerHA wannabe. The problem is that obviously, it's a lot more resource expensive, to put it mildly.

But yes, that's also it's weak point. You have nowhere to fail to during patching, nowhere to fail to if it's a software problem. Sure, you have hardware protection, but you have absolutely no software protection whatsoever. And the fact is that you are far, far, far more likely to have a software fault or failure than a hardware fault.

True, but VMware is a hardware company - virtualized hardware, mind you, but hardware nonetheless. It takes a lot of bandwidth, and the right cpu, but it doesn't take that much oomph, especially on Nehalem.

My experiences with MSCS is that it's far more of an issue than a solution, especially when combined with VMware products. The reservation conflicts (pre-vSphere), lack of HA/DRS (check the guide people, it's NOT supported in conjunction with those technologies, for good reason), limitations on number of nodes, etc...

I like MSCS - it has solid uses for things that need software redundancy, but to be really effective your software needs to be MSCS aware, which limits you not only to windows and certain software packages, but to a very narrow subset of those packages - sure, you can MSCS any process, but for it to truly play well it has to be aware. Most things aren't.

FT works on ANY VM, with ANY software package, on ANY supported hardware - it's totally transparent to the user and to the OS, which is the goal - it's HA at the hardware layer, which is simply awesome for stuff that must simply ~not~ have hardware related failures. And with a 10gE backplane, your key resource limitation is gone We support 10gE for FT

 

[sabregen]

brings up an interesting question...what 10GbE cards currently have drivers for ESX4? I can tell you that it's not Myricom, which is the favorite. Chelsio, maybe?

 

[AreEss]

My experiences with MSCS is that it's far more of an issue than a solution, especially when combined with VMware products. The reservation conflicts (pre-vSphere), lack of HA/DRS (check the guide people, it's NOT supported in conjunction with those technologies, for good reason), limitations on number of nodes, etc...

Actually, MSCS + HA is a supportable solution, provided you have specific controls and limitations in place. I've done it. It's complicated, and a royal pain, but it is possible to do in such a way that it doesn't blow the hell up every time you relocate one of the nodes with VMotion. Do I recommend it? Weeeeell, for file and print? Frankly, it just has too many advantages to argue against from a technical perspective. Anything else? Forget it.

I like MSCS - it has solid uses for things that need software redundancy, but to be really effective your software needs to be MSCS aware, which limits you not only to windows and certain software packages, but to a very narrow subset of those packages - sure, you can MSCS any process, but for it to truly play well it has to be aware. Most things aren't.

Anything that operates as a service can be put under MSCS. The problem of MSCS awareness only occurs when you have active-active configurations, really. For example, TSM isn't really all that MSCS aware. However, if you put the services (and optionally disk it's installed to) under MSCS, it works just fine that way. The problem is that most people simply ignore or don't understand how MSCS dependency chaining works. It's not MSCS specific either - I've seen it under Veritas frequently as well. They'll put a child completely outside the dependency chain, and predictably, things blow up on failover.

FT works on ANY VM, with ANY software package, on ANY supported hardware - it's totally transparent to the user and to the OS, which is the goal - it's HA at the hardware layer, which is simply awesome for stuff that must simply ~not~ have hardware related failures. And with a 10gE backplane, your key resource limitation is gone We support 10gE for FT

10GigE is, in my absolute expert opinion, a steaming pile of crap and always will be. It's a pathetic miserable hack that was thrown together from the steaming pile of wreckage that GigE has become. Look, I have written drivers for this stuff. It's utter garbage. The one true holy grail is InfiniBand; relying on 10GigE is the equivalent of entrusting your life to a seat belt that's been duct taped back together. If you wouldn't do that, then you shouldn't be wasting time on 10GigE.
InfiniBand is flat out superior in every single way, even despite frame size limitations. Jumbo frames were a hack in the first place, to some extent. InfiniBand is lower latency, higher bandwidth, higher reliability, higher throughput. Even a 1xSDR link is superior to 10GigE.

brings up an interesting question...what 10GbE cards currently have drivers for ESX4? I can tell you that it's not Myricom, which is the favorite. Chelsio, maybe?

Well, Myricom is utter crap anyways, so who the hell cares about them? (Hint; NOT ME.) Mellanox and Broadcom are the only 10GigE I'm aware of with support in ESX4. Not familiar with Chelsio. I believe ESX4 also supports the old Mellanox and PathScale HTX and HyperTransport Direct connect InfiniBand.

 

[sabregen]

Myricom has the closest possible throughput to raw line speed available on 10GbE @ 9.64.

 

[AreEss]

Myricom has the closest possible throughput to raw line speed available on 10GbE @ 9.64.

Mellanox wins this fight handily with 9.9Gbit/s half-duplex long frame, 17.6Gbit/s full-duplex. With early firmware.
http://www.analogzone.com/netp_052807.htm

 

[sabregen]

bah, all i can find is reports of fujitsu X700 and Myrinet 10GbE doing jumbo frame short bursts @ 9920mbps, and sustained full duplex @ 9.8 each direction, matching your specs on the Mellanox cards. Maybe I remembered it wrong. Nevertheless, they do not have ESX4 drivers, which sucks for me. I had to deploy our hosted environment (which I really wanted FT for) on 3.5 U4

 

[AreEss]

bah, all i can find is reports of fujitsu X700 and Myrinet 10GbE doing jumbo frame short bursts @ 9920mbps, and sustained full duplex @ 9.8 each direction, matching your specs on the Mellanox cards. Maybe I remembered it wrong. Nevertheless, they do not have ESX4 drivers, which sucks for me. I had to deploy our hosted environment (which I really wanted FT for) on 3.5 U4

Mellanox is also far cheaper. You should replace them anyways.

 

[lopoetve]

Actually, MSCS + HA is a supportable solution, provided you have specific controls and limitations in place. I've done it. It's complicated, and a royal pain, but it is possible to do in such a way that it doesn't blow the hell up every time you relocate one of the nodes with VMotion. Do I recommend it? Weeeeell, for file and print? Frankly, it just has too many advantages to argue against from a technical perspective. Anything else? Forget it.

Nope - as per the MSCS guide, it's not supported. HA is a less-than-smart agent, and is capable of powering up 2 MSCS nodes on a failure on the same host - due to how we do reservation passthrough, this will allow both vms to gain access to the quorum disk, causing data corruption, paycheck instability, and the VMware tech laughing at you for not reading the document. Don't do it. The ~only~ safe way to lock a vm down is to connect it to something like a local serial port - this will cause it to error out when it tries to power on the VM on a different host, but causes other issues (namely the lack of moving it to other hosts ever).

VMotion with MSCS is impossible. 1 - MSCS builds the ID of the node off of the cpu, so that would break it (we pass through that part), and 2, you cannot vmotion things with a shared SCSI bus. You did read the doc and set up the shared scsi bus, right?

Anything that operates as a service can be put under MSCS. The problem of MSCS awareness only occurs when you have active-active configurations, really. For example, TSM isn't really all that MSCS aware. However, if you put the services (and optionally disk it's installed to) under MSCS, it works just fine that way. The problem is that most people simply ignore or don't understand how MSCS dependency chaining works. It's not MSCS specific either - I've seen it under Veritas frequently as well. They'll put a child completely outside the dependency chain, and predictably, things blow up on failover.

That doesn't change that you're limited to Windows only applications. FT can work on a vapp - a combination of windows and linux machines, or any OS! The chained dependencies are just a pain, and talk about something that is horribly documented...

10GigE is, in my absolute expert opinion, a steaming pile of crap and always will be. It's a pathetic miserable hack that was thrown together from the steaming pile of wreckage that GigE has become. Look, I have written drivers for this stuff. It's utter garbage. The one true holy grail is InfiniBand; relying on 10GigE is the equivalent of entrusting your life to a seat belt that's been duct taped back together. If you wouldn't do that, then you shouldn't be wasting time on 10GigE.
InfiniBand is flat out superior in every single way, even despite frame size limitations. Jumbo frames were a hack in the first place, to some extent. InfiniBand is lower latency, higher bandwidth, higher reliability, higher throughput. Even a 1xSDR link is superior to 10GigE.

Maybe so, but Infiniband and 10gE are the only things that have the bandwidth that FT really needs, and no one has the money it seems for Infiniband, or the knowledge to actually use it. I'll agree that Infiniband is better - there's no doubt there, but ~no one~ ever has it. EVER. I've not seen a ~single~ infiniband deployment anywhere yet, except at NCAR in one of the super clusters! People don't use it - people use 10gE for whatever reason.

FT avoids all the issues that limit MSCS that everyone forgets really limit MSCS (btw, MSCS is 2 node ~only~ on VMWare - reservation issues again). You can ignore the HA/DRS/VMotion limitations, but it'll only get you badly burnt in the end.

Go read the MSCS doc / caveats. We have like 20 major limitations for MSCS clustering.

 

[AreEss]

Nope - as per the MSCS guide, it's not supported. HA is a less-than-smart agent, and is capable of powering up 2 MSCS nodes on a failure on the same host - due to how we do reservation passthrough, this will allow both vms to gain access to the quorum disk, causing data corruption, paycheck instability, and the VMware tech laughing at you for not reading the document. Don't do it. The ~only~ safe way to lock a vm down is to connect it to something like a local serial port - this will cause it to error out when it tries to power on the VM on a different host, but causes other issues (namely the lack of moving it to other hosts ever).

Like I said, I've done it. Step one, everything you read is wrong. Step two, nothing documented is right. Welcome to MSCS. Firstly, both systems should always have access to the quorum disk. That's the point of the quorum disk. Second, you have to play merry hell with MSCS to configure it correctly. Third, HA doesn't get to screw with the disk, period. You read that right; VMWare does NOT get to do anything other than pass the disks up. Once MSCS is configured to run with physical disk visible on both systems, it controls read/write access via communication and quorum disk. Attempts to access cluster disks on the wrong node will result in permission denied errors.

VMotion with MSCS is impossible. 1 - MSCS builds the ID of the node off of the cpu, so that would break it (we pass through that part),

It works. I've seen it done with multiple MSCS installations that I haven't even touched. Don't ask me how or why.

and 2, you cannot vmotion things with a shared SCSI bus. You did read the doc and set up the shared scsi bus, right?

No. I did it right. You take FC LUNs, and zone them to candidates. Then you do nothing more than pass the disk up, allowing MSCS to fully control disk access. It's just like VMotion; all candidates must see the same LUNs. No LUN, no VMotion target. No LUN, no MSCS target.

That doesn't change that you're limited to Windows only applications. FT can work on a vapp - a combination of windows and linux machines, or any OS! The chained dependencies are just a pain, and talk about something that is horribly documented...

No, I'm not. There's Veritas for Linux. And that doesn't change the fact that FT/HA does not work at software level, at all. I don't pity anyone stupid enough to rely entirely on FT/HA - I point and laugh. Chained dependencies are there for a reason. You should know better than to argue with mainframe, since that's what VMWare is aiming for in essence. Relying on FT/HA for all your high availability doesn't even rise to moronic - it's incompetence with a hefty dose of negligence.

Maybe so, but Infiniband and 10gE are the only things that have the bandwidth that FT really needs, and no one has the money it seems for Infiniband, or the knowledge to actually use it. I'll agree that Infiniband is better - there's no doubt there, but ~no one~ ever has it. EVER. I've not seen a ~single~ infiniband deployment anywhere yet, except at NCAR in one of the super clusters! People don't use it - people use 10gE for whatever reason.

Because they can now get 10GigE cards and try like complete idiots to run 10GigE over copper that could barely handle GigE. Then complain about bad throughput and link problems. The smart ones reuse existing FC infrastructure, since 10GigE uses the same multimode as a quality FC setup. People don't use InfiniBand because they're chained to Cisco like fools, and Cisco flat out rapes on their switches. Which are frankly, garbage. InfiniBand, long term, is cheaper than 10GigE. Install switch. Install FC to IB Gateway. Suddenly you only need an IB HBA, instead of IB+FC. I suspect though that some people are justifiably freaked out by the fact that IB is impedance-based link; if your grounding sucks, it doesn't work.

FT avoids all the issues that limit MSCS that everyone forgets really limit MSCS (btw, MSCS is 2 node ~only~ on VMWare - reservation issues again). You can ignore the HA/DRS/VMotion limitations, but it'll only get you badly burnt in the end.

No, the utter incompetents who think that FT/HA is a legitimate replacement for MSCS will get burnt very very badly yet again. The fact is that the vast majority of faults occur in software, not in hardware. And without any software FT/HA, you will get burned over and over and over again. And people like me will sit over here saying "told you so."
I've been doing HA design, implementation, and administration for many many years. I have seen time and time again where people prove themselves ignorant and negligent, or worse, by swearing up and down that hardware FT+HA is all any shop will ever need. Hardware FT is a very, very small but necessary piece of the puzzle. Hardware HA is a complete and total waste of time, money, and resources outside of mainframe. It doesn't work, it has never worked, and it will never work. That's an immutable FACT. When Oracle crashes in the middle of the night due to a bug, and screws up payroll, VMWare FT/HA is going to sit there looking like an idiot, because there's no hardware fault. But Oracle's still down. So, what, you're going to state fail over to the other node? Great. Oracle's still down. The only time hardware FT/HA will ever work is when Oracle goes down, because CPU0 just blew up and brought the whole system down. The number of times that happens versus the number of times an application crashes requires use of scientific notation for the app crash side.

 

[lopoetve]

Like I said, I've done it. Step one, everything you read is wrong. Step two, nothing documented is right. Welcome to MSCS. Firstly, both systems should always have access to the quorum disk. That's the point of the quorum disk. Second, you have to play merry hell with MSCS to configure it correctly. Third, HA doesn't get to screw with the disk, period. You read that right; VMWare does NOT get to do anything other than pass the disks up. Once MSCS is configured to run with physical disk visible on both systems, it controls read/write access via communication and quorum disk. Attempts to access cluster disks on the wrong node will result in permission denied errors.

Wrong. I know the VMWare software FAR better than you do, and I've assisted with the caveats and limitations with it for ESX4 for that matter. Yes, both systems have access, but since they're on different hosts, only one can get the SCSI-3 reservation that MSCS uses to lock the disk. When HA brings up the second node on the SAME system (which it is QUITE capable of doing), both VMs are allowed to place the reservation, causing problems. I don't care if you've done it - it's NOT supported, it does NOT work, and it is NOT safe, and you will get burned.

VMWare does a lot more than just pass the disks up - quite the contrary, in fact, as it works as a passthrough layer even in physical mode.

http://www.vmware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_mscs.pdf
Page 16.

It works. I've seen it done with multiple MSCS installations that I haven't even touched. Don't ask me how or why.

No. I did it right. You take FC LUNs, and zone them to candidates. Then you do nothing more than pass the disk up, allowing MSCS to fully control disk access. It's just like VMotion; all candidates must see the same LUNs. No LUN, no VMotion target. No LUN, no MSCS target.

Cannot be done, not as VMs. ESX must see the lun for the VM to see the lun - period. NPIV is not to that phase of development or release yet, and MSCS is only supported on FC with VMware ESX. Simply put, you zoned the lun to the host, and then gave it to the VM.

If you didn't set up the shared scsi bus, there's a good chance you'll have data corruption and other problems, if it even fails over correctly in the first place.

Vmotion does NOT work with a shared SCSI bus. Do you need me to get a screenshot?

No, I'm not. There's Veritas for Linux. And that doesn't change the fact that FT/HA does not work at software level, at all. I don't pity anyone stupid enough to rely entirely on FT/HA - I point and laugh. Chained dependencies are there for a reason. You should know better than to argue with mainframe, since that's what VMWare is aiming for in essence. Relying on FT/HA for all your high availability doesn't even rise to moronic - it's incompetence with a hefty dose of negligence.

I wouldn't rely on it, but I also wouldn't rely on any one system for it either - I'd have DR sites, backups, etc for everything critical. Given my experiences with MSCS on VMware, I certainly wouldn't rely on it alone either. They're all useful technologies - use the right tool for the job.

No, the utter incompetents who think that FT/HA is a legitimate replacement for MSCS will get burnt very very badly yet again. The fact is that the vast majority of faults occur in software, not in hardware. And without any software FT/HA, you will get burned over and over and over again. And people like me will sit over here saying "told you so."
I've been doing HA design, implementation, and administration for many many years. I have seen time and time again where people prove themselves ignorant and negligent, or worse, by swearing up and down that hardware FT+HA is all any shop will ever need. Hardware FT is a very, very small but necessary piece of the puzzle. Hardware HA is a complete and total waste of time, money, and resources outside of mainframe. It doesn't work, it has never worked, and it will never work. That's an immutable FACT. When Oracle crashes in the middle of the night due to a bug, and screws up payroll, VMWare FT/HA is going to sit there looking like an idiot, because there's no hardware fault. But Oracle's still down. So, what, you're going to state fail over to the other node? Great. Oracle's still down. The only time hardware FT/HA will ever work is when Oracle goes down, because CPU0 just blew up and brought the whole system down. The number of times that happens versus the number of times an application crashes requires use of scientific notation for the app crash side.

I'm a sysadmin - all I care about is the hardware side. If the software has an issue, that's a different group that fucked up. Relying on any one technology is a bad idea, but VMware's FT is a huge step towards adding reliability and stability for applications that have to withstand hardware failures, given that there is zero setup required for the application to be able to take advantage of it, and that it's totally automated for restoring full clustering after a failure.

It's great when the microcode update crashes your new system abruptly after a vmotion and none of the critical VMs that would have just died stay up instead. It's great when the idiot in the machine room pulls the wrong power cord. It's great when a ups blows and takes out a whole rack, or when ram goes bad and the system PSODs. You're going to be able to smile and laugh that none of your apps went down, even though you lost a box totally, even if you don't have the foggiest clue what those apps are or how they're configured (reality - most places have different people responsible for the VMs and the hosts). My experience is that hosts crash a lot more than you give them credit for.

FT lets you be blind to the app, which many many places are, and even more are heading towards. TBH, over a quarter of the places I work with now don't have access or knowledge of the VMs. The application admins/vm admins are totally separate from the server admins, as the environments are simply too large.

 

[Berg0]

I just have to say, although it's O/T, the preceding few posts are why I LOVE [H]

some good info!

 

[pissboy]

Because they can now get 10GigE cards and try like complete idiots to run 10GigE over copper that could barely handle GigE. Then complain about bad throughput and link problems. The smart ones reuse existing FC infrastructure, since 10GigE uses the same multimode as a quality FC setup. People don't use InfiniBand because they're chained to Cisco like fools, and Cisco flat out rapes on their switches. Which are frankly, garbage. InfiniBand, long term, is cheaper than 10GigE. Install switch. Install FC to IB Gateway. Suddenly you only need an IB HBA, instead of IB+FC. I suspect though that some people are justifiably freaked out by the fact that IB is impedance-based link; if your grounding sucks, it doesn't work.

Not sure I understand the point of people not using IB because they're chained to Cisco. All of the IB switches I've moved in the past few years were Cisco (formerly Topspin).

Maybe so, but Infiniband and 10gE are the only things that have the bandwidth that FT really needs, and no one has the money it seems for Infiniband, or the knowledge to actually use it. I'll agree that Infiniband is better - there's no doubt there, but ~no one~ ever has it. EVER. I've not seen a ~single~ infiniband deployment anywhere yet, except at NCAR in one of the super clusters! People don't use it - people use 10gE for whatever reason.

I can assure you people use IB in small installations. I've setup quite a few customers with the HCAs, Cables and Switches needed, for a fraction of what a 10gE setup would cost.

 

[AreEss]

Not sure I understand the point of people not using IB because they're chained to Cisco. All of the IB switches I've moved in the past few years were Cisco (formerly Topspin).

The new Cisco IB gear costs a lot more. They're also getting ready to start jacking up prices and making core functionality an additional license or five. Cisco's been going this route with everything these days. Started with SAN-OS, then the VoIP, and it proceeds apace. I'm actually seeing Cisco get thrown out of shops because of SmartNET costs more than doubling because of mandatory upgrades and new licensing.

I can assure you people use IB in small installations. I've setup quite a few customers with the HCAs, Cables and Switches needed, for a fraction of what a 10gE setup would cost.

Exactly. 10GbE is much more expensive than IB. QLogic has a few smaller switches, as does Voltaire. (I still prefer Voltaire, but that's mostly for homogenous management when you scale up to the big stuff.) The QLogic 9024 with 24 DDR IB ports can be had for around $5250; you can't even touch a Cisco with 48 GigE ports and basic IOS for that. Forget any 10GigE.
(Why did I say 24 ports? Feh.. meant 48. 3750G; with SmartNET list well over $6k per.)