HP BladeSystem c7000, Storage, Exchange 2013

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
I am just trying to get some opinions. I already know what was implemented is a bad design. Well, really just the choice of SAN that was a bad idea.

Customer has the c7000 blade system with a few 460c G8 blade servers and a few older 460c G6 blade servers. The blade system has 6Gb SAS switches for connectivity to the SAN. There are two SANs. One HP P2000 G3 MSA for the G8 blades and an old HP MSA2324sa for the G6 blades.

All the blade servers are running ESXi 5.5. There are a total of 15 VMs running, but my main concern is the two Exchange servers. Both Exchange servers have 3x 1 TB databases and are setup in DAG. So 6 TB total in databases running on the P2000 SAN. I don't know how 3PAR is setup, but these MSA's use vDisks and were setup in RAID 6 which I believe is limited to 16 disks per vDisk. So one vDisk is 6.5 TB with 1 TB free. The other vDisk is 3.8 TB with 1.35 TB free.

The Exchange server has about 2,000 users and is a healthcare company. Based on the past 2-3 months, they will be out of space in about 8 months at their current growth. Considering the amount of free space on the SAN, if a DB restore was required, I could barely fit one restored DB on the second vDisk.

You can add another disk enclosure to the P2000 SAN, but considering the size of Exchange and the growth, I don't think they should even be using a MSA (entry level) SAN.

Sticking with the 6Gb SAS switch, would you recommend using 3PAR? I would think at least 20 TB of capacity to accommodate for DB restores if required. And how do you feel about these MSA SANs?
 
Office365. :p

Please. I wish. This customer insists on owning their own gear. At the very least if we could put them in our cloud I would be happy, but they won't do that or use hosted Exchange through anyone.
 
Can we at least agree that the HP P2000 G3 MSA is not the appropriate SAN to use for an Exchange environment this large that is continuing to grow?
 
Just want to make sure I am not crazy. ;) The person that sold this seems to think it was perfectly acceptable and doesn't want to believe that both of those HP SANs should be replaced with a single 20 TB of more SAN.
 
The MSAs are for much smaller places than that. They're not bad boxes, but not for ~that~.

Seriously - I bet we could make a T820 work ;)
 
Yeah, but it is more about the connectivity and having everything under HP support. The SAS switches at least give decent throughput. Without them, I am stuck with 1 Gb. Plus I've never seen the compression for 6 TB of Exchange databases. All together there is about 8 TB used, but it should come down a bit by switching to thin provisioning. Just not sure 11 TB would really be enough for future growth and needing to possibly restore a DB.
 
MSA's aren't bad but the MSA's in use are pretty old. I"m obviously biased (I work for HP Storage) but HP 3PAR is some of the best storage out there for virtualization and IaaS. Lots of service provides are using 3PAR. 3PAR was in fact the exclusive FC development platform that VMware used for vVOLs.

But from what I'm reading, that would be the issue for you as 3PAR doesn't support SAS connect to the host.

We just announced the latest 3PAR platform, the HP 3PAR StoreServ 8000. The entry level all-flash starter kit which has 8 X 480GB SSDs is about $19K. I don't know what 20TB HDD 3PAR would cost but if that's a serious consideration for you, I can check. Here's a link to the HP 3PAR 8000 QuickSpec which has lots of technical detail.

The other option (though for me isn't as good an option) is looking at the latest generation MSA - the MSA2040. Again, I'm not a pricing guy but a base unit starts at under $9000. I say "not as good" is that for me, 3PAR is the best midrange SAN out there. It just past EMC and NetApp as the #1 midrange SAN (per latest IDC market data). Having been with HP Storage for a LONG time, 3PAR is the sweetest kit I've seen.

Happy to help with more info if you want to look closer at either of these.
 
There were two others on the FC side as well ;) (Disclaimer: I was the Global Tech Lead on VVOL and VSAN for 2 years at VMware). One of them was a total failure, granted, and the other stopped caring, so at the end 3PAR was it, but there ~were~ three to start with at least :p
 
MSA's aren't bad but the MSA's in use are pretty old. I"m obviously biased (I work for HP Storage) but HP 3PAR is some of the best storage out there for virtualization and IaaS. Lots of service provides are using 3PAR. 3PAR was in fact the exclusive FC development platform that VMware used for vVOLs.

But from what I'm reading, that would be the issue for you as 3PAR doesn't support SAS connect to the host.

We just announced the latest 3PAR platform, the HP 3PAR StoreServ 8000. The entry level all-flash starter kit which has 8 X 480GB SSDs is about $19K. I don't know what 20TB HDD 3PAR would cost but if that's a serious consideration for you, I can check. Here's a link to the HP 3PAR 8000 QuickSpec which has lots of technical detail.

The other option (though for me isn't as good an option) is looking at the latest generation MSA - the MSA2040. Again, I'm not a pricing guy but a base unit starts at under $9000. I say "not as good" is that for me, 3PAR is the best midrange SAN out there. It just past EMC and NetApp as the #1 midrange SAN (per latest IDC market data). Having been with HP Storage for a LONG time, 3PAR is the sweetest kit I've seen.

Happy to help with more info if you want to look closer at either of these.

If I am not mistaken the HP P2000 G3 MSA SAN they are using is actually a MSA 2040. It is less than two years old and personally I don't like it. I don't know how 3PAR works, but the Vdisks are so annoying. The SAN had 15 drives in it to begin with. One spare, and the other 14 in RAID 6. I wanted to expand it by adding 9 more drives and found out I would need to create another Vdisk because you can't really expand the current Vdisk without serious risk of data loss and also I believe RAID 6 is limited to 16 disks per Vdisk. I am used to enterprise storage like NetApp and Tintri and even worked with PowerVault and EqualLogic. The MSA is by far my least favorite.

With less than 2 TB remaining and growth expected to run out in about 6 months, we may be stuck just buying an additional disk shelf for the current MSA.
 
3PAR is another enterprise platform - raid groups, volumes, luns, etc. It's solid, great engineers too, but still a traditional platform. What's the cost on a shelf going to be?
 
Office365. :p

Show me a single customer who has thousands of accounts in O365 cloud based Exchange and is happy with the performance and we can talk. Cancer Treatment Centers of America just finished an eval of O365 for the second time since O365 was released and concluded that it makes zero sense to put Exchange in the cloud.

My current org (healthcare) has 15k mailboxes and we too looked very carefully at Exchange in the cloud (we have all of our users O365 licensed via E1 for clinical and E3 for knowledge workers). Exchange in the cloud is a big NO NO for larger organizations. It's great for your small business and anyone who only has a few hundred accounts. Once you get into the thousands and especially hundreds/thousands of public folders instead of Sharepoint based calendaring the Exchange online performance sucks extremely hard.

@Kaps you really need three copies of each DB in the DAG to fully protect yourself from DBs going offline. Our DBs are all between 1.6 - 1.8 TBs each, we run four hub transports and ten mailbox servers, still on Office 2010, skipped 2013, will do 2016 next year.

One thing you can do is limit the mailbox size to curb data growth. The employees will be like "WTF?!" but Ascension Health (150,000 employees) imposed the following limits on mailbox sizes:

Most employees: 150 MB
Supervisors, Managers, Directors: 300 MB
Executives & Marketing Team: 500 MB

Notice how in the memo that went out about this they use 150,000 KB instead of 150 MB, or 0.15 GB, to make it appear as if that's a lot of space. See Ascension Health email reduction memo page 1 and Ascension Health email reduction memo page 2 on imgur. It may say Ministry on the memo, but Ministry has been bought by Ascenson and the email reduction is a direct result of that acquisition.

Also notice how they say that they are at 10.37 BILLION (OMGWTFBBQ!!!!) KB of email. Well that's 10 TB, big deal even at FC disk prices. Whomever wrote that memo thinks he's really great at making inconsequential things look BIG.

At our org we do 2 GB as standard mailbox size, managers/directors get 5 GB, VP and up get 10 GB, and Marketing gets 25 GB. However, if someone runs out of space they can call the helpdesk which will increase the quota by 1 GB at a time.

What I do like about the Ascension approach is that if you only have 150 MB you can't keep a lot of email, which means that none of it is part of discovery in any legal proceedings. You should consider following the Ascension model, that way you don't have to buy more disk. ;)
 
Last edited:
@Kaps you really need three copies of each DB in the DAG to fully protect yourself from DBs going offline. Our DBs are all between 1.6 - 1.8 TBs each, we run four hub transports and ten mailbox servers, still on Office 2010, skipped 2013, will do 2016 next year.

One thing you can do is limit the mailbox size to curb data growth. The employees will be like "WTF?!" but Ascension Health (150,000 employees) imposed the following limits on mailbox sizes:

Most employees: 150 MB
Supervisors, Managers, Directors: 300 MB
Executives & Marketing Team: 500 MB

Notice how in the memo that went out about this they use 150,000 KB instead of 150 MB, or 0.15 GB, to make it appear as if that's a lot of space. See Ascension Health email reduction memo page 1 and Ascension Health email reduction memo page 2 on imgur. It may say Ministry on the memo, but Ministry has been bought by Ascenson and the email reduction is a direct result of that acquisition.

Also notice how they say that they are at 10.37 BILLION (OMGWTFBBQ!!!!) KB of email. Well that's 10 TB, big deal even at FC disk prices. Whomever wrote that memo thinks he's really great at making inconsequential things look BIG.

At our org we do 2 GB as standard mailbox size, managers/directors get 5 GB, VP and up get 10 GB, and Marketing gets 25 GB. However, if someone runs out of space they can call the helpdesk which will increase the quota by 1 GB at a time.

What I do like about the Ascension approach is that if you only have 150 MB you can't keep a lot of email, which means that none of it is part of discovery in any legal proceedings. You should consider following the Ascension model, that way you don't have to buy more disk. ;)

I wish. Unfortunately we are just the MSP and they don't listen so well about imposing limits. They have been talking about enabling retention policies to delete anything older than 6 months in the Sent Items, Deleted Items, and Inbox. But they only talk about it and it has been about 10 months now. So there is basically no restrictions and no cleanup and everything just keeps growing. When I worked at Pfizer is was a lot more like the model you are talking about above. Hell, you would need to have a good reason and management approval just to get a mailbox increase and it still wasn't guaranteed. This healthcare company lacks policies in a big way and I keep trying to explain to them that as a business they need policies, SOPs, etc. But instead they run the company like a small business with only 10 employees.
 
But from what I'm reading, that would be the issue for you as 3PAR doesn't support SAS connect to the host.

Are you sure about that? I mean, I would agree because all the 3PAR specs I have read on HP's website say FC and iSCSI, but a buddy of mine claims they use the same BladeSystem c7000 chassis with 3PAR and connected via SAS. Not saying he is right though.

I've been given the go ahead to spec something out for the customer. Being that the BladeSystem already has the SAS switches, I am trying to stick with that. Especially since the ESXi hosts that were probably $15k a piece only have two friggin 1Gb ports.

I used the HP Sizer for Microsoft Exchange 2013 and it recommends two of the HP StoreVirtual 4530 3TB MDL SAS Storage SANs. I don't know much about those SANs either, but they seem to come up under the entry level section as well as the MSA 2040 which is what is being used.

The only pro with the StoreVirtual 4000 series I see is that it integrates with Veeam which is something else I would like with a HP SAN and I don't see the MSA 2040 on that list. I would rather not try taking VMware snapshots of databases that are about 1.2 TB each every night and backup over the network, especially since it is DAG and I have had issues with much smaller databases when it comes to Veeam.
 
HP's models are so confusing. What is the difference between the P2000 G3 SAS MSA and the MSA 2040?
 
What I do like about the Ascension approach is that if you only have 150 MB you can't keep a lot of email, which means that none of it is part of discovery in any legal proceedings. You should consider following the Ascension model, that way you don't have to buy more disk. ;)

Having worked in legal for several years, this was definitely something we preached. The coincidental part is that lawyers never followed their own advice. Law firm mail servers are a disaster.
 
Having worked in legal for several years, this was definitely something we preached. The coincidental part is that lawyers never followed their own advice. Law firm mail servers are a disaster.

I've seen it first hand when a client came to us for help when their managed services provider accidentally formatted the Exchange LUNs presented to their physical mail servers with VMFS. Oh and, by the way, they also hadn't backed up the mail server LUNs in 18 months. Sorry!

Lawsuits galore!
 
I've seen it first hand when a client came to us for help when their managed services provider accidentally formatted the Exchange LUNs presented to their physical mail servers with VMFS. Oh and, by the way, they also hadn't backed up the mail server LUNs in 18 months. Sorry!

Lawsuits galore!

I could see that happening with the SAS connections. When installing ESXi, the SAN vdisk is available as a selection. All it takes is someone to think it is OK to proceed with the installation and formatting it.
 
I can't believe I overlooked this before. The three newer blade servers actually have 2x 10Gb NICs that are negotiating at 1Gb because of the old blade switches. So I will simply recommend getting 2x 10Gb switches. :D
 
Are you sure about that? I mean, I would agree because all the 3PAR specs I have read on HP's website say FC and iSCSI, but a buddy of mine claims they use the same BladeSystem c7000 chassis with 3PAR and connected via SAS. Not saying he is right though.

I've been given the go ahead to spec something out for the customer. Being that the BladeSystem already has the SAS switches, I am trying to stick with that. Especially since the ESXi hosts that were probably $15k a piece only have two friggin 1Gb ports.

I used the HP Sizer for Microsoft Exchange 2013 and it recommends two of the HP StoreVirtual 4530 3TB MDL SAS Storage SANs. I don't know much about those SANs either, but they seem to come up under the entry level section as well as the MSA 2040 which is what is being used.

The only pro with the StoreVirtual 4000 series I see is that it integrates with Veeam which is something else I would like with a HP SAN and I don't see the MSA 2040 on that list. I would rather not try taking VMware snapshots of databases that are about 1.2 TB each every night and backup over the network, especially since it is DAG and I have had issues with much smaller databases when it comes to Veeam.

It's a LeftHand. They're solid enough.
 
I can't believe I overlooked this before. The three newer blade servers actually have 2x 10Gb NICs that are negotiating at 1Gb because of the old blade switches. So I will simply recommend getting 2x 10Gb switches. :D

And... T820? ;) :D
 
And... T820? ;) :D

I would if it wasn't for price and capacity. This company is unbelievably cheap. The current SAN was only $10k. I hoping not to exceed $50k for a 20 TB SAN plus 2x 10 Gb blade switches.

LeftHand is less expensive than 3PAR, right?
 
The HP StoreVirtual 4730 900GB SAS Storage(B7E28A) looks like it could be worth while. Price online is $20k for about 20 TB. Also integrates with Veeam.
 
I would if it wasn't for price and capacity. This company is unbelievably cheap. The current SAN was only $10k. I hoping not to exceed $50k for a 20 TB SAN plus 2x 10 Gb blade switches.

LeftHand is less expensive than 3PAR, right?

Could look at Pure, too. :D Flash keeps getting cheaper, especially when you get a free controller upgrade every 3 years.
 
Could look at Pure, too. :D Flash keeps getting cheaper, especially when you get a free controller upgrade every 3 years.

LOL Forget that. I am going to recommend ripping out the entire blade system and SANs and get a Nutanix NX-3060. They actually are a colo customer of ours, so going to just 2U may save them money.
 
I would if it wasn't for price and capacity. This company is unbelievably cheap. The current SAN was only $10k. I hoping not to exceed $50k for a 20 TB SAN plus 2x 10 Gb blade switches.

LeftHand is less expensive than 3PAR, right?

LeftHand is less expensive, and less capable in general - but it's an older architecture that hasn't had as many updates.
 
3PAR is another enterprise platform - raid groups, volumes, luns, etc. It's solid, great engineers too, but still a traditional platform. What's the cost on a shelf going to be?

Sorry, I missed the replies to I'm late!

I guess we should define traditional platform but I'd say that 3PAR isn't traditional. A "traditional" dual controller (usually active/passive) is traditional. HP 3PAR uses a mesh active architecture for a shared everything fine-grained virtualization. It uses a custom ASIC that among other things is a warp drive data engine to move data (also does RAID calcuations, deduplication, zero reclaimation and a bunch of other things) to let the CPUs do their crunching.

What is unique in the architecture is that it does mixed workloads really, really well.
 
Are you sure about that? I mean, I would agree because all the 3PAR specs I have read on HP's website say FC and iSCSI, but a buddy of mine claims they use the same BladeSystem c7000 chassis with 3PAR and connected via SAS. Not saying he is right though.

I've been given the go ahead to spec something out for the customer. Being that the BladeSystem already has the SAS switches, I am trying to stick with that. Especially since the ESXi hosts that were probably $15k a piece only have two friggin 1Gb ports.

I used the HP Sizer for Microsoft Exchange 2013 and it recommends two of the HP StoreVirtual 4530 3TB MDL SAS Storage SANs. I don't know much about those SANs either, but they seem to come up under the entry level section as well as the MSA 2040 which is what is being used.

The only pro with the StoreVirtual 4000 series I see is that it integrates with Veeam which is something else I would like with a HP SAN and I don't see the MSA 2040 on that list. I would rather not try taking VMware snapshots of databases that are about 1.2 TB each every night and backup over the network, especially since it is DAG and I have had issues with much smaller databases when it comes to Veeam.

We can do something called a Flat SAN which will connect 3PAR to BladeSystem via virtual connect) without a FC switch but I'm 99.9% sure there's no SAS connect.

The StoreVirtual 4530 sits between the MSA and 3PAR. Really does somethings well. Not sure if you have heard of LeftHand Networks but that is the foundation for StoreVirtual. The 4530 has SSD and HDD with sub-LUN tiering. I just don't know pricing so not sure what the pricing of a StoreVirtual 4530 vs appropriately sized 3PAR would look like. BTW, 3PAR has the same snapshot integration with Veeam that StoreVirtual does. Nice way to have recovery built in.

If you want some help, I can get one of my technical guys to talk you through things. Feel free to drop me an email: calvin dot zito at hpe dot com (yes, HPE with the coming separation).
 
LeftHand is less expensive, and less capable in general - but it's an older architecture that hasn't had as many updates.

I keep meaning to do a blog post talking about what we've done with LeftHand (StoreVirtual) over the last few years. It's had more than it's share of updates - hardware has been updated to ProLiant Gen9, added sub-LUN tiering, 2 node quorum (with an NFS file share), and a lot more. It's a solid platform.
 
I keep meaning to do a blog post talking about what we've done with LeftHand (StoreVirtual) over the last few years. It's had more than it's share of updates - hardware has been updated to ProLiant Gen9, added sub-LUN tiering, 2 node quorum (with an NFS file share), and a lot more. It's a solid platform.

Sure, and if you knew me you'd know I'm huge fans of both (especially the LeftHand side) - but the fundamental architecture hasn't changed. you all definitely didn't pull an equallogic, or compellent :)
 
lop,

I guess I never noticed your signature before. You have a ZL1? How is it? I've had a few Camaros myself. Last was a 2011 SS with some work done to it. Around 500hp. Would love to get my hands on a Z/28 but I am interested to test drive the new Alpha platform.
 
lop,

I guess I never noticed your signature before. You have a ZL1? How is it? I've had a few Camaros myself. Last was a 2011 SS with some work done to it. Around 500hp. Would love to get my hands on a Z/28 but I am interested to test drive the new Alpha platform.

LOVE the thing. Was my dream "this is the end of the American V8 muscle beast era... again" car :) Saved for 3 years for it :D Sadly, I don't get to drive it nearly as much as I'd like.
 
I would if it wasn't for price and capacity. This company is unbelievably cheap. The current SAN was only $10k. I hoping not to exceed $50k for a 20 TB SAN plus 2x 10 Gb blade switches.

Let me be the first to suggest JetStor. Dual controllers, dual/triple PSU, shittons of capacity, no vendor lock on disk, no annual maintenance, no 4-hour or NBD onsite support. Buy with or without disks. Stock extra drives and an extra controller and you are golden. See: http://www.acnc.com

Significantly below $200/raw TB (depending on total capacity) even with spare disks and a spare controller. Your 50k will go a long long way and you'll still have cash left over after you are done with array and switching.

Ran a few of these at my old employeer, already bought one at the new place as well, will be buying more to store all the security video footage on them. No need to put that workload on expensive EMC gear.

Support is by email/phone. Three year warranty on array (can be extended for $$$). Five year warranty on drives. RMA requires you to send the broken part in or guarantee ship-first by credit card. Easier to just have a spare part on site and then send the broken part in.

In all these years I never hard any enclosure, backplane, PSU, or controller issues. Had a total of two drives go bad. RMA was easy and issue free.
 
Sure, and if you knew me you'd know I'm huge fans of both (especially the LeftHand side) - but the fundamental architecture hasn't changed. you all definitely didn't pull an equallogic, or compellent :)

That's a great compliment (about not pulling an equallogic or compellent). No doubt equallogic was THE iSCSI leader back when Dell bought them. But because of their architecture, they're stuck at a dead end. Compellent is definitely interesting tech but Dell trying to position it as enterprise was a big mistake. We'll see if they can recover or if they have to go buy someone else to have a decent go-forward storage strategy. I know several execs over there that worked with me at HP Storage - very smart guys and if anyone can right the ship, it would be them.

If you're on Twitter, hit me up. Would love to get to "know" you. I'm @CalvinZito there.
 
You got to love VARs. These are the options they come back to me with.

Option 1 – Lefthand: - Almost $100k
Adding 10GbE Switches to interconnect slots 3 and 4
Adding mezz cards to 3 – BL460s
5 StoreVirtual 4330 900GB SAS nodes to provide ~20TB of usable capacity


Option 2 – 3PAR: - Over $100K
Adding Brocade FC switches to interconnect slots 3 and 4
Adding FC Mezz cards to 3 - BL460s
3PAR 8200 with add-on self – 2 node system with 8 x 1.92TB SSDs – this is 15TB of raw capacity, we anticipate dedup to be turned on that will provide between 2:1 – 4:1 dedup ratios.

I am pretty sure I mentioned the primary use is for Exchange and I don't have a high IOPS requirement.

Even the HP Sizer for Exchange 2013 says to use the StoreVirtual-4530 3TB MDL SAS Storage. How they came about proposing an all flash 3PAR solution is beyond me.

I mean we are currently using an entry level HP SAN that cost a total of $10k without any latency issues. Granted I want to get rid of the entry level SAN, but these quotes are ridiculous.

Also, I am not an expert with these HP BladeSystems, but why do I need additional mezz cards for the blades? All the blades already have a 10Gb mezz card installed, but they are going to the 1Gb blade switches. Can't I just install 10Gb blade switches in slots 3 and 4 on the blade chassis, put one ESXi host in maintenance mode, log into the Onboard Administrator and change the Port Mapping? It looks like I can do it from here.

 
Back
Top