recommendations for a VMware server with DAS

[robfrog]

Hello Folks,

I would like to build a 2U server with the following parameters:
- 2-socket MB (16 cores)
- 64GB RAM
- 10-20TB disk

Budget is ~$8k.

Tentative component list:
Case: SuperMicro 826E26-R1200LPB with "dual expander" SAS backplane and 12x 3.5" bays
Mobo: SuperMicro X9DRi-F
CPU: 2x Xeon E5-2650v2 (8core)
RAM: 4x 16GB kingston DDR3L
HBA: LSI MegaRAID SAS 9260-8i
HDD: 10x Seagate 3TB NL-SAS

The server will host several VMs for various purposes: devlopment, testing, proofs-of-concept, file-server, teamspeak-server -- in a nutshell, with totally disparate workloads.

In my experience disk I/O can be the biggest bottleneck when running several VMs, so I'm keen to get the disk subsystem right.

My feeling is that using SAS, even if the disks are nearline and not "real" SAS, will give better I/O performance than pure SATA, plus a 2nd path to the HBA for a bit more redundancy. Also, the NL-SAS disks don't seem to be *that* much more expensive than SATA disks. A config consisting of true SAS disks would probably put me over budget, since they seem to be smaller (<1TB), and I'd need more, and probably 2.5" drives, and also a case to accommodate them.

The HBA I believe has two 4-lane SFF-8087 Connectors which can be connected to the primary and secondary backplane connectors. I would have considered two separate single-port HBAs, but VMware tells me that ESXi cannot support such a config (and that would also make RAID impossible at the HBA level, which I want to keep). Thus using a single dual-port HBA that handles the RAID and load balancing and offers a bit of multipathing seemed reasonable.

So the question is how to provide 10-20TB of disk that performs well (is relative) without spending a lot. Couldn't be simpler, eh?

Questions:
1) is my assumption about using SAS and NL-SAS disks reasonable?
1a) if not, I would be grateful for suggestions.
2) will the HBA automatically load balance over the two SAS cables?
3) are there better HBA choices? With more cache, for example?
4) any other comments?

And then a more general question about SAS: can a single SFF-8087 connection reasonably handle 12 disks? (wrt throughput and I/Os/second)

thanks,

Robfrog

 

[robfrog]

judging by the total silence, it seems I have breached the protocol for asking questions. Would anyone care to tell me how?

 

[Blue Fox]

I'd just say that it's above what most might have dealt with regularly. My recommendation for you would be to possibly get a newer LSI controller and if you want some really outstanding performance, get an SSD and then buy one of the CacheCade licenses (hardware token). You can look up some reviews to see if that will achieve the performance levels that you desire.

As for load balancing, yes, it's possible, but I'm unsure of the specifics of setting that up with LSI cards.

Cache I addressed already I think.

Finally, I'd get 8 x 8gb sticks of memory instead of 4 x 16gb. The processors are quad channel after all and for maximum performance you'll need 8 sticks minimum.

As for the single cable, each cable provides 4 lanes at 600mb/s, so 2.4gb/s. Hardly a limiting factor. Same goes for non-sequential IO.

 

[-Dragon-]

Since you mention a teamspeak server I'm assuming this is a personal project, I wouldn't recommend what I list below in a production environment really but it works great for home use.

My home setup consists of a dual socket SM box with 192GB RAM and an Adaptec 7Q caching RAID controller with 8 3TB WE Reds and 2 Intel DC S2700 400GB as cache. I use Hyper-V instead of VMware due to RAM limitations and the fact that I can deduplicate my VHD storage drive and get better performance out of the cache. With the WD's in a RAID-6 and the Intels in RAID-1 to enable RW caching, I can simultaneously start all ~30 of my VMs currently running on it at the same time and the majority of the guests will be up and fully functional within 60 seconds of hitting start, the only exceptions are long boot guests like SQL and Exchange.

I'm pretty sure I spent less than $8000 on this box, if it's more it's not MUCH more (and has 3x the RAM you're looking for so even if it is more, 128G less RAM would put it under easy).

 

[twistacatz]

I think you need to consider tiered storage. I&#8217;m not quite sure what your data set is going to be beyond VM&#8217;s but I would suggest a separate pool of SSD&#8217;s for the VM&#8217;s and SATA or SAS storage for everything else.

If this is a work enterprise server then by all means use SAS for anything else I would consider it overkill, others might disagree.

I think the HBA you picked is fine and I would not worry about load balancing too much (unless you feel like you hitting a bandwidth wall). To find out more info about load balancing check out the LSI HBA manual and also take a look at the SuperMicro chassis manual for more info on the built in SAS expander.

Ever thought about doing an all in one setup with Napp-it and OmniOS. This type of setup would allow you to do much more from a storage perspective like deduplication to help you maximize space on a SSD pool.

 

[Concentric]

The server will host several VMs for various purposes: devlopment, testing, proofs-of-concept, file-server, teamspeak-server -- in a nutshell, with totally disparate workloads.

I think it would help to have a little more information about exactly what your VMs will be doing and how many will be running at once. This will give us a better idea of your hardware requirements. "development", "testing" are quite vague terms that could mean anything. How many users will be hammering these virtual servers? Will there be lots of database load? Etc.

Depending on the answer to the above, I would question whether you really need 16 cores of CPU. It might be better to invest some of that cash in more RAM or the storage subsystem instead.

Also we need more info on how this "10-20TB" of data is comprised. For example, perhaps a lot of it is static content like media, with only a small proportion being IOPS intensive. In that case, separate arrays or a tiered system using an SSD for CacheCade would probably suit.

On the subject of dual SAS paths, this is slightly pointless if you have a single RAID controller. If the controller itself breaks then your redundancy between the disks and the controller was pointless.
If uptime is critical then I would look at duplicating the entire system in a different location. But again, I don't really know the context for the project.

 

[GMcDonnell]

I'd spend less on processors and more on SSDs.

 

[robfrog]

sorry about the late response, and thanks for all your comments.

Blue Fox: thanks for the tip about the DIMMs
-Dragon-: no, not for home. For a company I work for
Twistacatz: tiered-storage <- good idea; Napp-it/OmniOS - yes considered it, but it's outside the scope of this project.
Concentric: thanks for the HBA comments

I agree that a clearer requirements specification would be helpful. The problem is we will use the server for all sorts of tasks, which are mostly currently unknown. Things like:
- re-creating customer configurations for testing, debugging (so I/O perf mostly not a priority)
- file server (I/O perf not a priority)
- all sorts of internal servers

I'm leaning towards a 2U SuperMicro case that can accept 12x 3.5" HDDs, and then just stuffing in large (~4TB) enterprise-rated SATA drives, and maybe leaving a few slots free for SSDs later, if we find we need them.

Comments welcome...

Can someone recommend a good SuperMicro backplane expander/HBA combination for SATA drives?
What about a MegaRAID SAS card with s single SFF-8087 connector? Will that be able to handle the I/Os and throughput to 10x (or so) SATA drives? Or are there split backplanes that make half the HDDs available on one connector and the half on the other connector, and would be suitable for an HBA with 2x SFF-8087 connectors/channels? Or am I way off track?

I'm currently thinking I'll use RAID 10 for the performance.

cheers,

Robfrog

 

[patrickdk]

Does IO throughput matter?

Your going max out at around 150MB per disk, 4 disks to max out a channel, and you have 4 channels to max out.

But that only matters for streaming workloads, not most everything you described, then your limited to iops, and your likely be lucky to hit 3MB/sec. Those disks are going max out at 80iops per second, and if your using a raid6, it will be less, unless you hit the raid cache very optimally.

 

[Choucove]

Is this server business-critical? You're talking about a lot of performance, a lot of storage, and a lot of virtual services all running from a single physical piece of hardware. What if it goes down? Are you going to be alright without it for a week or more while replacement parts/upgrades/changes are made to the server? Are you OK with providing all of these maintenance tasks and responsibilities for the server?

You could consider splitting your workload instead across two servers and replicating between them. I've seen so many times customers looking at going overkill on their servers because simply they don't know what they need. In the long run it might have been better bang for the buck to get two lesser-performance servers and run half their intended load on each. This can then be replicated to the other server for backup if needed. Yes, this does require some additional storage capacity.

If you compare 7k RPM SAS drives to a 7k SATA drive, the throughput will be nearly the same between the two drives. However, SAS has greater IO capabilities because the SAS protocol on those hard drives is duplex and can simultaneously read and write. This is why hosts running multiple virtual machines will benefit from using SAS drives over SATA drives. However, like others have said, you may consider going with separate RAID arrays, one array that is capable of high throughput (15k SAS or SSDs) and slower but larger capacity storage ( 7k SATA drives, etc.) for long term storage or what doesn't need high throughput capabilities.

 

[MrGuvernment]

Just spend the money on a larger case and skip the DAS, why add another layer of failure or problems. If you can afford a DAS or NAS or SAN's you can afford to upgrade the local server and local storage will ALWAYS be faster, even if it is milliseconds

 

[robfrog]

Just spend the money on a larger case and skip the DAS, why add another layer of failure or problems. If you can afford a DAS or NAS or SAN's you can afford to upgrade the local server and local storage will ALWAYS be faster, even if it is milliseconds

um, DAS means everything that isn't NAS or SAN, and therefore also can refer to "local storage", which is in fact exactly what I'm asking about. But thanks for taking the time.

Robfrog