NetApp Flash Cach vs PernixData or other caching technology?

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
We have two FAS3240's running in HA. One of our shelves has 24 - 600 GB 15k SAS drives. Performance has been decreasing as more and more VMs are added. Plans are to purchase an additional shelf filled with the same drives and do some type of caching.

First, I have no idea how to figure out how many IOPS this shelf can handle. Based on our monitoring software it seems to average about 1.7k IOPS. There should be about 170 VMs running on this shelf. Quite a few of them running SQL.

We have a few new ESXi servers setup in vCenter 5.5 and started using VMware Flash Cache which you have to enable and allocate per VM. I'm not really sure how to even test the performance gains from this caching other than our clients are happy.

We were thinking about a more centralized SSD caching that could benefit all VMs running on our Tier 1 storage. One option is the NetApp Flash Cache. Last time we got a quote for this over a year ago, it was $17k for a 512 GB PCI Express card, and we would need two. We are going to get new pricing and hopefully it has at least dropped in half.

I have heard of PernixData, but never used it and don't know the pricing as of right now, but I would like more than just 512 GB of SSD caching since we have about 5 TB of Tier 1 storage currently in use.

How would you compare the SSD caching between NetApp and PernixData and maybe another product you have tested? Which would you go with or think is the best performance gain for the cost.
 
I am in the process of doing some testing with emc's pci xtremesf cards. Neat cards.. With cache cards, they only really help if you are heavy on reads and reading the same data over and over again.

Doing some basic math, 24 drives isn't enough for your iop requirements.. You'd need to double the amount of drives to hit those iops you need. Or, if your array supports it, using EFD/SSD internal for caching.

To test flash cache, you'll need to log into the guest vm, and run the native os performance tools.. windows, performance monitor, Linux, iostat/sar/nmon/top etc, and then look at disk response time. Reponses time is subjective based on what peoples expectations are.

if your array vendor supports it, look into installing ssd's on your array, and moving hot vm's/data stores to your ssds. Even better, if your array is smart enough, allowing ssds and sas drives to co-mingle in the same diskgroups allowing the array to auto tier what data needs to go where.
 
I have a pair of 3220s running in HA with flash cache in them. VMs mostly are on the one head, 3 shelves, 600GB 15k SAS in them, somewhere around 275 running guests including some SQL boxes and Exchange boxes. With the 512GB card we seem to average 14k-15k read requests and and average 8k-9k read hits, so it works pretty well, it's monitored in Read Ops/sec.
 
kdh said:
I am in the process of doing some testing with emc's pci xtremesf cards. Neat cards.. With cache cards, they only really help if you are heavy on reads and reading the same data over and over again.

Doing some basic math, 24 drives isn't enough for your iop requirements.. You'd need to double the amount of drives to hit those iops you need. Or, if your array supports it, using EFD/SSD internal for caching.

To test flash cache, you'll need to log into the guest vm, and run the native os performance tools.. windows, performance monitor, Linux, iostat/sar/nmon/top etc, and then look at disk response time. Reponses time is subjective based on what peoples expectations are.

if your array vendor supports it, look into installing ssd's on your array, and moving hot vm's/data stores to your ssds. Even better, if your array is smart enough, allowing ssds and sas drives to co-mingle in the same diskgroups allowing the array to auto tier what data needs to go where.
Click to expand...

I don't know how many IOPS we need. 1.7k is the average of IOPS we are currently using on 1 shelf. We are definitely picking up a second 600 GB 15k shelf which is another 24 drives. We have two SATA shelves already, but those are running fine.
 
KapsZ28 said:
I would hope so being that we can only install one per head. If the flash cache had writes on it and it suddenly failed, we would lose that data.
Click to expand...

What I thought. I sell a lot of EMC and their arrays do read and write caching with front-end SSDs. We still see great performance increases with PernixData FVP. You can put more flash in your hosts for a good bit less money than doing it in the array. We often will put Intel S3700s or Kingston E100s in the servers. Usually 200GB or 400GB per server. Add that up across all systems and you get a lot of flash right there local.

Usually at that much per server the size of the working set isn't a big deal and almost everything comes off flash and if you enable write caching all of your writes land on flash for VERY low latency. It's impressive and super easy to deploy and use.

Here is a writeup I did on FVP:

http://jasonnash.com/2013/06/24/a-look-at-pernixdatas-flash-virtualization-platform/

And a recent customer case study:

http://jasonnash.com/2014/03/25/cardinal-innovations-pernixdata-emc/
 
KapsZ28 said:
I don't know how many IOPS we need. 1.7k is the average of IOPS we are currently using on 1 shelf. We are definitely picking up a second 600 GB 15k shelf which is another 24 drives. We have two SATA shelves already, but those are running fine.
Click to expand...

But that's the nice thing about server-side caching like FVP. If those 15Ks are going to be used by VMs you probably don't need that. You could easily go 10K or even more 7.2K SATA. Let the flash do the real work. Save money on the back-end array and you still come out way ahead on performance.
 
NetJunkie said:
But that's the nice thing about server-side caching like FVP. If those 15Ks are going to be used by VMs you probably don't need that. You could easily go 10K or even more 7.2K SATA. Let the flash do the real work. Save money on the back-end array and you still come out way ahead on performance.
Click to expand...

That is a good point. I'll have to look into that too. But from what I heard the price difference between a shelf filled with SATA vs SAS isn't that much of a difference. I could be wrong though. Obviously you get a lot more storage from SATA.
 
NetJunkie said:
Is NetApp Fash Cache still read cache only?
Click to expand...

Does it matter?

Write cache goes to battery backed NVRAM. There is also flash pool which can use SSD as both read and write cache. Flash cache is global, flash pool is per aggregate and can be configure per volume.
 
dilidolo said:
Does it matter?

Write cache goes to battery backed NVRAM. There is also flash pool which can use SSD as both read and write cache. Flash cache is global, flash pool is per aggregate and can be configure per volume.
Click to expand...

So are you saying the flash cache for NetApp does support both reads and writes?

If it does, that is great, but the downside is our NetApp only supports a 512 GB flash cache which isn't a whole lot when we are already using 5 TB of Tier 1 storage for VMs.
 
NetApp Flash Cache doesn't do read and write. And keep in mind..there is no battery backup for flash cache. This isn't DRAM cache. If it did write cache without mirroring the data (like EMC VNX does...or PernixData can do) and that SSD dies you just lost that data that was being held in write cache.

Very big difference.
 
NetJunkie said:
NetApp Flash Cache doesn't do read and write. And keep in mind..there is no battery backup for flash cache. This isn't DRAM cache. If it did write cache without mirroring the data (like EMC VNX does...or PernixData can do) and that SSD dies you just lost that data that was being held in write cache.

Very big difference.
Click to expand...

Yeah, that is what I was saying before which is how I thought the NetApp worked. :D


And PernixData can do read/write on any setup? It doesn't have to be EMC?

Any idea what PernixData costs? And are there requirements for the SSDs in the ESXi hosts? As in, do they need to be connected to a RAID controller, HBA, direct to SATA port on motherboard, or PCI Express card?
 
PernixData doesn't care what the back-end storage is...all it cares is that it's block so iSCSI or FC. If you're doing NFS on that NetApp it won't help...yet.

You want good SSDs. Quality enterprise stuff. You want decent controllers. No RAID. I like the LSI9207 (as does Pernix). Those aren't much. Some servers have good onboard SATA controllers and some don't. It just depends.
 
NetJunkie said:
PernixData doesn't care what the back-end storage is...all it cares is that it's block so iSCSI or FC. If you're doing NFS on that NetApp it won't help...yet.

You want good SSDs. Quality enterprise stuff. You want decent controllers. No RAID. I like the LSI9207 (as does Pernix). Those aren't much. Some servers have good onboard SATA controllers and some don't. It just depends.
Click to expand...

Ouch. This environment is NFS. Everywhere else we are using iSCSI but they opted for NFS in our primary location for some reason.

Does VMware VSAN also work with the LSI 9207? Unfortunately I screwed up with our first three new servers by getting the LSI 9271-8i before I realized that all this VSAN stuff uses HBA instead of RAID.
 
NetJunkie said:
NetApp Flash Cache doesn't do read and write. And keep in mind..there is no battery backup for flash cache. This isn't DRAM cache. If it did write cache without mirroring the data (like EMC VNX does...or PernixData can do) and that SSD dies you just lost that data that was being held in write cache.

Very big difference.
Click to expand...

I think what he was saying is you can also use the flash pool with the netapp as well. That does support read and write and does automatic tiering.

If you're buying another shelf anyway, you might want to get pricing for the flash pool feature as well.
 
KapsZ28 said:
Ouch. This environment is NFS. Everywhere else we are using iSCSI but they opted for NFS in our primary location for some reason.

Does VMware VSAN also work with the LSI 9207? Unfortunately I screwed up with our first three new servers by getting the LSI 9271-8i before I realized that all this VSAN stuff uses HBA instead of RAID.
Click to expand...

Yes, VSAN also likes 9207 for the same reason. It's a good JBOD controller. You can use the 9271 just do RAID0 for FVP or VSAN.
 
ND40oz said:
I think what he was saying is you can also use the flash pool with the netapp as well. That does support read and write and does automatic tiering.

If you're buying another shelf anyway, you might want to get pricing for the flash pool feature as well.
Click to expand...

Yes, you can...but FVP is probably a MUCH better bang for the buck. We can do read/write cache on flash as well as auto tiering flash pools on arrays and I still like FVP. Go with a fatter array (SATA instead of SAS) for less money and still get better performance.
 
You're not comparing apples to apples when you look at Pernix's offering compared to NetApp FlashCache. Pernix is host based SSDs, FlashCache is storage based PCI cards that go into the array.

A closer approximation would be NetApp FlashPool (aka hybrid aggregate) where the SSDs front-end spinning disk and serve as a read/write caching tier. Writes get ack'd instantly and WAFL sequentializes those random writes for better performance, and hot blocks being read from disk first try to get read from SSD and if they exist there great instant IOP or if it needs to reach down to disk it will drop a copy off in SSD while serving it so it's ready for the next request.

NetApp also has storage side caching called Flash Accel that allow you to marry a non-NetApp enterprise SSD in your server and let Data ONTAP (the OS running on a NetApp filer) to manage that similarly to Flash Cache.
 
NetJunkie said:
Yes, you can...but FVP is probably a MUCH better bang for the buck. We can do read/write cache on flash as well as auto tiering flash pools on arrays and I still like FVP. Go with a fatter array (SATA instead of SAS) for less money and still get better performance.
Click to expand...

You can do the same with NetApp, just go SATA and SSD for the shelf and enable the flash pool. If you don't have the heads full of disk already, just add a recommended minimum of 5 SSDs and off you go with flash pool. Really depends on what you want to do, but the NetApp can do it as well.
 
Guys. I get it. I do storage for a living. :) I deal in arrays that do both front-end read/write cache with SSDs as well as auto-tiered pools of storage that have flash/SAS/SATA in them. I get it.

What you're missing is the cost per IOPS. You can't beat server-side cache for cost per IOPS. You just can't. It's just cheaper to buy standard good SSDs and put them in your vSphere hosts than it is to buy SSDs from your Tier 1 storage vendor...and that's with licensing included for the caching software.

You also get the benefit of cache RIGHT THERE ON THE SERVER. No network..no fabric. It's right there giving you sub-ms response time. If you need more cache you just throw more cheaper SSDs in the servers and go. No additional shelves. No changing pools. Etc. It works really, really well.

The reason I usually push PernixData for VMware is that they are the ONLY ones that do both read and write caching. Their biggest gap right now is no NFS support but that'll get fixed. Doesn't help OP this second but I can't help that. No one else does write caching, period.

I suggest anyone buying or adding storage to do the numbers themselves. You can usually get more capacity (bigger SATAs instead of small faster SAS) with a lot more performance (several times more usually) for less money.
 
One more point I forgot...

If the majority of your I/O is being serviced out of local cache what does that do for the backbend array? It takes a LOT of load off that array. So everything gets faster. It's a great way to extend the life of an aging array or keep you from doing a head upgrade due to load.
 
NetJunkie said:
No one else does write caching, period.
Click to expand...

Last I checked VSAN did write caching on host based SSD and you can overprovision SSDs to add more performance. I would expect someone who worships at the altar of EMC like yourself would be up to speed on what VMware is doing these days. Your points about high-end consumer/low-end enterprise drives giving superior IOPS/$ are completely valid though.
 
G4VEL said:
Last I checked VSAN did write caching on host based SSD and you can overprovision SSDs to add more performance. I would expect someone who worships at the altar of EMC like yourself would be up to speed on what VMware is doing these days. Your points about high-end consumer/low-end enterprise drives giving superior IOPS/$ are completely valid though.
Click to expand...

VSAN isn't server-side caching. It's kernel-integrated node-based storage. Yes, all writes land on flash with VSAN...but you can't put VSAN in front of another storage system. The persistent storage is spinning disks in each node with flash in front of them for read/write cache. So not really applicable.
 
NetJunkie said:
VSAN isn't server-side caching. It's kernel-integrated node-based storage. Yes, all writes land on flash with VSAN...but you can't put VSAN in front of another storage system. The persistent storage is spinning disks in each node with flash in front of them for read/write cache. So not really applicable.
Click to expand...

You were talking about server-side caching's benefits as it pertains to IOPS/$, VSAN seems extremely applicable in that scenario in a thread that's entirely about alternatives to dumping more money into Big Storage's take on caching. Of course you can't put VSAN in front of other storage, it is the storage. Simply pointing out that if your application would benefit from SSD write caching, and you can't afford enterprise storage, rolling your own VSAN is a great way to go.
 
G4VEL said:
You were talking about server-side caching's benefits as it pertains to IOPS/$, VSAN seems extremely applicable in that scenario in a thread that's entirely about alternatives to dumping more money into Big Storage's take on caching. Of course you can't put VSAN in front of other storage, it is the storage. Simply pointing out that if your application would benefit from SSD write caching, and you can't afford enterprise storage, rolling your own VSAN is a great way to go.
Click to expand...

Maybe. Maybe not. At $5K/host (two socket) plus drives and SSDs...plus you really want 10Gb...and servers that can hold more than just a couple disks... That all adds up and it all depends on what you have today. Have an existing array that you want to get more out of? Server-side caching. Have blades and can't do VSAN? Server-side caching.

I like VSAN a lot but it's going to take time for people to really trust and it's not a good fit for denser environments such as Cisco UCS. My expectation is that it'll be done in newly built environments, not bolted on to an existing one but we'll see.

My points above were targeted at those looking to add more IOPS or capacity to an existing array. If you are building completely new your options obviously open up. Traditional array... Scaled down array + server-side cache. Node-based storage such as VSAN, Nutanix, Symplivity...etc. Lots of options.
 
With NetApp flash pool you put the SSDs directly into the shelf? Would you be able to have SSDs just in one shelf with an aggregate spanned across two shelves both utilizing the flash pool? That's if you can even span an aggregate over multiple shelves, which I don't know.

We bought our new servers with VSAN in mind and they have 10gbe, but we don't have 10gbe switches yet. That is another budget. How bad would it be starting off VSAN with 1gbe?

We could always switch to iSCSI to use PernixData, but I think the iSCSI protocol is like $10k to enable on the NetApp.
 
1Gb w/ VSAN does limit you, especially on writes as it wants to mirror those writes to another node for redundancy. You can tell it not to do that but obviously you get N+0 redundancy at that point which is a "bad thing". If you need any real performance you want 10Gb.
 
KapsZ28 said:
With NetApp flash pool you put the SSDs directly into the shelf? Would you be able to have SSDs just in one shelf with an aggregate spanned across two shelves both utilizing the flash pool? That's if you can even span an aggregate over multiple shelves, which I don't know.
Click to expand...

Yes, you put the SSDs directly in the shelf with Flash Pool and you configure the Flash Pool per aggregate. Aggregates can be spanned over multiple shelves, or broken up across multiple shelves.
 
KapsZ28 said:
With NetApp flash pool you put the SSDs directly into the shelf? Would you be able to have SSDs just in one shelf with an aggregate spanned across two shelves both utilizing the flash pool? That's if you can even span an aggregate over multiple shelves, which I don't know.

We bought our new servers with VSAN in mind and they have 10gbe, but we don't have 10gbe switches yet. That is another budget. How bad would it be starting off VSAN with 1gbe?

We could always switch to iSCSI to use PernixData, but I think the iSCSI protocol is like $10k to enable on the NetApp.
Click to expand...

Talk to your NetApp rep by Friday. Adding protocols individually is "expensive" (you're way better off to buy a Complete bundle when you buy your controllers FYI) but this is NetApp's year end and they will drop their pants.
 
apnar said:
In particular the pants around their SSDs seem to be very low which could make a flash pool for the OP attractive.
Click to expand...

I hope so. We will find out.

What would be a good configuration for Flash Pool? So we have the 24 disks of Tier 1 storage. We are definitely buying another shelf with SAS disk. Would we get 18 SAS and 6 SSD? How does the SSD pool work? Is it also setup with dual parity?

Just trying to figure out how many and what size SSDs we would need. I would love to have at least 1 TB of SSD for read/write cache.
 
Keep in mind it won't be cache. It'll be Tier 0 (or 1..whatever you want to call it) storage. If all I/O lands on the SSDs and your data happens to be on the SSDs it'll be like cache...but not cache. Just keep that in mind.
 
KapsZ28 said:
I hope so. We will find out.

What would be a good configuration for Flash Pool? So we have the 24 disks of Tier 1 storage. We are definitely buying another shelf with SAS disk. Would we get 18 SAS and 6 SSD? How does the SSD pool work? Is it also setup with dual parity?

Just trying to figure out how many and what size SSDs we would need. I would love to have at least 1 TB of SSD for read/write cache.
Click to expand...

If you use RAID-DP for you aggregate, your SSD needs to use RAID-DP too. It has to match your raid type your aggregate uses, so minimal 3 SSDs per aggregate.
 
NetJunkie said:
Keep in mind it won't be cache. It'll be Tier 0 (or 1..whatever you want to call it) storage. If all I/O lands on the SSDs and your data happens to be on the SSDs it'll be like cache...but not cache. Just keep that in mind.
Click to expand...

No. There is no Tier 0. It works the same way as NVRAM and FlashCache, but use SSD inside aggregate instead.

IO lands on SSD first, then flushes to rotational disks. On read, it keeps a heat map on what blocks are accessed more frequently and caches them in SSD.

FlashPool is very similar as ZFS with zil/L2ARC with more limitations. It is very different as FAST which moves chunk of data into Tier 0.

You don't seem to understand NetApp much.
 
KapsZ28 said:
I hope so. We will find out.

What would be a good configuration for Flash Pool? So we have the 24 disks of Tier 1 storage. We are definitely buying another shelf with SAS disk. Would we get 18 SAS and 6 SSD? How does the SSD pool work? Is it also setup with dual parity?

Just trying to figure out how many and what size SSDs we would need. I would love to have at least 1 TB of SSD for read/write cache.
Click to expand...

BTW. You may take a look at http://www.infinio.com appliance. It only works with NFS datastore and does server side read caching only, but no SSD required as it uses RAM disk for caching.
 
dilidolo said:
If you use RAID-DP for you aggregate, your SSD needs to use RAID-DP too. It has to match your raid type your aggregate uses, so minimal 3 SSDs per aggregate.
Click to expand...

Sounds awesome. So we could technically start with three 512 GB SSDs if we wanted. Hopefully we will do more if we go this route, but still don't know about pricing yet.

Even if a shelf was $50k with 18 SAS and 6 SSDs, I think it would definitely be worth while to get that read/write caching. It sounds much better than getting a 512 GB Flash Cache PCI Express card installed in the head which would could never expand on.
 
One more question about FlashPool. Does it make sense to use it with 15k SAS drives? From what I am reading it seems more common on SATA drives since there is such a big improvement on performance.

Can you mix an aggregate with both SAS and SATA?
 
KapsZ28 said:
Sounds awesome. So we could technically start with three 512 GB SSDs if we wanted. Hopefully we will do more if we go this route, but still don't know about pricing yet.

Even if a shelf was $50k with 18 SAS and 6 SSDs, I think it would definitely be worth while to get that read/write caching. It sounds much better than getting a 512 GB Flash Cache PCI Express card installed in the head which would could never expand on.
Click to expand...

Yes and no.

FlashCache is global deduped, if you have multiple aggregate, it works better and it is faster than Flash Pool. If you only have 1 aggregate, it doesn't matter much.

One good thing about Flash Pool is it is more granular than FlashCache. You can turn it on/off per volume.

Now here is the limitation.
1. Once it's added, you can't remove it unlike ZFS ZIL/L2ARC.
2. You put 3 SSDs in RAID-DP, your cache size is only 1 SSD. So it makes more sense to put more SSDs, thus 5+1 hotspare.
 
KapsZ28 said:
One more question about FlashPool. Does it make sense to use it with 15k SAS drives? From what I am reading it seems more common on SATA drives since there is such a big improvement on performance.

Can you mix an aggregate with both SAS and SATA?
Click to expand...

1. yes, to improve latency
2. If same size, same rotation speed (7200RPM SAS/SATA), might be possible. But no, don't do it.
 
You must log in or register to reply here.
Back
Top