• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

SQL performance over 1GB iSCSI

Riccochet

Riccochet

Extremely [H]
2FA
Joined
Apr 11, 2007
Messages
33,334
\What's the general consensus around running SQL VM's with 1GBE iSCSI SAN's? These are HP P4530 SAN's with 15k drives. Would 10GBE make that much of a difference?
 
Depends on your environment.

Are the servers going to need more than 125 MB/s of bandwidth? Probably not during operation.

Also enabling Jumbo Frames on your iSCSI network is a good way to get free performance.
 
Jumbo frames are enabled. The reason I ask is we're having some performance issues. Two 1GB iSCSI channels in round-robin going to each SAN. Memory and CPU look great on the hosts, but storage latency spikes in to the 60-70ms range have me concerned. Reports of slowness from clients. Wondering if the jump to 10GBE will make much of a difference.
 
60-70ms is a high for a local connection, it suggests either a fully saturated link or that the problem isn't in the physical layer.

What are the performance metrics on the SAN itself? I wonder if you'd see that same 60-70ms delay there.
 
HydroBudz said:
Jumbo frames are enabled. The reason I ask is we're having some performance issues. Two 1GB iSCSI channels in round-robin going to each SAN. Memory and CPU look great on the hosts, but storage latency spikes in to the 60-70ms range have me concerned. Reports of slowness from clients. Wondering if the jump to 10GBE will make much of a difference.
Click to expand...

fire up esxtop and see what's going on - $5 it's IOPS contention or cache exhaustion on the array. Look at throughput and iops to see which is doing what.
 
What is the connection between nodes? I've run into issues in the past on HP LeftHand stuff with sync bandwidth between nodes during heavy write scenarios before if only using 1gb links. Another possibility would be to make sure that the RAID card in all nodes are set to have the cache enabled.
 
Last edited:
We have quite a few MS SQL DBs, Oracle DBs, and MYSQL DBs running over 1 Gb iSCSI and 1 Gb NFS. No performance complaints yet and we are not saturating the network. Our bottleneck ended up being spindles. So we bought more and added some SSD for flash cache. :)
 
Same, we run SQL 2008 and 2012 over 1 GB to iSCSI and except for one particular query that trashes the file system we don't have any performance issues during normal operations. Meaning we hit a storage performance bottleneck long before we saturate the network link.
 
We host roughly 180 DB's across 7 SQL VM's with over 1000 users accessing these DB's. The storage gets hammered during the day. I'll have to look at esxtop today, but from what I can tell on our switches it doesn't look like the network is being saturated. Though I've read some white papers saying SQL over 1GB iSCSI is a no-no. So, who knows.

Unfortunately I don't think the HP (LeftHand) P45xx series SAN's support SSD caching. That money would have been spend by now which I think would help tremendously.
 
So, when looking at the esxtop CMDS/s is there a way to see the total IOPS for a given datastore that's being accessed from multiple hosts?
 
Yeah, I'm at a loss since I'm not exactly sure what I'm looking at.

Might be time to bring in a consultant. :(
 
How large are the buffers on the switches connecting servers to SANs? Are they SAN switches physically separated from production/management traffic?
 
HydroBudz said:
Yeah, I'm at a loss since I'm not exactly sure what I'm looking at.

Might be time to bring in a consultant. :(
Click to expand...

Call VMware support - I'm away from real computers right now or I'd offer to help. You want to look at disk statistics for the Lun though (u key), and Iops/davg/kavg/gavg.
 
I installed VMWare I/O Analyzer vApp. See if I can get some performance over time stats from that which pulls directly from esxtop.
 
renixinq said:
Bump for wondering what the OP found.
Click to expand...

I don't know what I've found. Honestly, I can't find any information on where my IOPS should be opposed to what I'm seeing in esxtop. That and multiple hosts accessing the same datastores really throws me off.

In other words: fucked if I know.

We're looking to have a VMWare consultant come in and look at the environment.
 
NetJunkie said:
Any idea what size I/Os you're seeing on those databases?
Click to expand...

Not sure. I know reads and writes are all over the place. Have seen latency GAVG up in the 50-60 range at times. We're kind of leaning towards and I/O performance issue as memory and CPU are barely being taxed. Network, aside from being 1GBE, doesn't look too too bad. Might just be the amount of spinners we have. 12 disk 15k 600GB RAID 10's mirrored.
 
Did you ever answer my question about the SAN itself? What's the link between the nodes? And have you verified what performance that you ARE getting from it yet?
 
I just want to confirm your running some 180 databases from 7 VM with 1000+ users on 12 15k SAS spindles in a raid 10 that produces around 1400 IOPS (assuming 4k IO, 50% read, 50% write, and 10% cache hit) from the drives alone - not counting any caching from the SAN itself?

I'm going to go out on a limb here and say you need more spindles/SSD cache...

http://www.wmarow.com/strcalc/
 
Last edited:
Dytralis said:
I just want to confirm your running some 180 databases from 7 VM with 1000+ users on 12 15k SAS spindles in a raid 10 that produces around 1400 IOPS (assuming 4k IO, 50% read, 50% write, and 10% cache hit) from the drives alone - not counting any caching from the SAN itself?

I'm going to go out on a limb here and say you need more spindles/SSD cache...

http://www.wmarow.com/strcalc/
Click to expand...

Technically 24 spindles via 2 SAN's. Each array is mirrored, but that's not really going to increase performance. No SSD cache as I have yet to find any information that would lead me to believe the HP P4500 and P4530 support SSD cache. If they do I'll buy SSD cache drives today.

So, yes, I'm thinking we are I/O limited here.
 
HydroBudz said:
Technically 24 spindles via 2 SAN's. Each array is mirrored, but that's not really going to increase performance. No SSD cache as I have yet to find any information that would lead me to believe the HP P4500 and P4530 support SSD cache. If they do I'll buy SSD cache drives today.

So, yes, I'm thinking we are I/O limited here.
Click to expand...

PernixData FVP to the rescue!
It's not a silver bullet, but if your workload is such that you do not saturate your IO all the time but rather during times when certain queries run, then Pernix will make a huge difference.
 
Thuleman said:
PernixData FVP to the rescue!
It's not a silver bullet, but if your workload is such that you do not saturate your IO all the time but rather during times when certain queries run, then Pernix will make a huge difference.
Click to expand...

Interesting. I'll bring this up in our next meeting.
 
It's also extremely easy to POC and you'd find out very quickly if it's a fit for you.
 
Here's a screenshot of one of our SQL VM's waits and queue stats. Looks like a definite IO issue.

y5e573w.jpg
 
I had to spin up a few VMs for remote employees and after exhastive research by myself and about 4 other techs, I finally found that SQL performance in all test cases was mainly effected by the latency of the datastore. If its a decent NAS / SAN with low latency, it worked ok, but as soon as the physical disks/network were saturated to the point of network latency, SQL would slow to a crawl as the IO request que compounded. I solved it by using the SAN ds as the VM OS drive while using the ESX local ds for the SQL data drive inside the VMs
 
We'll find out the cause of the issues soon. Thanks to NetJunkie and Varrow for providing prompt attention to our needs. I can't wait to work with their engineer.
 
Looks like storage is our issue. Not enough IOPS. Uhhh boy. This is going to be an expensive resolution. I've been saying it for months.
 
You must log in or register to reply here.
Back
Top