Network Storage Performance Issues - Need Help

M

MountainX

Limp Gawd
Joined
Mar 2, 2008
Messages
254
Hi. I have a file server and a Terastation on my GigE LAN and I'm having severe performance problems when I back up from the file server to the Terastation. During the backup, none of the computers on my LAN can connect to the file server.

The last backup took around 12 hours (I think) to move about 250 GB and my whole LAN was just about unusable during that time. (Now, after the backup has completed, I still can't access the shares on the fileserver. I can ping it and I can even log in via remote desktop.)

I would like some advice on the performance issues. (I'll look into the share access issue separately but I mention it for completeness.)

The file server is a dual Xeon Windows Server 2003 machine with 4 GB of RAM and an Areca 1120 RAID controller with 8 WD RE2 HDDs. The motherboard is a Tyan S2720. I'm using the GigE NIC on the mb.

The Terastation is an older 1 TB model. Everything is hooked up through a higher-end Netgear GigE Switch.

During the backup, the fileserver shows about 11% CPU load and about 5% network utilization according to TaskManager.

I opened performance monitor and looked at a bunch of different performance counters and I couldn't find any that showed me where the problem was.

Questions:
1. which performance counters would be good to look at during a backup?
2. how long should it take to transfer 250 GB over a GigE LAN without much other traffic on it?
3. Am I overlooking anything obvious?

Thanks
 
Did I post this in the wrong forum or something? I wasn't sure where to post it.

How do you guys go about diagnosing performance problems related to NAS? Thanks
 
MountainX,

The problem here is likely not your server, but your Terastation. Those budget NAS devices are not bad for light use, but they can really struggle with writing operations of this much data, especially if it's configured in RAID 5. That's a lot of work for a relatively weak processor to handle.

It would help to have more information about your configuration but a possible reason your server shares appearing unavailable is that it's busy waiting on the NAS. You have one NIC and the server is trying to push a lot of data to a device that can't handle it, so there is a lot of waiting going on.

I don't know what other equipment you have to test with, but there are a couple of ways for you to test this. The first I would recommend if you have another server or decent workstation with a GbE NIC and a decently sized hard drive is connect it to the switch and point your backup job to it. If the problem goes away or improves, then it might be time to consider a more robust storage solution.
 
Hi Minsc - thanks for your helpful reply. I do not have any specialized test equipment. These computers are all in my home office.

My file server has a GigE NIC and a 100Mb NIC, but only the GigE is plugged in. I have another GigE card I could add, but I have always wondered how Windows balances traffic among multiple NICs. How would another computer on the LAN know to connect to the file server through the other NIC?

I can test the backup to another computer in the next couple days, but it will require a lot of setup work. I would prefer to try other tests in the mean time.

I may also build my own NAS, based on threads in this forum. But even then, I now realize that I need to learn something about how to find performance bottlenecks. Where can I find tips about which performance counters or other software tools to use? Thanks.
 
A couple of very complicated subjects so I'll start general and we can go in-depth from there if you like.

Very generally speaking, there are several reasons to run more than one NIC in a server. One is teaming/trunking/link aggregation, which is used to increase the link speed beyond the capability of a single NIC. You could team two 1Gb NICs together to effectively have a single 2Gb connection. Another is for fault tolerance. You might have a need to connect two NICs to redundant switches to protect against losing one NIC or one switch. Another reason is the need to segregate traffic for specific needs. For example, a server configured as a firewall could have at least one internal NIC connected to the LAN and one external NIC connected to the WAN. It will route traffic between the two separate networks based on some determined rule set.

In your particular situation, I don't think you really have a need for separate networks. With your current problem, you wouldn't even have a need to team two NICs since your problem is the end point is likely already overloaded. Providing more bandwidth in that case would not be an answer.

There are some great tools out there, but all of the ones I'm most familiar with are not free. Tools like Solarwinds or Spotlight are great, but way overkill for a small setup. Perfmon is actually pretty good and can tell you what you need to know. One could write an entire novel on perfmon, but generally there are some key counters to watch for in each area. For example, with disk utilization, %Disk Time, Current Queue Length, and Avg Transfer can provide a good window into what is going on. There are probably some decent guides on perfmon at TechNet or in the MSDN library, so I would start reading there if you are interested in learning more. Don't be surprised to find that many of them reference WinNT, and that's really not a big issue. Hardware performance principals are OS agnostic.

I think your test to a more capable device will tell you a lot, but I understand its a lot of work. Not everyone has spare equipment just lying around.

Another note: Unfortunately, I don't get a lot of time to hang out here and post, but there are a lot of smart guys here that can help.
 
Minsc - Thanks again. That's helpful.

As an intermediate step, would it help if I took the Terastation off the LAN and just hooked it up directly to the fileserver with a crossover cable (and probably just using the empty 10/100 NIC on the server)?

I would think that would free up the GigE NIC so that access to the fileserver would proceed normally.

BTW, the NAS I'm thinking about building will just use an old Pentium CPU, but I have an Alacritech GigE SLIC card I can put in it. And I also have an older hardware RAID card.
 
If your network switch has a management interface you should login and watch port traffic. Also I second using perfmon to analyze the file server. Have you disabled the network interface that you aren't using on file server? How about swap in a couple of brand new network cables? If you have another win machine you can install Aida32 on the file server and this other machine that lives on the same gig-e network and run a network benchmark between the two....that way you can at least rule out the file server and its path. GL; narrowing down an issue like this can be both time consuming and frustrating! :mad:
 
Shaggy3zx said:
Have you disabled the network interface that you aren't using on file server?
Click to expand...

No I have not. I take it that you are recommending I disable it.

Shaggy3zx said:
If you have another win machine you can install Aida32 on the file server and this other machine that lives on the same gig-e network and run a network benchmark between the two...
Click to expand...

OK, I see Aida32 is freeware. Nice.

Thanks for all your suggestions!
 
I wanted to give a quick update on this. I used two other boxes (each running Windows 2003 Server) on my LAN to test this today. Neither is a powerful server. Both are Pentium 4's.

I started a transfer of 106 GB from one to the other. During this, I am not having any of the performance issues I saw when doing the transfer to the Terastation. I tested all the same things -- and no problems.

I'm running performance monitor on both computers.
Code: 
                        Target       Source
% disk time       2000%+    60%
current Q lngth   0-20+       0-1
Avg Bytes/Trans 58000      65000
transfers/sec      0.03        0.001
CPU%               20-40      60-80+
I don't know exactly how to guage those numbers, so I'm just reporting them. I welcome any comments.

The 106GB transfer is estimated to take 71 minutes. And it has already progressed down to about 50 minutes.

This is in contrast to a 250GB transfer from the Terastation that took more than 12 hours (I can't say exactly how long). The conditions I'm testing at the moment are not all equal to the prior situation, so I'll report more if I find other factors that could be responsible for the differences.

For now, I would have to assume that the problem is the Terastation. A home built NAS will probably be my solution.

(Sorry for the double post, but hopefully this one has the data table formatted correctly)
 
You must log in or register to reply here.
Back
Top