Slow Multicasting on Cisco 3560G

K

KapsZ28

2[H]4U
Joined
May 29, 2009
Messages
2,114
I am not a network person, but I am not sure our Network Team is qualified either. For a long time we have been using a dying Cisco 2950 100/MB switch for imaging our computers using GhostCast. Multicasting on the 2950 was never an issue other than it was flooding the switch and making all the other ports unusable. But at least the multicast portion was imaging with decent times. We had two Intel Pro NICs teamed to get 200 MB/s and a typical multicast session was about 6.5 minutes. When using only 1 NIC, it takes about 8.5. Running two concurrent unicast or multicast sessions, well that is a different story. It then triples or quadruples the time.

Now we have a 3560G setup. A single session takes a little over 3 minutes. If we multicast off of this switch, the time is about 12.5 minutes. That is 4 minutes longer than the 2950! If I run 3 concurrent unicast sessions, it takes all of them between 3.5 to 4 minutes. So obviously there is an issue with multicasting.

The switch is in a lab separate from the production network. The server with our ghost images is plugged directly into the switch and so are the computers we are imaging. From what I was told from the Network Team is that IGMP snooping is on, but multicasting is off. He said that multicasting does not need to be turned on since all traffic is going through the local switch. Probably true since our old switch didn't have multicast turned on either, but it still imaged using multicast faster.

Below are some screen shots of the imaging. The first one is the 2950 using multicast to 3 laptops. The second one is the 3560 using multicast to 3 laptops. Finally is the 3560 using 3 unicast sessions to the same 3 laptops.

So, what exactly would you think is the problem that would cause the multicast to take so much longer on our new gigabit switch?

Cisco2950.jpg


Cisco3560G.jpg


Cisco3560G-3UnicastSessions.jpg
 
Really difficult to say without seeing the current configuration. There are no complicated commands required to make the multicast work.

Your network admin is correct in that IP multicast routing does not need to be enabled since there is no routing involved at all.

Ninja edit: never mind what I wrote before. If yours is a flat configuration (server and client are both in the same VLAN) then it should just work. No special configuration needed.

Why do you want multicast to work when unicast to multiple clients works fine? You aren't having a bottleneck problem so multicast isn't going to help anything.
 
Shadowspawn said:
Why do you want multicast to work when unicast to multiple clients works fine? You aren't having a bottleneck problem so multicast isn't going to help anything.
Click to expand...

When I am imaging 10-15 computers at a time, why would I not want to multicast? I am not going to setup 10-15 separate unicast sessions for imaging. That is a pain in the ass, time consuming, and will end up bottlenecking the server. This is the point in using multicast.
 
What are you doing is stupid, plain and simple.

1. Multicast inherently is a un-reliable delivery mechanism since it ONLY supports UDP. You can use it reliably but im not sure that you are, but that would be at the application level and its no longer considered a single stream multicast session.

2. I agree with Shadowspawn, you do not want to deliver your images over one stream, much due to what I mentioned above.

As far as performance goes, technically there is very little to be done on the switch at all since it sounds like its only forwarding frames(L2). What you can have your network team verify is that IGMP snooping is enabled on the switch so it can have some intelligence about how to forward multicast traffic. If you do not have IGMP snooping enabled on the switch it will simply flood traffic out all ports, which is less desirable.
 
I guess network people in general don't have a clue as to how imaging is done. Telling IT that using multicast is stupid is well, stupid. I supposed when we do our DR events and have to image 100 PCs, we should use unicast as well, right? That shouldn't take any longer than usual. :rolleyes:

We have been using multicast for years and never once had a problem with the image being delivered.
 
xphil3 said:
What are you doing is stupid, plain and simple.

1. Multicast inherently is a un-reliable delivery mechanism since it ONLY supports UDP. You can use it reliably but im not sure that you are, but that would be at the application level and its no longer considered a single stream multicast session.
Click to expand...

Technically, multicast can use whatever protocol you want. Though putting TCP on top of it would be really stupid. I'm glad you mentioned the 2nd part .. I was gonna jump on you with feedback groups :)

Anyway Kaps, you shouldn't be seeing this performance loss. The 3560 isn't something I'd use for L3 multicast, so there's a possiblity you have this turned on by mistake and the app is sending out TTL=1 multicast packets punting your box bringing it to its knees.

Take a look at "sh proc cpu his" during your multicast transfer and see if your CPU goes through the roof when you do the transfer. If so, you could troubleshoot to see what you have enabled, but it's probably easier to make a NEW vlan, DO NOT configure an IP on the vlan interface, and that should work out better.

Finally, make sure you don't have storm control configured.
 
KapsZ28 said:
I guess network people in general don't have a clue as to how imaging is done. Telling IT that using multicast is stupid is well, stupid. I supposed when we do our DR events and have to image 100 PCs, we should use unicast as well, right? That shouldn't take any longer than usual. :rolleyes:

We have been using multicast for years and never once had a problem with the image being delivered.
Click to expand...
Its not that we dont have a clue, it's we generally know how to do IT's job better and understand how the technology works. We're not mine sweeper experts like 99.999% of IT guys are.

Feel free to use "multicast" though to get images to your laptops(again, I doubt that its doing this anyways.. perhaps it is Xcast)... and since you're not a network person its abundantly clear why us network guys think you're stupid for wanting to using it, based solely on your lack of knowledge of the underlying technology that makes your "Magic" work. Have fun with your pixie dust my friend! :rolleyes:

Justin, of course i knew someone would mention xcast or PGM and its specifically why I mentioned that its at the application layer. I deal with application goons all day and they CONSTANTLY have something about a "network issue". Not saying that you're an application goon :)
 
Last edited:
Well, I will be sure to call Symantec and tell them to remove the multicast option from GhostCast since it is unreliable and nobody should be using this feature.
 
KapsZ28 said:
Well, I will be sure to call Symantec and tell them to remove the multicast option from GhostCast since it is unreliable and nobody should be using this feature.
Click to expand...
lol, guy. Listen.

What I was trying to covey was that you need to better understand the application that you're working with, and not resort to calling it a network issue just becuase you see an increase in the time it takes to deliver your images. How do you do this?

1. Verify that it is using UDP or a TCP hack
2. If it is using UDP, verify that you are seeing negative ACKs sent via the ghost server.. This is one form of reliable UDP transport. If you see SIMPLY UDP its a terrible way to deliver images
3. Call symantec and open a case, if this is truly an issue that is impactful then it should be treated as such.
4. Dont come on these forums bitching at people that are providing sound information just because we tossed in a jab or so. It happens far too often on here, people come and bitch about "network issues" just to bitch.
 
xphil3 said:
4. Dont come on these forums bitching at people that are providing sound information just because we tossed in a jab or so. It happens far too often on here, people come and bitch about "network issues" just to bitch.
Click to expand...

I am not coming here to bitch. I am coming here for suggestions since our Network Team is saying "everything is configured properly." If that was the case, then the multicasting would work just like it did on our old switch, only faster. But it doesn't. It takes 4 minutes longer than it does on a 100/MB switch, but yet unicast is done incredibly fast.

Prior to your last post you said "What are you doing is stupid, plain and simple." and you also suggested to check if IGMP snooping is enable, which it is as I mentioned that in the original post.

Nothing has changed server side, and if you look into using GhostCast there is pretty much no configurations that can be done.

This is an old article, but it shows how Ghost communicates over the network in unicast and multicast.

http://service1.symantec.com/SUPPORT/ghost.nsf/ppfdocs/2002101612025325

Sorry if you think I am bitching, I just need something worth while to bring back to the network team to get them to work with us, rather than just popping in a switch that can't multicast.
 
Also, this is how everything is currently setup. We are waiting to have a fiber run to our downstairs closet. So right now the 3560 is plugged into the 2950 which is plugged into a 2600 series router, which is plugged into the production network. All PCs and servers have been moved to the new switch. Ignore the 5 cables plugged into the old switch as that was just for testing. And also ignore the cable modem and Linksys router. They are only connected to Comcast.

IMG-20110218-00063.jpg
 
Take back what I said to your network dudes.

I'll bet you a night of booze and bitches that making a new L2 vlan with only those ports in it will fix it, provided that storm-control is OFF and igmp-snooping is ON.

Then we can work further to resolve what you guys have misconfigured.
 
just2cool said:
Take back what I said to your network dudes.

I'll bet you a night of booze and bitches that making a new L2 vlan with only those ports in it will fix it, provided that storm-control is OFF and igmp-snooping is ON.

Then we can work further to resolve what you guys have misconfigured.
Click to expand...

Yes, I am not knocking other network guys, but our team seems special and also the hardest to deal with in the company. Supposedly after the fiber is run, the old router and switch will be removed and we will only have the 3560G in our room.

We are on a separate VLAN, but I guess you are saying to make sure it is a layer 2 VLAN?
 
KapsZ28 said:
I am not coming here to bitch. I am coming here for suggestions since our Network Team is saying "everything is configured properly." If that was the case, then the multicasting would work just like it did on our old switch, only faster. But it doesn't. It takes 4 minutes longer than it does on a 100/MB switch, but yet unicast is done incredibly fast.

Prior to your last post you said "What are you doing is stupid, plain and simple." and you also suggested to check if IGMP snooping is enable, which it is as I mentioned that in the original post.
Click to expand...

1. I didn't see that IGMP in your OP, so if its already been verified then great. Though even if it wasn't turned on i HIGHLY doubt you would see any performance impact with only 5 PCs.

2. I said its stupid only becuase the assumption that Ghost is adhearing to the original standard of ONLY sending UPD traffic. If this is the case, like Ive stated in previous posts, its a horrible way to deliver data which should be sent complete and intact. There is no arguing this intelligently.

3. Just becuase you have implemented a gigabit network switch doesn't mean that your performance will increase. You shouldn't assume this, though you are correct that you should have the SAME performance.

KapsZ28 said:
Nothing has changed server side, and if you look into using GhostCast there is pretty much no configurations that can be done.

This is an old article, but it shows how Ghost communicates over the network in unicast and multicast.


http://service1.symantec.com/SUPPORT/ghost.nsf/ppfdocs/2002101612025325

Sorry if you think I am bitching, I just need something worth while to bring back to the network team to get them to work with us, rather than just popping in a switch that can't multicast.
Click to expand...
I actually read that same link before posting originally, and it appeared that it used UDP with no negative acks by the ghost server, which again is NOT a good way of tranport but you dont have to listen to me.

I assuming that you're going to be redundant in asking these questions(which is why I left them out) to the network team:

speed duplex settings configured correctly?
No errors on the interface(overruns, drops, giants, etc)
Jumbo frames turned off server/client side(assuming switch is not configured to support them)
As just2cool stated, storm control but it would have to be enabled for broadcast/multicast specifically as you dont see the issue with unicast traffic.

I know im restating what you've already heard but its vanilla from from a network standpoint, you should really open a case with the vendor as well.
 
I'm curious how this works. Multicasting for imaging. This doesn't compute in my brain. There is no verification of the data and even if one system has a problem with the data it can't request the missed data because the server has moved on.

So lets say you must image 10 systems. You would have to visit each system and prepare it for the multicast stream and only once all systems are setup can you then tell the server to start...and pray nothing goes wrong with any of the clients because they will have to be completely redone.

Individual sessions makes much more sense to me. Setup a system, start it. Setup the next system, start it, etc, etc. By the time you get to the end, the first systems should be finishing. Assuming your ghost server is meaty enough it should be able to handle the load. The switch certainly can.

What am I missing?
 
Shadowspawn said:
I'm curious how this works. Multicasting for imaging. This doesn't compute in my brain. There is no verification of the data and even if one system has a problem with the data it can't request the missed data because the server has moved on.

So lets say you must image 10 systems. You would have to visit each system and prepare it for the multicast stream and only once all systems are setup can you then tell the server to start...and pray nothing goes wrong with any of the clients because they will have to be completely redone.

Individual sessions makes much more sense to me. Setup a system, start it. Setup the next system, start it, etc, etc. By the time you get to the end, the first systems should be finishing. Assuming your ghost server is meaty enough it should be able to handle the load. The switch certainly can.

What am I missing?
Click to expand...

For one, you don't have to visit any systems if you are using PXE. However we aren't currently using that. Two, during multicast, a PC can fail imaging and it doesn't affect the other clients. The ghostcast session keeps going. Three, the point in multicasting when imaging is so that the image file is only be read once. This way there is no affect on server performance when compared to unicast. Four, the more unicast sessions you run, the longer and longer it takes. Look at the third screen shot. The times are not consistent. Normall it only takes 3 minutes and 15 seconds to deploy the image I was testing. If you add more and unicast sessions, you we affect server performance especially with disk I/O. Lastly, like I mentioned before, imaging 100 PCs one by one will take a long time. During DR, time is not something with have.
 
Is autonegotiation used or is the speed/duplex configured manually?
What multicast address is Ghost using?
Which MAC addresse(s) are the IGMP membership requests and reports being sent to?
What does the cpu/memory/# of interrupts on the switch look like during multicasting?
Do you see any strange traffic such as ttl=1 packets as another poster mentioned?
Are any non-CEF features enabled on the switch such as policy based routing?

There are a lot of things that can go wrong with multicast..."the switch is correctly configured" is not a reasonable answer.
 
thefreeaccount said:
Is autonegotiation used or is the speed/duplex configured manually?
What multicast address is Ghost using?
Which MAC addresse(s) are the IGMP membership requests and reports being sent to?
What does the cpu/memory/# of interrupts on the switch look like during multicasting?
Do you see any strange traffic such as ttl=1 packets as another poster mentioned?
Are any non-CEF features enabled on the switch such as policy based routing?

There are a lot of things that can go wrong with multicast..."the switch is correctly configured" is not a reasonable answer.
Click to expand...

Yeah, we really need them to troubleshoot with us. So far that hasn't happened. But at least now I have some decent information to go off of. Although I barely understand what the hell you guys are talking about. :)
 
Yeah, we really need them to troubleshoot with us. So far that hasn't happened.
Click to expand...
If you can't get them to do much, have them turn IGMP snooping on the switch completely off. If performance either remains the same or increases, you know that there is an IGMP configuration issue between Ghost and the switch (most likely everything getting sent to the all-hosts address). You may even find that the performance with IGMP snooping off is sufficient and you can just leave it that way.

If performance decreases when IGMP snooping is turned off, you know that there is some kind of general configuration problem with the switch such as network card auto-negotiation not working or a very slow feature turned on.
 
For those that don't remember my original post about replacing our old switch, it is located here.

http://hardforum.com/showthread.php?t=1562917

All we wanted to do was replace our broken 2950 100/MB switch with a gigabit switch. They told us we needed a L3 switch for IP routing between two separate VLANs, but they also spec'd out a $4k gigabit switch with a 10GB uplink. 10GB uplink? :confused: We have an older Dell PowerEdge 2900 server that is primarily for ghosting, but also used outside of out work area around the building to install software and is backed up to our datacenter. Not sure how 10GB would help. Not to mention it was out of our budget. So we ended up with the 3560G which is L3. It is connected through the old switch and router because they said that the copper needs to be replaced with fiber. Although we don't use 1GB connections anywhere in the building. Even with all the newish gigabit switches being installed, they are all hard coded for 100/MB. Once again, another expense we don't need especially since we are planning on using Altiris down the road to deploy images which will not be done through our server or switch.

I am definitely not knocking network guys in general, but the network in our building is bad. We couldn't even ingest .PST files over the network into our Enterprise Vault. This was specifically tracked down to network issues.

When deploying a NAC server in a different location, it would not communicate properly over the network. One of the network guys was nice enough to turn all ACLs off as a test and then everything worked properly. After turning the ACLs back on, it stopped working again. Now this is a Symantec server setup identically to another one in a different location. One works fine, the other won't at all unless the ACLs are removed, but yet the network team won't work with our engineers to reconfigure the switch so the server works.

In our building port security is supposed to be turned on everywhere, and our network team claims it is. I've only ran into port security 3-4 times in the past year.

If you talk to one network person, you get one answer. If you talk to another network person, you get another answer. :rolleyes:

Yeah, now I am just complaining. But in all seriousness, of all the IT groups, everyone complains about working with the Network Team. Not because we think they are dumb, but because they don't work as a team with us. Just like my situation. The network guy said it is configure correctly and left it at that. Didn't even bother doing any testing with me. At a DR event I worked with a different network guy and we were having imaging issues. He checked and monitored the switch until it was configured right.
 
KapsZ28 said:
But in all seriousness, of all the IT groups, everyone complains about working with the Network Team. Not because we think they are dumb, but because they don't work as a team with us. Just like my situation. The network guy said it is configure correctly and left it at that. Didn't even bother doing any testing with me. At a DR event I worked with a different network guy and we were having imaging issues. He checked and monitored the switch until it was configured right.
Click to expand...

If you're not going to call them dumb, let me be the first to do so.

Just because a switch is configured and it magically passes a few frames doesn't mean it's configured correctly; hence your problem. Network gear needs to be optimized/tuned for the environment/role that it is placed in. It really sucks that there are so many bad network engineers out there...

I think you're better off using your Cisco support contract and giving TAC a call. If you tell them the symptoms and give them remote access, they'll go through similar steps we went through here (start at L1, then L2, L3 ...)
 
just2cool said:
I think you're better off using your Cisco support contract and giving TAC a call. If you tell them the symptoms and give them remote access, they'll go through similar steps we went through here (start at L1, then L2, L3 ...)
Click to expand...

Sounds like a good idea, but unfortunately this would need to be done by our Network Team. Too much politics where I work. Hell, I wanted to rebuild one of our servers we use for backups and was told it would need to be done by Windows Engineering.
 
Here is an update.

Email communication with WAN Tech.

From:
Sent: Thursday, February 24, 2011 8:56 AM
To:
Cc:
Subject: Re: Multicast issue with our new switch

We have moved back to the old switch and it performs as normal. The issue is only going through the new switch.

From:
Sent: Thursday, February 24, 2011 08:46 AM
To:
Cc:
Subject: RE: Multicast issue with our new switch

My suggestion is for you to move your connections back to the old switch and see if the problem follows, as the new switch is configured (LAN-wise) the same as the old switch.




From:
Sent: Wednesday, February 23, 2011 7:07 PM
To:
Cc:
Subject: Multicast issue with our new switch


How are you? First I wanted to thank you for setting up our new gigabit switch. It has helped quite a bit with data backup and imaging using unicast. (Takes about 3 minutes to image a PC.) However, as Nelson mentioned last week, we are having an issue with multicast. I attached screen shots of the old switch verse the new switch when doing multicast. As you can see, the new switch is taking about 4 minutes longer when using multicast. This feature is very important to use as we have about 50 computers that need to be imaged and SBL may be replacing about 100 PCs any day now.

I was hoping you could work with us to monitor the switch during multicast imaging to see what may be causing the problem and also have some questions.


Are any L3 settings configured?
Is auto negotiation used or is the speed/duplex configured manually?
Are any non-CEF features enabled on the switch such as policy based routing?
Is Storm Control configured?
Are Jumbo Frames turned off? (This is disabled on the server NIC.)

When monitoring, I would like to look for the following.

What does the CPU, memory, and # of interrupts on the switch look like during multicasting?
Is there any abnormal traffic such as TTL=1 multicast packets?
Are there any errors on the interface, (overruns, drops, giants, etc?)


Please let me know when you will have some time to troubleshoot this issue.
Click to expand...

He then stops by and wants me to move everything back to the old switch test there first and states that everything is LAN based, the two switches are configured the same and either multicast will work or it won't work.

Do you understand why I get frustrated?
 
Also, is it normal for all the ports to blink quickly while doing a multicast session? Even if I am only multicasting to 3 PCs, all the ports that have active connections blink quickly in synchronous.
 
Vito_Corleone said:
Sounds like it's being flooded. Put a sniffer on a port and find out. See if CGMP is on,
Click to expand...

Can a sniffer be an application installed on a laptop that is connected to a port on the switch? If so, what would you suggest I use?

I don't have access to the switch to see the configuration unfortunately.
 
Yea, put wireshark on your PC.

Have one of those idiots look at it. You need to escalate this to someone if you have to. Bitching to people on the internet isn't going to fix your issue.
 
Vito_Corleone said:
Bitching to people on the internet isn't going to fix your issue.
Click to expand...

I know, but I am trying to get information. We are taking the nice path right now and giving the technician the opportunity to make it right. Even providing him with configurations we would like checked. He is supposed to look at it tomorrow. If he gives us the same BS answer that the multicasting just doesn't work, then we go to his manager.
 
Well, the WAN tech should be here in about an hour to troubleshoot. I captured the bandwidth of the switches while imaging.

The first two pictures are of the 2950 in unicast and multicast. As you can see, it is putting out about the same exact bandwidth which is how it is supposed to work. The next two pictures are of the 3560 in unicast and mulicast. The unicast is nice and fast. The multicast puts out a whole 70mbps.

2950Unicast.jpg


2950Multicast.jpg


3560Unicast.jpg


3560Multicast.jpg
 
Hi,

having the same problem. Great unicast performance, bad multicast with Cisco 3560X switches.
According some other forums, jumbo frames should be activated to get much better performance. (Would be my next step...)
Have your problems ever been resolved?
 
You must log in or register to reply here.
Back
Top