t_ski
Supreme [H]ardness
- Joined
- Jun 13, 2006
- Messages
- 7,503
I've got a weird issue my organization has been battling for a couple years, with slowness on our print server. Here are a couple of background details:
1. I am not the print admin, but the VMware/storage/backup admin. I don't deal with this directly, but indirectly through my VMware environment.
2. The print server is running Windows Server 2012 and has been completely stable for about 5-6 years. This is a VM, currently on clustered hosts running ESXi 6.0 (which are usually up to date on VMware patching, hardware firmware and appropriate driver levels to match the firmware revisions). Currently over 850 "printers" with about 500-600 devices (some devices have multiple trays and are presented as different printers).
3. After implementing a new EHR two years ago, which is remote hosted and presented through Citrix, we immediately saw slowness during our busy time of day. This is primarily 8:00 am to 4:00-4:30 pm, and the slowness does not occur in the evenings or on weekends. We immediately found CPU usage was higher during these periods than the usage had ever been. The vCPU was increased from 2 cores to 4 and the slowness seemed to go away.
4. About a year in, the slowness gradually came back, and vCPU was increased again from 4 cores to 6.
5. Now we are in at the 2 year mark, and the slowness is returning. This was just pointed out to me again today, as the VM was migrated off one host to another for host maintenance and the server slowed down.
6. During our initial troubleshooting, we discovered that the IPC$ share count would peak when the slowness was occurring. Support requests with our EHR vendor and with Microsoft went nowhere. Of course, our EHR vendor states that no other customer has ever had an issue like this.
7. Citrix farm is on the vendor side: we have no control or access to manage these servers. We have tried multiple revisions of the Citrix Receiver, currently running 4.9.2.
8. Printers are deployed to end user workstations through group policy, and then these are being redirected through Citrix to the session.
9. Our last estimate was that we had approximately 1100 concurrent users peak through on the EHR application.
10. The network is 10Gb on the server side and 1Gb on the client side and IIRC we have a dedicated 40Mb connection to the vendor.
11. There is some discussion about making all the printers available to Citrix "backend" printers, which are mapped straight to the print devices instead of through the print server. Current jobs that use this method don't seem to be affected by slowness. However, there is no concrete plan to move to this method, not is there a timeline to push forward with this.
The issue is annoying in the least, especially considering that we never had any issues with printing before this EHR. We can continue to throw more and more hardware resources at this, but that's only a band-aid fix and not a solution. Our virtual infrastructure has been solid enough that typically I do host maintenance during the day as needed, so maybe I'm just spoiled.
If you have questions, just ask. I may have to reference others on my team for the answers.
1. I am not the print admin, but the VMware/storage/backup admin. I don't deal with this directly, but indirectly through my VMware environment.
2. The print server is running Windows Server 2012 and has been completely stable for about 5-6 years. This is a VM, currently on clustered hosts running ESXi 6.0 (which are usually up to date on VMware patching, hardware firmware and appropriate driver levels to match the firmware revisions). Currently over 850 "printers" with about 500-600 devices (some devices have multiple trays and are presented as different printers).
3. After implementing a new EHR two years ago, which is remote hosted and presented through Citrix, we immediately saw slowness during our busy time of day. This is primarily 8:00 am to 4:00-4:30 pm, and the slowness does not occur in the evenings or on weekends. We immediately found CPU usage was higher during these periods than the usage had ever been. The vCPU was increased from 2 cores to 4 and the slowness seemed to go away.
4. About a year in, the slowness gradually came back, and vCPU was increased again from 4 cores to 6.
5. Now we are in at the 2 year mark, and the slowness is returning. This was just pointed out to me again today, as the VM was migrated off one host to another for host maintenance and the server slowed down.
6. During our initial troubleshooting, we discovered that the IPC$ share count would peak when the slowness was occurring. Support requests with our EHR vendor and with Microsoft went nowhere. Of course, our EHR vendor states that no other customer has ever had an issue like this.
7. Citrix farm is on the vendor side: we have no control or access to manage these servers. We have tried multiple revisions of the Citrix Receiver, currently running 4.9.2.
8. Printers are deployed to end user workstations through group policy, and then these are being redirected through Citrix to the session.
9. Our last estimate was that we had approximately 1100 concurrent users peak through on the EHR application.
10. The network is 10Gb on the server side and 1Gb on the client side and IIRC we have a dedicated 40Mb connection to the vendor.
11. There is some discussion about making all the printers available to Citrix "backend" printers, which are mapped straight to the print devices instead of through the print server. Current jobs that use this method don't seem to be affected by slowness. However, there is no concrete plan to move to this method, not is there a timeline to push forward with this.
The issue is annoying in the least, especially considering that we never had any issues with printing before this EHR. We can continue to throw more and more hardware resources at this, but that's only a band-aid fix and not a solution. Our virtual infrastructure has been solid enough that typically I do host maintenance during the day as needed, so maybe I'm just spoiled.
If you have questions, just ask. I may have to reference others on my team for the answers.