Issue with Windows print server slowness as IPC$ shares increase (with Citrix)

Discussion in 'Operating Systems' started by t_ski, Apr 4, 2019.

  1. t_ski

    t_ski [H]ardness Supreme

    Messages:
    7,491
    Joined:
    Jun 13, 2006
    I've got a weird issue my organization has been battling for a couple years, with slowness on our print server. Here are a couple of background details:

    1. I am not the print admin, but the VMware/storage/backup admin. I don't deal with this directly, but indirectly through my VMware environment.
    2. The print server is running Windows Server 2012 and has been completely stable for about 5-6 years. This is a VM, currently on clustered hosts running ESXi 6.0 (which are usually up to date on VMware patching, hardware firmware and appropriate driver levels to match the firmware revisions). Currently over 850 "printers" with about 500-600 devices (some devices have multiple trays and are presented as different printers).
    3. After implementing a new EHR two years ago, which is remote hosted and presented through Citrix, we immediately saw slowness during our busy time of day. This is primarily 8:00 am to 4:00-4:30 pm, and the slowness does not occur in the evenings or on weekends. We immediately found CPU usage was higher during these periods than the usage had ever been. The vCPU was increased from 2 cores to 4 and the slowness seemed to go away.
    4. About a year in, the slowness gradually came back, and vCPU was increased again from 4 cores to 6.
    5. Now we are in at the 2 year mark, and the slowness is returning. This was just pointed out to me again today, as the VM was migrated off one host to another for host maintenance and the server slowed down.
    6. During our initial troubleshooting, we discovered that the IPC$ share count would peak when the slowness was occurring. Support requests with our EHR vendor and with Microsoft went nowhere. Of course, our EHR vendor states that no other customer has ever had an issue like this.
    7. Citrix farm is on the vendor side: we have no control or access to manage these servers. We have tried multiple revisions of the Citrix Receiver, currently running 4.9.2.
    8. Printers are deployed to end user workstations through group policy, and then these are being redirected through Citrix to the session.
    9. Our last estimate was that we had approximately 1100 concurrent users peak through on the EHR application.
    10. The network is 10Gb on the server side and 1Gb on the client side and IIRC we have a dedicated 40Mb connection to the vendor.
    11. There is some discussion about making all the printers available to Citrix "backend" printers, which are mapped straight to the print devices instead of through the print server. Current jobs that use this method don't seem to be affected by slowness. However, there is no concrete plan to move to this method, not is there a timeline to push forward with this.

    The issue is annoying in the least, especially considering that we never had any issues with printing before this EHR. We can continue to throw more and more hardware resources at this, but that's only a band-aid fix and not a solution. Our virtual infrastructure has been solid enough that typically I do host maintenance during the day as needed, so maybe I'm just spoiled. :D

    If you have questions, just ask. I may have to reference others on my team for the answers.
     
  2. t_ski

    t_ski [H]ardness Supreme

    Messages:
    7,491
    Joined:
    Jun 13, 2006
    Is this thread in the wrong location, or is everyone else stumped, too?
     
  3. schizrade

    schizrade [H]ardness Supreme

    Messages:
    4,683
    Joined:
    Feb 15, 2003
    Hey just seeing this. I have a similar server with 400 "printers" and 4 vcpus but never experienced anything like that.

    It sounds like the introduction of EHR over Citrix started it, and that configuration needs to be scrutinized. Is the EHR host attempting to keep in constant contact with the printers on the host? As in are they all assigned to it as well as your end users?
     
    t_ski likes this.
  4. t_ski

    t_ski [H]ardness Supreme

    Messages:
    7,491
    Joined:
    Jun 13, 2006
    Thanks for the reply. IIRC, all the PCs redirect all of their printers through Citrix, which I think is where the issue lies. Working with Microsoft and the EHR vendor both came up empty-handed. We don't have a Citrix contract, so no direct support there either.

    Whenever the print server migrates, it seems like it's grabbing new connections while the old ones are still open, and I believe the reboot drops all those connections and forces new ones. Reboot has been the only "fix" if the shares get too high.
     
  5. schizrade

    schizrade [H]ardness Supreme

    Messages:
    4,683
    Joined:
    Feb 15, 2003
    Ok so way back in the Presentation Server 3.0/4.0 days I shared excel DBs and a few other things including printers out over a WAN. I recall having those printers be managed by PS, and not letting then touch the local print servers. This was a smaller env than either of us are at now, but I do recall when I first arrived at that place that printing to these specific printers was extremely slow and the work around I came to with the help of the Citrix folks was to remove them from the Print Server and allow PS to manage them. I think the tug of war between the print server and the remote Citrix host is (obviously) causing your issue. Can you push the printers through Citrix instead of GP?
     
    t_ski likes this.
  6. t_ski

    t_ski [H]ardness Supreme

    Messages:
    7,491
    Joined:
    Jun 13, 2006
    Unfortunately, we have no access, and therefore no control, over the Citrix environment. We can make suggestions through SR's, but that's about it. We do have a tech on our team that is slowly learning about Citrix and has about 5-10 users accessing a few apps so far, with plans to grow. I can mention this to him and see if he can look into the details on what we have, then we can discuss with the vendor.

    Thanks!
     
  7. schizrade

    schizrade [H]ardness Supreme

    Messages:
    4,683
    Joined:
    Feb 15, 2003
    Yeah keep the thread updated. It will be interesting to see what solutions you can create.
     
  8. Steve1123

    Steve1123 n00b

    Messages:
    11
    Joined:
    Mar 12, 2016
    This is known issue with Citrix. Please refer to article: https://support.citrix.com/article/CTX213455



    Solution:

    Deleted below registry key.
    HKEY_LOCAL_MACHINE\SOFTWARE\Citrix\Print\DefaultPrnFlags

    Log off and login again.