Hi and welcome to this second part about troubleshooting Citrix XenApp.
Last november I had the honor to speak at two well known EUC and Citrix events. First of all at E2EVC in Rome, Italy and the day after at DCUG TecCon in Kassel, Germany. The topic of both sessions was “Troubleshoot XenApp with Style”.
I am writing this blog for all of you that couldn´t attend one of these great conferences. The third part will cover general troubleshooting tools that didn´t fit in the session and part 1 and 2 of the blog series.
The first part is about the Citrix tools you can use.
This post is not about the usual problems you might have with Certificates, Firewalls or similar things where you might be able to find an answer googling around or checking the knowledge base.
It is more about the methodology and the tools to use when you are not able to find anything in the whole world wide web. Check the first part to get an idea of what questions to ask and how to start the troubleshooting process.
Let´s have a look at some useful tools from Microsoft you can (and should) use when troubleshooting Citrix XenApp issues. Some of them need to be used in combination with the Citrix troubleshooting tools because otherwise you won´t have useful results or you might not be able to read and analyze the information, others can be used themselves. Always keep in mind the more information you get from different sources the better you will understand the issue.
Let´s go…
The Services Trace Viewer is a very handy tool if you need to analyze Citrix StoreFront verbose logs. The general recommendation for the creation of StoreFront verbose logs can be found here. Typically you would zip the files and send the logs to Citrix support if you wouldn´t troubleshoot on your own. But because we are trying to get the issue on our own we need SvcTraceViewer.exe.
In order to get the tool you need to install the Windows 10 SDK. The download link for that is https://developer.microsoft.com/en-US/windows/downloads/windows-10-sdk.
The Trace Viewer itself can be found after installation at this location:
C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.2 Tools
Now you are able to read the logs and you can try to find the issue on your own.
If you need to dig deeper you can open Sysinternals DebugView during tracing to get more information. As I said this is typically asked from Citrix Support to get a better picture of the problem. The moment you start DebugView it will collect debug output from the system it is running on.
Use SVCTraceViewer to check StoreFront Verbose log files.
Another simple but very effective tool ist the Steps Recorder. It comes preinstalled in every Windows version and can be found in the start menu. I prefer to use it every time a user needs to explain or clarify an issue. Often users are not able to express perfectly what the problem is and steps recorder gives you the opportunity to get an idea of the problem they might have.
Let the user open Steps Recorder.
By hitting “Start Record” every single click a user does is recorded by a screenshot and the date and time it happens along with the application name the user is acting with. This is perfect to match this data to eventlogs and similar things you might look into in order to resolve an issue.
When the user finishes he simply hits “Stop Record” and he´s able to add some extra comments. In the end he can send it to you for further assistance.
Use Steps Recorder for situations where a user is not able to explain the issue or if you want additional information about the environment.
This is the second most used tool in troubleshooting scenarios. It is the perfect tool to monitor and watch all of the processes an a Windows system or for a specific user or program.
You can see very clearly what kind of child processes are being created and if you enable the columns with User Name, Process ID and the corresponding command line you have a good overview of what is going on in a session or during interaction with the system.
If you like you can replace the Task Manager an your system with Process Explorer. I really like this feature for my homelab servers or systems in development environments.
If you want to know more about a process you can show additional information like Performance, Performance Graph, GPU Graph, Threads or the Enironment variables. The environment variables tab is sometimes helpfull when troubleshooting scripts that set and use variables and you are unsure if they have the correct value.
Replace Task Manager with Process Explorer to get more information about running processes on your systems.
If you want to replace Task Manager with Process Explorer there is a very simple and effective way using PowerShell:
New-Item -path ‘HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\taskmgr.exe’
Set-ItemProperty -path ‘HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\taskmgr.exe’ -name “Debugger” -value “””C:\Program Files\Tools\SysinternalsSuite\procexp.exe”” /ACCEPTEULA”
Process Monitor is one of the most famous (if not the most most most famous) tools out there. This is really the tool were you should be able to track down most of the issues you can have in a XenApp environment.
You can filter for registry activity, file activity, network activity and process and thread activity.
What I personally like is the ability to show the “relative time” column. Especially when you are analyzing slow logons it shows you the time from the beginning of the trace. That makes it easy to keep track of the time a process needs to run.
Additonally you can show columns like Architecture, Comand Line and User and Session.
If you are searching for files and registry keys that you need to tweak with your UEM solution, this is the tool to go because you will be able to find the files, registry keys and entries you need to set for a specific application.
The following example shows you the creation of a shortcut through SelfService.exe from Citrix Receiver.
You should be aware of the fact that ProcMon can use a lot of system resources when it is running and sometimes, when you are trying to analyze timing issues during logon you will see that they are gone during analysis because of Procmon slowing down the system.
This tool can have a massive impact on the performance of the system you are starting it. Therefore don´t let it run without having a look into it, and always have a look at the ammount of data you are gathering.
ProcMon is creating a lot of data…really a lot!
If you need to check communication from a host to specific systems and ports Netstat is the simple tool you can start from the commandline.
If you also want to know which Process is creating connections from your host to another and the specific port it is using try TCPview from the Sysinternals Suite.
TCPview will show you the process behind a connection.
I know that this is a no-brainer but it is simple and very effective to get some information from the system you are logging on. You can specify and extend it with custom PowerShell scripts. The next picture shows you an example where I added Fields for the PowerShell, Citrix Receiver, App-V and Internet Explorer Version.
The integrated PowerShell scripts simply gather the version information from the registry.
If you are out of budget or need a very simple solution to get the information about the users that are logging on the a system you can create a textfile through BGinfo that documents these logons.
The logfile creates entries with a variety of information you can import to Excel to create filters and search for specific users. This is an additional source to gather information to use in combination with other troubleshooting sources.
BGinfo – Simple but often effective.
If you want to start BGinfo in every session on a Workstation or Server there is a very simple and effective way using PowerShell:
Set-ItemProperty -path ‘HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Run’ -name “BgInfo” -value “””C:\Program Files\Tools\SysinternalsSuite\Bginfo.exe”” “”C:\Program Files\Tools\SysinternalsSuite\demolab.bgi”” /TIMER:0 /NOLICPROMPT”
The command tells BGinfo to load a custom bgi file that I created before. The files need to be present on the system. I wouldn´t recommend placing the files on a fileshare because this might have impact on your logon time.
Also take care of the fact that this BGinfo background can grow large depending on the size of the screen and a potential background image you might be using. Such a file can easily grow 3 MB and more and will reside in the user profile.
This tool is part of the System Center Configuration Manager Tools. If you work with Configuration Manager you know how challanging log files can be (Configuration Manager has more then 70 different logs). The cool thing about CMTrace is that it is highlighting lines with errors in red, lines with warnings in yellow. This makes checking logfiles a much more simpler task, because you are quicker reading and searching through them.
You can download the System Center 2012 R2 Configuration Manager Toolkit here.
Use CMTrace to quickly find errors and warnings in log files.
Group Policies or better said the configuration of those often create problems we need to check. In big companies usually more then one team is responsible for Group Policies and other teams might screw your environment without even knowing or noticing it.
In such cases it might be suitable to take a look at the Group Policies that affect a user or a system. The cool thing about rsop.msc is that it will show you the configured settings and the Group Policy name it is comming from.
The user himself can start rsop.msc (if he is able to start a cmd or PowerShell console). This way you could have a quick look shadowing a user and getting some insights into his configuration.
As administrator you get a lot more more information.
This will help you finding the Group Policy that is changing a specific configured setting as last. But this won´t tell you how many times a setting might have changed before back and forth. This might be something you should additionally consider having a look at. Many policies create many problems
rsop.msc shows you a configured setting and the name of the policy it belongs to and it saved my ass more than once.
Sometimes you need to dive deeper into the logon process of a user. This is the moment you should consider enabling UserEnvDegugLogging on a system.
Be aware of the fact that using ProcessMonitor in combination with UserEnvDebugLogging can have a huge impact on the performance of a system and on the log file UserEnvDebugLogging is creating.
Also be aware of the fact that it won´t solve all of your problems. I had situations were I tried to get a picture of a slow logon but I couldn´t find any information regarding the cause of the issue.
In such a case it makes sense to have a look at Process Monitor that will show you more information.
Enable logging by setting the above Registry Key. Logging will immediately start at the following location…
If we look into it we will see a lot of information in it. I won´t explain it in detail because there are other very good articles out there.
Do not use UserEnvDebugLogging with Process Monitor at the same time.
This is it for the second part about troubleshooting Citrix XenApp. I hope you found something useful for your next issue.
Always remember – it needs a lot of practice and experience to find some of the issues you might run into and you need to decide if it makes sense in means of the time to find the issue and the impact it might have on your business if you are contacting Citrix directly or not.
Cheers,
Sinisa
Stay in touch with us and find out what is going on right now.
