Hi and welcome to this first part about troubleshooting Citrix XenApp. Last november I had the honor to speak at two well known EUC and Citrix events.
First of all at E2EVC in Rome, Italy and the day after at DCUG TecCon in Kassel, Germany. The topic of both sessions was “Troubleshoot XenApp with Style”.
I am writing this blog for all of you that couldn´t attend one of these great conferences. This first part is about the Citrix tools because otherwise it would grow too big if I would integrate all parts. The second part is about some of the Microsoft tools. The third part is about general troubleshooting tools that didn´t fit in the session.
This post is not about the usual problems you might have with Certificates, Firewalls or similar things where you might be able to find an answer googling around or checking the knowledge base.
It is more about the methodology and the tools to use when you are not able to find anything in the whole world wide web.
You can check two good troubleshooting guides from Citrix regarding NetScaler, StoreFront, XenApp and slow logons following these both links:
Troubleshooting Methodology for NetScaler, StoreFront with XenApp and/or XenDesktop
How to Troubleshoot Slow Logons on XenApp
If you are in charge of your Citrix XenApp environment or if you are a consultant arriving at a customer site for troubleshooting purposes you should ask you and your customer the following questions to get a picture of the problem:
There are a lot more questions you could ask to get more details about what is going on but you should start simple.
Questions should lead you as near as possible to the component that is creating problems.
Because Citrix XenApp is complex in it´s nature because of the dependencies to a lot of other infrastructure components you should go on asking about infrastructure details. Ask questions about (not a complete list):
After you´ve asked all the questions you should be able to get a more precise picture of the problem and after that it is time to check the technological aspects.
Always start simple.
Try to get as much information as you can get but always keep an eye on the amount of data you might get during verbose and debug logging.
Keep the amount of data you need to analyze as low as possible.
You should try to reproduce the problem to minimize data. If it is not reproducable you will have to go through a lot of data. It is really no fun to analyze 15 GB of ProcessMonitor data. now let´s dive into the tools.
The recent Citrix Supportability Toolkit consists of 51 tools that deal with all kinds of problems regarding Citrix infrastructures. I won´t go through all of these here but I will choose some of them that might be usefull during troubleshooting of your XenApp environment and I will start from simple to complex
The Citrix Health Assistant is a very simple tool that you can copy to your VDA and that you run directly. This is how it looks like:
And as soon as you click on Start a VDA registration check it will check the VDA software installation, the domain member ship of your machine, the communication with the required ports, the VDA services, the communication with the DDC and if your VDA is synced in time.
Use Citrix Health Assistant for a first quick check of your VDA installation.
Now you have a first impression if your VDA is missbehaving or not.
This is a more advanced commandline tool that you can run on the VDA and the DDC.
Again you simply copy the binaries to the target system and you´re good to go. You´ll get a lot more information about the specific machine it is run on.
Like:
Use xdping.exe to gather additional information about your VDA or Controller.
The information you get will give you a lot more detail then Citrix Health assistant and it might show you the way to a source of a failure.
Both tools run on the same system might deliver different results. Citrix Health Assistant is created intelligently enough to check the Policies hive in the registry for the ListofDDCs value:
You can check this with ProcessMonitor:
xdping.exe shows the option as not configured and tells us that it is not possible to enumerate the list of DDCs:
You can check this with ProcessMonitor:
This might simply be because of the fact that these tools are written by Citrix support guys and they might have started with xdping.exe as 32-bit tool and never checked it again. Although I would expect this error to be observed by a few customers.
Be carefull… Don´t trust anybody. Even not the tools you use…
Let´s check another tool to gather information of a system.
The next great tool to get an idea of your environment and to start digging into the configured options is HDX Monitor. It can be installed directly on one system via msi or you can run an online install through a browser by visiting: https://cis.citrix.com/hdx/download
After the installation you are able to choose your local system or a remote system for analysis.
You´ll get an iTunes-style star rating of your system with regards to specific topics.
By clicking on the topics you can dive deeper into the settings. For example Network:
Or for example VDA settings:
Use HDX Monitor to get a quick overview over an environment.
Try a little bit for yourself to get used to the things you can check with this handy tool. That´s enough for the VDA. Now let´s see what you can use to troubleshoot Citrix Receiver.
I prefer the Diagnostics Toolkit over How to Enable Logging on Receiver for Windows Using Registry Entries because I had not much luck in my initial tests with Receiver 4.5. The tool is simple to use and only needs to be copied to the VDA or a client with Receiver installed.
Click on Start Tracing and the tool will immediately start collecting data. Now you should reproduce the steps that lead to the problem you have.
You can configure the tool to check for updates and you can set the location were Receiver diagnostic logs are stored as well as the Trace level and the Log size. If you want you can even configure the Event logs (System and Application) and the amount of days that should be collected.
If you stop the trace you will find a folder with the Eventlogs, some CDF traces and a PackingList.csv for further analysis.
The lazy one´s can upload the files to Citrix Inside Services (CIS), were the collected files will be analyzed automatically. In my tests the feedback I got from CIS left room for improvements. Check the logfiles on your own to find specific hints for problems you might have.
You can dig deeper into the logfiles by using tools like the Configuration Manager Trace Log Tool that you can download here. It will show you very quickly in which lines to look at by marking lines with warnings in yellow and errors in red.
If you have suggestions for the tool let the creators know. Only by providing feedback they will have the chance to improve it.
Now let´s have a look at CDF Control that you can use to check the above created traces or to create traces for specific Citrix modules.
CDFControl is an event tracing controller/consumer, geared towards capturing Citrix Diagnostic Facility (CDF) trace messages that are output from the various Citrix tracing providers.
This is an advanced analytics and debugging tool with a lot of options. If you are experienced with troubleshooting tools it will be self explanatory after a few tries.
Before you start to trace have a look at Recommendations for Collecting the CDF Traces
After the trace you are able to filter for Modules, Functions or for example Errors. The next screenshot set to trace Deliver Controller Services shows us very quick that the database server is unavailable. This should tell you to call the database guys and tell your boss you´re not guilty 
CDF Control is an advanced troubleshooting tool with a lot of potential to resolve errors.
If you need to create CDF control traces during startup check the following support article:
How to Collect a Citrix Diagnostic Facility (CDF) Trace at System Startup
Citrix Scout doesn´t seem complex from a user´s point of view but it will gather tons of data that might lead you to the root of a problem.
Scout is there on your Delivery Controllers. Use it!
You can add 10 machines (if you haven´t changed it) for data gathering and that makes it perfect to gather data about machines that might behave differently although configured and installed identically.
After you added your machines to trace click on CONTINUE and start the trace.
You are now in charge to reproduce the problem and to keep the amount of data low or you can leave it running and wait for the problem to happen. This depends on the problem you are trying to resolve.
Options you can configure are:
You´ll get an overview of your site and of every machine you entered for the trace.
And you´ll get a folder with a lot of information…
With a lot of logs for Citrix services…
And a lot of Registry entries to look for…
Although this tool creates a lot of data it is perfect to check for differences through machines in your deployment.
The Citrix Call Home feature – you remember that thing that you can enable during installation – if enabled uploads periodically information about your environment to CIS.
After logging in with your Citrix account you get an overview over your uploaded reports and information of your Site. Diagnostic reports will show you isssues you have and will give you hints on how to resolve them.
Although it might not be usefull to report issues like only one DDC in my site (because it is a homelab and small) and additionally lacking the hint of only one StoreFront server or an unclustered Site database it gives you an idea were this is going in the future.
In the lower left corner you will see additional information regarding your environment.
We have checked the VDAs, Receivers and Controllers and the last thing that you should check for errors is StoreFront.
The first check leads us to the Eventlog:
If you are not able to see errors you should enable verbose logging. This is done via PowerShell:
If you look at the Trace Folder it looks this way during normal operation:
After enabling verbose logging it will contain additional logs:
Don´t try to open the logs with Notepad because it will not show you data that is readable. Use DebugView from Sysinternals instead. It will look more friendly.
You should disable verbose logging when you´re done:
Set-DSTraceLevel –All –TraceLevel Off
And this is it for the first part of the Troubleshooting tools for XenApp series. In the next part we will have a look at Microsoft tools you can use. I hope you found it usefull.
Cheers,
Sinisa
Bleiben Sie mit uns in Kontakt und finden Sie heraus, was gerade jetzt passiert.
