Load testing is necessary when deploying any new system or redistributing data streams among existing systems. I think everybody knows this or at least would guess this is the case.
Everything seems simple: grab your load testing tool, record some traffic, get some scripts, run your tests, and enjoy the process. At least that’s the theory…
Approaches to loading
Actually, in most cases engineers are faced with testing information systems that have a classic three-tiered architecture:
The standard approach to performance testing is to emulate user traffic from clients to the back end. This works, but only if the end client applications use standard protocols such as HTTP/HTTPS, SOAP, ODBC/JDBC, SAP GUI, and a few others.
But what do you do if the exchange between client and server uses a closed protocol or, even worse, a proprietary protocol, and the traffic cannot be recorded using standard tools?
You can try to figure out the protocol, decompile the application, reverse engineer it, and develop a plug-in or sampler based on the data received. But the likelihood of success is extremely small. And if you also consider the time required and translate that to money, then the entire undertaking is very difficult to justify.
In other words, the task is far from trivial and not always solvable. In fact, this is akin to hacking. It’s like a TV show where the players being told the story before being asked what’s in the black box. But you may not even have access to the story (read: source code for the application’s client end).
Of course, you can use various traffic analyzers (Wireshark, TcpDump, and others), but the volume of traffic may be so large that you’ll need months to analyze it!
In our experience, it is not uncommon situation when a customer starts to think about load testing only one week before a scheduled release.
Of course, you still have to write a wrapper so it works correctly with the chosen loading tool.
Well, we can only salute those who choose this path and succeed. Your results will be truly grand if you can overcome all of the “stumbling blocks”, but we in general we do not recommend this approach.
What can be done if you aren’t willing to expend so many resources on such lengthy investigations?
At this point, an experienced load tester will, of course, say, “Hey, guys, try CITRIX!”
Using Citrix virtualization technologies
One way to handle this task is to use Citrix virtualization products, which may it possible to gain remote access to an application on almost any device. What’s at the heart of this approach?
The client application that end users interact with is published on a dedicated Citrix server (farm).
Unlike the first approach where the loading tool is “inserted” into the client-server link and parametrized traffic is used, here our tool sends commands to copies of the client application using the ICA protocol.
To tell the truth, we actually manage the applications through a terminal server.
This lets us emulate not the behavior of the client application through replaying traffic, but rather the behavior of a large number of users of the client application. It’s no secret that Citrix has been used in performance testing long enough to emulate user behavior on a client application. But by no means can every loading tool work with this protocol.
The most popular loading tool for working with the Citrix protocol is HP’s LoadRunner.
It’s popularity is a result of the fact that there is no other choice. Looking at JMeter, we see nothing, empty space, anguish, pain, and somebody’s forgotten broken crutches on an overgrown path through a forest that leads who knows where.
Let’s returning to Load Runner. This product recommends itself as massive and functional. However, for all of its pluses there are minuses.
The most obvious is the product’s cost and complexity when expanding functionality.
Anyway, we’ve chosen a tool, so everything ought to work out, but now we count up the cost of the licenses. In current market conditions, when converting dollar prices to rubles, the cost of licenses for testing may exceed the value of the solution being tested.
What’s more, the closest analog, Apache JMeter, is entirely free and easy to extend, thanks to open source code and a simple, understandable API for writing plugins.
So what am I driving at? Yes, Load Runner is good, but our customers’ desire for a more budget-friendly tool prompted us to develop our own JMeter plugin to link with CITRIX, OCR, and synchronization.
Our plugin work produced a highly versatile and easy-to-use tool for working with Citrix, which almost entirely covers HP LoadRunner’s functionality when working with this protocol, and in some instances surpasses it. But the most important advantage for our customers is the fact that it is free.
Differences from LoadRunner-based solutions
If you’ve ever written terminal scripts for LR, then you are familiar with the agony of debugging, which requires the synchronization of every little thing, every image, and where being off by a couple of pixels can break everything. Tedious, slow, unreliable, devilishly torturous – these are the epithets you want to cry out during coding and debugging. The pain and suffering literally permeate you when you have to support this model.
Knowing this, when developing the plugin to work with Citrix we placed great emphasis on achieving the following benefits:
- flexibility and easy of use similar to LoadRunner;
- automation mechanisms to save time when implementing certain difficult algorithms directly in load scenarios;
- integrated optical character recognition (OCR), which gives the virtual user the ability to respond to any text that appears on the user’s screen and, as a result, makes it possible to handle more complex test scenarios.
The most important difference is that it is free of charge for our customers. Using JMeter instead of LoadRunner makes considerable savings possible, especially during lengthy testing, because using HP LoadRunner requires the purchase of licenses whose cost is based on the number of virtual users and the duration of testing.
- Record user actions (keyboard input, mouse clicks and movement) with the ability to choose the required input sources.
- Playback recorded scripts.
- Start sessions using *.ica files.
- Start sessions with selected parameters (size of the Citrix workspace, color depth).
- Specify whether to save screenshots made during recording and playback.
- Parametrize any command/parameter in samplers.
- Specify a multiplier for all ThinkTimes.
- Start sessions without blocking user input.
- Configure extensive synchronization settings.
- Specify a (theoretically) unlimited number of checksums for synchronization.
- Recognize text with the integrated OCR (Optical Character Recognition) engine.
- Configure whether the tool starts a new session with each iteration.
- Enjoy Remote Support.
And now we’ll briefly explain how it all works.
Anybody not directly associated with load testing can skip this section in order to avoid brain trauma.
What you need to get started
You’ll need the following in order for the plugin to work properly:
- JMeter for the plugin to run;
- The com4j library for working with COM (required by the plugin; you can download it here);
- JRE x86 v7+ for COM4J and the plugin to work properly (compiling with JRE v6 is also possible);
- Citrix Receiver, used for recording and playback.
Installing the plugin
I won’t describe how to install JMeter and JRE. I assume everybody is already familiar with these procedures. Installing Citrix Receiver is also straightforward, so we’ll get right to installation of the plugin. To install the plugin itself, simply copy the plugin’s .jar files and com4j library files to the lib folder in JMeter’s root directory. But to be able to record and playback you also need to add a registry entry (the path given is for Windows x64; for 32-bit versions, remove the Wow6432Node subdirectory from the path):
This text can be saved in a text file with the .reg extension and then added to the registry by executing the file.
To record scripts, you must add the Citrix ICA Recorder element (located in the Non-Test Elements category) to WorkBench and there indicate the correct connection parameters:
- ICA file location – The path to the .ica file for initializing the session. If an ICA file is specified, the remaining connection parameters will be ignored. You can use the GUI to choose the file by clicking Browse;
- Host – Address of the server;
- Port – Connection port. If no port is specified, the standard port will be used;
- User Name – The user name of a user authorized to access the published application;
- User Password – The user’s password;
- Domain – The user’s domain. Optional;
- Application – The name of the published application;
The recording parameters must also be specified:
- Save screenshots – If enabled, then screenshots saved during recording will be saved;
- Screenshot folder – The path to the folder where the screenshots will be saved. The Browse lets you use the GUI to choose the path;
- Resolution (WxH) – Indicates the dimensions of the application’s Citrix workspace (maximum window size). Specified as x, separated by a lowercase Latin “x”;
- Color 24 bit – Indicates which color depth to use;
- Record: – Indicates which events to record and which to not record. Possible values: Keyboard – Record keyboard events, Mouse Clicks – Record mouse clicks, Mouse Move – Record movements of the mouse cursor;
- Default ICA Config – Indicates which ICA Config Element to use by default in recorded samplers (more details about this later);
- Select controller – Indicates which controller to put the recorded samplers in;
Recording a session
Once all of the settings have been entered, you can connect to the a session and start recording. To start/stop recording, use the Recording controls button group in the recorder. The Connect button simply connects to the session. This does not start recording. You can start recording later. The Record button connects and immediately starts recording the selected actions. The Stop button stops recording and disconnects from the session. The session itself (Citrix Receiver window) is not closed. Moreover, when the session is exited (for example, by closing the application) recording is stopped automatically based on the LogOut event.
The Step controls button group is for managing steps. It becomes available once recording has begun. The Add Step button saves the current step and adds it to the selected controller. Additionally, the recording begins to be received by a new sampler. The Restart Step button simply clears all actions from the current step (helpful if you make a mistake). The Start Tag button inserts a tag so that all commands recorded until the End Tag is pressed will be ignored and instead the text entered by the user will be inserted. Accordingly, the End Tag button inserts a tag that terminates the effect of the Start Tag command. The Synchronize button creates a screenshot of the Citrix session’s entire workspace and brings up a dialog with synchronization settings, which I would like to discuss in more detail.
There are two 2 synchronization options available at present: screenshot-based synchronization and text-based synchronization (using OCR).
To synchronize using screenshots, select the synchronization area (red semitransparent rectangle) and specify the right parameters.
- Wait sync – Indicates whether to wait until an area with the indicated hash is found and how long (at most) to wait. The second parameter , try every, indicates how often to check. The parameters are specified in milliseconds using whole numbers.
- Delta x, y – Indicates the maximum delta along the x and y axes, respectively, to search for the area. This is helpful if the element used for synchronization may shift slightly. But it is utterly useless if the element changes its dimensions, because the screenshot’s checksum will be different.
- Set sampler error if failed – Indicates whether to make a note of a sampler that fails to find the specified area within the specified time. If this is not enabled, then the sampler will considered always successful and the only way to discern whether the area appeared is to check the value of the variable in the following item.
- Write result in variable named – Indicates the name of a variable used to record whether synchronization is successful. The value of the variable will be true if the area is found. Otherwise, it will be false.
This type of synchronization involves comparing checksums of the specified screenshot areas during recording and playback. As a result, changing the color of a single pixel during playback will change the screenshot’s checksum, which will cause the comparison to fail and the area will be considered missing.
For text-based synchronization, you also must choose the area of the screen in which text will be recognized. You also need to specify two parameters: OCR config handle – A string indicated in the OCR Config test element in the OCR Handle field (more about this later), used to select the customized OCR engine to be used for text recognition; and OCR result variable – The name of a variable used to record the result. If nothing is recognized, the variable will be created and initialized with an empty string.
Preparing for playback
Once the script has been recorded, you should have a test plan with a controller and Citrix ICA Samplers inside. At least one Citrix ICA Config element is required for playback.
A Citrix ICA config looks very similar to a Citrix ICA Recorder. It has exactly the same connection parameters, so there’s no point in describing them here. I’ll only mention that all fields in the connection configuration can be parametrized, and these values will be used when initializing the session, i.e. when executing the first Citrix ICA Sampler encountered. For example, don’t be afraid to use CSV Data Set Config to parametrize fields. The Citrix Config Handle field must be unique for all such configs. The samplers use it to determine which connection settings to use, and to connect to the session to continue the test.
The settings to playback sessions differ from those in the recorder. I’ll point out the differences:
- New session each iteration – If a value is specified, a new session will be created for each iteration. The old session, if it is not closed, will remain active but will not do anything.
- Replay mode – If a value is indicated, then all input to the started sessions will be blocked. Otherwise, the user can use the mouse/keyboard to enter data into the open application. Most likely needed for debugging.
- Output mode – The window display mode. If Normal, then the window will be displayed on the screen. If Windowless, then the window will not be displayed, which saves resources on load benches (not working in the current version; currently the Receiver ignores this setting).
- Use sleep times – If a value is indicated, all recorded or manually entered delays will be replayed. You can also indicate a multiplier used to increase wait times (only for SleepTime commands).
- Unlock all sessions and Lock all sessions – These parameters respectively unblock and block input for all active sessions, i.e. changes the Replay mode parameter to “nope”. They only works with sessions started on an active JMeter. They do not work for sessions started on remote servers.
If text recognition actions were recorded during recording, then an OCR Config element must also be added. It is quite simple and has only two parameters:
- OCR handle – This parameter differs from Citrix ICA Handle only in that it identifies an OCR Config rather than a Citrix ICA Config. It must be unique for all OCR Configs.
- Table – The left column indicates the path to an image used for training. The right column indicates the range of symbols in the image (specified as – without the angled brackets and separated by a hyphen).
The training images are images with sequentially printed symbols (in a line or in a table). The range of codes for the specified symbols must be uninterrupted and in increasing order. A single range should be indicated for each image.
Description of samplers
In addition to the standard names and comments, the Citrix ICA Sampler has two fields. First, Citrix Config Handle indicates which settings to use to start a session or the pool from which to take an existing active session. The second field contains a list of all the commands it will execute. This field may be fully parametrized using JMeter variables.
To playback a recorded and parametrized script, simply run the test. After each sampler has run, you can view the result of each command in Response Data.
If an .ica file is used to initialize sessions, it’s best to parametrize the field that corresponds to it. Because the file is single-use, the second stream (iteration) cannot be started using the same file. To solve this problem, you can get these files immediately before starting a session, e.g. using a web protocol.
If a sampler is unable to gain control of a session within 120 seconds, it will attempt to logout and disconnect the session for another 60 seconds. Then another attempt is connect will be made automatically.
If in rare instances an error occurs during screenshot-based synchronization, don’t worry. Try adding sampler the checksum obtained during playback (you can view the checksum on the Response Data tab in the View Results Tree, if the error occurred during synchronization).
If one of several actions is required based on what is displayed on the screen, you can use synchronization without a timeout and conditional controllers in the cycle in order to execute the desired action.
When using Citrix Receiver v4.4+, you cannot start a session with “direct” connection parameters. Only through .ica files.
Advantages of using Citrix during performance testing
Using Citrix technologies in performance testing has its advantages over traditional approaches.
During this type of testing, interaction with the system takes place through the client application. Data sent from the client to the server are not emulated, but rather the user’s actions on the client. As a result, there is no need to analyze the details of the client’s interaction with the system. And this increases information security, makes it possible to test systems with an extremely complex/closed protocol, and saves time on development of test scenarios. This approach also provides the ability to assess the client application’s perfomance (for example, the time require to render information, the time required to write data to disk, etc.). What’s more, you can test the scalability of the Citrix farm itself.
The main deficiency is that to achieve universality you have to pay for load bench capacity and a Citrix farm. This technology works well when you have hundreds or thousands of users, but when there are tens of thousands or hundreds of thousands — alas, you have to think of something on the protocol level. (We’ll tell you about this another time.)
Choosing a performance testing tool is crucial when creating a testing strategy. The individual specifics of the system being tested, the protocols being used, and the load on the system are highly important.
If a testing project involves Citrix, then we can boldly state that we can perform the testing using our in-house tools just as well as HP LoadRunner, which can yield significant savings on testing costs.
This extension is a truly “fresh” solution, which was developed at the end of 2015 but has already proven itself well in performance testing at a major bank.
A comparison with other solutions revealed no material disadvantages, so if the difference is trivial, why pay more?
About the author
Alex Makarov is an IT expert with more than 15 years of experience. Alex is the Head of load and performance testing department at Performance Lab, a global company specializing in software testing and quality assurance. His main focus is to ensure that complex information systems perform well and to deploy new technological solutions in the area of software testing.