How to speed up JMeter. Part 1

Apache JMeter load testing doesn’t need to be advertised; nonetheless, not enough attention is paid to the load script speed. In this article, we take a look at different approaches to writing load scripts in an optimal way that allows saving money on the load testing machines.

This article will be useful for load testing engineers and Apache JMeter users (beginners and advanced), as well as for developers using JVM/JDK, profiling and code optimization.

After reading the first part of the article, we suggest you enjoy the second part.

More about Apache JMeter

Apache JMeter is a popular load testing tool that provides a wide range of components and settings. An operation can be executed in several ways during load testing using JMeter.

Load script performance comes to light in projects with high loads. We need a system of measurement for the Apache JMeter component performance and approaches to load script writing that optimize the load script performance.

We’ve used Java app profiling tools such as Java Flight Recorder, jVisualVM, and SJK for this Apache JMeter load testing tutorial. We also had access to the tool’s source code. Additionally, we have written synthetic tests and analyzed project experience.

Here are the most important facts about Apache JMeter:

  • It’s one of the best known and most used tools on the market. It has been widely used since 2003.
  • There are plugins for more than 50 systems, formats, and result representation modes.
  • It saves statistics in ClickHouse, InfluxDB, or Graphite. Many statistics collection services display the statistics in Grafana or in an HTML report.
  • It’s easy to introduce JMeter into CI/CD systems.
  • It is well integrated with different development tools.

Load agent

Whether you run the tests locally or in the cloud, for a quick run you’ll need to prepare a station and configure JMeter for load testing. The preparation process consists of several important steps, which we will discuss in our Apache JMeter load testing tutorial.

Agent monitoring

Telegraf, InfluxDB, Grafana

Grafana website for the agent monitoring
Dashboard 1902

To begin, you need to configure the agent monitoring. If you haven’t done this yet and have only monitored the target load system, there is a simple solution for Windows OS. Go to the Grafana’s site, dashboard 1902. There is a Telegraf tutorial on how to configure metrics in the dashboard’s documentation.

Telegraf, InfluxDB, Grafana

Grafana website for the agent monitoring
Dashboard 2381

Board 2381 suits Linux well because it allows the creation of handy graphics for quick analysis of load station problems.

Network in Windows

If you work in Microsoft Windows, don’t forget to configure the network subsystem. A Windows network subsystem is configured by default in such a way that your system can’t pass a virus or a worm to another computer. The restriction takes place on both client and server platforms. There are three simple parameters that must be adjusted to lift the restrictions and configure JMeter for load testing. They can be found in the register …\Services\Tcpip\Parameters for the network subsystem.

…\Services\Tcpip\Parameters

Key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Value
MaxUserPort
Data Type
REG_DWORD
Recommended
65534
Default
5000
Range
5000 to 65534

The first parameter is called MaxUserPort. It increases the port range that can be accessed from your IP address. Each Internet network connection is defined by four numbers: the sender’s IP address (your address), the sender’s port (your port), the receiver’s IP address, and the receiver’s port. The maximal number of connections is limited by the range of ports that can be opened from one station. It is within the range of 1024 to 5000 by default. We need to set this value to its maximum so that more connections are allowed. As a result, the performance will be increased for Apache JMeter load testing.

…\Services\Tcpip\Parameters

Key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Value
TcpTimedWaitDelay
Data Type
REG_DWORD
Recommended
30
Default
240
Range
30 to 300

The second parameter that influences the performance is TcpTimedWaitDelay. It allows faster closing of half-opened connections.

…\Services\Tcpip\Parameters

Key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
Value
TcpFinWait2Delay
Data Type
REG_DWORD
Recommended
30
Default
120
Range
330–294,967,295

The third parameter is TcpFinWait2Delay. It allows the freeing of opened connections that are no longer required. As a result, your system’s performance on Windows will double after applying the settings above.

Network in Linux

Please pay attention to the first rule about changing the settings: we change them one by one to configure JMeter for load testing. If changing one of them has not helped to achieve your goal, we revert the parameter value.

  • If you work in Linux Ubuntu, then the first parameter to consider is tcp_tw_reuse. Its default value is not limited. We recommend setting it to 1 for load stations. Afterwards, your station will reestablish a new connection 3.5 times faster.
  • In RedHat Enterprise Linux, the parameter is called tcp_tw_recycle. We recommend setting it to 1, which will increase the speed of connection establishment performance by 3.5 times.
  • Let’s increase the port range on Linux not in the register but in files from 25 to 65 thousand addresses. Setting the parameter ip_local_port_range to 1025 65535 increases the speed of the connection establishment performance by up to 1.5 times.

Gatling tutorial

Gatling tutorial for the network in Linux

If you’d like to read some valuable advice, it’s worth checking out the Gatling tutorial. It explains how to configure everything, including the network connections and the operational system’s limitations.

Limitations on Linux

All is a file

The description of the limitations on Linux

It’s important to keep in mind that in the Linux OS everything is a file. If you have established a network connection and opened a socket, then you have used a disk. You may reach the maximal number of opened files and processes. We will need to optimize them. This is the first step in preparing JMeter tests.

JMeter settings

The file JMeter Properties or User Properties is with JMeter configuration files in the bin folder. You need to pay attention to two parameters that will help you to solve some problems and optimize JMeter load testing. The first parameter is httpclient.reset_state_ …; the second is httpsampler.max_bytes_ ….

JMeter 5.1 and older
HTTP Request with default settings closes connections after every iteration of the script. Connections don’t have time to open again on short fast tests.

JMeter 5.2 and newer
New settings in Thread Group “Same user on each iteration.” If this option is set, connections are not closed after each iteration.

JMeter, in particular HTTP Request, closes the connections after each script iteration by default. Consequently, the connections do not have enough time to reopen on short, quick tests.

Generally, JMeter runs a test iteration by default in the following way: after the connection has been established, the script makes requests and closes the connection. In the next loop, the OS sends the request to establish the connection again. In this way, you are also testing the load balancer. The parameter “Same user on each iteration” changes that behavior. It’s available in JMeter versions 5.4 and newer in the standard Thread Group.

The parameter httpclient.reset_state_ …

Your station with one IP address and a limited port range may not sustain the desired load. The connections may not have enough time to open. The simple solution is to leave open the connection between two iterations. This action can be applied if it’s not necessary to test the load balancer with respect to connection establishment performance.

After you’ve created a thread, a connection has been opened. If you have set the parameter to False (httpclient.reset_state_on_thread_group_iteration=false), then you will have a correspondence of 1000 threads to 1000 connections. Your test performance will improve by 3.9 times, and load testing using JMeter will become faster.

httpclient.reset_state_ …
If the intensity changes

Results of setting the JMeter parameter

The intensity function on the synthetic tests fluctuated and had a mean of 4.84 thousand.

httpclient.reset_state_ …
Speeds up a test up to 3.9 times

Results of setting the JMeter parameter

After changing the parameter, the result was 19 thousand, and it was less fluctuated. If you don’t close the connections, a graph will be generated that is not fluctuated, has high intensity, and has no errors.

If there are no HTTP errors, neither the HTTP response code ‘java.net.NoRouteToHostException’ nor the HTTP response message ‘Address not available’ will be returned.

JMeter load testing, in particular HTTP Request, requires a great deal of memory by default (the buffer is four times bigger than the file itself). It follows that you can’t download 1100 MB with a heap of 4 GB while generating objects with 1.8 GB/sec. JMeter downloads the complete response body, even when the post-processors won’t use it. So, you don’t necessarily need the 4 GB of heap.

The parameter httpsampler.max_bytes_

This property defines the number of bytes that will be used by post-processors. It’s not limited by default. Don’t store the entire response when downloading megabytes. We can limit it to a reasonable size (e.g., 555 000 bytes (httpsampler.max_bytes_to_store_per_request=555000)). This is enough for the post-processor. It will find everything needed without wasting the memory. This change will allow you to achieve significant results in speeding up Apache JMeter load testing.

httpsampler.max_bytes_ …
If there wasn’t enough memory

The error in the JMeter code

Firstly, the “OutOfMemoryError: Java heap space” error will be remedied. Secondly, you’ll speed up the response download time owing to the shorter response body.

httpsampler.max_bytes_ …
Speed will be increased by 3 times

The duration of the download 1 GB 200 times (seconds)

Results of setting the JMeter parameter

httpsampler.max_bytes_ …
Memory will be decreased by 22 times

Memory used when downloading 1 GB 200 times (Megabytes)

Results of setting the JMeter parameter

The download speed of 1 GB increases up to 3 times, and the memory utilization decreases by up to 22 times on a synthetic test. In other words, you decrease the buffer and JMeter wastes 180 MB. If you don’t, it wastes 4 GB while downloading 1 GB.

JMeter components

You can combine certain JMeter components and component configurations to achieve a better performance. They should suit each other and be correctly configured for load testing using JMeter.

Samplers

We don’t recommend the BeanShell sampler and its components, opting instead for JSR-223 with Groovy and cache.

HTTP Request and Body Data
File Path and Request Body

Parameters of the JMeter Samplers

Some users create a large XML variable and insert it into Body Data HTTP Request to send a large request with a big request body. This takes a huge amount of memory and slows down processes.

HTTP Request and Body Data
File Path and Request Body

Parameters of the JMeter Samplers

Instead, we recommend indicating a static request body loaded from the file. This approach doesn’t load the data into the memory and allows reading from file even a request body with a size of 300 MB. To use this approach, fill in File Path in File Upload, leave other parameters empty (Parameter Name and MIME Type), and don’t activate the Use multipart/form data check-box. Your JMeter won’t fire the OutOfMemory error.

HTTP Request and Resources
load an average of a third

Results of the JMeter Samplers

Sampler JMeter doesn’t recognize links to statistics well; it recognizes about 33% of the links on an average site. At this stage, we recommend performing optimization and configuring JMeter for load testing.

All sites use images from other domains, such as Google, and Facebook, as well as different kinds of metrics. If you don’t have enough resources in your load station networks, we recommend economizing on the requests to search engines and social networks. You need to remove them from the list of loaded resources and keep only the CSS that addresses your host. It’s the most evident and quickest fix possible.

HTTP Request and Resources
with domain filtering

Parameters of the JMeter Samplers

You can use a regular expression that chooses the host list from the URLs Must Match parameter, which can be found in the “Embedded Resources from Html File” list. Here is an example of the regular expression:

\/\/((${host1})|(${host2}))\/.*

This way you won’t overload either JMeter or the broadband, and you will save resources and provide a higher load capacity on the tested server.

HTTP Request and Resources
with domain filtering

Domain filtering of the JMeter Samplers

Logic Controllers

Transaction Controller
with the Generate Parent Sample

Parameters of the JMeter Logic Controllers

It’s convenient when the requests are grouped to transactions and are displayed as the View Results Tree. This is handy for debugging, but you must deactivate this functionality before starting a test.

Transaction Controller
with the Generate Parent Sample

Parameters of the JMeter Logic Controllers

If you don’t deactivate this functionality, JMeter will save everything in the memory before sending all saved results with the request bodies and statistics to the next recipient. For example, if a transaction contains 10 requests, all of them are saved in the memory and only afterwards discarded. However, the number of requests can be 5 times as big. Each thread will save multiple responses and discard them slower, which will slow down the test itself.

Transaction Controller
without the Generate Parent Sample

Parameters of the JMeter Logic Controllers

Therefore, we recommend deactivating the Generate Parent Sample in the Transaction Controller before starting Apache JMeter load testing.

Listeners

File with Save as XML is excess
CSV format is enough

Parameters of the JMeter Listeners

It used to be good practice to use Save as XML and save all response bodies when the server responded with an error. However, we don’t recommend doing so (by ticking the checkbox) because reviewing errors in the CSV format is sufficient.

Assertions

We recommend using assertions to configure JMeter for load testing. If you send requests and don’t check the responses, then your load test is faulty. Another reason to implement them is if those assertions don’t weigh much.

Use Assertions
but not with Continue

Parameters of the JMeter Assertions

All thread groups continue test execution after an error by default. This way, although you have set an assertion and it has fired, the test execution continues and your results are mixed with the errors from the other 10 requests. As a result, you receive a log file with 50 errors but need only one of them to find a functionality bug. It’s possible that you’ll waste time solving this problem.

Use Assertions
with Start Next Thread Loop

Parameters of the JMeter Assertions

The better option is to activate the Start Next Thread Loop checkbox when using assertions. If there is an error, the test will go to the next iteration. This way you will avoid unnecessary data during JMeter load testing.

Post-Processors

Boundary and RegExp are fast

Max RPS in a text

Results of the post-processors

More – better

Let’s take a look at the post-processors available for HTML and their work intensity when they are used on their own. Boundary and RegExp turned out to be the fastest, then JSR223, BeanShell, and CSS; XPath was the slowest. Therefore, we recommend removing XPath from your services.

XPath with Use Tidy for HTML

Parameters of the post-processors

The XPath post-processor works slowly because it has to use a tolerant parser called Use Tidy, which transforms HTML to valid XML, on which XPath is then applied. Unfortunately, Use Tidy works extremely slowly: it reserves a great deal of memory, loads HTML into it, and then cleans it all. We don’t recommend it for load testing using JMeter.

Miscellaneous Features

Debug Sampler at the start

Parameters of the Miscellaneous Features

There is a Debug Sampler under Miscellaneous Features in the JMeter documentation. It is used while debugging JMeter scripts and is sometimes kept in the test during the test run. The problem is that the Debug Sampler throws into the response body all variable values and generally everything that is kept in the memory by default. So, if your test is highly intensive and you are using the Debug Sampler, the test will slow down significantly.

Disable the Debug Sampler

Parameters of the Miscellaneous Features

In this case, don’t forget about the shortcut CTRL+T to select all Debug Samplers and disable them while debugging.

Summary

In the first part of our Apache JMeter load testing tutorial, we looked at how to increase the speed of the service using its load agent, settings, and components.

First of all, we advise you to install agent monitoring. In order to do this, follow these steps:

  1. Set up monitoring of your configuration agent. This will allow you to make convenient graphs for quick analysis of problems with the load station.
  2. For Windows OS, apply network subsystem settings that will as much as double the performance speed:
  • MaxUserPort increases the range of ports that are allowed from a single IP address, which increases performance;
  • TcpTimedWaitDelay allows you to close semi-open connections more quickly;
  • TcpFinWait2Delay makes it faster to release connections that are open and no longer need to be reused.
  1. For Linux, refer to the following settings, depending on the system type:
  • For Linux Ubuntu, set tcp_tw_reuse to 1 for load stations, and your station will be 3.5 times faster in restoring a new connection.
  • For RedHat Enterprise Linux, set tcp_tw_recycle to 1 to increase the speed of opening connections by up to 3.5 times.
  • ip_local_port_range 1025 65535 increases the speed of opening connections by up to 1.5 times.
  1. Linux OS limits: optimize the maximum number of open files or processes.

Furthermore, you should refer to the JMeter settings, the use of which will help you to avoid important errors and increase the speed of the system.

  1. httpclient.reset_state_ … Use this setting so that connections are not closed even between two iterations. As a result, you will get a smooth graph with high intensity and zero errors.
  1. httpsampler.max_bytes_ … This setting determines how many bytes from the beginning of the response can be left for post-processors to use. Limiting the bytes will reduce the frequency of the lack of memory error (“OutOfMemoryError: Java heap space error”), and also speed up the response download time.

In addition, optimize the JMeter components:

  1. Samplers.
  • Choose JSR-223 with Groovy and cache.
  • We recommend indicating a static request body loaded from the file. Select all Debug samplers and set them to Disable mode. Due to this, JMeter won’t fire the OutOfMemory error.
  • We recommend economizing on the requests to search engines and social networks. You need to remove them from the list of loaded resources and keep only the CSS that addresses your host. This way, you will overload neither JMeter nor the broadband, and you will save resources and provide a higher load on the tested server.
  1. Logic Controllers
  • We recommend deactivating the Generate Parent Sample in Transaction Controller before stating a test.
  1. Listeners
  • Reviewing errors in the CSV format is sufficient.
  1. Assertions
  • The better option is to activate the tick “Start Next Thread Loop” when using assertions. This way you will protect your results from any other errors.
  1. Post-Processors
  • We recommend using the fastest postprocessors, Boundary and RegExp, as well as removing the slowest post processor, XPath.
  1. Miscellaneous Features
  • Disable all Debug Samplers before debugging.

The report is based on materials from the PERFORMANCE CONF conference. Author: Smirnov Vyachelav.

Got a project in mind?

There is no better place for a QA solution than Performance Lab.
Drop us a line to find out what our team can do for you.

Latest posts from us

Company Growth and Strategic Development within COVID-19 Pandemic. Interview with Performance lab Founder & CEO, Yurii Kovalov
Tasked With Staying Ahead of a Global Pandemic, Performance Lab Founder & CEO, Yurii Kovalov Talks About Growth and Strategic Development of the Firm
Contact center and IVR performance testing intro
Contact center and IVR performance testing
Performance Lab Receives Top Industry Award from Clutch
Performance Lab Receives Top Industry Award from Clutch
Performance Lab is among Top QA and Software Testing companies
Performance Lab is among Top QA and Software Testing companies​
Speed up a WordPress Website
How to Speed Up a WordPress Website – Part 2