Experience testing Wi-Fi service for multimedia broadcasts during air travel

Customer satisfaction with our services largely accounts for the success of the company. And nothing is trivial in this industry. Knowing this, the S7 aviation group contacted Performance Lab with a request to conduct load testing of BoardConnect, a service designed for Wi-Fi broadcasting of audio and video content to airplane passengers.

A similar service was used by another airline, Germany's Lufthansa, which had previously tested BoardConnect. However, S7 decided to double-check these results. Our customer's management was well aware that if passengers experienced problems with the service, the carrier's image would suffer.

Performance Lab launched a project to load test BoardConnect in early spring 2019. Our team worked on it throughout March.

We faced the task of checking the stability of the Wi-Fi-based media broadcast service. And this was a real challenge, because we had not previously load tested hardware, only software.

We had not previously load tested hardware

Got a project in mind?

There is no better place for a QA solution than Performance Lab.
Drop us a line to find out what our team can do for you.

Request a quote

How were we supposed to do this?

By simulating the behavior of 50 independent passengers whose mobile devices can connect to the tested service after the plane's ascent. Their devices should provide them with a choice of various audio and video content: movies, TV series, music, audio books, and cartoons.

How did we do it?

We used laptops to perform the task. The problem was in scaling up: it was impossible to emulate the work of more than seven mobile devices from one personal computer. As a result, we had to set up an entire "fleet". Each laptop used six Wi-Fi adapters: one built-in and six USB (two connected directly, four connected via a USB hub). In this way, we followed the principle "One adapter, one user".

To facilitate the operation of connected adapters, we created virtual machines (VMs) running Ubuntu. Here we applied the rule "One adapter, one virtual machine". Each VM was configured to work with a single Wi-Fi adapter, and all the VMs were combined into a local network.

We analyzed traffic in Fiddler 4. We wrote scripts, debugged, and tested using JMeter 5.0, including in remote testing mode. Finally, we collected and analyzed data using InfluxDB and Grafana.

Now let's walk through the stages of the project and consider individual aspects of each.

Fine-tuning the workbench. Working with the Wi-Fi network

BoardConnect connects at 2.4 GHz (channels 1 and 5) and 5 GHz (channels 36 and 40). In our case, channels 1 and 36 should have been used in equal measures.

However, the planned twenty-five 2.4-GHz connections failed. This was probably due to the fact that many other nearby Wi-Fi networks were operating at the same frequency. We worked around this problem by creating 34 connections at 5 GHz. We managed to create the remaining 16 connections at 2.4 GHz.

To view data about the Wi-Fi networks in real time, we used the program Acrylic Wi-Fi Scanner.

Script creation

Four scripts (use cases) were created to emulate user actions in the BoardConnect service:

01

UC01 - view random videos (movies, cartoons, TV series);

02

UC02 - listen to random audio (music, audiobooks);

03

UC03 - read books (open pdf files);

04

UC04 - view transfer information.

At the customer's request, we used the following testing profile: 35 simultaneous users performing UC01, 10 simultaneous users performing UC02, three simultaneous users performing UC03, and two simultaneous users performing UC04. The use cases were implemented in four separate JMeter files.

Additionally, traffic analysis showed that, in addition to downloading content, short messages are constantly exchanged between the client and server. Some of them represent flight information, while others maintain the connection. To emulate this mechanism, each script created a separate thread consisting of HTTP GET requests.

The third and fourth use cases were implemented as a sequence of HTTP GET requests. The first and second use cases, in addition to GET requests, implemented a media stream emulator using the DASH protocol.

Virtual machine image

Virtual machines (VMs) were created using Oracle VM VirtualBox. We chose Ubuntu 16.04 (32-bit) as the operating system for the VMs, since it supports Java 8, which is required to run JMeter Server. In addition, this version of Ubuntu is easy to configure and has low CPU requirements.

In addition to the operating system, the VM image contained JMeter 5.0.

Creation and configuration of a network of virtual machines

The test bench consisted of ten laptops connected to a switch by patch cords. Moreover, one computer (we'll call it the "master") was used to start and monitor the load, while the rest ("hosts") were used to create a load on the service under test. The virtual machines (VMs) were started on the hosts. As mentioned above, each VM used a single Wi-Fi adapter.

The master, hosts, and VMs were joined to a local network. To set up this network, we assigned IP addresses in the form of 192.168.2.110 with a subnet mask of 255.255.255.0 for the master's and hosts' network adapters.

To connect a VM to this network, we created a network bridge to the RJ45 network adapter for each machine's host. After the network was started, the IP address in the Ubuntu network connection settings was changed so that the VM had an address similar to the host's, but differing in the values of the unit digit. For example, if the IP address of the host was 192.168.2.110, then the VM on this host had 192.168.2.111, 192.168.2.112, and so on.

To provide Wi-Fi connectivity to the service, a network bridge was created to one of the Wi-Fi adapters for each VM. Thus, the VMs had access to two adapters: the first — for connecting to the local network, the second — for interacting with BoardConnect.

As a result, we achieved a network consisting of 50 VMs and one master.

Using JMeter in remote testing mode

After we created a network of VMs, we were able to start load testing using JMeter in remote testing mode. This mode involves one control machine (master) and several servers, to which the JMX file and launch command are sent.

01

The RMI and SSL protocols were used to connect the master and servers. This made it necessary to copy the rmi_keystore.jks file, which contains the required SSL certificate, to the "Bin" folder. This file is generated by running create-rmikeystore.bat.

02

Before applying the load, we still had to solve a few more issues. To allow the master to find all the servers, we had to add their IP addresses to the remote_hosts property in the jmeter.properties file: remote_hosts= 192.168.2.111, 192.168.2.112 (and so on).

03

Then we launched jmeter-server.bat from the "Bin" folder on all servers, with an indication of the VM's IP address: jmeter-server.bat - Djava.rmi.server.hostname=192.168.2.111

04

The final step was to launch JMeter on the master: jmeter - Djava.rmi.server.hostname=192.168.2.40 -n -t seven_UC01.jmx -r

Here:

  • -Djava.rmi.server.hostname indicates the IP address of the master;
  • -n starts JMeter in no-GUI mode;
  • -t file.jmx indicates which test file should be run;
  • -r tells JMeter that remote testing must be started on the servers indicated in the jmeter.properties file in the remote_hosts parameter.

However, in our case, we used -R {list of IP addresses} in order to ignore the settings in jmeter.properties and instead run testing on the VMs whose addresses we indicated directly in the run command.

We had four JMX files: one for each of our use case scripts. This made it possible to satisfy the testing profile and at the same time support simple adjustments and script launching.

Accordingly, on the master, we ran the following four lines in different command prompts from different copies of the JMeter folder:

jmeter -Djava.rmi.server.hostname = 192.168.2.40 -n -t
seven_UC01.jmx -R {IP addresses of 35 virtual machines}
jmeter -Djava.rmi.server.hostname = 192.168.2.40 -n -t
seven_UC02.jmx -R {IP addresses of 10 virtual machines}
jmeter -Djava.rmi.server.hostname = 192.168.2.40 -n -t
seven_UC03.jmx -R {IP addresses of 3 virtual machines}
jmeter -Djava.rmi.server.hostname = 192.168.2.40 -n -t
seven_UC04.jmx -R {IP addresses of 2 virtual machines}

An Excel spreadsheet was used to compile and adjust the long IP addresses. The final line was formed using concatenation, with the result being used as the value of the -R option.

Data collection and analysis

For this, we used a classic bundle: JMeter + InfluxDB + Grafana.

As mentioned above, we used four JMX files, each a separate script running in remote testing mode on predefined virtual machines.

Given this implementation, we had to create two Backend Listeners in each JMeter script when configuring data writing to the InfluxDB database. The first was equal to the value of the "Application" field ("General"). The second was equal to the name of the use case.

Generalized data for all scripts was written to "General", while data for each specific use case was written to the "Application" equal to the name of the use case.

This approach has at least two advantages:

experience testing Wi-Fi service

the ability to monitor the number of active threads for each use case;

experience testing Wi-Fi service

during real-time analysis of the execution of a use case, the volume of data sent to Grafana is much less.

In all scripts, the InfluxDB server's IP address was identical: it was the master laptop, which did not run any load scripts.

We imported a dashboard as the basis for configuring data monitoring and analysis (Dashboard #5496 Apache JMeter Dashboard using Core InfluxdbBackendListenerClient from Philippe M. Updated on 2018-09-18 17:20:33). However, we changed its basic panels significantly.

For example, for quick analysis of the total load created by each use case, panels at the top of the dashboard were configured to monitor the number of transactions and errors, traffic, and average data transfer rate, and to display the number of active users and data received and sent.

Rapid switching between data sets was carried out by selecting the desired value of the "Application" variable at the top of the dashboard.

Using the Backend Listener made it possible to configure the "Active Threads" panel for each use case, which allowed us to select the number of active users.

At the bottom of the dashboard, we placed panels that recorded the volume of transferred data and response time for each transaction. To display this last metric, we used both the original panel and a heatmap panel, which allows us to track the change in a value over time.

What did the project provide to the customer?

The testing conducted by our team showed that BoardConnect is unable to stably provide multimedia content over Wi-fi to 50 plane passengers.

Our results did not agree with the results of the Lufthansa study, since it involved emulation of fewer devices. In other words, we were able to identify a previously undetected problem and report it to the customer. S7 decided to send the device for reworking in order to avoid reputational harm.

What did the project provide to our company?

Our team gained valuable experience in stress testing hardware. And now we are prepared for the challenges that may arise in such projects.

Download
brochures

More information about QA solutions we provide is available in our brochures

PL CSV SOLUTION
TEST AUTOMATION SERVICES
CASE STUDY: TOP 10 BANK AT
PERFORMANCE TESTING BROCHURE
AGILE PERFORMANCE TESTING
IVR LOAD TESTING SERVICES
CASE STUDY TOP 10 BANK LT
CASE STUDY RETAIL
CASE STUDY GOVERNMENT
QA OUTSOURCING WHITEPAPER

Latest posts from us

Unit Testing vs Integration Testing
X Reasons to Start Website Testing
Things You Need to Know Before Starting E-commerce Website Testing
Software Test Life Cycle (STLC) Importance
What is User Acceptance Testing (UAT) for Websites? Definition & Examples