FIS Profile Banking System Load Testing

The IT infrastructure of any organization requires some key elements in order to function. The most important system for our client, a large state-owned bank, is FIS Profile.

This system was created by Fidelity Information Services, Inc. (or FIS), an American organization included in the Fortune 500 rundown that offers a wide range of financial products and services.

FIS is headquartered in Jacksonville, Florida, and employs about 55,000 people worldwide. Following the completion of its recent — and by far its largest — acquisition of Worldpay, Inc. for $35 billion in the third quarter of 2019, FIS became the largest processing and payments company in the world.

FIS has a product portfolio for the financial services sector that encompasses both retail and investment banking. This includes Profile, a financial application based on the open-source G.T.M, a transaction processing database engine powered by FIS. This product is the fastest NoSQL solution utilized in many banks around the world. Profile serves millions of operations per minute and has been running 24/7 for decades.

The Profile system is loaded with user actions, batch operations as per banking regulations, and Visa and Mastercard transaction processing. There are at least 11 releases per year for banking application testing. The PFLB team has been conducting performance testing of each of these system releases every year for 7 years, with each load test profile including 50 operations. Each release starts with standard steps:

collecting statistics and compiling a profile;

creating test plans;

constructing a methodology.

In addition, each test run includes a series of standard launches of the required software, hardware resource utilization checks, and the termination of the test (stopping the banking software testing, collecting the required post-test data, etc.).

Plus, it is essential to run performance tests when updates are released or changes are made to products and hardware.

What are the limitations of Fidelity Profile testing?

Profile is no ordinary database. Its history in the United States dates back to about 70 years ago. Profile was mainly used in medical institutions. Therefore, there is little data on the workflow of this system.
Full support for Profile is only offered by FIS, which makes it complicated to exchange information quickly.
Profile monitoring is a challenge in itself. It may take some time for a new user to understand how the system’s tables and records are organized and learn how to control the processes running in Profile.

All these elements are critical to the issue of developing experience and an understanding of the working structure of Profile. Maintaining the acquired competence and increasing the knowledge of Profile is one of the main objectives of PFLB.

Banking software testing is a necessity not only for the client, one of the largest banks in the world, but also for its millions of customers. Without it, the system may not work properly after an update. This could put millions of people at risk, as the bank serves hundreds of government organizations and large corporations. Without the stable and high-quality work of the client’s IT infrastructure, employees could go without being paid or not receive necessary cash payments on time.

Task

Our client pays special attention to the stability of the Profile system, which serves more than 65 million customers, 55 million contracts, 9 million loan agreements, and 45 million cards.

Fidelity Profile is not the most user-friendly system based on the IBM POWER architecture. The project is a large non-relational Database Management System (DBMS), and it consists of many individual executable objects (scripts, procedures, batches, etc.) that are called on-demand. The system works based on the principle of a black box: there is an input and an output. Among other challenges, the following points are worth mentioning:

a programming language that is not very popular (MUMPS);
console interface;
lack of integration with common automation tools and frameworks;
data volume that takes up dozens of terabytes;
functioning of the system that is critical to the client.

PFLB has conducted banking application testing for each of the system releases. On average, a release contains about ten changes to the functionality of the system, each of which requires detailed elaboration and specific testing.

Want to Learn More about Our Performance Testing Services?

Find out what’s included and how to start working with us

As part of this task, our testing team had to take these steps:

analyze operational execution statistics and performance requirements;

automate the load test execution phases;

develop load profiles and outline the list of emulated operations and the intensity of their performance;

develop tools for load testing: scripts, emulators of external systems, test data generators;

perform comprehensive load tests of the Profile system:

define maximum and peak performance;

check tests’ reliability;

compare test results with previous system releases;

evaluate the performance impact of each system change;

analyze the test results and prepare reports for the client.

Solution

We decided to make the testing process more agile by implementing the following:

introducing the banking software testing process into the system development life cycle;

automating the performance testing process to complete tasks on time and meet all the client’s requirements;

developing an approach to collect a profile and create load profiles;

developing tools for load testing.

Once the load testing tools are developed, we assemble the load configuration based on the analysis of statistics of queries served by Profile.

Each time, our team conducts 24-hour testing for system reliability for more than 50 business cases, including account management operations, loans, payroll projects, card processing, and customer data management.

To test all these business cases, the experts at PFLB developed a structure that provides a quick record of changes in the requests caused by system updates. The framework also allows the automation of the connecting of test data.

PFLB engineers developed an emulator to simulate the load caused by card operations using the ISO 8583 protocol. External system emulators have also been developed as a part of the project. They generate an additional load using Java, JDBC, and SOAP technologies.

To emulate the load, we use several tools: Apache JMeter, MicroFocus Performance Center, and unique PFLB products.

At the beginning of each release, we generate data to increase the size of the database. This is done to perform more accurate tests, taking into account the growth and development of the company in the future. For each release, additional objects are created in the database: 160,000 clients, 160,000 accounts, 240,000 loans, and 560,000 cards.

Our current achievements:

our team conducts performance testing of 11 system releases each year;

the load testing is automated;

the system is tested under a load of approximately 6 million operations per hour;

the client receives recommendations on how to optimize the Profile system settings to function with maximum performance after each testing iteration.

Profile automation

The Fidelity Profile system is a large project, and for its load testing, it is essential to automate processes. For this reason, our engineers have automated the following actions:

creating test plans;

preparing tests;

receiving test results from Grafana and Influxdb, collecting post-test data, and creating charts based on these data;

preparation of the test report.

Test plans in this project have been created according to a specific algorithm:

We create a test profile in advance, based on the collected project statistics;

From this profile, we generate a file to transfer the script for making the test plan;

We take the values for XML compilation from the file;

We send XML via Rest-API to MicroFocus Performance Center and create the test plan.

Also, it is important to always review the collected data, because the amount of data needed for the same type of tests is constantly changing, depending on the load profile obtained. Moreover, an incorrect manual calculation can lead to the loss of test results and, more importantly, the amount of time for its implementation and the re-generation of data.

Benefits for the client

The project has enabled the client to implement a process of regular comparative performance testing of different configurations of the system with a real industrial load of about 6 million operations per hour.

Due to the load-testing project, the bank has minimized the risk of failures in a production environment where the system is unavailable and customers cannot perform their operations. Effective stress testing from our company has allowed the client to save money and avoid system downtime.

Due to automation, we have obtained the following results:

Previously, it took three to four hours to create the test plan; now it only takes five to ten minutes.

The automation of testing pools has made it possible not to waste time on testing all the data on the Virtual Table Server and has made the checking process more accurate. For seven years, PFLB staff have been testing the system’s reliability in 24/7 mode, solving problems in the production environment, and helping to detect performance degradation since the release time.

For PFLB, this project is also valuable in terms of the opportunity to develop skills in dealing with a complex and original banking system.