FIS Profile Banking System Load Testing
The IT infrastructure of any organization has key elements that are required for it to function. The most important system for our client, a large state-owned bank, is FIS Profile.
This system was created by Fidelity Information Services, Inc. (or FIS), an American organization included in the Fortune 500 rundown, which offers a wide range of financial products and services.
FIS is headquartered in Jacksonville, Florida, and employs about 55,000 people worldwide. Following the completion of its recent — and by far its largest — acquisition of Worldpay, Ink. for $35 billion in the third quarter of 2019, FIS has become the largest processing and payments company in the world.
FIS has a product portfolio for the financial services sector that spans both retail and investment banking.
This includes “Profile” — a financial application based on the open-source GT.M, a transaction processing database engine powered by FIS. This product is the fastest NoSQL solution and is utilized in many banks around the world. Profile serves millions of operations per minute and has been running 24/7 for decades.
The Profile system is loaded with user actions, batch operations as per banking regulations, and Visa and Mastercard card transaction processing. There are at least 11 releases per year for banking application testing. Meanwhile, the load test profile includes 50 operations. Each release starts with standard operations:
- collecting statistics and compiling a profile;
- creating test plans;
- constructing a methodology.
In addition, each test run includes a series of standard launches of the required
software, hardware resource utilization checks, and the termination of the test
(stopping the banking software testing, collecting the required post-test data, etc.).
It is essential to run performance tests when updates are released or changes are made to products and hardware.
What are the limitations of Fidelity Profile testing?
- Profile is no ordinary database. Its history in the United States dates back to about 70 years ago. Profile was mainly used in medical institutions. Therefore, there are not many data on the workflow of this system.
- Full support for Profile is only offered by FIS, which makes it complicated to exchange information quickly.
- Profile monitoring is a challenge in itself. It may take some time for a new user to understand how the system’s tables and records are organized. It is necessary to develop extensive expertise to understand these complicated data fields and learn how to control the processes running in Profile.
All these elements are critical to the issue of developing experience and an understanding of the working structure of Profile. Maintaining the acquired competence and increasing the knowledge of Profile is one of the main objectives of Performance Lab.
Banking software testing is a necessity not only for the client, one of the largest banks, but also for its millions of customers. If it is absent, the system may not work properly after an update. This could put millions of people at risk. It is important to remember that the bank serves hundreds of government organizations and large corporations. Without the stable and high-quality work of the client’s IT infrastructure, employees could go without being paid and not receive necessary cash payments on time.
Our client pays special attention to the stability of the Profile system, which serves more than 65 million customers, 55 million contracts, 9 million loan agreements, and 45 million cards.
Fidelity Profile is not the most user-friendly system based on the IBM POWER architecture. The project is a large non-relational Database Management System (DBMS), and it consists of many individual executable objects (scripts, procedures, batches, etc.) that are called on-demand. The system works based on the principle of a black box: there is an input and an output. Among other challenges, the following are worth mentioning:
- A programming language that is not very popular (MUMPS);
- A console interface;
- A lack of integration with common automation tools and frameworks;
- A data volume that takes up dozens of terabytes;
- A functioning of the system that is critical to the client.
Performance Lab conducts banking application testing for each of the system releases. On average, a release contains about ten changes to the functionality of the system, each of which requires detailed elaboration and specific testing.
As part of this task, our testing team performs the following steps:
- analyzes operational execution statistics and performance requirements;
- automates the load test execution phases;
- develops load profiles and outlines the list of emulated operations and the intensity of their performance;
- develops tools for load testing: scripts, emulators of external systems, test data generators;
- performs comprehensive stress tests of the Profile system:
- defines maximum and peak performance;
- tests reliability;
- compares test results with previous system releases;
- evaluates the performance impact of each system change.
- analyzes the test results and prepares reports for the client.
As part of the project, we have to implement the following:
- introduce the banking software testing process into the system development life cycle;
- automate the performance testing process to complete tasks on time and meet all the client’s requirements;
- create an approach to collect a profile and create load profiles;
- develop tools for load testing.
Once the load testing tools are developed, the load configuration, based on the
analysis of statistics of queries served by Profile, is assembled.
Our team conducts 24-hour testing for system reliability and more than 50 business cases: account management operations, loans, payroll projects, card processing, and customer data management.
To test all these business cases, the experts at Performance Lab develop a structure that provides a quick record of changes in the requests caused by system updates. The framework also allows the automation of the connecting of test data.
Performance Lab engineers developed an emulator to simulate the load caused by card operations using the ISO 8583 protocol. External system emulators were also developed as a part of the project. They generate an additional load using Java, JDBC, and SOAP technologies.
To emulate the load, we use several tools: Apache JMeter, MicroFocus Performance Center, and unique Performance Lab products.
At the beginning of each release, data are generated to increase the size of the
database. This is done to perform more accurate tests, taking into account the
growth and development of the company in the future. For each release, additional objects are created in the database: 160,000 clients, 160,000 accounts, 240,000 loans, and 560,000 cards.
The consequences are as follows:
- our team conducts the performance testing of 11 system releases each year;
- the load testing is automated;
- the system is tested under a load of approximately 6 million operations per hour;
- the client receives recommendations on how to optimize the Profile system settings to work with maximum performance.
The Fidelity Profile system is a large project, and for its load testing, it is essential to automate processes. For this reason, our engineers have automated the following actions:
- the process of creating test plans;
- the process of preparing tests;
- the process of receiving test results from Grafana and Influxdb, collecting post-test data, and creating charts based on these data;
- the preparation of the test report.
Test plans are created according to a specific algorithm:
- We create a test profile in advance, based on the collected project statistics;
- From this profile, we generate a file to transfer the script for making the test plan;
- From the file described above, we take the values for XML compilation;
- We send XML via Rest-API to MicroFocus Performance Center and create the test plan.
Also, it is important to review the collected data. This is because the amount of data needed for the same type of tests is constantly changing, depending on the load profile obtained. Moreover, an incorrect manual calculation can lead to the loss of a test result and, more importantly, the time of its implementation and the re-generation of data.
Benefits for the client
The project enables the client to implement a process of the regular comparative performance testing of different configurations of the system with a real industrial load of about 6 million operations per hour.
Thanks to the stress-testing project, the bank minimizes the risk of failures and
incidents in a production environment when the system is unavailable and customers cannot perform their operations. Effective stress testing from our company allows the client to save huge amounts of money and avoid system downtime.
Thanks to automation, we have obtained the following results:
Previously, it took 3-4 hours to create the test plan, but now it only takes 5-10 minutes.
The automation of testing pools has made it possible not to waste time on testing all the data on the Virtual Table Server and has made the checking process more accurate. For seven years, the Performance Lab staff has been testing the system’s reliability in 24/7 mode, solving problems in the production environment, and helping to detect performance degradation since the release time.
For Performance Lab, this project is also valuable in terms of the opportunity to develop skills in dealing with a complex and original banking system.
Have a project in mind?
There is no better place for a QA solution than Performance Lab.Drop us a line to find out what our team can do for you.