There are aspects of software testing that often plant confusion in those new to the process — such as drawing the line between stability and reliability testing. The two are often used interchangeably and share a common goal of ensuring a system can perform steadily over a chosen time frame. In this post, we’ll take… Read more…
When launching a product, the project team needs to ensure users can install it effortlessly. That’s why it’s crucial not to miss out on tasks like installation testing. These tests check that the installation process is bug-free, regardless of the device or operating system that runs it. This post will offer business owners, QA specialists,… Read more…
Performance testing is one of the most challenging components of web application testing. Irrelevant web performance test data or wrong metrics, chosen during testing, cost companies a ton of money and effort. If done right, performance testing has all the tools needed to forecast product behavior and its response to user’s actions with an impressive… Read more…
Automated Testing Strategy: How to Build & Examples
Automation is a crucial part of all modern development processes. Be it Waterfall, Agile, or any other methodology used, they all play a part in continuous development, integration, testing, and more. For developers, this means 10, 20, 50 releases per year. Pushing out so many updates and builds with no automation would be a pain in the neck.
Automated testing is a way out for all companies who are already using or are looking forward to implementing Agile into their workflows. On top of that, the quality of testing and the number of detected errors is higher when the workflow is automated.
How to approach automated testing? What should a developer do to create an efficient ecosystem for running scripts? In this post, we’ll touch on automated strategies in detail.
By definition, automated testing is the process of conducting a set of tests that run on their own. Unlike manual tests, reliant on the workforce, the automated ones depend on tools for management and reporting.
This approach to testing has become extremely popular as companies start to transition towards DevOps. Opposite to a traditional development environment, companies have started performing testing not at the end of the cycle but early on.
In order to be able to run dozens of small tests simultaneously without creating a bottleneck for developers and stalling the process, QA professionals have picked up on automated testing. Essentially, it allows them to:
Ensure a steady coverage of tests all the way down the development pipeline;
Create a set of tests that could be repeatedly re-used;
Deploy tests 24/7 and speed up product delivery to the market;
Reduce the cost of end-to-end testing by catching bugs early on;
Create an environment for parallel testing.
Automated Testing Strategy
In general, an automated testing strategy includes the overall approach to testing and reporting. All the processes are outlined and planned along with the expectations for the testing cycle.
When working on an automation strategy for testing sessions, testers will start from outlining the following points:
Test environment. A good testing environment should, first and foremost, be fully controlled and provide a developer with the freedom to modify all the necessary data. There are cases when running a test requires several environments to work in tandem – in this case, a QA professional has to ensure both systems are compatible with one another.
Testing approach. This is the part where a tester defines the scope of the test, creates a list of activities, and assigns team members to oversee each cluster. A standard automated testing pipeline would normally look as follows: Initiation, Automation Test Planning, Execution, Reporting, Close-out.
Testing tools. A tester has to choose and outline the range of used tools as well as their scope.
While the structure of the strategy is highly depending on the structure of the company, the number of people involved, and the methodology a business has adopted, the number of steps and actions to take remains fairly steady.
Here’s an example of automated testing strategy you can use for executing scripts.
Step 1. Choose cases for automated testing
As much as developers would want automated testing from A to Z, there are bottlenecks that can only be assessed manually to the fullest of their extent. So, before figuring out the logistics of automated testing, you will have to decide which cases will be up for automation.
If your test case matches one of the following categories, you can be confident about automating it:
Runs on multiple configurations. In case you are testing an application using different operating systems and target processors, writing automated scripts will come in handy. Testing each configuration manually will take a ton of time and stall the development process.
Requires a large data input. If a case requires a large data set, consider automating it. Moreover, in order to not collect data manually, testers normally use data generation tools.
Takes a long period of time to perform. In case a test needs to be supervised continuously for more than 10–12 hours, it’s smart to consider automation. A script can be executed on its own without any human supervision – all a tester should do is to analyze the report generated by the testing tool after the execution of the script.
Performance tests. In order to get meaningful insights from a performance test, a manager would have to ask thousands of employees to access a website or an app simultaneously. Logistically, it would be a nightmare – nor is it likely that your company has thousands of employees to spare for such mundane tasks. That’s why all aspects of performance testing are recommended for automation.
Data-driven tests. In case your case involves changing data sets without changing the body of the test itself, automation is a go-to option. You’ll then be able to run multiple tests simultaneously, improving both the quality and the speed of the session.
Step 2. Hire an automation engineer or a team
After you have determined a set of cases that will be up for testing, it’s time to choose a team to help you with the execution. Hiring an automation engineer is not an easy task especially if you want him to be versatile and capable of creating multi-faceted scripts.
Here are a few tips to reduce the number of challenges down the road and help you find a fit candidate in no time.
Look for someone who’s capable of assessing whether or not a case should be automated. Automating all suites and cases is a bad practice that will only make the management more complex and the results of the execution – less insightful for the company. A person you want on board is capable of determining which cases are critical for automation, those that could be automated down the line, or stay manual altogether.
Choose engineers with experience in multiple tools. Having a tester who has mastered one tool and is loyal to it has its own upsides – however, you’d be losing in terms of versatility and freedom. Make sure an engineer you consider a hire stays open to tools, monitors the landscape and is updated on the latest solutions – both open-source and commercial.
Specify your requirements. You need to know what you’re looking for. Instead of simply browsing through CVs of engineers, specify the type of test you want to perform and ensure a candidate is not out of his depth in this particular area. By narrowing down the list of job requirements, you’re more likely to find the right person for the job.
While skills are crucial when it comes to looking for an automation engineer, salaries are still a factor to consider. In case you’re not familiar with the job market, here are the salaries of a junior, middle, and senior automation professional.
Choosing a suite of tools for testing is crucial when it comes to a automated testing strategy. You need to ensure all the software you use for script execution would improve, not limit the process. At the moment, there are dozens of solutions with a similar range of features – choosing the one to go with is, by no means, easy.
That’s why a developer needs a set of criteria to use for support. Here are a few factors that influence the set of tools a QA specialist chooses to go with:
Complexity. If your team does not possess considerable experience in automation, learning a new programming language in order to navigate the tool would be a concern. Make sure to find out the reach of your QA’s abilities, the languages they are comfortable with, and the tools they have previously used. Also, be sure to not choose powerhouse software for simple testing tasks – the abundance of features will slow the process down instead of improving the pace.
Technical feasibility. Make sure that the tool allows you to reap the main benefits of automation testing – runs tests with no supervision, collects automatically generated reports, doesn’t rely heavily on the exact coordinates of the screen, is capable of running multiple tests at once, and so on.
Test data. It’s best to go for a tool that has a set of data management features. This way, you won’t have to rely on different software products for dataset generation and script execution.
Application stability. This includes an established community along with a broad range of documentation articles. Make sure the tool you choose for testing has a well-established reputation and is not at the risk of shutting down.
Reusability of scripts. Make sure a tool allows you to keep a database of all written scripts and reuse them for a new project
A possibility to execute scripts in various environments. It’s important for a chosen tool to be capable of transferring a script to a different environment and automatically changing application-specific data.
Step 4. Create the test automation framework
When creating a test automation framework, a QA specialist has to ensure it is scalable. Keep in mind that the framework is rarely used for a single project – instead, it is a template of test automation strategy, guidelines, and principles your testers should be capable of using for any kind of automated testing.
You need to make sure a testing framework covers not just the execution but a system of reports as well as a retry mechanism to re-execute tests that have previously failed.
While your testing framework will depend on the scope of testing and the tools you choose to handle executions, here are a few general tips to take into account.
Use the PageObjects model (POM). As a tester is increasing the test coverage, they may end up with an unmaintainable structure. By adhering to POM and keeping tests and element locators separately, a QA specialist will be able to avoid code duplicates as well as the time they would normally need in order to set up locators.
Implement Object Repositories for GUI testing. Instead of having test objects scattered all over test scripts, it’s better to keep them in a centralized location. That’s why using Object Repository comes in handy as it allows UI testers to store and declare all the elements of the app.
Define the scope of testing. Make sure you’re clear on the testing requirements – this way, you’ll be able to define whether or not a test was a success. In case you don’t have a testing scope set in stone, major insights, and application issues can be overlooked.
Conduct frequent framework code reviews. This way, a team manager is capable of bringing testers closer together allowing them to know each other’s strengths better – also, you’ll be able to seamlessly monitor juniors. By reviewing code, a team lead will ensure that every member of the team understand the unified testing standards and complies with them.
Provide the team with all the needed documentation. This includes all the tool-related articles that allow QA specialists to solve problems during script execution as well as test reports. Make sure team members can share reports with each other – this way, everyone on the team will get an understanding of the bigger picture.
Step 5. Getting your team up to speed with automation
Establishing an agile and comprehensive testing framework is crucial – but it’s even more important to ensure a testing team adheres to it. If you’re a business owner or a project manager, it’s your job to establish a link between testers and developers as well as within the testing team itself.
Here are a few tips that will help you onboard QA specialists faster and build teamwork within the department.
Test and improve the team’s expertise regarding domain knowledge, various software systems, and software testing phases.
Establish the communication and reporting system within the team. You can use project management tools as well as shared code editors to increase the amount of collaboration during test sessions. It’s a good idea to put testers in a small group – this way, if one team member were to leave the team, another one would step in, and the development process remains undisturbed
Update your team about the requirements and test cases on a regular basis. As a manager, it’s your job to ensure not a single change in the test objective remains unnoticed. It’s a good idea to schedule regular status update calls to make sure everybody on the team is on the same page.
Be sure to get regular reports from each team member to have better control of the overall process. There are managers who adopt a report day strategy – during a fixed day of the week, a team lead collects reports from each team members. Such a strategy allows you to bring structure to the communication system and track testers’ progress.
Step 6. Setting up the testing environment
The testing environment is a combination of software and hardware needed to test the performance of the system. Setting it up the right way is the bedrock of a successful testing session. Here’s a short checklist for creating a favorable environment for automated tests.
Make sure all the needed equipment is available. This includes the hardware as well as additional tools – working PCs that supported the needed operating system, the needed amount of monitors, and so on.
Ensure that the system has all the applications needed for testing. You might need to see how your program complies with other software a user may have installed. If that’s the case, be sure to prepare all the additional applications and file sets beforehand. Also, check that your team knows how to use all the apps needed for successfully navigating the environment.
Check the availability of standard datasets. Remember that test data should be collected before script execution – otherwise, the test will take too much time to complete.
Assign the environment maintenance to a specific person on the team. This team member will be responsible for the state of servers, hardware, and software.
Plan the scheduled use of the testing environment. In case you share an environment with the development team, ensure both departments occupy separate time slots. Having both QA specialists and software engineers working in the same environments simultaneously could provide you with false test results and corrupt the entire process.
Ensure that the connectivity is achieved before initializing the testing. This is especially important for cases that require a complex environment configuration.
Step 7. Developing An Execution Plan
When a team lead is understood by their team, understands its expertise and the range of cases and tools used for testing, and the environment is up and running, it’s time to elaborate on a detailed execution automated testing plan.
Unlike the testing framework or the outline developed on the initial stages of a testing strategy, an execution plan is a daily list of activities described in detail.
Here’s what it normally consists of:
A description of day-to-day tasks. The automation plan should include time slots for status emails, progress updates, room to learn, and so on.
Using pipeline orchestrators for test executions. These tools allow testers to automate sequences of tasks and ensure they are completed in a fixed order. Some of the most popular options in the market include TFS, Bamboo, Jenkins, TeamCity, and so on.
Define a set of practices to keep tests change-resistant. These guidelines should cover the way test cases interact with the elements of the tested application.
Step 8. Writing Scripts
One of the crucial steps of the automated testing strategy is writing scripts for execution. In order to make sure your automated tests are efficient and assess the functionality of the application to the best extent possible, consider paying attention to the following guidelines:
Prioritize scenarios. List the most important user flows and focus on ensuring they are bug-free. A list of features with the highest values normally includes the login page, checkouts, or adding items to a shopping cart. In case you have an Analytics Account, use this data to find out which user flows are used most extensively on your website.
Break scenarios down to atomic flows. Rather than writing a script for a complex sequence of events, break them down to a set of simple tests. When all of these actions are tested, run a sequence test to ensure the performance of the whole is no different than that of its parts.
Create single-purposed tests. Depending on the element and the action you’re testing, most common testing purposes include validation (check the visibility and the look of the element), setup, and actions (testing a user flow).
Be sure the initial state of the test is consistent. Due to the fact that automated tests consist of a fixed number of actions, it’s crucial that they start in the same state – otherwise, the flow of the test will be compromised.
Compose a complex test as a whole of simple ones. This way, testers will be able to save time and human resources as there’s no need to record a new workflow from scratch.
Reduce the number of occurrence conditions. In order to ensure the test is predictable and easy to manage, try to avoid if-statements when writing a script.
Write isolated scripts. Running self-contained flows improves the quality of parallel testing. As opposed to connected suites, if there’s a hold up with an independent test, the entire session will not be affected.
Step 9. Reporting
After a series of tests is executed, a QA manager has to capture insights and put them into a readable and concise document that will help developers improve the product. While there are tools that create reports automatically – you’d only need to revise the content – there are a few tips that make test results reporting easier and way more efficient.
Analyze the results collectively, not in isolation. That’s why experienced testers tend to keep logs of all the old session. By comparing the data between testing sessions, you’ll be able to keep track of progress, uncover repeated issues, identify tests that need further improvement, and so on. Also, in case you have detected an issue that was not indicated by the previous report, it’ll be easier for a developer to understand what triggered the change.
Get broader logs than pass-or-fail. Make sure team members provide you with access to screenshots, system logs, and all the data captured during testing. Instead of tracking progress with pass-or-fail logs, rely on the step-by-step description of the execution process.
Improve data visualization. The insights will have more meaning if you group them together in a table, a diagram, or a graph – this way, you’ll be able to get a clearer comparison between the builds of the program. Visualizing testing reports in a visually compelling way is highly important for team leads in case they have to use them as a status update for a client.
Step 10. Script maintenance
Creating a low-maintenance reusable test suite is a struggle for a fair share of testing teams. There are QA specialists who put no emphasis on test suite maintenance whatsoever – as a result, their scripts end up stale and underused.
Here are a few tips for testers who are eager to improve script maintenance.
Exclude browsers and operating systems with low usage rates in order to decrease the number of combinations for testing. If you need to test an application in different browsers across all versions of Windows, that would leave you with over 20 server-side combinations. By eliminating rarely used Windows versions or browsers, you’ll be able to significantly save both time and effort otherwise spent on testing.
Avoid complicated scenarios. Test suites that are easy to maintain are, most of all, flexible and simple. This means excluding ‘IF-THEN-ELSE’ patterns and avoiding complex flows that mix users’ behaviors. Keep in mind that complex tests with nested conditionals are harder to debug and read – thus, their maintenance cost will grow.
Get to know the system’s asynchronous actions. Be sure to keep an eye on client-side asynchronous actions and avoid hard-wired delays. While handling callbacks can be challenging for testers, in cooperation with developers, you’ll be able to determine the flow of the page and create helper functions.
Cooperate with developers by asking them to create backing APIs.
Automated Testing at Performance Lab
In case you’re looking for a team of experienced testers and automation engineers, be sure to contact Performance Lab. Our team will create and execute an automated testing strategy that will comply with your company’s development methodology and will be non-invasive to developers.
Here are the types of automated testing we normally handle:
Automated regression testing;
Test automation during development;
Test automation for DevOps;
Test automation in a production environment;
Embedded system test automation.
Do you want a team of certified testers to handle your next big project? Contact us today – Performance Lab will be happy to get on board. You can also take a look at the full list of services we offer to choose which type of testing your project needs.