6 steps to successfully test

Due to the widespread popularization of mobile apps, we’re beginning to use mobile apps more and more. Taxis, weather, news, ordering food, etc. There a mobile service for nearly everything.
And as the number of apps grows, so does the need for quality in the apps being produced. It’s true! More and more organizations are coming to the point of testing their mobile apps. Of course, you are often doing this testing not with us (testers), but rather with users and customers. However, if an app has a large audience, then the price of a defect grows. This means you can’t just poke keys on the keyboard. Thought and knowledge must be employed to find bugs quickly.

So what am I getting at?

Oh yeah! There are loads of materials that tell us what mobile testing is, how it is to be done, how the process should be arranged, and then there are piles and piles of all sorts of rubbish. After all, we’re not simply testing mobile phones, we’re testing apps on different platforms!
It’s against this background that we’ll discuss the specific features of testing mobile apps and where attention must be focused before testing begins. And the subject of our discussion will be the Android platform.

Well, since I’ve begun talking about the steps to releasing a high-quality mobile app, I’ll mention that one of the first steps is to prepare a mobile laboratory. 

1. Devices.

Due to the fact that the number of mobile devices running Android is constantly growing, it is important to test an app’s ability to work on a large number of devices. According to official Android statistics, today more than 190 countries have access to Android devices, and the total number of devices is in the hundreds of millions. Given such a massively popular platform, each device naturally has its own specific features, such as the physical design and user interface for various attributes.
If you are testing, remember that your set of devices should include not one, not two, and not even ten devices.
Often mobile app testers approach me to talk, and when the conversation turns to the test set of mobile devices, 90% of the time I hear they use at most 4-5 devices in testing at their companies.


When choosing devices you can’t set aside the version of the platform!!!

Many people believe that if we test an app on Android 5.0 Lollipop on a Samsung Galaxy S6, then it means the app works.
That’s crazy!

In addition to the platform, there are a number of components, such as the CPU, diagonal, screen resolution, and RAM, that can seriously affect your app’s ability to work.
On average, based on statistics we put together following our research, full-fledged testing of Android mobile apps requires roughly 69 Android devices.

Moreover, we must always remember that many apps may be available to an even larger number of devices, including cheap smartphones, which are increasingly influencing many markets, especially in China 
If you have plans to grow and enter the international market, you absolutely must consider this factor and conduct a continuous analysis to expand the set of test devices.

2. Screen size and density.

While screen size may be simple enough to understand, typical testers don’t always encounter the concept of screen density.

Anyway, screen density is the number of physical pixels on the display.

Compared with the quite small number of screens on iOS devices, the variations of screen size and density for the Android platform adds additional complexity to testing. Despite the fact that there are more than 12,000 different devices, Android officially divides them into 4 classes based on screen size and 6 classes based on density.

As we can see, the most popular screen size is “Normal” with the following densities: hdpi, xhdpi, xxhdpi, and mdpi. This is important to consider when putting together a set of devices to test mobile apps.
I will also note that this information is constantly updated on the official Android developer website, so, before testing, occasionally check the accuracy of the information, since the mobile market is always growing.

Link to Android dashboard

3. Versioning.

Officially, Android supports 10 platforms/versions, beginning with Froyo 2.2 and ending with Marshmallow 6.0. But today not all mobile apps support all versions and platforms. Thus, when planning testing, you must absolutely account for running test cases on several versions of Android.

For example, if we look at the current market, Lollipop 5.0-5.1 (released back in December 2014) and Marshmallow 6.0 have been out for a long time, but the leader is still KitKat 4.4.
Why is it important to use different platforms?

On the one hand, you shouldn’t test on the latest versions of the platform, because this sort of testing literally prevents a huge number of users from working enjoyably with your product.
On the other hand, if your app isn’t updated to the newest versions of Android, then you risk losing current users after they update their version of Android.

I once took a look at the Play Market and was amazed to see that for virtually every app users write that the app does not work properly on a particular device!

And sometimes this really touches a nerve. Like when you download an app (especially not over Wi-Fi), launch it 900 meters underground, and the app won’t open!
Accordingly, I can say that it’s important to test your product on at least 4, or better yet 5, versions of Android.
This concludes my creative narrative about putting together a test set of mobile devices. 

Let’s move forward!

Everything I talked about above is part of the preparation phase. Now let’s talk about the most pleasant part: mobile testing. I will describe the main problems you may face when testing mobile apps.

4. General functionality.

In actuality, many people may feel that this consists of typical and easy tasks. I don’t deny that this is the case. But if we don’t test well, then a huge disaster awaits us when the app is released/updated.
Let’s start at the beginning.

  • Registration and signing in to the system.

Perhaps a more obvious case for testing can’t be found, but I can’t help but mention it. Testing any mobile app must fully cover the process of registering and signing in to the system. You can pay special attention to the fact that when testing these processes you need to run them from beginning to end. Moreover, you need to generate as many negative tests as possible, everything from invalid passwords to SQL injection.

  • Menu items.

If you look at mobile testing on a large number of devices with different screen resolutions, then you can also encounter the situation where the standard menu items, such as Help, About, etc., are simply inaccessible, both visually absent and physically impossible to tap with the finger. This is especially true for devices with small screens when trying to put your fat finger on a link. In the end, you have to take aim and press carefully. 

  • Key functions.

Any functions, especially those tied to entering text, scrolling, clicking “Back”, etc., that are essential for a normal user to actually be able to use the app absolutely must be verified during testing. What’s more, you’ve got to verify that the app works properly given consecutive use of the physical buttons and the touchscreen.
Recently, I downloaded the KFC app. My rage knew no bounds! Never mind the fact that I was simply unable to press the “add to cart” buttons, it takes a minute to load! And I had to find my restaurant on my own, because the app stubbornly told me that I was on the opposite size of the city!

  • Interruptions. 

How does the app behave when the phone’s battery level is full, medium, and low? Can the user make calls and send and receive text messages? You should always know how the operation of other apps (often built-in apps) can affect the operation of your app. Studying the crashlog is helpful here. For example, I have a Lenovo that constantly reboots itself when music is playing over Bluetooth and Yandex Navigator is launched a second time or the GPS button is pushed. This is terrible, especially when you’re going 70 mph, lose your GPS, and need to restart it. The panic begins. Where do I go…?

And this is where the lottery begins! The touchscreen is physically running in portrait mode, but the screen is clipped in half and displayed in landscape mode. When I first hit this type of bug I tried for 30 minutes to get at the button to reboot the phone!
But let us return to mobile testing.

  • Auto-rotate.

It is very important to remember that an app may often be used in both portrait and landscape modes. Many users frequently use portrait mode, but personally I prefer to use navigation apps and video players in landscape mode. Thus, during testing you need to verify that your mobile app’s functionality and usability do not change when switching from portrait to landscape.

When testing a mobile app you must be certain to check how app settings affect the app’s performance, including the app’s effect on the battery level. Many users want to use their mobile devices for at least one day while enjoying any app. If your app has a large audience, then you should certainly know how it affects the phone’s battery. I recommend running usage tests for this.

  • Affect on operation of the device.

Speaking of usage, this assumes the app is operated in a normal manner. Testing requires launching the app and using it for 6-12 hours. During use, you need to measure the battery level every 30 minutes to 1 hour. This sort of check is frequently automated so testers don’t have to spend time collecting data. If the test results show the battery level dropping before 6 hours while all of the phone’s background functions are running, it’s a clear indication of problems in the app.
I think everyone has had the experience of downloading an app and then seeing it burn through 20% of your battery in 30 minutes. I don’t remember their names, but I’ve definitely seen them!

5. Additional checks.

Apart from the main functionality, there are a series of problems you may face when testing mobile apps on Android. When testing be sure to focus on:

Special symbols If the app you’re testing includes a search box or a text entry field, you need to test for the ability to enter special symbols.

Long press on the screen.You must be sure to verify that the standard Android copy and paste functions work when you tap and hold down the touchscreen in text entry fields. Make sure, first of all, that these functions work and are not mixed up and, second, that a long press on the screen does not cause the app to crash.

Virtual keyboard.Most devices running Android have virtual keyboards. If the user changes the keyboard’s size and skin, you need to be certain that these changes do not affect the use of your mobile app and, worse, make it unusable.

Additionally, always put yourself in the position of a real user and test the app from this perspective, running through real user scenarios.
Remember that any malfunctions will make the app’s users unhappy.

6. Android security.

In generally, I don’t consider myself an expert at testing security, but there are nevertheless certain checks that must be performed to protect end users.
Any system must ensure that data requirements are observed:
Privacy. Is the user’s personal information secure?
Integrity. Is it possible for an outside party to alter personal data as it is sent?
Authentication. How does the app verify that I’m actually the account owner?
Availability. How and under what circumstances is it possible to break the app or make it inoperable?
Fault tolerance. How are all system failures recorded?

If security specialists examined all of these points, verified the encryption methods used during data transfer, and tested the ability to tap into the device’s standard functions through an API (contacts, camera, or GPS), then you can consider your app to be ready for customers to use.

And remember that an overly obsessive privacy policy can also kill the desire to use a mobile app!

And finally…

I had to dig into Google Play quite a bit, but it proved to be very helpful.

I was able to determine the relative occurrence of the problems faced by users:

  • 55% complain about installation
  • 24% complain about performance
  • 13% complain about app crashes
  • 5% write that the app freezes or hangs
  • 2% complain about the UI
  • 1% complain about problems with privacy or security.

And, as always, the negative reviews outnumber the positive. This speaks to the fact that very few companies in Russia fully test their mobile apps. Poor reviews can shoot down an app’s release before the app starts working properly.

Test mobile apps both on emulators during development and on real devices during full-blown testing.  Functions such as the accelerometer, touch screen, and location mapping are simply impossible to verify on emulators.

Don’t chase after cheap emulators. A keyboard and mouse will never replace actual use of a real device in a person’s hands.

I hope this was helpful!

Alex Meshkov, Head of Service Delivery and Operations at Performance Lab.

Got a project in mind?

There is no better place for a QA solution than Performance Lab.
Drop us a line to find out what our team can do for you.

Submit a Request

Latest posts from us

Android apps performance and load testing: tips and tricks
Application Performance Monitoring vs Crash Reporting preview
Application Performance Monitoring vs Crash Reporting
10 Steps to SAP Performance Testing
10 Steps to SAP Performance Testing
Speed up a Wordpress Website
How to Speed Up a WordPress Website – Part 2
SAP Load and Performance Testing using Loadrunner
SAP Load and Performance Testing Using Loadrunner