Automated quality assurance (QA) testing has been common practice on web applications for years. Mobile platforms, on the other hand, present some unique challenges that have made automated testing difficult to implement. The advantages of automated testing are big enough that Deloitte Digital has worked hard to understand the practices and technology that can bring its benefits to our clients’ mobile products.
What are the challenges?
First, mobile platforms are not only a relatively young, constantly evolving technology, but many are—in contrast to the web—closed platforms. Cracking these devices open (figuratively speaking) so that a computer can drive them like a user takes special technology and expertise.
Second, device and display fragmentation is so dramatic that the most effective testing framework is cross-platform. This allows a testing program to be easily executed on any mobile device without the core structure having to be re-engineered. Because they work cross-platform, test frameworks such as the open source Calabash and Appium frameworks are flexible and effective solutions for multi-platform projects.
The last challenge with automating a mobile application is as much about your mindset as it is about technology: Users expect a different quality of responses from a mobile device than they do from the web experience. For automated testing, this sometimes means writing more nuanced test code, but often it means wisely deciding when and where automated testing should be applied—and where only the hands and eyes of a live tester will do.
What’s the benefit of overcoming these challenges?
Automated testing can be applied many ways to bolster the quality of the product and contribute to a more effective development process.
An automated build verification test can check the basic stability of the product: does it build, run, allow sign-in, and have a functional main view? When run by a continuous integration build server after every code change, this kind of verification test provides instant feedback when the build is broken and makes it easier to isolate the source of any problem.
Automated verification can go further, to quickly and regularly walk through the "happy path"—the most common states and activities within the application—and check for regressions as new features are added and bug fixing takes place.
Data-driven automation is a strategy that can very quickly process a large number of minor business requirements like giving appropriate error messages for certain input. Without this automation, a tester has to manually work through a spreadsheet of inputs and expected output. An automated script can easily pull that data out of the spreadsheet and execute the tests in quick succession. And expanding the scope of a data-driven test is as simple as adding more rows to the spreadsheet.
These are only some of the useful applications of automation. It can also be used to test web services regularly, to perform ongoing regression testing of previously fixed bugs, or to verify technical acceptance criteria.
Can all testing be automated?
The answer is no. Automated tests are essentially designed to catch expected failures. Only a human being can do exploratory testing to uncover that strange sequence of actions that breaks the app. On mobile platforms in particular, touchscreen interactions are so nuanced and open-ended that, while they can be automated, a live tester is far more likely to find problematic edge cases or unexpected states. An automated test can verify that the text in an app is being displayed as intended, but it certainly can't determine whether that text is semantically clear and user-friendly. Usability in general is an area that simply can't be tested without an experienced critical human eye.
Deloitte Digital’s QA team has spent many hours tackling the inherent challenges of mobile test automation, but perhaps our most valuable contribution to the practice of mobile QA is the good judgment that comes from deep, full-spectrum quality assurance experience. Being able to execute automated tests is a skill that is valuable when judiciously applied. When it is, automated testing saves time, money, and lets you have greater confidence in an application’s quality.
Chris Floyd is a QA Engineer at Deloitte Digital, Deloitte Consulting LLP’s Denver Studio with 15 years’ experience testing and designing experiences on diverse platforms including web, mobile, and video game consoles.