Automated testing in software development is a great tool to ensure high quality solutions, especially when maintaining an application over a long period. Testing in our apps helps us cover cases that would take a very long time to test manually. You have mobile devices with many different versions of the operating system, many different screen sizes and vendor specific implementations that differs from standard operating system behavior, just to name a few.
Since we began writing tests at a larger scale for the apps the number of bugs has gone down significantly while the end user satisfaction has increased. This increases trust in our solutions which allows us to build long standing business relationships with our partners.
The purpose of this article is to explain how we view and are working with automated testing in our mobile apps.
Types of tests
We have tests split into three categories.
Unit tests
Small function level tests that validate business logic. These tests are very fast to run and easy to debug. They are by far the most numerous tests in the apps, and they will rarely break. Unit tests will have almost every single external dependency mocked away to ensure the assertion is not polluted by something that is out of the scope for the test.
In our apps unit tests covers logic such as filtering of data from backend, ensuring translation text keys have a value and input validators.
Unit tests do not require an emulator to run since they only have dependencies on code specific implementations and the rest is mocked away.
@Test
fun testNameToInitial() {
var participant = VisibaParticipant.getDefault(displayName = "Jan")
Assert.assertEquals(participant.getNameInitals(), "J")
participant = VisibaParticipant.getDefault(displayName = "Jan ")
Assert.assertEquals(participant.getNameInitals(), "J")
participant = VisibaParticipant.getDefault(displayName = "Jan-Erik Gode")
Assert.assertEquals(participant.getNameInitals(), "JG")
participant = VisibaParticipant.getDefault(displayName = "")
Assert.assertEquals(participant.getNameInitals(), "?")
participant = VisibaParticipant.getDefault(displayName = " ")
Assert.assertEquals(participant.getNameInitals(), "?")
participant = VisibaParticipant.getDefault(displayName = "Jan Turesson")
Assert.assertEquals(participant.getNameInitals(), "JT")
}Example of one our unit tests
Integration tests
Larger tests where almost nothing is mocked away. These tests cover multiple architectural layers and come with a larger burden of maintenance since they cover multiple classes. Small changes somewhere can break them, and it takes longer to debug exactly where things went wrong. They can even break without there being anything wrong with the code, but the test itself has been broken.
These tests are a great compliment to the regression testing that is performed by QA before each new release of the apps. When done right they can catch subtle bugs long before they become an issue.
For the apps integration tests can take a long time to run because they often contain a lot of code that must run on an emulator.
In our apps integration tests covers major flows like booking a timeslot with everything except API-calls being non-mocked code.
@Test
fun testEntireTimeslotBookingFlow() {
initTestVariant()
testRobot
.clickBookAppointmentButton()
timeslotGroupActivityTestRobot
.selectTimeslotByResourceName("name")
.clickNextButton()
timeslotBookingActivityTestRobot
.setReasonText("reason")
.clickBookButton()
timeslotBookingConfirmationActivityTestRobot
.scrollToConfirmButton()
.clickConfirmButton()
appointmentActivityTestRobot
.clickStartAppointmentButton()
waitingRooActivityTestRobot
.clickAnswerCall()
videoCallActivityTestRobot
.clickExpandPortMenu()
.clickLeaveCall()
.clickYesInLeaveCallDialog()
}Example of an integration test from the apps landing page all the way to a video call
UI-tests
All tests that validate visual elements in the user interface fall under this category. These tests require an emulator to run and are notorious for breaking without anything being wrong. If you for example run the test on an emulator with animations turned on, the test can fail because it thinks the UI-state is wrong when it is just waiting for the animation.
These tests are most useful when we need to validate that a view looks correct according to the current configuration. The configuration might contain a flag that says if we should show/hide some elements in the interface. Manually testing all different combinations of configuration flags quickly becomes unfeasible and therefore automating it saves us time.
@Test
fun canFillAndSubmitSurvey() {
initActivityVariant(SurveyViewTestVariant.Standard())
testRobot
.setFreeStyleText("text")
.clickSurveyStar2()
.clickSubmitSurvey()
}Example of a UI-test limited to the scope of one view
Why tests?
Knowing why we write tests is an important factor to make people take them seriously and want to write and maintain them.
Functional documentation
Writing tests is a sort of documentation for expected behaviors in our apps. You can for many screens in our mobile apps get a good understanding of all the functionality by simply looking at the tests for that screen. The tests also guarantees that actual documentation stays accurate by enforcing business requirements. For example, we might have an input field that takes a maximum of 4000 characters which is also written down in a user guide. By having a test for it then it will stay correct until intentionally changed by the business requirements.
Edge cases
Edge cases can be time consuming to test manually and there is a burden of knowledge to even know they exists. By covering edge cases in tests: it ensures that new developers and developers coding in parts of the application they are not familiar with do not break non-obvious behaviors. Edge cases can also to be noted with comments in the code since their very nature makes it difficult for them to be self-documenting.
Refactoring safety harness
For long living software refactoring is an inevitability to combat technical debt and keep up with the latest useful tooling and coding patterns. If you already have tests in place before starting the refactoring it automatically becomes a test-driven development type of working pattern. This speeds up the refactoring process significantly since you will spend less time ensuring you are not breaking existing functionality and more on improving the code.
Faster debugging
A sometimes-overlooked aspect of having a solid automated test framework it is that it increases velocity for debugging and implementing new features. If tasked with implementing a new function that filters data in a specific way, it is much faster to just write a unit test and validate that it works that way than deploying the apps on a real device and test it in an actual flow.
This applies to debugging also. If you know in which class or function that bug occurs, you can write tests to force the bugged state and quickly validate that your solution solved it.
Architecture
A solid architecture is the foundation for good tests that are readable and easy to maintain.
Robot pattern
Robot pattern makes UI-tests much more readable and easier to maintain by defining all possible user actions on a screen in one class. Here is a code snippet of how the test for Bank-id login is implemented
@Test
fun testCanCancelBankIdLogin() {
initActivityVariant(LoginViewTestVariant.LoginViewBankIdPollingTestVariant())
testRobot
.selectBankIdAuthMethod()
.setNationalIdNumber("XXXXXXXXX")
.clickBankIdLogin()
.clickCancelLoading()
.clickBankIdLogin()
}It is very easy for someone unfamiliar with the login flow to follow what is happening in this test thanks to the test robot. The test robot is also very easy to reuse in integration tests which chains multiple test robots together when testing flows.
Test variants
Test variants helps to keep track of all different permutations and allows reusing of common setups between tests. We use the test variant to say which state and data we want the view to show when we run the test. When we want to test how a list is displayed, we use one variant with many items, one variant with no items and one variant where the items failed to load. You can then verify that the interface correctly handles all these cases and reuse the test variant in an integration test.
Dependency injection
Dependency injection is crucial when writing good, isolated tests since it allows you to use mocked objects instead of real implementations for external dependencies. This is very important for unit tests which otherwise might test things out of scope or not work at all because of behavior outside the function itself.
Dependency injection also helps making code that normally would not be testable by discouraging the use of things like global static helper classes. For example, in a function that writes a file to disk you can mock the file-writing so you can check how it handles error or success handling rather than the actual writing itself.
Writing and running tests
When writing and running tests there are a few things we always keep in the back of our mind.
Avoid happy paths
A happy path is when you go through a flow where you do everything correctly and nothing goes wrong. While developing you often test happy paths a lot just to get through a flow and end up at your intended destination. This makes writing tests for happy paths far less important than tests for error handling or the less common paths through the flow in our apps.
Continuous integration
Tests should run automatically on a regular basis to ensure no one has broken any behavior without even noticing. For us this is done in Azure every time someone opens a pull request or code is pushed into develop or a release branch. The unit tests are very simple to run this way while the UI and integration tests are much trickier because of the emulator requirement. It is also painfully slow to run emulator tests this way so as of now we only do it manually. We do however verify that emulator-based tests still build successfully.
Multiple devices and configs
For the apps there is support for running tests in parallel on multiple devices at the same time. This very useful since you can on a single test run cover many different screen sizes if you have written a UI-test for some new elements you added. Both XCode and Android Studio which are the tools we use to develop our apps have built-in support for this.
Summary
Tests have helped us reduce the number of bugs in our apps and increased the satisfaction of end users. We have categorized test into integration tests, UI-tests and unit tests which have their pros and cons. Together they cover all important behaviors of the apps. We write tests to share knowledge and speed up development while having high quality solutions. We believe having a solid architecture in the code is a requirement to write good tests that can be easily maintained. When writing tests, you want to focus on edge cases and things that are not normally tested in the standard workflow. We must also ensure the tests run on a regular basis and on as many different devices and configurations as possible.
Philip Sandegren
App Developer at Visiba Care
