With the evolution of web and mobile applications, the building of maintainable and reliable automated tests has become a crucial need to ensure high quality software. But what if these tests do not have the same outcome for the same code and the same environment and they are failing for no obvious reason? This behavior is called the « flakiness » issue which is one of the most challenging problems in software development as they can waste a significant amount of time and resources and impede the release of new features. They can also undermine the credibility of the test suite and decrease the confidence of developers and stakeholders in the software. According to some studies, flaky tests can consume up to 50% of a development team’s time and they can cause delays in software releases and increased costs. Afrequent example of the impact of flaky tests is when a test suite that is designed to be run daily starts to fail frequently, this will slow down the development process as the developers will need to investigate and debug the test failures before they can continue working on new features. Another example is when a flaky test causes a regression in the software and it goes unnoticed, this can lead to customer complaints and the development team will have to spend extra time fixing the issue and testing the software again. The goal of this article is to explain the problem of flaky tests and provide strategies for preventing, identifying, and fixing them to ensure a stable and the reliable test suite, and to provide best practices for maintaining a stable test suite.
Flaky Tests Are Like A Canary In A Coal Mine. They
Are A Warning That Something Is Wrong And Needs
TO BE ADDRESSED. Causes of Flaky Tests It can be hard to point out the exact cause of flaky tests as there can be a variety of reasons why these tests are failing. The common cause of flaky tests includes changes in the test environment, test design issues, and test data dependencies. The test environment can include things such as the operating system, software dependencies, and configurations. An automated test that runs correctly on Windows might fail on Linux, or an update to a dependency might cause a test that previously passed to fail. Also, changes to the test environment can also include changes to the test infrastructure, such as updates to testing frameworks or libraries. Test design issues may cause unreliable tests that can be poorly written, tests that are tightly coupled to the product implementation or tests that make assumptions about the state of the system can fail when the product is updated with new versions or when the state of the system changes. Besides, tests should properly handle exceptions and errors to avoid test failures. Tests that rely on external data or services, such as databases or APIs, can become flaky if that data or service changes. For example, a test that relies on a specific dataset might fail if that dataset is modified or deleted, or a test that relies on an external API might fail if the API’s response format changes. Automated tests that have timing-related issues, such as tests that rely on hard-coded sleep or wait times, can lead to flaky tests as the
test results maybe dependent on the time it takes to execute the test. Additionally, the developer should avoid using a lot of sleep time and use instead implicit waits because it makes the test slower and cause it to fail when the test suite is run on a different environment with different resources. Shared resources can lead to Interference with other tests by modifying the the same resource can cause flay tests as it may be dependent on the order in which they are executed. All these causes can make automated tests flaky and unreliable, and they can be frustrating for developers to investigate but understanding the causes can help to better prevent this issue. How to reduce flaky tests? Not having reliable tests in place can be a major issue for any development the team as it affects its ability to create quality code and prevent finding bugs. To eliminate the flaky tests, we need first to isolate those tests, keep them in another automated test suite or mark them as flaky in the original test suite so that we can keep the remaining deterministic tests aside. In this case, it is better to use a test framework that provides built-in support for retrying falling tests. Then what to do with the isolated tests? Isolating tests means decreasing the regression test coverage, so these tests should be treated quickly. Isolating test environments from external dependencies can help prevent flakily tests. This can include using virtualization or containerization technologies to create isolated test environments or using service virtualization to simulate external dependencies. In addition, Developers should consider the best practices in test design. This includes designing tests that are independent of the implementation, designing tests that handle exceptions and errors properly, and designing tests that can handle concurrency and timing issues. Managing the test data and resources in a controlled way can reduce flaky tests. This can include using test data that is less prone to change, such as data that is generated by the application or using test data management tools to version and manage test data. Besides, using continuous integration tools and testing can help to detect flaky tests by running tests regularly and detecting failures as soon as they occur:automated tests should be run every time the test suite code is changed, or the product has been modified so that developers can detect test failures early and address them before they become more difficult to fix. Tests Reports and test logs are indispensable for investigating test failures. Sometimes, tests are not deterministic due to the user testing frameworks and libraries which are not well maintained. We should ensure that the tests are running on a stable and well-supported platform. Additionally, code review is one of the most crucial factors that can ensure a stable test suite by catching the test design issues, identifying test data, and detecting concurrency issues: Reviewers can check if the tests are properly designed to handle concurrency, and if not, they can suggest changes to fix it. Also, they can check if the tests are dependent on external data or ser-
vices. By having other developers review the tests, it is more likely to catch issues early and to have a better understanding of the tests and their purpose. This can help to maintain a stable test suite and ensure the quality and reliability of the software. Finally, the causes of flaky tests are essentially related to imperfections in the software development process which is revealed by failed tests. By investigating the cause of flakiness, we are taking a step back to call into question some practices and software processes that need to be reviewed. This is what makes flakiness an anomaly indicator for automated tests that should be carefully considered and not ignored. It’s worth mentioning that Artificial Intelligence may have the potential to help prevent flaky tests by providing new tools and techniques for tests and maybe one day. Rabeb Mnani Iam a QA Automation Engineer with six years of software development and testing experience, specializing in automating Financial applications. Ihave a proven background in Software Testing in Client-Server Applications and Web-based applications using Manual Testing Techniques and Automated open source Testing Tools. Ihave extensive Knowledge of Quality Assurance standards, methodologies, and strategies and am certifi ed by ISTQB® Foundation, Professional Scrum Product Owner, and A4Q Selenium. a shift-right approach to support E2E test automation
By embracing DevOps mindsets, software development
teams can continuously deliver new value to their customers. The testing process followed the lead and became continuous. Teams don’t just test during development but also in the context of the application in operation, the so-called shift-right. Testing in production has many advantages. By monitoring an application in its users’ hands, we can quickly discover errors, bugs, and performance issues. We can fix them before causing any more damage to our customer’s satisfaction. We can also do A/Btesting on a feature’s design to see what works better for our users and learn how they interact with our application. Knowing our users’ behavior can help us design more accurate tests and targeted test suites. We can keep improving quality and reducing the release cycle simultaneously. In this article, we will discuss how Usage-centric Testing can strengthen our end-to-end test automation. End-to-end test automation, a love/hate relationship Before we dig further into Usage-centric Testing, let’s recall why it is important (and complicated) to design and automate the most relevant end-to-end (E2E) tests. E2E tests focus on validating the right functioning of the system/application. This includes testing business processes (from beginning to end) under production-like circumstances from the user’s perspective. E2E test automation generally aims to detect regressions on the most critical user flows. Let’s position E2E testing on our good old test pyramid: E2E test position on the pyramid hints at two facts about those tests:
- E2E tests are expensive to produce and maintain.
First, designing them requires good business knowledge, and their automation requires some technical bootstrapping (setting up dependencies, database flushing, a set of first automated steps for the basic actions in your application like the login, etc.). Then, even if writing new test cases becomes smoother with time (once you have a proven library of automated steps), maintenance remains complicated and time-consuming. Every minor change in the UI results in tweaking the E2E test code to correct it in case of test code breakage.
- But these tests give us reasonable confidence
to release the application. This value is the positive counterpart of the previous point: E2E tests provide us with confidence that a tested user journey won’t break. E2E tests seek to simulate the real world as closely as possible to prevent anomalies during actual use. E2E tests are expensive but provide value, so we want a few of them, but which one? Sizing the E2E test suite: an arduous task « The number of tests in our test suite grows exponentially… but the number of bugs doesn’t decrease accordingly! » « We have too many tests, and a lot of them cover functionalities that are not relevant anymore… » « We have limited resources to develop and maintain our E2E tests; we don’t know how to prioritize our test effort… » This is the kind of statement we can hear when conducting interviews with QA engineers, test managers, and developers we get in touch with. At some point during the development of software, the number of E2E tests grows to become a bottleneck that slows down our ability to deliver quickly. How do we fix our love/hate relationship with E2E tests, so it becomes a “love/love” relationship? Finding the proper minimal set of E2E tests is critical to balancing test suite execution speed and feature coverage. But how? By focusing our tests on our users’ most essential journeys, which are, therefore, the most critical ones to our business. As we can’t do this before delivering new features, we have to “shift right” a bit, so we can incrementally and iteratively improve our E2E test suite by learning from the way our application is used in production. This is precisely the Usage-Centric Testing goal. Usage-centric Testing: a shift-right approach Learning from our users’ behavior to continuously optimize our E2E tests The main idea of Usage-centric Testing (UcT) is to leverage users’ behavioral data to continuously optimize our E2E test suite. As we saw previously, designing a suitable E2E test suite is more complicated than it looks. Collecting anonymized user actions supports the design of E2E tests according in order to:
- Measure and improve E2E test coverage. User flow analysis increases the fidelity of E2E tests but also enables
design according to the representativeness of test action flows. Application usage analysis enables the detection of frequent usage patterns, or critical from a business point of view, and particular cases that deserve to be tested. The collection of user actions allows a new indicator: the measurement of usage coverage by E2E tests.
- Clean up our E2E test suite. Learning how features are
used is also valuable. It is the perfect opportunity to eliminate useless tests and reduce our E2E test suite maintenance effort.
- Prioritize test environments. Demographic data can also
be leveraged. Devices we use to interact with software multiplied up, making testing even harsher. There are a lot of cross-browser test automation tools on the market, making cross-device testing easier. By getting insights into our users’ most common environments, we can effectively prioritize the test runs for these environments and shorten our feedback loop. Collecting and analyzing anonymized user actions thus allows better design decisions for E2E tests and their optimization over time. Now, let’s see how we can concretely do this. Ausage-centric test automation process Usage-centric test automation process is composed of four main activities:
- Collect anonymous data about user actions on the application and figure out its actual usage;
- Evaluate the coverage of these usages by the current E2E
tests using their execution traces;
- Determine which usages to cover and which usage sessions
are relevant to increase the coverage. Identify E2E test cases with less value;
- Generate test scripts for your favorite test automation
These activities must be supported by tools to be carried out effectively. The first step is tracking usage data from web analytics tools (commonly used by marketing teams). But the support is very partial, as these tools don’t measure the coverage of the tests with respect to usage. Anew segment of test automation tools has thus emerged to support the UcT approach. Appvance, Datadog RUM, Gravity, and Prodperfect, for example, support a UcT-like approach. These tools respect the GPDR rules by collecting anonymized user behavior data and by generating test automation code to cover the usage. They also use Data Mining and Machine Learning techniques to provide services such as suggesting tests to be performed and analyzing test coverage on a given scope of use. The following figure shows a screenshot of the analysis of usage coverage by tests in Gravity. Common criticisms of the UcT approach Let’s have a look at some criticism of this new approach: “by collaborating with business representatives (PO / BA), Ican define accurate E2E test cases, no need to analyze the usage in operation”. Yes, E2E tests benefit strongly from being discussed with business representatives. But in practice, we only really know what our users do by analyzing monitored data (see Figure). UcT suggests discussing E2E test scenarios with business representatives based on monitored usage data to enable data-informed test automation choices. Another criticism comes from the fact that the UcT approach requires usage data of the tested features. Which means that these features are already released. This is true. But let’s remember that our automated tests target regression tests, i.e. the verification of the impact of changes on existing features. The UcT concept is therefore to continuously adapt the coverage of our E2E regression tests based on usage analysis. Also, a constraint and a benefit come by design from the UcT approach. It is necessary to be able to monitor the actual usage (this is the constraint). And by this observation, we can ensure a high fidelity to the actual usage by our E2E tests (this is the benefit). We build software for humans. It is now common to adopt a user-centered design approach to create useful, desirable, and delightful services. As we saw, such an approach can also be beneficial for the efficiency of our delivery process and prevent regressions in our customers’ journeys. In this article, we presented Usage-Centric Testing as an answer to the tricky questions: “What should my E2E test suite cover?” and “How to decide which E2E test scenarios to automate?”. Tools supporting the UcT approach will continue to evolve. They will take advantage of collecting user actions to improve their ability to optimize and maintain E2E tests by leveraging Data Mining and Machine Learning techniques. Bruno Legeard is a Professor of Software Engineering at the University of Franche-Comté (France), Secretary of the ISTQB ® -French Testing Board, and Scientifi c Advisor at Smartesting. Bruno has more than 20 years of expertise in Test Automation and Model- Based Testing. His research activities are currently focused on AI-powered test automation. He is the author of three books on test automation and Model-Based Testing in the industry. He is also the working-group leader for ISTQB ® MBT and ISTQB ® Acceptance Testing syllabus Joan Racenet is a software product manager working on a “Usage-Driven testing” platform, Gravity, developed by Smartesting. He started his career as a software developer and co-founded HipTest in 2015, later acquired by Smartbear in 2018. He worked on multiple software quality assurance tools, like Hiptest, Cucumber, and Zephyr.