Mutation Testing with PITest

1. Overview

Software testing refers to the techniques used to assess the functionality of a software application. In this article we’re going to discuss some of the metrics used in the software testing industry, such as code coverage and mutation testing, with peculiar interest on how to perform a mutation test using the PITest library.

For the sake of simplicity, we’re going to base this demonstration on a basic palindrome function – Note that a palindrome is a string that reads the same backward and forward.

2. Maven Dependencies

As you can see in the Maven dependencies configuration, we will use JUnit to run our tests and the PITest library to introduce mutants into our code – don’t worry, we will see in a second what a mutant is. You can always lookup the latest dependency version against the maven central repository by following this link.

<dependency>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-parent</artifactId>
    <version>1.1.10</version>
    <type>pom</type>
</dependency>

In order to have the PITest library up and running, we also need to include the pitest-maven plugin in our pom.xml configuration file:

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.1.10</version>
    <configuration>
        <targetClasses>
            <param>com.baeldung.testing.mutation.*</param>
        </targetClasses>
        <targetTests>
            <param>com.baeldung.mutation.test.*</param>
	</targetTests>
     </configuration>
</plugin>

3. Project Setup

Now that we have our Maven dependencies configured, let’s have a look at this self-explanatory palindrome function:

public boolean isPalindrome(String inputString) {
    if (inputString.length() == 0) {
        return true;
    } else {
        char firstChar = inputString.charAt(0);
        char lastChar = inputString.charAt(inputString.length() - 1);
        String mid = inputString.substring(1, inputString.length() - 1);
        return (firstChar == lastChar) && isPalindrome(mid);
    }
}

All we need now is a simple JUnit test to make sure that our implementation works in the desired way:

@Test
public void acceptsPalindrome() {
    Palindrome palindromeTester = new Palindrome();
    assertTrue(palindromeTester.isPalindrome("noon"));
}

So far so good, we are ready to run our test case successfully as a JUnit test.

Next in this article, we are going to focus on code and mutation coverage using the PITest library.

4. Code Coverage

Code coverage has been used extensively in the software industry, to measure what percent of the execution paths has been exercised during automated tests.

We can measure the effective code coverage based on execution paths using tools like Eclemma available on Eclipse IDE.

After running TestPalindrome with code coverage we can easily achieve a 100% coverage score – Note that isPalindrome is recursive, so it’s pretty obvious that empty input length check will be covered anyway.

Unfortunately, code coverage metrics can sometimes be quite ineffective, because a 100% code coverage score only means that all lines were exercised at least once, but it says nothing about tests accuracy or use-cases completeness, and that’s why mutation testing actually matters.

5. Mutation Coverage

Mutation testing is a testing technique used to improve the adequacy of tests, and identify defects in code. The idea is to change the production code dynamically, and cause the tests to fail.

Good tests shall fail

Each change in the code is called a mutant, and it results an altered version of the program, called a mutation.

We say that the mutation is killed, if it can cause a fail in the tests. We also say that the mutation survived if the mutant couldn’t affect the tests behavior.

Now let’s run the test using Maven, with the goal option set to: org.pitest:pitest-maven:mutationCoverage.

We can check the reports in HTML format in the target/pit-test/YYYYMMDDHHMI directory:

• 100% line coverage: 7/7
• 63% mutation coverage: 5/8

Clearly our test sweeps across all the execution paths, thus, the line coverage score is 100%. In the other hand the PITest library introduced 8 mutants, 5 of them were killed – Caused a fail – but 3 survived.

We can check the com.baeldung.testing.mutation/Palindrome.java.html report for more details about the mutants created:

These are the mutators active by default when running a mutation coverage test:

• INCREMENTS_MUTATOR
• VOID_METHOD_CALL_MUTATOR
• RETURN_VALS_MUTATOR
• MATH_MUTATOR
• NEGATE_CONDITIONALS_MUTATOR
• INVERT_NEGS_MUTATOR
• CONDITIONALS_BOUNDARY_MUTATOR

For more details about the PITest mutators, you can check the official documentation page link.

Our mutation coverage score reflects the lack of test cases, as we can’t make sure that our palindrome function rejects non-palindromic and near-palindromic string inputs.

6. Improve the Mutation Score

Now that we know what a mutation is, we need to improve our mutation score by killing the surviving mutants.

Let’s take the first mutation – negated conditional – on line 6 as an example. The mutant survived because even if we change the code snippet:

if (inputString.length() == 0) {
    return true;
}

To:

if (inputString.length() != 0) {
    return true;
}

The test will pass, and that’s why the mutation survived. The idea is to implement a new test that will fail, in case the mutant is introduced. The same can be done for the remaining mutants.

@Test
public void rejectsNonPalindrome() {
    Palindrome palindromeTester = new Palindrome();
    assertFalse(palindromeTester.isPalindrome("box"));
}
@Test
public void rejectsNearPalindrome() {
    Palindrome palindromeTester = new Palindrome();
    assertFalse(palindromeTester.isPalindrome("neon"));
}

Now we can run our tests using the mutation coverage plugin, to make sure that all the mutations were killed, as we can see in the PITest report generated in target directory.

• 100% line coverage: 7/7
• 100% mutation coverage: 8/8

7. PITest Tests Configuration

Mutation testing may be resources-extensive sometimes, so we need to put proper configuration in place to improve tests effectiveness. We can make use of the targetClasses tag, to define the list of classes to be mutated. Mutation testing cannot be applied on all classes in a real world project, as it will be time consuming, and resource critical.

It is also important to define the mutators you plan to use during mutation testing, in order to minimize the computing resources needed to perform the tests:

<configuration>
    <targetClasses>
        <param>com.baeldung.testing.mutation.*</param>
    </targetClasses>
    <targetTests>
        <param>com.baeldung.mutation.test.*</param>
    </targetTests>
    <mutators>
        <mutator>CONSTRUCTOR_CALLS</mutator>
        <mutator>VOID_METHOD_CALLS</mutator>
        <mutator>RETURN_VALS</mutator>
        <mutator>NON_VOID_METHOD_CALLS</mutator>
    </mutators>
</configuration>

Moreover, the PITest library offers a variety of options available to customize your testing strategies, you can specify the maximum number of mutants introduced by class using the maxMutationsPerClass option for example. More details about PITest options in the official Maven quickstart guide.

8. Conclusion

Note that code coverage is still an important metric, but sometimes it is not sufficient enough to guarantee a well-tested code. So in this article we’ve walked through mutation testing as a more sophisticated way to ensure tests quality and endorse test-cases, using the PITest library.

We have also seen how to analyze a basic PITest reports, while improving the mutation coverage score.

Even though mutation testing reveals defects in code, it should be used wisely, because it is an extremely costly and time-consuming process.

You can checkout the examples provided in this article in the linked GitHub project.