Mutation Testing of Maven Project using Pitest (PIT)

Setup your system for mutation testing of a basic Java project using pitest testing tool.

Image Credits: testbytes from Pixbay

Scope

In this blog, we will initially cover how to set up your system for running mutation tests. A well-known tool called Pitest or PIT will be utilized throughout this article. From there on, we will also see how to increase the mutation coverage score.

Pre-requisite

• You should be aware of the theoretical concepts of mutation testing. Check out [1] if you would like to learn about mutation testing.
• Java should be installed on your system. If you need any help with that do check out [2] regarding setup. In the current article, Visual Studio Code has been used for editing and running the code. You may however choose the editor of your choice.
• Make sure that maven is set up in your system path. In case if you face any error, I highly recommend you check this StackOverflow answer.

Create a Maven Project

1. Open the VS Code
2. Press Ctrl+Shift+p
3. Then select “java: create java project” → “Maven, provided by Maven for Java” → “Maven-archetype-quickstart” → select a version
4. Follow along and provide the package name of your choice and wait for the project to be created.
5. For your reference, find GIF 1 below to understand the above-mentioned steps.

GIF 1: Creating a Maven Project in Visual Studio Code

Run your Project

1. At this point in time, you should ensure that your code is running correctly.
2. Make sure your default java programs (App.java and AppTest.java) are up and running.

Adding Pitest to the Project

1. Navigate to the Pitest website for a while. It will give you a brief insight as to what this tool does.
2. Since this blog will cover mutation testing for maven, we will use this link to copy the code as shown in the snippet below. Also note that at the time of writing this article, 1.5.0 was the latest version.
3. Paste this code snippet in the pom.xml file in your project directory.

<plugin>
    <groupId>org.pitest</groupId>
    <artifactId>pitest-maven</artifactId>
    <version>1.5.0</version>
 </plugin>

Running Pitest plugin

Figure 1: Automatic pop-up after pom.xml is updated

1. After making necessary changes to pom.xml, you might be asked automatically to build the XML file. Select ‘Now’ and the rest will be handled by your intelligent plugins (figure 1).
2. Ideally speaking, the pitest plugin should be visible under the plugins directory of your project. But for me, this did not happen.
3. Go to the command line in VS Code and typemvn install to compile your project.
4. After this run PIT using the following command:

mvn eu.stamp-project:pitmp-maven-plugin:run

5. The output will be as shown in figure 2 below.

Figure 2: After running Pitest command for mutation testing

Understanding the Output

If you see the output so far, it is certainly not easy to understand. Fortunately, when we run the above code, a folder namely “target” is created in the project directory. There you can find the “pit-reports” folder where all your runs will be stored.

Figure 3: Every time you run the Pitest, more folders will be created under “pit-reports” as shown on the left-hand side of the picture.

Go to the “pit-reports” folder and find the latest run folder. Under that folder open the “index.html” file in your web browser (Chrome, Firefox, Edge, etc.). You will see something like GIF 2. Note that Light pink shows a lack of line coverage, dark pink shows a lack of mutation coverage.

GIF 2: Mutation test report generated by Pitest for your default (hello world) maven project

Increase the Mutation Coverage Score

As per the report generated above, we observe that the line coverage and mutation coverage are 0%. That is expected behavior since App.java and AppTest.java files are the most basic ones and do not cover anything that will kill the mutants (again, ensure you know theoretical concepts of mutation testing).

Write the following in your App.java file:

public class App{
public static String hello(){ //simply returns "Hello World"
return "Hello World";
}
public static void main( String[] args) {
App.hello();    //calling hello() method
}
}

Modify your AppTest.java file to test the asserted value as shown below:

package com.example;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import org.junit.Test;
public class AppTest {
@Test
public void shouldAnswerWithTrue()
{
assertTrue( true );
}
@Test      //new test case added
public void shouldReturnHelloWorld(){
String value = App.hello();
assertEquals(value, "Hello World");
}
}

Now run the pitest as you did previously and observe the HTML test report.

GIF 3: Mutation coverage score increased to 100% after adding new test cases

Light green shows line coverage, dark green shows mutation coverage. Note how the mutation score increased from 0 to 100%. The line coverage also increased to 50%. In the upcoming tutorial, we will see a relatively complex problem and work on increasing the line and mutation coverage.

Conclusion

In this tutorial, we learned how to integrate pitest (PIT) in a maven project. We also saw how to generate very convenient mutation test reports. PIT is useful for providing automated mutants and therefore we can focus on improving the quality of unit test cases that will provide higher scores for mutation coverage. This will recognize the more vulnerable pieces of the code.

Also, you can ask me a question on Twitter and LinkedIn!

Reference

[1] Rungta, K. (2021, May 10). Mutation Testing in Software Testing: Mutant Score & Analysis Example. Guru 99. https://www.guru99.com/mutation-testing.html

[2] Rajwal, S. (2021, June 10). Software Testing: Unit Test Cases for Simple Java Programs. Medium. https://medium.com/geekculture/unit-testing-of-simple-java-programs-b785a164b440

[3] S. (n.d.). STAMP-project/pitmp-maven-plugin. GitHub. Retrieved June 28, 2021, from https://github.com/STAMP-project/pitmp-maven-plugin