Mutation Testing of Maven Project using Pitest (PIT)
Setup your system for mutation testing of a basic Java project using pitest testing tool.
Scope
In this blog, we will initially cover how to set up your system for running mutation tests. A well-known tool called Pitest or PIT will be utilized throughout this article. From there on, we will also see how to increase the mutation coverage score.
Pre-requisite
- You should be aware of the theoretical concepts of mutation testing. Check out [1] if you would like to learn about mutation testing.
- Java should be installed on your system. If you need any help with that do check out [2] regarding setup. In the current article, Visual Studio Code has been used for editing and running the code. You may however choose the editor of your choice.
- Make sure that maven is set up in your system path. In case if you face any error, I highly recommend you check this StackOverflow answer.
Create a Maven Project
- Open the VS Code
- Press Ctrl+Shift+p
- Then select “java: create java project” → “Maven, provided by Maven for Java” → “Maven-archetype-quickstart” → select a version
- Follow along and provide the package name of your choice and wait for the project to be created.
- For your reference, find GIF 1 below to understand the above-mentioned steps.
Run your Project
- At this point in time, you should ensure that your code is running correctly.
- Make sure your default java programs (App.java and AppTest.java) are up and running.
Adding Pitest to the Project
- Navigate to the Pitest website for a while. It will give you a brief insight as to what this tool does.
- Since this blog will cover mutation testing for maven, we will use this link to copy the code as shown in the snippet below. Also note that at the time of writing this article, 1.5.0 was the latest version.
- Paste this code snippet in the pom.xml file in your project directory.
<plugin>
<groupId>org.pitest</groupId>
<artifactId>pitest-maven</artifactId>
<version>1.5.0</version>
</plugin>
Running Pitest plugin
- After making necessary changes to pom.xml, you might be asked automatically to build the XML file. Select ‘Now’ and the rest will be handled by your intelligent plugins (figure 1).
- Ideally speaking, the pitest plugin should be visible under the plugins directory of your project. But for me, this did not happen.
- Go to the command line in VS Code and type
mvn install
to compile your project. - After this run PIT using the following command:
mvn eu.stamp-project:pitmp-maven-plugin:run
5. The output will be as shown in figure 2 below.
Understanding the Output
If you see the output so far, it is certainly not easy to understand. Fortunately, when we run the above code, a folder namely “target” is created in the project directory. There you can find the “pit-reports” folder where all your runs will be stored.
Go to the “pit-reports” folder and find the latest run folder. Under that folder open the “index.html” file in your web browser (Chrome, Firefox, Edge, etc.). You will see something like GIF 2. Note that Light pink shows a lack of line coverage, dark pink shows a lack of mutation coverage.
Increase the Mutation Coverage Score
As per the report generated above, we observe that the line coverage and mutation coverage are 0%. That is expected behavior since App.java and AppTest.java files are the most basic ones and do not cover anything that will kill the mutants (again, ensure you know theoretical concepts of mutation testing).
Write the following in your App.java file:
public class App{public static String hello(){ //simply returns "Hello World"
return "Hello World";
}public static void main( String[] args) {
App.hello(); //calling hello() method
}}
Modify your AppTest.java file to test the asserted value as shown below:
package com.example;
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import org.junit.Test;public class AppTest {@Test
public void shouldAnswerWithTrue()
{
assertTrue( true );
}@Test //new test case added
public void shouldReturnHelloWorld(){
String value = App.hello();
assertEquals(value, "Hello World");
}
}
Now run the pitest as you did previously and observe the HTML test report.
Light green shows line coverage, dark green shows mutation coverage. Note how the mutation score increased from 0 to 100%. The line coverage also increased to 50%. In the upcoming tutorial, we will see a relatively complex problem and work on increasing the line and mutation coverage.
Conclusion
In this tutorial, we learned how to integrate pitest (PIT) in a maven project. We also saw how to generate very convenient mutation test reports. PIT is useful for providing automated mutants and therefore we can focus on improving the quality of unit test cases that will provide higher scores for mutation coverage. This will recognize the more vulnerable pieces of the code.
Also, you can ask me a question on Twitter and LinkedIn!
Reference
[1] Rungta, K. (2021, May 10). Mutation Testing in Software Testing: Mutant Score & Analysis Example. Guru 99. https://www.guru99.com/mutation-testing.html
[2] Rajwal, S. (2021, June 10). Software Testing: Unit Test Cases for Simple Java Programs. Medium. https://medium.com/geekculture/unit-testing-of-simple-java-programs-b785a164b440
[3] S. (n.d.). STAMP-project/pitmp-maven-plugin. GitHub. Retrieved June 28, 2021, from https://github.com/STAMP-project/pitmp-maven-plugin