Skip to content

Lab 09

In this lab sessions you will be practicing two advanced software testing concerns: Mutation testing and Mocking. For the exercises in this last lab session, you will be working with the PrimeNumbers project and the MockServerExercise project, both available on Gitlab.

Mutation testing

  • Mutation testing is an automated stress test for existing unit and integration tests.
  • The key thought is to inject errors in a supposed-correct program and assert if at least one tests fails, per injected error.
  • Each variant of the original software injected with an error is called a mutant.
  • If no test is able to detect the artificial program modification, the associated tests are assumed to contain zombies, i.e. pointless of insufficient tests.

Pitest

  • The pitest plugin is a convenient extension to the maven build process.
  • Installation is as simple as adding the following dependency and plugin to a pom.xml configuration:

  • Dependency:

      <dependency>
      <groupId>org.pitest</groupId>
      <artifactId>pitest-junit5-plugin</artifactId>
      <version>1.2.3</version>
      <scope>test</scope>
  </dependency>

  • Plugin:

    <!-- Mutation test report-->
<!-- HTML report available at:  target/pit-reports/index.html  -->
<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>1.17.1</version>
  <configuration>
    <mutationThreshold>65</mutationThreshold>
  </configuration>
  <!-- Mutation tests are expensive (slow), and there's no point in executing them if common tests fail. Therefore we execute them later, in the verify phase.-->
  <executions>
    <execution>
      <id>mutation-tests</id>
      <goals>
        <goal>mutationCoverage</goal>
      </goals>
      <phase>verify</phase>
    </execution>
  </executions>
</plugin>

  • Afterwards, creation of a mutation test report is triggered by the test phase, i.e. mvn clean test

  • The report is available in target/pit-reports/index.html and can be inspected with any web browser.

Your turn

Sample code setup:

  • Clone the prime number checker project: (sub-dir l11/PrimeChecker)
git clone git@gitlab.info.uqam.ca:inf2050/labs.git
  • Open the project in IntelliJ
  • Run the provided tests, using mvn clean test

All tests will pass, but does that mean your tests are good ? Let's find out:

  • Activate the mutation test plugin, by integrating PIT into your pom.xml.
  • Run the tests again and create a mutation test report: mvn clean test
  • Inspect the mutation test report in your browser.

The report will tell you about a negation mutation that can be added to your code, without the tests noticing.

  • Identify which mutation this is, i.e. which line, which changement.
  • Verify the mutation, by actually mutating the code the exact same way (your code will now count non-primes instead of primes)
  • Run the tests again.
What should happen ?

The test results should still be all passing. The mutation test results said your tests are not able to identify the modification to your source code.

Finally, it is time to fix the problem and improve your tests!

  • Revisit your tests. Identify the issue with your existing tests.
  • Either modify the existing tests or add more tests.
  • Run the mutation tests again and verify the mutation coverage is now 100%.

Integration tests

  • In the last course lecture you've learned that the maven default lifecycle defines two different testing phases:
    • test for executing unit tests.
    • verify for executing integration tests.

Unit tests vs Integration tests

Unit tests verify the behaviour of individual classes of methods. Integration tests verify the behaviour of component interplay, ensuring the application in its integrity serves its purpose.

Mad Max

One way to integration test game implementations, e.g. the XoxInternals project, is to write robot players (or if you want to use hype language you can call them "AIs").

  • Where unit tests verify individual functions of the game, competing robot players attempt to play the game and this way verify if it works reliably.
  • The simplest way of such an integration test is the "Mad Max" robot player.
    • Mad Max uses a random generator to pick a random action, whenever it is their turn.
    • Two competing Mad Max instances must eventually maneuver the game into a game over state.
    • The integration test is successful if the game ends without errors within 5 seconds of CPU time.
  • A seeded random number generator can be used to ensure deterministic execution paths of the robot players.
  • For advanced testing, 1000 game instances can be tested, each with an individual seed (e.g. simple incrementing seed value 1, 2, 3, ...)

Integration tests are also tests, and belong into the src/test/java folder structure.

  • To distinguish between standard unit tests (which must be executed at test phase) and integration tests (which must be executed in the verify phase), a special file name syntax can be used.
  • Integration tests should end with ...IT.java.
  • An additional pom.xml configuration for the test plugin then splits execution by file name: 
    <project>
  ...
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>3.5.2</version>
        <configuration>
          <excludes>
            <exclude>**/*IT.java</exclude>
          </excludes>
        </configuration>
        <executions>
          <execution>
            <id>integration-test</id>
            <goals>
              <goal>test</goal>
            </goals>
            <phase>integration-test</phase>
            <configuration>
              <excludes>
                <exclude>none</exclude>
              </excludes>
              <includes>
                <include>**/*IT.java</include>
              </includes>
            </configuration>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</project>

Your Turn

  1. Add the above maven-surfire-plugin to your XoxFrontend project.
  2. Create a new test class MadMaxIT.java.
  3. In MadMaxIT, create an @Test decorated function that runs 1000 different Xox games with two competing MadMax robot players.
  4. Verify the integration tests passes through the 1000 game instances without crashing.
    • mvn clean test must not run the integration tests.
    • mvn clean verify must run the integration tests.

You can use the below code snippets to implement the requested functionality:

  /**
 * Provides a random number between 0 and the provided value (exclusive).
 *
 * @param upperBoundExcluded as the maximum value that is guaranteed not to be reached.
 */
public int pickRandomAction(int upperBoundExcluded) {
  return Math.abs(randomNumberGenerator.nextInt()) % upperBoundExcluded;
}

/**
 * Initialize seeded random number generator. Each game test-run of two competing MadMax players
 * should use a different seed.
 */
public void initializeRandomNumberGenerator(int seed) {
  // For seed 42, will always generate:
  // -1170105035 234785527 -1360544799 205897768 ...
  randomNumberGenerator = new Random(seed);
}

/**
 * The integration test creates 1000 test games and verifies that each of them is successfully
 * finished by two competing MadMax robot players. Every game uses a different random generator
 * seed.
 */
@Test
public void madMax() {
  // Run 1000 random games. All must conclude in valid game ending
  for (int i = 0; i < 1000; i++) {
    runSeededRandomGame(i);
  }
}

private void runSeededRandomGame(int seed) {
  // TODO: implement missing functionality to finish seeded game
  long gameId = XoxGameInitiator.createNewGame(playerNames[0], playerNames[1]);
  while (!gameOver) {
    // select a random move for the current player
    XoxClaimFieldAction[] allPossibleActions =
        gameManager.getActions(gameId, playerNames[currentPlayer]);
    // pick and play the random move
    gameManager.performAction(gameId, playerNames[currentPlayer], randomAction);
    // advance current player and update game over status
    gameOver = gameManager.getRanking(gameId).isGameOver();
  }
}

Mocking

  • In the last lecture you've seen an example of a SUT dependency (database), which needed to be replaced by a Mock to facilitate testing.
  • In the following you are going to apply the same concepts on a different application, the TagCounter.
  • The use case is identical, we have a SUT (class to test), which does just one thing: Count HTML tags.

Example

This HTML sample

<html>
<h1> Hello, World! </h1>
</html>

has 4 tags:

  • <html>
  • <h1>
  • </h1>
  • </html>

Dependency illustration

Unfortunately, we cannot test our TagCounter in isolation. To instantiate, we also need a ServerFileDownloader object, which provides the html code, by downloading if from a real server: Wikipedia

---
title: SUT needs a ServerFileDownloader object
---
classDiagram
    TagCounter *--> ServerFileDownloader: has a

    class ServerFileDownloader {
        <<Class>>
        +getWebpageContent() void
    }

    class TagCounter {
        <<Class>>
        +TagCounter(ServerFileDownloader) TagCounter
        +countTags() int
    }

Your turn

  • Download the prepared ServerMockExercise project.
  • Run the application, familiarize yourself with the existing code.
  • Modify only the test code, to enable mock testing
    • Add Mockito as test dependency to the pom.xml
    • Open the prepared test class: TagCounterTest and use the annotations seen in class to replace the server dependency by a mock dependency.
    • Add a when-thenReturns statement to prepare a test class. Return a minimal webpage, e.g. the one listed above.
    • Add an assert statement to your test, to verify the TagCounter functions correctly.
    • Use a Captor to verify the Mock serverFileDownloader object is correctly invoked.