We are the Dev Teams of
  • brands
  • ebay_main
  • ebay
  • mobile
<
>
BLOG

Exception testing pitfalls

by Manuel Kaufmann
in How We Do It

Exceptions are an essential part of modern Java applications. Whether they communicate errors to users of an API or bubble up through the layers of code and are translated by the GUI layer into human readable error notifications. Exceptions are important business logic and thus worth testing, even when they are sometimes just tested to ensure the magic 100% test coverage is reached.

JUnit offers various methods to test exception handling. Before JUnit 4 the way to test exceptions was a cumbersome try/catch block pattern. In it the Assert class's fail methods would throw an AssertionError in the event of the code progressing after the expected exception throwing code:

public void testMyExpectedException() {
    prepareTest();

    try {
        service.causeException();
        Assert.fail("Test failed - MyExpectedException was not thrown");
    } catch (MyExpectedException mee) {
        // The test passed when the exception was thrown
    }
}

With JUnit v4.x came the @Test annotation and its optional parameter ‘expected’, together they provide a simple way to test whether an exception was thrown or not. The parameter takes classes extending Throwable as an argument that will determine whether the annotated method threw the right exception or one at all. Using this technique the unit test code from above would be reduced to this:

@Test(expected = MyExpectedException.class)
public void testMyExpectedException() {
  prepareTest();
  service.causeException();
}

This method works well under two conditions. The first is that your test preparations do not throw the expected exception before the execution of the actual tested code. In this case the test stops immediately and it cannot be confirmed if the code to be tested was the cause or if it was run at all. Unlikely with uncommon exceptions but more likely when testing for common ones like the NullPointerException. The second condition is that not much detail can be validated, for example it’s not possible to verify the exceptions message and in other cases you miss additional specified information like the http response status code in Jerseys WebApplicationException. Since version 4.7 JUnit offers the ExpectedException rule to handle the latter case.

The rule is initialized by expecting no exception to be thrown in the classes unit tests by default:

@Rule
public ExpectedException thrown = ExpectedException.none();

Tests focusing on the exception’s details can use the thrown object now to set their expectancies, while for all others nothing changes. The various expect methods of the rule are great for cases where you are not just interested in the type of the exception but their message and/or cause as well. One expect method can even be called with a Hamcrest matcher argument to analyze the exception even further.

@Test
public void testMyExpectedException() {
  prepareTest();

  thrown.expect(MyExpectedException.class);
  thrown.expect(is(instanceOf(RuntimeException.class)));
  thrown.expectMessage("Important reason");
  thrown.expectCause(is(new OriginalThrownException()));

  service.causeException();
}

The ExpectedException rule is convenient for verifying similar exceptions with different messages. However, similar to the @Test annotation, some expertise is essential to write this test to ensure that the exception is acutally tested and not just the preparation. For example, with common exceptions like the NullPointerException it is possible that it may be thrown by the test set up code and thus rendering the test useless. Another problem can be the lack of options to verify the message of nested exceptions.

Another pitfall when testing exceptions has been found when working with mocks. Unit tests covering exceptions are evaluated by JUnit in a try/catch block (for both the @Test annotation and the ExpectedException rule) that catches the exception and checks for the expected conditions. Thus the code will fail the test when the wrong or no exception is thrown. For example, starting with a typical test pattern with the mock Framework EasyMock.

@Test
public void testMyExpectedException() {
  prepareAndSetupMocks();
  replayAll();

  // for this test the rule expects an exception to be thrown
  thrown.expect(MyExpectedException.class);

  // code to test that now should throw an exception
  service.causeException();

  // never called
  verifyAll();
}

A common order while using EasyMock is to configure the expected mocks behavior after creating or resetting them. As seen in the example code above, replayAll() sets them into replay state afterwards. In this state the code to be tested can interact with the mocks. But before the service to test is run, the expectations on the ExpectedException rule must be set. This prevents the rule's exception catching mechanisms from being triggered too early by the test setup code.

To confirm everything is working as intended the verifyAll() call step checks all mocks to make sure their configured expectations are met. But as previously mentioned JUnit stops the execution flow as soon as the tested code throws the exception by catching it in a try/catch block and preventing everything afterwards to be called. That means for our example above the EasyMock's verifyAll() method is never called on the mocks and so the information about the recorded behavior is lost.

With this in mind, what counter measures could be used? The obvious and most logical seems to be using a precise try/finally block. However this solution appears to be excessive, so other measures may be more suitable. One option is to put the verification step into an @After annotated method to make sure the mocks are verified. But that seems to be an anti-pattern - what if some tests in the same testing class use the mocks only partially or are not interested in them at all?

The EasyMockVerification rule

The conclusion is that the most convenient result was found by combining JUnit's test rule mechanisms with a useful feature of EasyMock. By using EasyMockSupport it is easy to access the registered mocks in the form of a list of IMocksControl instances. While a rule can not simply run verifyAll() in the scope of the current test method after Junit's exception catching, it can certainly imitate this functionality when having access to the mocks. The usage pattern is similar that of JUnit's ExpectedException. The rule is initialized expecting nothing to be verified by default:

public class ExceptionUnitTest extends EasyMockSupport {
  @Rule
  public ExpectedException thrown = ExpectedException.none();
  @Rule
  public EasyMockVerificationRule verifier = EasyMockVerificationRule.none();

  // Tests
}

Only tests which need the rule’s functionality would pass the list of IMockControl objects to the current rule. They are exposed by EasyMockSupport via the protected "controls" field. The number of code lines stays the same when using this technique since the rule renders the call of verifyAll() redundant:

@Test
public void testMyExpectedException() {
  prepareAndSetupMocks();

  thrown.expect(MyExpectedException.class);
  verifier.verify(controls);
  replayAll();

  service.causeException();
  // no verifyAll() necessary
}

The rule itself is designed to be as simple as possible. As seen in the code already, the rule is instantiated with an empty list of controls and a method to set the mock controls:

public class EasyMockVerificationRule implements TestRule {

  private List<IMocksControl> controls = Collections.emptyList();

  private EasyMockVerificationRule() {}

  public static EasyMockVerificationRule none() {
    return new EasyMockVerificationRule();
  }

  public void verify(final List<IMocksControl> controls) {
    this.controls = controls;
  }

  // apply method code follows down below
}

Finally the overridden apply method is used to return a wrapper around the given Statement base parameter. Within it a try/finally block validation of the mocks is done by calling verify on each control.

@Override
public Statement apply(final Statement base, final Description description) {
  return new Statement() {

    @Override
    public void evaluate() throws Throwable {
      try {
        base.evaluate();
      }
      finally {
        for (final IMocksControl control : controls) {
          control.verify();
        }
      }
    }

  };
}

This concept is still being tested and works quite well so far. Potentially the next step is to unite the functionality of both rules by delegating the verification functionality to the ExpectedExceptionRule and reduce the lines of code even further.

What about you? Do you use similar or completely different concepts to verify your mocks? Have you tried other frameworks in your code, like the catch-exception library? Maybe you still prefer to use try/finally? Or are Java 8 lambdas already your way to go? Please enter your ideas or any suggestions on how we can improve the ExpectedExceptionRule in the comments box below.

testing, programming, exceptions, junit, easymock

?>