Error and invalid state handling in Robolectric

Sometimes when developing code there are times when calling a method with certain parameters or while the program is in a certain state should be considered a programming error (see android.media.MediaPlayer for a good example). Although it is sometimes possible to write code in such a way that these errors can be caught by the compiler, this is not always the case, and when we are dealing with a third-party API (such as Android) we have to work with what we are given.

Where it is not possible/practical for code to be structured in a way that a certain error can be caught be the compiler, the next best thing is for the code to throw an exception at the earliest opportunity (fail fast), rather than have the invalid parameter or state cause issues later down the track. This policy sets a standard going forward for how we can do this in Robolectric in a way to help Android developers write more robust code.

Background

This policy has been developed out of experience in developing the ShadowMediaPlayer shadow. MediaPlayer is one of the most complicated and important components of the Android environment, and hence its shadow serves as a good reference implementation for the policy outlined below. Subsequent experience with other shadows (eg for the Looper) revealed that the same policy might also be usefully applied to many other shadows too - hence the motivation for producing this policy.

Types of programming error

There are two distinct types of programming error that you could encounter while working on a project using Robolectric:

1. Programming error in the application code.
2. Programming error in the test code.

These two cases need to be treated differently, as discussed below.

Error in application code

Your application may have programming errors in the way that it uses the Android API. Robolectric has the potential to help in checking for such errors through its shadow implementations of the Android API.

Three levels of error handling can be distinguished:

1. SILENT: In this mode, programming errors are silently ignored by the shadow implementation. The justification for this mode is that, historically, shadow implementations have started off simple (perhaps empty stubs) and often behave in this way. The SILENT error handling mode is provided (where appropriate) to preserve backward compatibility with such implementations. It should be considered a stop-gap until such time as the AUT and tests can be improved to handle one of the two more rigorous error handling modes below.
2. EMULATE: In this mode, programming errors are handled (as far as is possible) in the same way that Android would handle them under the same circumstances. Typically (though not always) Android handles programming errors by throwing an appropriate exception. Typically (though not always) this exception will be unchecked (eg, IllegalStateException or IllegalArgumentException). This mode would be appropriate for checking that the AUT will gracefully handle the error situation should it ever occur.
3. ASSERT: In this mode, programming errors are indicated to the test harness by throwing an assertion with a descriptive error message. The theory behind this mode is that it is better to eliminate programming errors from your AUT than it is to try and handle them. The thrown exception will be subsequently caught & reported by the test harness (JUnit) as a test failure, and the error message will hopefully be useful enough for the developer to track down the cause of the programming error and eliminate it.

Ideally, only legacy tests would rely on the SILENT mode and most tests would use at least the EMULATE mode to test their AUT. Ideally we'd all use the ASSERT mode so that our tests help us to eliminate all programming errors. It can be an ongoing discussion as to which of these modes are required and which one should be the default.

For examples of this, please see any of ShadowMediaPlayer's playback method implementations, which implement three levels of error handling along the above lines.

Error in test code

This considers the case where a test itself contains what constitutes a programming error.

In this case, the best course of action is to do what you would do in any application: throw an appropriate runtime exception (eg, IllegalStateException or IllegalStateException) with an appropriate error message. It is not appropriate to assert because in the test harness assertions are caught and reported as bugs in the AUT.

Example: see doSetDataSource() in ShadowMediaPlayer - ShadowMediaPlayer requires that you pre-populate the media source information before setDataSource() is called. If you do not do this, it will not know the media information to associate with the given data source when the AUT invokes it. Such is not an error in the AUT, but in the test itself. Hence when the shadow detects that this has happened it throws an IllegalStateException.