Contract-based testing |
![]() ![]() |
General idea
Testing a software system completely automatically (without any intervention from the user) is only possible when the specification of the software is available and when we can automatically check that the implementation conforms to the given specification. Contracts fulfill both these requirements: they contain the intended semantics of the software, are integrated with the source code, and are executable. Hence, they can be used as an oracle (the part of the testing system that decides if a test case has passed or failed).
Based on this insight, in a simplified version, testing a routine becomes a succession of the following steps:
- Generate the target and the argument objects of the call. (This is done in a random manner. See section Random input generation for more information.)
- If the precondition of the routine is satisfied, we execute its body.
- If a contract violation occurs (in the body of the tested routine or further down the call chain), we have found a bug. If no such violation occurs, the test case has passed.
Example
Let's suppose we are testing the following routine:
class MATH_OPERATIONS ... sqrt (x : REAL): REAL is require x >= 0 do -- Implementation ensure abs (Result * Result - x) <= epsilon end ... endIn order to call the routine, we must first create a target object (an instance of the enclosing class of the routine, MATH_OPERATIONS) and an argument (a real number). If our input value generator provides us value -1, the precondtion is violated, so we can't run the routine. We call this an invalid test case. However, let's suppose that our input data generator gives us value 9 for the argument. This satisfies the precondition, se we execute the body. Supposing that routine sqrt calls other routines, if any contract is violated in those routines, we have found a bug. Also, if routine sqrt does not fulfill its postcondition or breaks the class invariant, we will again conclude that there is a bug in its implementation.
All these operations are performed by AutoTest. A user must only specify that he wants to test class MATH_OPERATIONS, and all this class's routines will be tested in the manner illustrated by the example of the sqrt routine. When testing is over, the user will see the results of the test cases, as illustrated in the next section.
Test result types
AutoTest classifies the outcomes of running test cases into the following categories:
![]() ![]() ![]() ![]() |