Software Testing Basics

During Software Construction

Gerson Sunyé gerson.sunye@univ-nantes.fr

Preamble

Figure 1. Where are we?

Plan

Introduction
Dynamic Testing
Test Cases
Unit Testing
Integration Testing
Test Automation
Conclusion

Definition

Software Testing is the activity of finding errors in a software application, i.e. that the application functions according to the user’s requirements.

Test Classification

Different criteria according to:

The quality factor: functional, performance, etc.
The level: Unit, Component, Integration, System, Acceptance
The development phase: no-regression, acceptation, etc.
The execution type: dynamic, static

Testing During Software Construction

Unit (functional, dynamic)
Integration (functional, dynamic)
Static Analysis (functional, static)

Dynamic Testing

In a nutshell:

Dynamic Testing relies on the execution of the Software Under Test to verify whether it complies with its specification.
This requires executing the same program a large amount of times to cover all relevant cases from the specification.

In practical terms

Dynamic Testing consists in:
1. executing a Script that drives the Software into a given state
2. providing Data to the Software
3. executing an Oracle that tells whether the test fails or succeeds
The Script, the Data, and the Oracle form a Test Case
A suite of Test Cases is called a Test Suite

Test Cases

Dynamic Test Cases

Figure 2. Test Case Execution

Example: The ArrayList class

How to test class ArrayList?

What should be tested?
How many Test Cases?

How to…

describe the Test Cases?
name them?
code the Test Scripts?
code the Oracles?

What should be tested?

Test behaviors, not methods!

Single responsibility: one test should be responsible for one scenario only.
- One method, multiple behaviors: multiple tests
- One behavior, multiple methods: one test

A method: add()
Behaviors:
1. Add to an empty list
2. Add several times the same element
3. Add a null element
4. etc.

How many Test Cases?

Less is more!

Test cases are also code: they should be maintainable
If there are too many tests for a single behavior, changing this behavior will broke too many tests
Tests should simple and small: when then fail, it should be easy to localize the error

How to name test cases?

There are several naming conventions

Test Case Naming Guidelines

Allow freedom of expression!

No rigid naming policy: a complex behavior cannot fit in a narrow name!
Name the test as if you were describing the scenario to a non-programmer domain-expert.

The_size_of_an_empty_list_is_zero

A_list_contains_an_element_added_previously

Remove_an_element_doesnt_remove_others

How to Describe a Test Case?

The Given-When-Then Style:

GIVEN a context
WHEN some condition
THEN expect some output

Test Cases:

Given an empty list, When an element is added, Expect list to contain element
Given an empty list, When three elements are added Expect size to be 3
Given a list with elements a, b, and c, When b is removed, Expect list should only contain a and b

How to Code a Test Script?

The script should contain 3 blocks, separated by comments or one empty lines:

Given (Input): Test preparation like creating objects or configuring mocks
When (Action): Call the method under test with the prepared objects
Then (Output): The Oracle, usually an assertion that verifies the correct behavior of the method.

// Given:
List<Integer> list = new List<>();
int expectedSize = 0;

// When:
int actualSize = list.size()

// Then:
assert actualSize == expectedSize;

If the blocks are too complex, use auxiliary (private) methods!

Test Script Guidelines

Keep it Simple, Stupid!

Write readable tests

Test scripts must maintainable.
Variable, method, and class names should be self-descriptive
Use actual and expect to name the value returned by the method under test and the expected value.

Test Script Guidelines (Cont.)

Write single flow code:

No conditional logic or loops
Split in to two tests rather than using “If” or “Case”
Tests should not contain “While”, “Do While” or “For” loops.
If test logic has to be repeated, it probably means the test is too complicated.
Call method multiple times rather than looping inside of method.

Single Flow Example

ArrayList<String> expected = new ArrayList<>();
ArrayList<String> actual = new ArrayList<>();
expected.add("one");
expected.add("three");
actual.add("one");
actual.add("two");
actual.add("three");

actual.remove("two");

assert actual.equals(expected);

Test Script Guidelines (Cont.)

Test should have no uncertainty:

All inputs should be known
Method behavior should be predictable
Expected output should be strictly defined

Do not handle exceptions

Indicate expected exception with attribute.
Catch only the expected type of exception.
Fail test if expected exception is not caught.
Let other exceptions go uncaught.

How to Code an Oracle?

Use assertions to describe the expected behavior

assert actual.equals(expected) : "After a remove, other elements remain unaltered"

Test case must contain at most one assert:

When the test fails, you will know why.

Oracle Guidelines

Use informative assertion messages

By reading the assertion message, one should know why the test failed and what to do.
Include business logic information in the assertion message (such as input values, etc.)

Oracle Guidelines (Cont.)

Good assertion messages:

Improve documentation of the code
Inform developers about the problem if the test fails
Helps to localize the error

A bad assertion messages makes developers spend more time diagnosing!

What are Assertions?

An Assertion is a statement that detects an unexpected behavior in the code.
In other terms, a boolean expression that must be true when the assertion executes.

Assertions ≠ Exceptions:

Exceptions handle conditions that are expected to occur
Assertions handle conditions that should never occur

Assertions Can Replace Tests

Some Test Cases can easily be turned into assertions.
Consider for instance the ArrayList::add() method, from OpenJDK:

public boolean add(E e) {
    ensureCapacityInternal(size + 1);  // Increments modCount!!
    elementData[size++] = e;
    return true;
}

Assertions Can Replace Tests (Cont.)

The same method, with an Assertion:

public boolean add(E e) {
    ensureCapacityInternal(size + 1);  // Increments modCount!!
    elementData[size++] = e;

    assert this.contains(e) : "An element added to the list is " +
            "contained by the same list"
    return true;
}

The behavior is verified by every test case that uses the add() method:
- unit, integration, system, etc.

Final Test Case Guidelines

Ensure Isolation

To ensure testing robustness and simplify maintenance, tests should never rely on other tests nor should they depend on the ordering in which tests are executed.

Ensure reproducibility

Multiple test executions must consistently yield the same result, provided no changes were made on the software under test.

Ensure atomicity

Tests must either pass or fail, they cannot be partially successful.

Unit Testing

Definition

Unit testing is a software testing approach that consists in testing each unit of the software under test individually.

Units

Intuitively, a unit is the smallest testable part of a software.
In procedural programming: a function or a procedure.
In object-oriented programming: a method, a class, or a component.

Goals

Check that each unit works as designed
Isolate each software part and show they are corrects

Rationale

The reliability of a system is equal to its less reliable part.

A trustworthy system is made of trustworthy units.

Tested properties

Functional behavior
Error handling
Input values check
Performance

Benefits

Avoid the «Developer’s Block»:

Developers do not fear to change existing code
Improve developer’s confidence on his code

Are a living documentation

Tests are usage examples of each unit
Work as the unit specification

Software construction simplification

Unit tests

Simplify debugging:
- Reduce the search space
- Avoid useless "prints"
Simplify evolution:
- Avoid regression
- Work as a «safety net»
Simplify integration:
- If developers trust on each unit, integration errors are easier to find

Integration Testing

Definition

Integration testing verifies whether many separately developed units work together as expected.

In object-oriented programming, integration happens at different levels: method, class, and component.

Goals

Test the interaction between units
Determine in which order the units must be tested
Identify dependencies between units

Rationale

In the real word:

different developers are assigned to different modules of the same application
each developer has its own logic, which is passed to the code
integration testing checks whether the different modules is in accordance with the specification

Modules may interact with third party tools:

integration testings checks whether the data send to that tool and the generated response are correct.

Challenges

Test management is complex: integration testing often involves interacting with the environment: operation systems, hardware, database, middleware
Integration with legacy system my require lot of changes and testing efforts.
Integration with third party software is even harder.

Benefits

Increases confidence on the developed code
Detects higher level issues: broken database, integration mistakes, etc.
Helpful when there are frequent requirement changes.

Integration Approaches

Figure 3. Dependency Tree

Big Bang: all units are tested in the same time
Bottom-up Integration: build a dependency tree, start with the leaves and go up until the root
Top-down Integration: the inverse; start with the root and go down until the leaves

Integration Testing Best Practices

Don’t wait for a complete unit testing to start coding integration tests: be flexible
Separate unit tests from integration tests: execute unit before integration tests
Log as much as possible: diagnostic is more difficult than in unit testing.

Test Automation

With Maven, JUnit, and Truth

What is Test Automation?

Roughly, the execution of a suite of test cases:

Automates some repetitive yet necessary tasks in a testing process already in place
Performs additional testing that cannot be done manually

Critical for continuous integration!

Why?

Software projects are often complex: small changes tend to affect multiple parts of the software
When adding new features, developers usually only check whether the code works as intended
The same applies to correcting errors (maintenance)

Benefices

Test cases become easy to run: every developer is able to run all the tests in their development environment.
Test cases become a part of the build: in an ideal situation, every build automatically launches all the tests and displays their results.
Every change is checked by testing the entire system.

An investment in the future

Automatic tests

Will be reused countless times during the software lifetime
- During corrective and evolutive maintenance
Force developers to write testable code
Improve design
- Tests are clients of the unit under test, helping developers to write simple interfaces

Truth

A library for simplifying assertion coding

Truth

A library for coding assertions in tests (or elsewhere)
Owned and maintained by the Guava team.
Used in the majority of the tests in Google’s own codebase.
Web site: https://truth.dev

Examples

assertThat(text).contains("testuser@google.com");
assertThat(aList)
      .containsExactlyElementsIn(anotherList)
      .inOrder();
assertThat(aMap).containsEntry("one", 1L);
assertThat(aComparable).isIn(Range.closed(1, 10));

Why ?

Prevents writing complex assertion logic
More descriptive failure messages

Failure messages

Assertions:

assert "one".equals("two") : "One and two are different";

assertThat("one").isEqualTo("two"); // Truth

Java Failure Message:

Exception in thread "main" java.lang.AssertionError: One and two are different
	at fr.unantes.sce.Main.main(Main.java:13)

Truth Failure Message:

Exception in thread "main" expected: two
but was : one
	at fr.unantes.sce.Main.main(Main.java:14)

JUnit

Java Framework for writing dynamic unit tests
Open source, available at http://www.junit.org
Origins:
- eXtreme Programming (XP)
- Smalltalk Test Framework (Kent Beck)
- First implementation by Erich Gamma 1997

Usage

JUnit is not only for Unit Testing!

Beyond its name:

JUnit is a framework for automatically executing methods (test cases), with several features
JUnit can be used to perform system, integration, unit, etc.

Example: The Interval Class

public class Interval<T extends Comparable> {
    private final T begin;
    private final T end;

    protected Interval(T begin, T end) {
        this.begin = begin;
        this.end = end;
    }

    public boolean includes(T i) {
        return i.compareTo(begin) >= 0 && i.compareTo(end) <= 0;
    }
}

Use JUnit to test the Interval class.
The Interval class represents simple intervals
- Method includes() checks if a value belongs to the interval

Testing the Interval class

Create a test class
Implement a test method that:
- Instantiates the Interval 1-10
- Check if 5 belongs to this interval

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class IntervalTest {

    @Test
    @DisplayName(value = "An interval contains a number between its boundaries")
    void inverval_contains_number_between_boundaries() {
        Interval<Integer> interval = new Interval<>(0, 10);

        assertTrue(interval.includes(5));
    }
}

Some Implementation details

JUnit automatically executes all methods adorned with @Test
Assertions.assertTrue(<exp>) makes the test fail if <exp> evaluates to false.

Improving the tests

Check if -1 and 11 do not belong to the interval

import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class IntervalTest {

    @Test
    @DisplayName(value = "An interval contains a number between its boundaries")
    void includes() {
        Interval<Integer> interval = new Interval<>(0, 10);

        assertTrue(interval.includes(5));
    }

    @Test
    @DisplayName(value = "An interval doesnt contain a number greater thans its upper boundary")
    void doesntIncludesGreaterThanUpperBoundary() {
        Interval<Integer> interval = new Interval<>(0, 10);

        assertFalse(interval.includes(11));
    }

    @Test
    @DisplayName(value = "An interval doesnt contain a number lesser than its lower boundary")
    void doesntIncludesLesserThanLowerBoundary() {
        Interval<Integer> interval = new Interval<>(0, 10);

        assertFalse(interval.includes(-1));
    }
}

Assembling common code

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import static org.junit.jupiter.api.Assertions.*;

class IntervalTest {
    private Interval<Integer> interval_1_10;

    @BeforeEach
    void setup() {
        interval_1_10 = new Interval<>(0, 10);
    }

    @Test
    @DisplayName(value = "An interval contains a number between its boundaries")
    void includes() {
        assertTrue(interval_1_10.includes(5));
    }

    @Test
    @DisplayName(value = "An interval doesnt contain a number greater thans its upper boundary")
    void doesntIncludesGreaterThanUpperBoundary() {
        assertFalse(interval_1_10.includes(11));
    }

    @Test
    @DisplayName(value = "An interval doesnt contain a number lesser than its lower boundary")
    void doesntIncludesLesserThanLowerBoundary() {
        assertFalse(interval_1_10.includes(-1));
    }
}

More implementation details

JUnit automatically executes the method adorned with @BeforeEach before each @Test method.
In the example, setup() is executed 3 times.

Parameterized tests

JUnit automatically executes methods adorned with @ParameterizedTest several times: one time for each value

import org.junit.jupiter.api.BeforeEach;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.params.ParameterizedTest;
import org.junit.jupiter.params.provider.ValueSource;

import static org.junit.jupiter.api.Assertions.*;

class IntervalTest {
    private Interval<Integer> interval_1_10;

    @BeforeEach
    void setup() {
        interval_1_10 = new Interval<>(0, 10);
    }

    @ParameterizedTest
    @ValueSource(ints = {1, 5, 10})
    @DisplayName(value = "An interval contains a number between its boundaries")
    void includes(int value) {
        assertTrue(interval_1_10.includes(value));
    }

    @ParameterizedTest
    @ValueSource(ints = {Integer.MIN_VALUE, -1, 11, Integer.MAX_VALUE})
    @DisplayName(value = "An interval doesnt contain a number outside its boundaries")
    void outsideBoundaries(int value) {
        assertFalse(interval_1_10.includes(value));
    }
}

Running Unit and Integration Tests with Maven

Build Lifecycle

Figure 4. Maven Build Lifecycle

Plugins:

Surefire: manages the Test phase
Failsafe: manages the Integration Test phase

Surefire

Executes all unit tests included in the src/main/test directory
Automatically include all test classes with the following wildcard patterns:

**/Test*.java, **/*Test.java

but not:

**/*Abstract.java, **/Abstract*.java

Test classes must be named
Test classes can be any combination of the following:
- TestNG
- JUnit (3.8, 4.x or 5.x)
- POJO

Failsafe

Executes all integration tests included in the src/main/test directory
- but this can be changed for src/main/it
Automatically include all test classes with the following wildcard patterns:

**/IT*.java, **/*IT.java, and **/*ITCase.java

Test classes can be any combination of the following:
- TestNG
- JUnit (3.8, 4.x or 5.x)
- POJO

Conclusion

It is impossible to test a program completely.
Testing cannot prove the absence of bugs.
Specifications are never final: software is always evolving

Conclusion (Cont.)

Test cases are also software: use software engineering best practices:
- modularity, factorization, reuse, etc.
Maintaining old tests up to date is as important as writing new ones.

Final Word

Tests don’t improve quality: developers do!