Testing

This section covers running and writing tests for the h codebase.

Running the tests, linters and code formatters

To run the unit tests (both backend and frontend) run:

make test

To run the functional tests:

make functests

To format your code correctly:

make format

To run the linter:

make lint

For many more useful make commands see:

make help

Running the backend tests only

To run the backend test suite only call tox directly. For example:

# Run the backend unit tests:
tox

# Run the backend functional tests:
tox -qe functests

# Run only one test directory or test file:
tox tests/h/models/annotation_test.py
tox -qe functests tests/functional/api/test_profile.py

# To pass arguments to pytest put them after a `--`:
tox -- --exitfirst --pdb --failed-first tests/h
tox -qe functests -- --exitfirst --pdb --failed-first tests/functional

# See all of pytest's command line options:
tox -- -h

Running the frontend tests only

To run the frontend test suite only, run the appropriate test task with gulp. For example:

make gulp args=test

When working on the front-end code, you can run the Karma test runner in auto-watch mode which will re-run the tests whenever a change is made to the source code. To start the test runner in auto-watch mode, run:

make gulp args=test-watch

To run only a subset of tests for front-end code, use the --grep argument or mocha’s .only() modifier.

make gulp args=test-watch --grep <pattern>

Writing tests

Sean Hammond has written up a guide to getting started running and writing our tests, which covers some of the tools we use (tox and pytest) and some of the testing techniques they provide (factories and parametrization).

Unit and functional tests

We keep our functional tests separate from our unit tests, in the tests/functional directory. Because these are slow to run, we will usually write one or two functional tests to check a new feature works in the common case, and unit tests for all the other cases.

Using mock objects

The mock library lets us construct fake versions of our objects to help with testing. While this can make it easier to write fast, isolated tests, it also makes it easier to write tests that don’t reflect reality.

In an ideal world, we would always be able to use real objects instead of stubs or mocks, but sometimes this can result in:

• complicated test setup code
• slow tests
• coupling of test assertions to non-interface implementation details

For new code, it’s usually a good idea to design the code so that it’s easy to test with “real” objects, rather than stubs or mocks. It can help to make extensive use of value objects in tested interfaces (using collections.namedtuple from the standard library, for example) and apply the functional core, imperative shell pattern.

For older code which doesn’t make testing so easy, or for code that is part of the “imperative shell” (see link in previous paragraph) it can sometimes be hard to test what you need without resorting to stubs or mock objects, and that’s fine.