Testing¶
ChimeraX currently has both a hand-rolled test suite and a Pytest suite; it is preferred that additional tests use Pytest.
pytest
¶
The pytest
test suite can be invoked from the command line, but it is easier to invoke
it using ChimeraX’s top level Makefile
. To run the tests, simply run:
make pytest
in the root directory. This runs three rounds of tests:
tests/test_env.py
with both Python and the ChimeraX binary, to ensure there’s no difference between running ChimeraX and callingpython -m chimerax.core
.Wheel tests; i.e. all tests that are marked with
@pytest.mark.wheel
All other tests, using a production session produced in
conftest.py
In the latter two cases, the appropriate tests/test_imports_{app,wheel}.py
is run before
the rest of the suite, since if a module is unimportable then that is a critical error that
must be addressed before other failures.
Test Order¶
In general it is good practice if tests do not depend on each other, but in our case different runs of the test suite do have an order. Wheel tests must be run first, for several reasons:
Creating the session that is used for testing the ChimeraX app changes values found in the top-level
chimerax
namespaceDuring session creation, a global
chimerax.core.toolshed.Toolshed
object is produced, which has its own problems detailed below
It would be possible to run these tests at the same time if these issues were addressed.
Configuration¶
The configuration for pytest
is found in the top level pyproject.toml
.
Pytest is configured to look for tests in both the top-level tests
folder and any
bundle’s tests
folder.
Custom markers include:
wheel
: used to mark tests that should be run in the same environment that would be present if a user had downloaded the ChimeraX PyPi package.
Please send email to chimerax-programmers@cgl.ucsf.edu
before changing the configuration.
Uploading Test Data¶
ChimeraX deals with a huge variety of data, much of which is held in large files. To avoid making the
ChimeraX repo too large, test data is not stored in the repo. It is instead stored on Plato, the RBVI’s
server cluster. To add test data to Plato, place it in the test-data
folder, which is next to the
prereqs
folder, in a subdirectory that has the same name as the bundle directory in the ChimeraX
repository.
Downloading Test Data¶
There is a script at utils/sync-test-data.sh
that will download test data from Plato given the
name of a bundle’s folder, e.g.
./utils/sync-test-data.sh -b dicom
will download the test data for the dicom
bundle to src/bundles/dicom/tests/data
.
Adding Tests¶
First, add a folder called tests
in your bundle’s directory. Any Python file in this directory
named test_*.py
or *_test.py
will be picked up and run by Pytest. Then, in those files,
any function that starts with test_
will be run.
def test_something():
# test code here
If your test requires a session, the recommended way to do this is to use the test_production_session
fixture.
There is no need to import this fixture, as it is automatically available in all test files.
def test_something(test_production_session):
# test code here
If you need to parametrize the test to run over several different cases, or need to mark the test as for the wheel,
you’ll need to import pytest
import pytest
@pytest.mark.parametrize('input,expected', [('a', 'b'), ('c', 'd')])
def test_something(test_production_session, input, expected):
# test code here
If your test uses the data directory, you can enumerate test cases using this pattern so that when the test case is printed in the Pytest log it shows just the name of the file or folder and no the whole absolute path to the data:
import os
import pytest
test_data_dir = os.path.join(os.path.dirname(__file__), 'data')
test_cases = os.listdir(test_data_dir)
@pytest.mark.parametrize('data', test_cases)
def test_something(test_production_session, data):
test_file = os.path.join(test_data_dir, data)
# test code here
Finally, if you have a set of tests that do require data, it’s best to mark those tests to be skipped if the data is not present.
import os
import pytest
test_data_dir = os.path.join(os.path.dirname(__file__), 'data')
if not os.path.exists(test_data_dir):
pytest.skip("Skipping <BUNDLE> tests because test data is unavailable.",
allow_module_level=True,
)
test_cases = os.listdir(test_data_dir)
@pytest.mark.parametrize('data', test_cases)
def test_something(test_production_session, data):
test_file = os.path.join(test_data_dir, data)
# test code here
Wheel Import Tests¶
The wheel import tests attempt to import every module in ChimeraX that is destined for inclusion in the
PyPi package. The list of modules that are not included can be found in utils/wheel/filter_modules.py
.
The wheel is created by copying the chimerax
directory out of a built ChimeraX and then running that
script to delete excluded modules. To keep data centralized, add exclusions to that file instead of
tests/test_imports_wheel.py
.