Testing elm

py.test Unit Tests

These testing instructions assume you have cloned the elm repository locally and installed from source.

Note: Many tests are skipped if you have not defined the environment variable ELM_EXAMPLE_DATA_PATH (referring to your local clone of `elm-examples`_)

Run the faster running tests:

py.test -m "not slow"

Running all tests:

py.test

or get the verbose test output

py.test -v

and cut and paste a test mark to run a specific test:

py.test -k test_bad_train_config

When running py.test the environment variables related to dask determine whether dask-distributed or thread pool client or serial evaluation is used (See also dask-distributed).

Longer Running Tests

Deprecated with elm-main temporarily

The elm-run-all-tests console entry point can automate running of some or all python scripts and yaml elm-main config files in elm's examples and/or the py.test unit tests.

Here is an example that is run from inside the cloned elm repository with elm-examples cloned in the current directory (see the first two arguments: ./ - cloned elm repo and examples - the location of cloned elm-examples)

ELM_LOGGING_LEVEL=DEBUG elm-run-all-tests ./ elm-examples/ --skip-pytest --skip-scripts --dask-clients SERIAL DISTRIBUTED --dask-scheduler 10.0.0.10:8786

Here is the full help on elm-run-all-tests entry point:

$ elm-run-all-tests --help
usage: elm-run-all-tests [-h] [--pytest-mark PYTEST_MARK]
                         [--dask-clients {ALL,SERIAL,DISTRIBUTED,THREAD_POOL} [{ALL,SERIAL,DISTRIBUTED,THREAD_POOL} ...]]
                         [--dask-scheduler DASK_SCHEDULER] [--skip-pytest]
                         [--skip-scripts] [--skip-configs]
                         [--add-large-test-settings]
                         [--glob-pattern GLOB_PATTERN]
                         [--remote-git-branch REMOTE_GIT_BRANCH]
                         repo_dir elm_examples_path

Run longer-running tests of elm

positional arguments:
  repo_dir              Directory that is the top dir of cloned elm repo
  elm_examples_path     Path to a directory which contains subdirectories
                        "scripts", "scripts", and "example_data" with yaml-
                        configs, python-scripts, and example data,
                        respectively

optional arguments:
  -h, --help            show this help message and exit
  --pytest-mark PYTEST_MARK
                        Mark to pass to py.test -m (marker of unit tests)
  --dask-clients {ALL,SERIAL,DISTRIBUTED,THREAD_POOL} [{ALL,SERIAL,DISTRIBUTED,THREAD_POOL} ...]
                        Dask client(s) to test: ['ALL', 'SERIAL',
                        'DISTRIBUTED', 'THREAD_POOL']
  --dask-scheduler DASK_SCHEDULER
                        Dask scheduler URL
  --skip-pytest         Do not run py.test (default is run py.test as well as
                        configs)
  --skip-scripts        Do not run scripts from elm-examples
  --skip-configs        Do not run configs from elm-examples
  --add-large-test-settings
                        Adjust configs for larger ensembles / param_grids
  --glob-pattern GLOB_PATTERN
                        Glob within repo_dir
  --remote-git-branch REMOTE_GIT_BRANCH
                        Run on a remote git branch