Test Harness Command Line Interface

When you run your test harness from the command line, certain command line arguments have significance, and are parsed when interact.core.Harness.start() is called.

You can find a brief description of the command line arguments available by running your harness with the --help option (ex: ./my_harness.py --help). This document attempts to be a more thorough description of behavior.

Basic Usage

When you run your harness (that you created using the Galah Interact library) you use the following flags to control the behavior of the harness.

-m, --mode MODE
Use this flag to set the execution mode of the harness. The default mode is galah. See Execution Modes for more information.
-s, --set-value KEY VALUE
Use this flag (multiple times if desired) to set “configuration” values when in test mode. See Configuration Values for more information.

Examples

If we want to start a harness in test mode and let the harness guess all of the values...

./my_harness.py --mode test

If we want to start a harness in test mode and set the testables directory (where the student’s code is) to another directory...

./my_harness.py --mode test --set-value testables_directory ./student1/

If we want to start a harness in test mode and set the testables directory along with providing a fake submission object...

./my_harness.py --mode test --set-value testables_directory ./student1/ --set-value raw_submission "{'id': 'junk', 'user': 'john'}"

Execution Modes

The mode of execution (set with the --mode option) determines how the test harness will gather the information it needs to run, and how it outputs the results once they are available.

galah mode

In galah mode, when the test harness starts it will try to read in JSON from standard input, and when it finishes the output will be sent to standard output as JSON. This is not a very human-friendly process and generally you don’t want to set it to this mode during development of a test harness.

If you harness seems like it’s not responding when you start it, it’s probably running in this mode and is waiting for an entire JSON object to be placed into stdin.

This mode exists for when the test harness is being run within Galah.

test mode

In test mode, the test harness will try to guess any values it needs (and you can specify values explictly using --set-value) and then print out the results in a human-friendly way. You should always use this mode during the development of your test harness.

Configuration Values

Test harnesses need certain information in order to function properly. In galah mode, you must specify them all with JSON, but in test mode the harness will attempt to guess some values, and you can override any of the guesses by using the --set-value command line argument. If you use --set-value with any argument that expects a list or dictionary, the value you set will be assumed to be JSON and will be deserialized.

These values are available in interact.core.Harness.sheep_data from within the test harness.

Note

The below keys are case-sensitive.

KEY (DEFAULT VALUE):
DESCRIPTION
testables_directory (current directory):
The directory that contains the student’s code.
harness_directory (directory of the running test harness):
The directory that contains the test harness itself.
raw_submission (None):

A submission object with meta data on the student’s submission. In Galah, this is a dictionary with at least the following fields:

{
    "id": "ID of submission in database",
    "assignment": "ID of assignment in database",
    "user": "username of student",
    "timestamp": "submission time in ISO format"
}

If specified with –set-value, the string supplied will be assumed to be a valid JSON object and will be deserialized.

raw_assignment (None):

An assignment object with meta data on the assignment this harness is attached to. In Galah, this is a dictionary with at least the following fields:

{
    "name": "The name of the assignment",
    "due": "due date in ISO format",
    "due_cutoff": "cutoff date in ISO format",
    "hide_until": "hide until field in ISO format"
}
raw_harness (None):

A harness object with meta data about the test harness. In Galah, this is a dictionary with at least the following fields:

{
    "config": "dictionary supplied when harness was uploaded",
    "id": "ID of harness in database"
}
actions (instance of interact.core.UniverseSet):
A list of actions that the test harness should perform. This is not yet fully supported within Galah. As such there’s not full support for it in Galah Interact yet.