Interact Framework¶

Galah Interact is a framework for creating Test Harnesses that grade students’ programming assignments. For general project information, check out the project page on GitHub: https://github.com/galah-group/galah-interact-python

Galah Interact was originally created by Galah Group LLC and is licensed under the Apache license version 2.0. A copy of the license is available in the root directory of the git repo in the file LICENSE, and online at http://www.apache.org/licenses/LICENSE-2.0. Please ensure your use of this library is within your rights per that license.

Other contributers have given their valuable time to this project in order to make it better, and they are listed in the CONTRIBUTERS file in the root directory of the git repo.

`interact.core`¶

Functions and classes that are essential when using this library. Is imported by interact/__init__.py such that interact.core.x and interact.x are equivalent.

class interact.core.Harness[source]¶

An omniscient object responsible for driving the behavior of any Test Harness created using Galah Interact. Create a single one of these when you create your Test Harness and call the start() method.

A typical Test Harness will roughly follow the below format.

import interact
harness = interact.Harness()
harness.start()

# Your code here

harness.run_tests()
harness.finish(max_score = some_number)

Variables:

sheep_data – The “configuration” values received from outside the harness (either from Galah or command line arguments). See Test Harness Command Line Interface.
execution_mode – The mode of execution the harness is running in. Is set by Harness.start() and is None before it is set. For information on the different modes, check out Test Harness Command Line Interface.
tests – A dictionary mapping test functions to Harness.Test objects. This is of type ORDERED_DICT.

class FailedDependencies(max_score=10)[source]¶

A special TestResult used by Harness.run_tests() whenever a test couldn’t be run do to one of its dependencies failing.

>>> a = interact.Harness.FailedDependencies()
>>> a.add_failure("Program compiles")
>>> a.add_failure("Program is sane")
>>> print a
Score: 0 out of 10

This test will only be run if all of the other tests it depends on pass first. Fix those tests *before* worrying about this one.

 * Dependency *Program compiles* failed.
 * Dependency *Program is sane* failed.

add_failure(test_name)[source]¶: Adds a new failure message to the test result signifying a particular dependency has failed.

class Harness.Test(name, depends, func, result=None)[source]¶: Meta information on a single test.

Harness.finish(score=None, max_score=None)[source]¶: Marks the end of the test harness. When start was not initialized via command line arguments, this command will print out the test results in a human readable fashion. Otherwise it will print out JSON appropriate for Galah to read.

Harness.run_tests()[source]¶

Runs all of the tests the user has registered.

Raises:	`Harness.CyclicDependency` if a cyclic dependency exists among the test functions.

Any tests that can’t be run due to failed dependencies will have instances of Harness.FailedDependencies as their result.

Harness.start(self, arguments = sys.argv[1:])[source]¶

Takes in input from the proper source, initializes the harness with values it needs to function correctly.

Parameters:	arguments – A list of command line arguments that will be read to determine the harness’s behavior. See below for more information on this.
Returns:	`None`

See also

Test Harness Command Line Interface

Harness.student_file(filename)[source]¶: Given a path to a student’s file relative to the root of the student’s submission, returns an absolute path to that file.

Harness.student_files(*args)[source]¶: Very similar to student_file. Given many files as arguments, will return a list of absolute paths to those files.

Harness.test(name, depends=None)[source]¶: A decorator that takes in a test name and some dependencies and makes the harness aware of it all.

interact.core.ORDERED_DICT¶

An OrderedDict type. The stdlib’s collections module is searched first, then the module ordereddict is searched, and finally it defaults to a regular dict (which means that the order that test results are displayed will be undefined). This is the type of Harness.tests. This complexity is required to support older versions of Python.

alias of OrderedDict

class interact.core.TestResult(brief=None, score=None, max_score=None, messages=None, default_message=None, bulleted_messages=True)[source]¶

Represents the result of one unit of testing. The goal is to generate a number of these and then pass them all out of the test harness with a final score.

Variables:

brief – A brief description of the test that was run. This will always be displayed to the user.
score – The score the student received from this test.
max_score – The maximum score the student could have received from this test.
messages – A list of TestResult.Message objects that will be joined together appropriately and displayed to the user. Use add_message() to add to this list.
default_message – A string that will be displayed if there are no messages. Useful to easily create a “good job” message that is shown when no problems were detected.
bulleted_messages – A boolean. If True, all of the messages will be printed out in bullet point form (a message per bullet). If False, they will simply be printed out one-per-line. You can set this to False if only one message will ever be displayed to the user, otherwise bullet points usually look better.

class Message(text, *args, **kwargs)[source]¶: A message to the user. This is the primary mode of giving feedback to users.

TestResult.add_message(*args, **kwargs)[source]¶: Adds a message object to the TestResult. If a Message object that is used, otherwise a new Message object is constructed and its constructor is passed all the arguments.

TestResult.calculate_score(starting_score=None, max_score=None, min_score=None)[source]¶

Automatically calculates the score by adding up the dscore of each message and setting the score of the TestResult appropriately.

Parameters:	starting_score – This score is added to the sum of every message’s `dscore`. If `None`, `max_score` is used. max_score – The `max_score` field of the object is set to this value. If `None`, the current `max_score` is used, i.e. no change is made. min_score – If the calculated score is less than this value, this value is used instead.
Returns:	`self`. This allows you to return the result of this function from test functions.

>>> a = TestResult(max_score = 4)
>>> a.add_message("Foo", dscore = -1)
>>> a.add_message("Bar!", dscore = -5)
>>> print a.calculate_score().score
-2
>>> print a.score
-2
>>> print a.calculate_score(min_score = 0).score
0
>>> print a.calculate_score(starting_score = 8, max_score = 6).score
2
>>> print a.max_score
6

TestResult.is_failing()[source]¶

Returns:	The inverse of what `is_passing()` returns.

This function is most useful when dealing with a TestResult that you want to consider either passing or failing, and never anything in between.

See also

set_passing() and is_passing().

TestResult.is_passing()[source]¶

Returns:	`True` if the score is not `0` (note this function will return `True` if the score is negative).

This function is most useful when dealing with a TestResult that you want to consider either passing or failing, and never anything in between.

See also

set_passing() and is_failing().

TestResult.set_passing(passing)[source]¶

Parameters:	passing – Whether the test is passing or not.
Returns:	`self`. This allows you to return the result of this function directly, leading to more concise test functions.

This function sets score to either 1 (if passing is True) or 0 (if passing is False). It also sets the max_score to 1.

See also

is_passing() and is_failing().

class interact.core.UniverseSet(iterable=None)[source]¶

A special set such that every in query returns True.

>>> a = UniverseSet()
>>> "hamster" in a
True
>>> "apple sauce" in a
True
>>> 3234 in a
True
>>> "taco" not in a
False

interact.core.json_module()[source]¶

A handy function that will try to find a suitable JSON module to import and return that module (already loaded).

Basically, it tries to load the json module, and if that doesn’t exist it tries to load the simplejson module. If that doesn’t exist, a friendly ImportError is raised.

`interact.execute`¶

interact.execute.compile_program(files, flags=[], ignore_cache=False)[source]¶

Compiles the provided code files. If ignore_cache is False and the program has already been compiled with this function, it will not be compiled again.

Parameters:	files – A list of files to compile. flags – A list of flags to pass to `g++`. See `create_compile_command()` for information on how exactly these are used. ignore_cache – If `True`, the cache will not be used to service this query, even if an already compiled executable exists. See below for more information on the cache.
Returns:	A two-tuple `(compiler output, executable path)`. If the executable was loaded from the cache, the compiler output will be `None`. If the program did not compile successfully, the executable path will be `None`.

Note

Note that this function blocks for as long as it takes to compile the files (which might be quite some time). Of coures if the executable is loaded from the cache no such long wait time will occur.

This function caches its results so that if you give it the same files to compile again it will not compile them over again, but rather it will immediately return a prepared executable. The cache is cleared whenever the program exits.

interact.execute.create_compile_command(files, flags)[source]¶

From a list of files and flags, crafts a list suitable to pass into subprocess.Popen to compile those files.

Parameters:	files – A list of files to compile. flags – A list of flags to pass onto `g++`. `-o main` will always be passed after these flags.
Returns:	A list of arguments appropriate to pass onto `subprocess.Popen`.

>>> create_compile_command(["main.cpp", "foo.cpp"], ["-Wall", "-Werror])
["g++", "-Wall", "-Werror", "-o", "main", "main.cpp", "foo.cpp"]

interact.execute.default_run_func(executable, temp_dir, args=[])[source]¶

Used by the run_program() to create a Popen object that is responsible for running the exectuable.

Parameters:	executable – An absolute path to the executable that needs to be run. temp_dir – An absolute path to a temporary directory that can be used as the current working directory. It will be deleted automatically at the end of the `run_program()` function. The executable will not be in the directory. args – A list of arguments to give the executabe.

This function may be overriden to override the default run_func value used in the run_program() function.

Warning

You must pass in subprocess.PIPE to the Popen constructor for the stdout and stdin arguments.

You can use this function as a reference when creating your own run functions to pass into run_program().

interact.execute.run_program(files=None, given_input='', run_func=None, executable=None, timeout=None, args=[])[source]¶

Runs a program made up of some code files by first compiling, then executing it.

Parameters:

files – The code files to compile and execute. compile_program() is used to compile the files, so its caching applies here.
given_input – Text to feed into the compiled program’s standard input.
run_func – A function responsible for creating the Popen object that actually runs the program. Defaults to default_run_func().
executable – If you don’t need to compile any code you can pass a path to an executable that will be executed directly.
timeout – Specifies, in seconds, when process should be terminated. returncode will be None if terminated forcefully.
args – Gives arguments to the executable.

Returns:

A three-tuple containing the result of the program’s execution (stdout, stderr, returncode).

`interact.parse`¶

This module is useful when attempting to roughly parse students’ code (ex: trying to check that indentation was properly used). This module does not attempt to, and never will, try and fully parse C++. If such facilities are added to Galah Interact they will probably be added as a seperate module that provides a nice abstraction to Clang.

class interact.parse.Block(lines, sub_blocks=None)[source]¶

Represents a block of code.

Variables:	lines – A list of `Line` objects that make up this block. sub_blocks – A list of `Block` objects that are children of this block.

interact.parse.INDENT_EXCEPTED_LINES = ['public:', 'private:', 'protected:']¶: Lines of code to ignore when looking for bad indentation. See find_bad_indentation() for more information.

class interact.parse.Line(line_number, code)[source]¶

Represents a line of code.

Variables:	code – The contents of the line. line_number – The line number.

indent_level()[source]¶

Determines the indentation level of the current line.

Returns:	The sum of the number of tabs and the number of spaces at the start of the line. Iff the line is blank (not including whitespace), `None` is returned.

static lines_to_str(lines)[source]¶

Creates a single string from a list of Line objects.

Parameters:	lines – A list of `Line` objects.
Returns:	A single string.

>>> my_lines = [
    Line(1, "int main() {"),
    Line(2, "   return 0;"),
    Line(3, "}")
]
>>> Line.lines_to_str(my_lines)
"int main() {\n    return 0\n}\n"

static lines_to_str_list(lines)[source]¶

Creates a list of strings from a list of Line objects.

Parameters:	lines – A list of `Line` objects.
Returns:	A list of strings.

>>> my_lines = [
    Line(1, "int main() {"),
    Line(2, "   return 0;"),
    Line(3, "}")
]
>>> Line.lines_to_str_list(my_lines)
[
    "int main() {",
    "    return 0;",
    "}"
]

classmethod make_lines(lines, start=1)[source]¶

Creates a list of Line objects from a list of strings representing lines in a file.

Parameters:	lines – A list of strings where each string is a line in a file. start – The line number of the first line in `lines`.
Returns:	A list of line objects.

>>> Line.make_lines(["int main() {", "   return 0;", "}"], 1)
[
    Line(1, "int main() {"),
    Line(2, "   return 0;"),
    Line(3, "}")
]

interact.parse.cleanse_quoted_strings(line)[source]¶

Removes all quoted strings from a line. Single quotes are treated the same as double quotes.

Escaped quotes are handled. A forward slash is assumed to be the escape character. Escape sequences are not processed (meaning “ does not become “, it just remains as “).

Parameters:	line – A string to be cleansed.
Returns:	The line without any quoted strings.

>>> cleanse_quoted_strings("I am 'John Sullivan', creator of worlds.")
"I am , creator of worlds."
>>> cleanse_quoted_strings(
...     'I am "John Sullivan \"the Destroyer\", McGee", fear me.'
... )
"I am , fear me."

This function is of particular use when trying to detect curly braces or other language constructs, and you don’t want to be fooled by the symbols appearing in string literals.

interact.parse.find_bad_indentation(block, minimum=None)[source]¶

Detects blocks of code that are not indented more than their parent blocks.

Parameters:	block – The top-level block of code. Sub-blocks will be recursively checked. minimum – The minimum level of indentation required for the top-level block. Mainly useful due to this function’s recursive nature.
Returns:	A list of `Line` objects where each `Line` had a problem with its indentation.

Note

Lines that match (after removing whitespace) lines in INDENT_EXCEPTED_LINES will be ignored.

>>> my_block = Block(
...     lines = [
...         Line(0,  "#include <iostream>"),
...         Line(1,  ""),
...         Line(2,  "using namespace std;"),
...         Line(3,  ""),
...         Line(4,  "int main() {"),
...         Line(15, "}")
...     ],
...     sub_blocks = [
...         Block(
...             lines = [
...                 Line(5,  '    cout << "{" << endl;'),
...                 Line(6,  "    if (true)"),
...                 Line(7,  "    {"),
...                 Line(9,  "    } else {"),
...                 Line(12, "    }"),
...                 Line(13, "        pinata"),
...                 Line(14, "    return 0")
...             ],
...             sub_blocks = [
...                 Block(
...                     lines = [
...                         Line(8, "        return false;")
...                     ]
...                 ),
...                 Block(
...                     lines = [
...                         Line(10, "        return true;"),
...                         Line(11, "oh noz")
...                     ]
...                 )
...             ]
...         )
...     ]
... )
>>> find_bad_indentation(my_block)
[Line(11, "oh noz")]

interact.parse.grab_blocks(lines)[source]¶

Finds all blocks created using curly braces (does not handle two line if statements for example).

Parameters:	lines – A list of `Line` objects.
Returns:	A single `Block` object which can be traversed like a tree.

>>> my_lines = [
...     Line(0, "#include <iostream>"),
...     Line(1, ""),
...     Line(2, "using namespace std;"),
...     Line(3, ""),
...     Line(4, "int main() {"),
...     Line(5, '    cout << "Hello world" << endl;'),
...     Line(6, "    return 0"),
...     Line(7, "}")
... ]
>>> grab_blocks(my_lines)
Block(
    lines = [
        Line(0, "#include <iostream>"),
        Line(1, ""),
        Line(2, "using namespace std;"),
        Line(3, ""),
        Line(4, "int main() {"),
        Line(7, "}")
    ],
    sub_blocks = [
        Block(
            lines = [
                Line(5, '    cout << "Hello world" << endl;'),
                Line(6, "    return 0")
            ],
            sub_blocks = None
        )
    ]
)

(Note that I formatted the above example specially, it won’t actually print out so beautifully if you try it yourself, but the content will be the same)

`interact.pretty`¶

Module useful for displaying nice, grammatically correct output.

interact.pretty.craft_shell_command(command)[source]¶: Returns a shell command from a list of arguments suitable to be passed into subprocess.Popen. The returned string should only be used for display purposes and is not secure enough to actually be sent into a shell.

interact.pretty.escape_shell_string(str)[source]¶: Escapes a shell string such that it is suitable to be displayed to the user. This function should not be used to actually feed arguments into a shell as this function is not secure enough.

interact.pretty.plural_if(zstring, zcondition)[source]¶

Returns zstring pluralized (adds an ‘s’ to the end) if zcondition is True or if zcondition is not equal to 1.

Example usage could be plural_if("cow", len(cow_list)).

interact.pretty.pretty_list(the_list, conjunction='and', none_string='nothing')[source]¶

Returns a grammatically correct string representing the given list. For example...

>>> pretty_list(["John", "Bill", "Stacy"])
"John, Bill, and Stacy"
>>> pretty_list(["Bill", "Jorgan"], "or")
"Bill or Jorgan"
>>> pretty_list([], none_string = "nobody")
"nobody"

`interact.standardtests`¶

This module contains useful test functions that perform full testing on input, returning TestResult objects. These are typical tests that many harnesses need to perform such as checking indentation or checking to see if the correct files were submitted.

interact.standardtests.check_compiles(files, flags=[], ignore_cache=False)[source]¶

Attempts to compile some files.

Parameters:

files – A list of paths to files to compile.
flags – A list of command line arguments to supply to the compiler. Note that -o main will be added after your arguments.
ignore_cache – If you ask Galah Interact to compile some files, it will cache the results. The next time you try to compile the same files, the executable that was cached will be used instead. Set this argument to True if you don’t want the cache to be used.

Returns:

A TestResult object.

>>> print interact.standardtests.check_compiles(["main.cpp", "foo.cpp"])
Score: 0 out of 10

This test ensures that your code compiles without errors. Your program was compiled with g++ -o main /tmp/main.cpp /tmp/foo.cpp.

Your code did not compile. The compiler outputted the following errors:

```
/tmp/main.cpp: In function 'int main()':
/tmp/main.cpp:7:9: error: 'foo' was not declared in this scope
/tmp/main.cpp:9:18: error: 'dothings' was not declared in this scope
/tmp/main.cpp:11:19: error: 'dootherthings' was not declared in this scope

```

interact.standardtests.check_files_exist(*files, **extra)[source]¶

Checks to see if the given files provided as arguments exist. They must be files as defined by os.path.isfile().

Parameters:	files – The files to check for existance. Note this is not a list, rather you should pass in each file as a seperate arugment. See the examples below. *extra – extra parameters. If extra[“basename”] is True, then os.path.basename is applied to all filenames before printing.
Returns:	Returns a TestResult object that will be passing iff all of the files exist.

# The current directory contains only a main.cpp file.
>>> print check_files_exist("main.cpp", "foo.cpp", "foo.h")
Score: 0 out of 1

This test ensures that all of the necessary files are present.

 * You are missing foo.cpp and foo.h.

(Note that this function really does return a TestResult object, but TestResult.__str__() which transforms the TestResult into a string that can be printed formats it specially as seen above)

interact.standardtests.check_indentation(files, max_score=10, allow_negative=False)[source]¶

Checks to see if code is indented properly.

Currently code is indented properly iff every block of code is indented strictly more than its parent block.

Parameters:	files – A list of file paths that will each be opened and examined. max_score – For every improperly indented line of code, a point is taken off from the total score. The total score starts at `max_score`. allow_negative – If True, a negative total score will be possible, if False, 0 will be the lowest score possible.
Returns:	A `TestResult` object.

>>> print open("main.cpp").read()
#include <iostream>

using namespace std;

int main() {
    if (true) {
    foo();
    } else {
        dothings();
        while (false) {
    dootherthings();
        }
        cout << "{}{}{{{{{}}}{}{}{}}}}}}}}{{{{"<< endl;
}
return 0;
}

>>> print open("foo.cpp").read()
#include <iostream>

using namespace std;

int main() {
return 0;
}

>>> print check_indentation(["main.cpp", "foo.cpp"])
Score: 6 out of 10

This test checks to ensure you are indenting properly. Make sure that every time you start a new block (curly braces delimit blocks) you indent more.

 * Lines 14, 13, and 10 in main.cpp are not indented more than the outer block.
 * Line 5 in foo.cpp is not indented more than the outer block.

`interact.unittest`¶

This module contains very useful functions you can use while unittesting student’s code.

Note

In order to use the unittest module, you need to make sure that you have SWIG installed, and that you have Python development headers installed, both of which are probably available through your distribution’s package manager (apt-get or yum for example).

exception interact.unittest.CouldNotCompile(message, stderr)[source]¶

Exception raised when a student’s code could not be compiled into a single library file.

Variables:	message – A short message describing the exception. stderr – The output that was received through standard error. This is output by `distutils.core.setup`.

interact.unittest.load_files(files)[source]¶

Compiles and loads functions and classes in code files and makes them callable from within Python.

Parameters:	files – A list of file paths. All of the files will be compiled and loaded together. These must be absolute paths, see `Harness.student_files`.
Returns:	A `dict` where every file that was passed in is a key in the dictionary (without its file extension) and the value is another `dict` where each key is the name of a function or class in the file and the value is a callable you can use to actually execute or create an instance of that function or class.
Raises:	`EnvironmentError` if swig is not properly installed.
Raises:	`CouldNotCompile` if the student’s code could not be compiled into a library file.

Warning

During testing, oftentimes the execution of loaded code’s main() function failed. We haven’t determined what the problem is yet so for now don’t use this function to test main() functions (the interact.execute module should work well instead).

>>> print open("main.cpp").read()
#include <iostream>

using namespace std;

class Foo {
    int a_;
public:
    Foo(int a);
    int get_a() const;
};

Foo::Foo(int a) : a_(a) {
    // Do nothing
}

int Foo::get_a() const {
    return a_;
}

int bar() {
    Foo foo(3);
    cout << "foo.get_a() = " << foo.get_a() << endl;
    return 2;
}

int main() {
    return 0;
}
>>> students_code = interact.unittest.load_files(["main.cpp"])
>>> Foo = students_code["main"]["Foo"]
>>> bar = students_code["main"]["bar"]
>>> b = Foo(3)
>>> b.get_a()
3
>>> rvalue = b.bar()
foo.get_a() = 3
>>> print rvalue
2

If you want to test a function that prints things to stdout or reads from stdin (like the bar() function in the above example) you can use the interact.capture module.

interact.unittest.swig_path = None¶: The absolute path to the swig executable. When this module is imported, the environmental variable PATH is searched for a file named swig, this variable will be set to the first one that is found. This variable will equal None if no such file could be found.

`interact.capture`¶

This module provides the tools for easily running a Python function in a seperate process in order to capture its standard input, output, and error.

class interact.capture.CapturedFunction(pid, stdin_pipe, stdout_pipe, stderr_pipe, returnvalue_pipe)[source]¶

The type of object returned by capture_function(). Provides access to a captured function’s stdin, stdout, stderr, and return value.

Variables:

pid – The process ID of the process that is running/ran the function.
stdin – A file object (opened for writing) that the captured function is using as stdin.
stdout – A file object (opened for reading) that the captured function is using as stdout.
stderr – A file object (opened for reading) that the captured function is using as stderr.
return_value – Whatever the function returned. Will not be set until CapturedFunction.wait() is called. Will contain the value CapturedFunction.NOT_SET if it has not been set by a call to CapturedFunction.wait().

The correct way to check if return_value is set is to compare with CapturedFunction.NOT_SET like so:

if my_captured_function.return_value is CapturedFunction.NOT_SET:
    print "Not set yet!"
else:
    print "It's set!"

NOT_SET = <interact.capture._NotSet instance>¶: A sentinel value used to denote that a return_value has not been set yet.

wait()[source]¶

Blocks until the process running the captured function exits (which will be when the function returns). Sets return_value.

If the function raised an exception, this function will raise that exception.

interact.capture.capture_function(func, *args, **kwargs)[source]¶

Executes a function and captures anything it prints to standard output or standard error, along with capturing its return value.

Parameters:	func – The function to execute and capture. args,*kwargs – The arguments to pass to the function.
Returns:	An instance of `CapturedFunction`.

>>> def foo(x, c = 3):
...     print x, "likes", c
...     return x + c
>>> a = capture_function(foo, 2, c = 9)
>>> a.stdout.read()
"2 likes 9\n"
>>> a.wait()
>>> print a.return_value
11

Interact Framework¶

`interact.core`¶

`interact.execute`¶

`interact.parse`¶

`interact.pretty`¶

`interact.standardtests`¶

`interact.unittest`¶

`interact.capture`¶

Table Of Contents

Previous topic

Next topic

This Page