Scheme-robo -- automatic assessment system for Scheme exercises

Scheme-robo is an automatic assessment system for programming exercises written in the Scheme programming language. It is meant for programming courses where students write many small programs in Scheme.

Downloads

• Scheme-robo version 2.2 (138 KB)
• Scheme-robo version 2.1 (131 KB)
• Scheme-robo version 2.0 (123 KB)
• Scheme-robo version 1.0 (103 KB)
• PostScript version of the current documentation (179 KB)

Description

Scheme-robo is an automatic assessment system for programming exercises written in the Scheme programming language. It is meant for programming courses where students write many small programs in Scheme.

Basically, Scheme-robo takes as input a solution to an exercise (usually written by a student and submitted to the system in some way) and some exercise-specific configuration. Scheme-robo does various kinds of tests (e.g., test runs) on the solution and returns a number of points and a set of comments to be given to the student.

Scheme-robo is not a complete automatic assessment system. It does not handle course management tasks such as collecting points from different exercises or receiving submissions from students. Scheme-robo should be thought of as a module to be plugged in to some kind of course management system. As a minimum, the course management system needs to receive submissions from the student, send them on to Scheme-robo, maintain a database of points received for various q exercises, and give the points and comments from Scheme-robo back to the student in some form. (We are also creating a generic course-management system where Scheme-robo could be plugged in. This will also be available as free software once we get it finished.)

Scheme-robo executes student code in a metacircular Scheme interpreter. Thus, test runs are specified as Scheme procedure calls, and the actual return value is usually compared to a set of correct solutions. The comparison is usually done with a standard equality predicate. Thus, there is no need for student code to read input or print out values, and we avoid the problem of parsing output written by student code. The metacircular interpreter also gives us a ``sandbox'' where student code can be safely executed: the interpreter does not include potentially dangerous features such as file I/O.

Some other features of Scheme-robo are:

• A possibility for randomised test runs.
• Rough execution time analysis for student code, for example, to detect O(n^2) algorithms where a linear algorithm was required.
• Structural analysis of student code: Scheme-robo can do a kind of pattern-matching for Lisp-like syntax. This can be used, for example, for checking that the code conforms to a template given in the problem statement for the exercise.
• Searching for special keywords like set!, for example to prevent using them for particular exercises.
• Scheme-robo also supports multiple-choice exercises and other simple textual exercises. This is done by simple regular-expression searching that can be used to match the correct answers and give points for them.
• In addition to a standard Scheme interpreter, Scheme-robo supports the non-deterministic amb evaluator in Abelson and Sussman's Structure and Interpretation of Computer Programs (SICP).

Requirements

Scheme-robo is written as a combination of Unix shell scripts and Scheme code. The shell scripts are very simple and easily reimplemented in any other language. The Scheme code uses some of the special features provided by the Scheme implementation MzScheme.

Thus, Scheme-robo currently requires:

• a Unix operating system (though it is quite easy to port to other platforms, and the Scheme part should run as is in Windows and Macintosh platforms)
• MzScheme version 200 or above
• some sort of course-management software, as described above