This page is being prepared...

ReCodEx is an open-source system for automated evaluation of coding assignments. It is being developed at School of Computer Science, Faculty of Mathematics and Physics, Charles University in Prague (Czechia). Its main objective is to assist with practical coding classes, evaluating lab assignments, homework, and lab tests.

ReCodEx is implemented as a distributed application with a web interface. Its core module implements business logic, manages data, coordinates submission evaluation, handles security, and exposes a REST API for both the client and all other modules. The web client is a single-page application implemented in React. An additional CLI application is provided to simplify scripting. Communication between the core module and backend workers (which are responsible for evaluating the submissions) is handled by a ZeroMQ broker, which is also responsible for queuing the evaluation jobs. To provide a direct connection between the broker and web clients, a separate Monitor module is associated with the broker, providing a WebSocket API for progress reporting (so the UI can visualize the evaluation in real-time). The data are stored in a MySQL database, except for the files (submissions, testing data, ...), which are stored on the file system.

Course organization

The key organizational entity in ReCodEx is called group. A group has members (students of the same lab group or enrolled for a particular course) and assignments that members can solve. The group also defines grading (e.g., how many points the members need to collect from the assignments). One student may be a member of multiple groups (as students usually have multiple courses).

Groups form a tree-like hierarchy. Each group has exactly one parent, except for the root group (which corresponds to an instance, the institution that uses ReCodEx). Besides regular groups, ReCodEx also supports organizational groups, which cannot have members or assignments. Organizational groups act like folders in a file system (parents for other groups), so they help in creating the overall hierarchy. An example of group hierarchy is depicted on the following figure:

The 1st level group (note that the root group is not visualized here) corresponds to a Programming course. The second level represents a semester (which is practical, so we can archive all groups of a semester once it concludes), and the two groups at the third level correspond to two lab groups (one being taught on Friday, the other on Wednesday).

In the UI, a user needs to select a group for most interactions (e.g., to list its assignments, its members, ...). This is usually done on the Dashboard or the instance page. However, since this is a frequent operation, a quick selection can be performed by clicking the top panel, where the selected group is also displayed. Let us note that the name in the figure is hierarchical (i.e., containing the names of parent groups, similarly to an absolute path to a file).

Teachers, who need to access the group, have to be associated with a group as

• administrators (users with full control),
• supervisors (who can manage only the students, assignments, and solutions),
• or observers (read-only access).

Administrators are special in two ways -- their names are also displayed in the group listings, and administrator privileges are transitively applied on all sub-groups (unlike supervisors or observers). The teachers' affiliations can be managed on the Group Info page.

Solving assignments

The assignments are instances of exercises, where an exercise is like a template for evaluation (holding the specification, tests, and resource limits), and an assignment adds additional group-specific constraints (like max. amount of points, or deadlines for students). When created, an assignment holds a copy of all exercise data, so it is not affected when exercise changes. However, changes in the exercise are indicated, and the teacher may synchronize the data (overwrite the assignment data with new exercise data).

Page Group Assignments shows a list of assignments for both teachers and students. An assignment is hidden from students when created and must be explicitly set to a visible state (or has its visibility scheduled from a given time). Assignments also have one or two deadlines that the students must meet if they wish to receive points. Students may submit after the deadline (and their submissions are evaluated), but no points are awarded for them. An example of assignments from Arduino classes is depicted in the following:

Students can access the assignment specification (rendered Markdown) and submit their solutions by uploading source code (or data) files. The specification, assignment details, submit button, and list of past submissions are conveniently located on the Assignment Detail page:

Once a solution is submitted by a student, it is immediately enqueued for evaluation. Once the evaluation is completed, the solution is graded (assigned points), and the student can inspect the results. Most notably, the results table summarizes the outcomes of individual tests (judge results and consumed resources). In case of judge failures, the student can read the log from the judge to get more detailed information on why the solution is not working properly (so the user can fix the solution and resubmit it). An example of test results, where two of three tests have failed (the judge indicated wrong outputs):

The overall correctness of the solution is computed as the (weighted) average of correctness values from individual tests. The points are computed simply by multiplying the correctness by max. points set for the given assignment (for solutions that are submitted before the deadline). The points may be overridden by the teacher, or additional bonus points may be awarded (e.g., points 1+2 indicate 1 point awarded from the evaluation and +2 bonus points awarded by the teacher). Points achieved from the best solutions are summarized on the Group Students page:

Apart from automated assessment, ReCodEx also integrates means for the teachers to inspect the solutions. Submitted files page shows the contents of all files in one place and allows the teachers to compare them to other solutions (like the previously submitted solution of the same student) or conduct regular code reviews. In the code reviews, the teacher can comment on individual lines of code (similarly to code reviews in GitHub):

Finally, the group has a way to provide additional points for accomplishments that reach beyond ReCodEx (like active participation in the class). For this purpose, a teacher may assign shadow assignments in a group. A shadow assignment has a specification, and the teacher may assign points for it (manually) to individual students, but the students do not submit anything into the system (thus, nothing is automatically evaluated).

Exercises

An exercise is a template for an assignment. It comprises the specification (text in Markdown), attachment files (which can be referenced from the specification), supplementary files (which can be used in tests), configuration for tests and limits (for time and memory), and reference solutions.

Each exercise is associated with at least one group (referred to as group of residence). The group of residents restricts which teachers can see (and assign) the exercise; in particular, only admins and supervisors of the group of residence (or any of its sub-groups) can do that. The expected utilization is that exercises of a particular course are associated with the top-level group representing the course, so they can be accessed in any sub-group (corresponding to a student's lab group) of that course. This way, exercises from different courses will not mix up, but sharing of an exercise is still possible (it can be placed in multiple groups of residence).

Testing

An exercise defines a list of tests. Each test is executed independently and contributes to the overall correctness score. A test comprises the following steps:

• compilation (for compiled languages)
• execution (running the tested solution in a safe environment using test inputs and arguments)
• judge (evaluating the results of an execution)

The compilation can be skipped (in case of interpreted languages) or executed just once for all tests (if they share the compilation parameters). Execution is performed in a sandbox that measures and limits consumed time and memory. The result of the execution is collected either from stdout or a predefined file and passed to a judge. Judge is a script that verifies the correctness of the result (usually by comparing it with a reference output) and optionally yields a log that may help the user find the error. It can be anything from a common diff tool to a custom script provided by the teacher. An example of a test configuration is in the following figure:

Runtime environments

An exercise can be configured for multiple runtime environments (programming languages). Students may choose any of these languages to solve the assignment. Some courses aim at a single language (like Programming in C++), others (like the CS1 programming course or Algorithms and data structures) offer the students a choice. Some specialized environments (data-only, Haskell, ...) are not compatible with others (due to severe differences in test configurations), so they cannot be combined. A selection form with currently configured environments is depicted below:

Limits

An exercise is configured for a particular hardware group. The hardware group identifies a group of workers that can evaluate the assignment. Workers in the same group must share the same configuration of the environment (compiler versions, installed libraries) and have identical hardware (so the time measurements are comparable).

The exercise holds time and memory limits, which are set to be valid on the selected hardware group (usually based on measuring the performance of reference solutions). A time and memory limit needs to be set for every test in every runtime environment (as depicted in the previous figure).

Pipelines

Other features