CLICS Validator Implementation - pc2ccs/pc2v9 GitHub Wiki

Overview

PC2V9 provides an implementation of Output Validators as defined by the CLICS Validator specification.
The implementation supports all of the specified options for CLICS output validators, including

  • ''case_sensitive''
  • ''space_change_sensitive''
  • ''float_relative_tolerance E''
  • ''float_absolute_tolerance E''
  • ''float_tolerance E''

where ''E'' is a float value in either decimal notation (e.g. ''0.01'') or scientific notation (e.g. ''1e-02'').

In addition, the PC2V9 implementation accepts keywords ''case-sensitive'', ''space-change-sensitive'', ''space-sensitive'', ''float-relative-tolerance'', ''float-absolute-tolerance'', and ''float-tolerance'' as replacements for the corresponding keywords using underscores.

The operation of all of the above options is exactly as described in the CLICS Validator specification.

Implementation

The source code for the PC² V9 CLICS Validator implementation is contained in class ClicsValidator.java in the clicsValidator package within the edu.csus.ecs.pc2.validator package of the PC2V9 Source Code. The ClicsValidator implementation complies with the CLICS Output Validator specification: it expects three command line arguments (the input data file name, the judge's answer file name, and the name of a writable feedback directory), and it expects the output of the team's program to be presented to the validator's standard input channel. It returns Exit Code 42 if the team's output is deemed correct, and Exit Code 43 if not. It returns Exit Code -39 if an error occurs during validator execution. (The Exit Code values are defined in constants at the top of the ClicsValidator.java class.)

Note that when a Contest Problem is configured to use the Clics Validator, PC2V9 automatically invokes the validator utilizing the above interface.

Validator Feedback

The CLICS Output Validator specification states that an output validator may optionally write additional "feedback information" into the feedback directory. The PC2V9 Clics Validator implementation does this. Specifically, it creates a judgement feedback file in the feedback directory containing a description of the validation result. If the team's output is deemed correct then it uses method outputSuccess() to write a "correct answer" message into the judgement feedback file, whereas if the team's output is deemed incorrect then it uses method outputFailure() to write a failure message into the judgement feedback file.

If the team's output is deemed incorrect, then the PC2V9 CLICS Validator also creates a second file, called the judgement details file, in the feedback directory. Examples of judgement details file messages are things like

  • "Expected float token in team output, got "
  • "Float tolerance output of range" (plus some specific details on the expected and actual float values)
  • "String token mismatch", followed by the judge's (expected) token and the team's (actual) token
  • "Space change error", followed by the judge's (expected) token/spacing and the team's (actual) token/spacing
  • "Incomplete output", followed by the judge's token which was expected but missing from the team's output
  • "Team has trailing output beyond what judge answer file contains"

These feedback files are used by PC2V9 to display detailed information to the judges regarding what happened during the output validation process. (The detail messages are displayed on the Judge's GUI during manual judging of problems.)

The actual exit codes used by the PC2V9 Clics Validator, the names of the files used for judgement feedback and judgement details, and the actual strings written into the judgement feedback files, are defined in constants at the beginning of the ClicsValidator class. As of this writing (1 September 2019), the values are:

static public final int CLICS_VALIDATOR_JUDGED_RUN_SUCCESS_EXIT_CODE = 42;
static public final int CLICS_VALIDATOR_JUDGED_RUN_FAILURE_EXIT_CODE = 43;
static public final int CLICS_VALIDATOR_ERROR_EXIT_CODE = -39;

//the judgement messages which can be returned in the judgement feedback file in the feedback directory
static public final String CLICS_INCOMPLETE_OUTPUT_MSG = "Incomplete output";
static public final String CLICS_EXCESSIVE_OUTPUT_MSG = "Excessive output";
static public final String CLICS_INCORRECT_OUTPUT_FORMAT_MSG = "Incorrect output format";
static public final String CLICS_WRONG_ANSWER_MSG = "Wrong Answer";
static public final String CLICS_CORRECT_ANSWER_MSG = "accepted";

static public final String CLICS_JUDGEMENT_FEEDBACK_FILE_NAME = "judgement.txt";
static public final String CLICS_JUDGEMENT_DETAILS_FEEDBACK_FILE_NAME = "judgementdetails.txt";
⚠️ **GitHub.com Fallback** ⚠️