Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 37 additions & 14 deletions spec/2025-09.md
Original file line number Diff line number Diff line change
Expand Up @@ -642,12 +642,13 @@ For each test case:
- `output_validator_args` defines arguments passed to the output validator for the test case.
- `input_visualizer_args` defines arguments passed to the input visualizer for the test case.
- `output_visualizer_args` defines arguments passed to the output visualizer for the test case.
- When `full_feedback` is `true`, somebody whose submission didn't pass case should be shown:
- When `full_feedback` is `true`, the following should be made available to a solver that didn't pass the test case:
- the given input,
- the produced output (stdout),
- the produced output (stdout),
- any error messages (stderr),
- the illustration created by the output visualizer (if applicable),
- the expected output.
For a submission that fails a test case with `full_feedback` set to `true`, the relevant feedback files for that test case should be made available. These files are not made available otherwise; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback).
- `hint` provides feedback for solving a test case to, for example, somebody whose submission didn't pass.
- `description` conveys the purpose of a test case.
It is an explanation of what aspect or edge case of the solution the input file is meant to test.
Expand Down Expand Up @@ -714,11 +715,8 @@ It may be missing (for problems with no samples) or empty.

#### Samples shown in the problem statement

By default, the `.in` and `.ans` pairs from `data/sample` are shown in the problem statement.
If a `.out` file exists the `.out` file is shown instead of the `.ans` file in the problem statement.
This behavior can be customized by creating files with extension `.in.statement` and `.ans.statement`.
If one of these files exists, its contents replaces that of the file with the same name -- except the `.statement` extension -- for purposes of the problem statement.
Note that it is an error to provide both a `.out` and a `.ans.statement` file.
By default, the `.in` and `.ans` pairs from `data/sample` are shown in the problem statement. This can customized by using `.out`, `.in.statement`, and `.ans.statement` files. See more in [Displayed test cases](#displayed-test-cases).


##### Interactive problems

Expand Down Expand Up @@ -747,12 +745,7 @@ Note that if you are using a `.interaction` file you must not provide a `.in.sta

#### Samples available for download

By default, the `.in`, `.ans`, and `.files` files in `data/sample` are available for download.
Note that the content of `.in.statement` replaces that of `.in` and that the content of `.out` or `.ans.statement` replace that of `.ans` for the download.
This behavior can be further customized by providing files with the extension `in.download` or `ans.download`.
If one of these files exists, its contents replaces that of the file with the same name -- except the `.download` extension -- for the problem download.

If you want to make other files -- like testing tools -- available for download, you can use [attachments](#attachments).
By default, the `.in`, `.ans`, and `.files` files in `data/sample` are available for download. This can customized by using `.out`, `.in.statement`, `.ans.statement`, `.in.download`, and `.ans.download` files. See more in [Downloadable test cases](#downloadable-test-cases).

#### Validation

Expand All @@ -766,6 +759,14 @@ However, since only the `.out` files are validated it is advised to use these if

Validation can be customized by specifying `input_validator_args` and `output_validator_args` in `data/sample/test_group.yaml`.

### Test Cases with Full Feedback

If a test case has `full_feedback` set to `true` (see [Test Case Configuration](#test-case-configuration)), then by default, the `.in` and `.ans` pair for this particular testcase should be shown as feedback, if the submission fails this test case.
This can customized by using `.out`, `.in.statement`, and `.ans.statement` files. See more in [Displayed test cases](#displayed-test-cases).

By default, the `.in` and `.ans` files for the test cases with full feedback available for download, for the submission that fails this test case.
This can customized by using `.out`, `.in.statement`, `.ans.statement`, `.in.download`, and `.ans.download` files. See more in [Downloadable test cases](#downloadable-test-cases).


## Generators

Expand Down Expand Up @@ -948,7 +949,29 @@ The system may warn when this makes other constraints redundant and should error

It is required that at least one submission is used to lower bound the time limit.

## Input validators
### Priority of test case files

This section summarizes which files take precedence for display and download. Each priority order defines the search order; use the first file that exists. These priorities do not affect what is executed during judging unless explicitly noted.

#### Displayed test cases

For samples and full-feedback test cases, the priority of displayed test cases is as follows.
- Input shown: `.in.statement` > `.in`.
- Output shown: `.out` > `.ans.statement` > `.ans`.
Note that it is an error to have both `.out` and `.ans.statement`.

For interactive and multi-pass problems, if `.interaction` exists, it is the only thing shown for the sample. Do not provide `.in.statement`, `.ans.statement`, or `.out` alongside `.interaction`.

#### Downloadable test cases

For the files shown to the team, if the sample or test case with full-feedback is downloaded, the priority is as follows.
- Input downloaded: `.in.download` > `.in.statement` > `.in`.
- Output downloaded: `.ans.download` > `.out` > `.ans.statement` > `.ans`.
The `.files` files is included in downloads when present.

If you want to make other files available for download, you can use [attachments](#attachments). One example of such files are testing tools.

## Input Validators

Input Validators, verifying the correctness of the input files, are provided in `input_validators/`.
Input validators can be written in three formats: as a program (as specified [above](#programs)),
Expand Down