diff --git a/spec/2025-09.md b/spec/2025-09.md index 201a119b..6b7aff80 100644 --- a/spec/2025-09.md +++ b/spec/2025-09.md @@ -642,12 +642,13 @@ For each test case: - `output_validator_args` defines arguments passed to the output validator for the test case. - `input_visualizer_args` defines arguments passed to the input visualizer for the test case. - `output_visualizer_args` defines arguments passed to the output visualizer for the test case. -- When `full_feedback` is `true`, somebody whose submission didn't pass case should be shown: +- When `full_feedback` is `true`, the following should be made available to a solver that didn't pass the test case: - the given input, - - the produced output (stdout), + - the produced output (stdout), - any error messages (stderr), - the illustration created by the output visualizer (if applicable), - the expected output. + For a submission that fails a test case with `full_feedback` set to `true`, the relevant feedback files for that test case should be made available. These files are not made available otherwise; the rules determining which files are provided are specified in [Test Cases with Full Feedback](#test-cases-with-full-feedback). - `hint` provides feedback for solving a test case to, for example, somebody whose submission didn't pass. - `description` conveys the purpose of a test case. It is an explanation of what aspect or edge case of the solution the input file is meant to test. @@ -714,11 +715,8 @@ It may be missing (for problems with no samples) or empty. #### Samples shown in the problem statement -By default, the `.in` and `.ans` pairs from `data/sample` are shown in the problem statement. -If a `.out` file exists the `.out` file is shown instead of the `.ans` file in the problem statement. -This behavior can be customized by creating files with extension `.in.statement` and `.ans.statement`. -If one of these files exists, its contents replaces that of the file with the same name -- except the `.statement` extension -- for purposes of the problem statement. -Note that it is an error to provide both a `.out` and a `.ans.statement` file. +By default, the `.in` and `.ans` pairs from `data/sample` are shown in the problem statement. This can customized by using `.out`, `.in.statement`, and `.ans.statement` files. See more in [Displayed test cases](#displayed-test-cases). + ##### Interactive problems @@ -747,12 +745,7 @@ Note that if you are using a `.interaction` file you must not provide a `.in.sta #### Samples available for download -By default, the `.in`, `.ans`, and `.files` files in `data/sample` are available for download. -Note that the content of `.in.statement` replaces that of `.in` and that the content of `.out` or `.ans.statement` replace that of `.ans` for the download. -This behavior can be further customized by providing files with the extension `in.download` or `ans.download`. -If one of these files exists, its contents replaces that of the file with the same name -- except the `.download` extension -- for the problem download. - -If you want to make other files -- like testing tools -- available for download, you can use [attachments](#attachments). +By default, the `.in`, `.ans`, and `.files` files in `data/sample` are available for download. This can customized by using `.out`, `.in.statement`, `.ans.statement`, `.in.download`, and `.ans.download` files. See more in [Downloadable test cases](#downloadable-test-cases). #### Validation @@ -766,6 +759,14 @@ However, since only the `.out` files are validated it is advised to use these if Validation can be customized by specifying `input_validator_args` and `output_validator_args` in `data/sample/test_group.yaml`. +### Test Cases with Full Feedback + +If a test case has `full_feedback` set to `true` (see [Test Case Configuration](#test-case-configuration)), then by default, the `.in` and `.ans` pair for this particular testcase should be shown as feedback, if the submission fails this test case. +This can customized by using `.out`, `.in.statement`, and `.ans.statement` files. See more in [Displayed test cases](#displayed-test-cases). + +By default, the `.in` and `.ans` files for the test cases with full feedback available for download, for the submission that fails this test case. +This can customized by using `.out`, `.in.statement`, `.ans.statement`, `.in.download`, and `.ans.download` files. See more in [Downloadable test cases](#downloadable-test-cases). + ## Generators @@ -948,7 +949,33 @@ The system may warn when this makes other constraints redundant and should error It is required that at least one submission is used to lower bound the time limit. -## Input validators +### Priority of test case files + +This section summarizes which files take precedence for display and download. Each priority order defines the search order; use the first file that exists. These priorities do not affect what is executed during judging unless explicitly noted. + +#### Displayed test cases + +For samples and full-feedback test cases, the priority of displayed test cases is as follows. + - Input shown: `.in.statement` > `.in`. + - Output shown: `.out` > `.ans.statement` > `.ans`. +Note that it is an error to have both `.out` and `.ans.statement`. + +For interactive and multi-pass problems, if `.interaction` exists, it is the only thing shown for the sample (and for any test case with full feedback). Do not provide `.in.statement`, `.ans.statement`, or `.out` alongside `.interaction`. + +#### Downloadable test cases + +For the files shown to the team, if the sample or test case with full-feedback is downloaded, the priority is as follows. + - Input downloaded: `.in.download` > `.in`. + - Output downloaded: `.ans.download` > `.out` > `.ans`. +The `.files` files is included in downloads when present. + +For the same base name, `.in.download` and `.in.statement` must both be present or both absent; likewise for `.ans.download` and `.ans.statement`. + +For interactive and multi-pass problems, if `.interaction` exists, it is the only thing made available for download for the sample (and for any test case with full feedback). Do not provide `.in.download`, `.ans.download`, or `.out` alongside `.interaction`. + +If you want to make other files available for download, you can use [attachments](#attachments). One example of such files are testing tools. + +## Input Validators Input Validators, verifying the correctness of the input files, are provided in `input_validators/`. Input validators can be written in three formats: as a program (as specified [above](#programs)),