-
Notifications
You must be signed in to change notification settings - Fork 71
Mutation testing functionality #306
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
This can be reverted if ocaml-sf#300 is fixed.
| * | ||
| * Learn-OCaml is distributed under the terms of the MIT license. See the | ||
| * included LICENSE file for details. *) | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add a documentation header to present the purpose of this module.
src/grader/mutation_test.mli
Outdated
| * Learn-OCaml is distributed under the terms of the MIT license. See the | ||
| * included LICENSE file for details. *) | ||
|
|
||
| type 'a test_result = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks like a quite general type definition. Shouldn't it be in another module?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hmm, it is probably not necessary to expose this type and the associated run_test_against functions from this module at all, since it is basically just a weaker version of the test functions in Test_lib.
src/grader/mutation_test.mli
Outdated
| | Err of exn | ||
|
|
||
| (** A mutant is a pair of a string describing the mutant and the | ||
| mutant function itself. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please start with a definition of what a mutant function is, and the problem it solves.
src/grader/mutation_test.mli
Outdated
| ?compare: ('b -> 'b -> bool) -> | ||
| ('a -> 'b) -> ('a * 'b) -> 'b test_result | ||
|
|
||
| (** Run a test (a pair of input and expected output) on a mutant. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Given this first sentence, it is a bit weird not to see a mutant as an argument of this function.
| equality ([(=)]). | ||
| *) | ||
| module type S = sig | ||
| val test_unit_tests_1: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could profit from #302 to avoid the multiple versions of this function.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The module signature was designed around the test functions we used most often, which are the only ones documented in the tutorials (test_function_1_against_solution, etc.). I'm not familiar with the version that uses function prototypes so I am not sure I can write a version like test_unit_tests using function prototypes at this moment...
I think regardless it is useful to keep the multiple versions to match the Test_lib functions, right?
|
Updated, addressing comments from the review and a couple other points:
|
|
Thanks a lot for this contribution! |
This PR adds mutation testing functionality for requiring students to write test suites for their functions, and automatically grading the quality of these test suites based on desired test scenarios provided by the instructor.
When this extension is enabled, students can write test cases (input/expected output pairs) for a function
foo: 'a -> 'bin a variable calledfoo_tests: ('a * 'b) list. These tests are first run against the solution code to ensure that they are correct tests. They can optionally also be run against the student's own submission. Finally, the tests are run against a selection of buggy programs specified by the instructor. The quality of the student's tests is judged by how many of the buggy programs fail at least one of the student's tests. (The ideal is for each buggy program to fail at least one of the student's tests, meaning that the student's test suite is rigorous enough to catch each of the given bugs.)The mutation test functions are designed to look similar to the correctness testing functions in
Test_lib; a typical invocation looks like this:The demo exercise has been updated to demonstrate this functionality.
I suppose it is probably preferable to add this to the
Test_libfile instead of having it in its own file; let me know.