Skip to content

Conversation

@adhameer
Copy link
Contributor

This PR adds mutation testing functionality for requiring students to write test suites for their functions, and automatically grading the quality of these test suites based on desired test scenarios provided by the instructor.

When this extension is enabled, students can write test cases (input/expected output pairs) for a function foo: 'a -> 'b in a variable called foo_tests: ('a * 'b) list. These tests are first run against the solution code to ensure that they are correct tests. They can optionally also be run against the student's own submission. Finally, the tests are run against a selection of buggy programs specified by the instructor. The quality of the student's tests is judged by how many of the buggy programs fail at least one of the student's tests. (The ideal is for each buggy program to fail at least one of the student's tests, meaning that the student's test suite is rigorous enough to catch each of the given bugs.)

The mutation test functions are designed to look similar to the correctness testing functions in Test_lib; a typical invocation looks like this:

Mutation_test.test_unit_tests_2
    [%ty: int -> int -> int] "plus"
    (* List of buggy implementations *)
    [("Subtracts instead of adding", fun x y -> x - y)]

The demo exercise has been updated to demonstrate this functionality.

I suppose it is probably preferable to add this to the Test_lib file instead of having it in its own file; let me know.

*
* Learn-OCaml is distributed under the terms of the MIT license. See the
* included LICENSE file for details. *)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a documentation header to present the purpose of this module.

* Learn-OCaml is distributed under the terms of the MIT license. See the
* included LICENSE file for details. *)

type 'a test_result =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a quite general type definition. Shouldn't it be in another module?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, it is probably not necessary to expose this type and the associated run_test_against functions from this module at all, since it is basically just a weaker version of the test functions in Test_lib.

| Err of exn

(** A mutant is a pair of a string describing the mutant and the
mutant function itself.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please start with a definition of what a mutant function is, and the problem it solves.

?compare: ('b -> 'b -> bool) ->
('a -> 'b) -> ('a * 'b) -> 'b test_result

(** Run a test (a pair of input and expected output) on a mutant.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given this first sentence, it is a bit weird not to see a mutant as an argument of this function.

equality ([(=)]).
*)
module type S = sig
val test_unit_tests_1:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could profit from #302 to avoid the multiple versions of this function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The module signature was designed around the test functions we used most often, which are the only ones documented in the tutorials (test_function_1_against_solution, etc.). I'm not familiar with the version that uses function prototypes so I am not sure I can write a version like test_unit_tests using function prototypes at this moment...

I think regardless it is useful to keep the multiple versions to match the Test_lib functions, right?

@adhameer
Copy link
Contributor Author

Updated, addressing comments from the review and a couple other points:

  • Allow instructors to specify scores on a per-mutant basis instead of weighting them all equally
  • Rename the type mutant to mutant_info which is more accurate
  • Respect the timeout value by enclosing calls to student code in run_timeout instead of calling the functions directly
  • Add documentation and reorganize the module signature a little

@yurug
Copy link
Collaborator

yurug commented Sep 4, 2019

Thanks a lot for this contribution!

@yurug yurug merged commit 4818795 into ocaml-sf:master Sep 4, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants