First build of learn-ocaml-corpus fails

[erikmd]

Following @yurug's suggestion #302 (comment), I've tried compiling learn-ocaml/master = 718d1f1 + learn-ocaml-corpus and noticed a subtle bug: the first build of learn-ocaml-corpus fails, but the second succeeds.

For the record, I summarize below the opam dependencies I used to reproduce this.

OPAM dependencies

$ opam list
# Packages matching: installed
# Name                  # Installed  # Synopsis
asak                    0.1          Partitioning OCaml codes
astring                 0.8.3        Alternative String module for OCaml
base                    v0.9.4       Full standard library replacement for OCaml
base-bigarray           base
base-bytes              base         Bytes library distributed with the OCaml compiler
base-num                base         Num library distributed with the OCaml compiler
base-threads            base
base-unix               base
base64                  2.3.0        Base64 encoding for OCaml
bigarray-compat         1.0.0        Compatibility library to use Stdlib.Bigarray when possible
biniou                  1.2.0        Binary data format designed for speed, safety, ease of use and backw
bos                     0.2.0        Basic OS interaction for OCaml
checkseum               0.0.3        Adler-32, CRC32 and CRC32-C implementation in C and OCaml
cmdliner                1.0.3        Declarative definition of command line interfaces for OCaml
cohttp                  1.1.1
cohttp-lwt              1.1.1        An OCaml library for HTTP clients and servers
cohttp-lwt-unix         1.1.1        An OCaml library for HTTP clients and servers
conduit                 1.3.0        An OCaml network connection establishment library
conduit-lwt             1.3.0        An OCaml network connection establishment library
conduit-lwt-unix        1.3.0        An OCaml network connection establishment library
conf-git                1.0          Virtual package relying on git
conf-libev              4-11         High-performance event loop/event model with lots of features
conf-m4                 1            Virtual package relying on m4
conf-openssl            1            Virtual package relying on an OpenSSL system installation
conf-pkg-config         1.1          Virtual package relying on pkg-config installation
conf-which              1            Virtual package relying on which
cppo                    1.6.6        Code preprocessor like cpp for OCaml
cstruct                 3.3.0        Access C-like structures directly from OCaml
decompress              0.8.1        Implementation of Zlib in OCaml
digestif                0.7.1        Hashes implementations (SHA*, RIPEMD160, BLAKE2* and MD5)
dune                    1.6.3        Fast, portable and opinionated build system
easy-format             1.3.1        High-level and functional interface to the Format module of the OCam
eqaf                    0.4          Constant-time equal function on string
ezjsonm                 0.6.0        An easy interface on top of the Jsonm library
fieldslib               v0.9.0       Syntax extension to define first class values representing record fi
fmt                     0.8.6        OCaml Format pretty-printer combinators
fpath                   0.7.2        File system paths for OCaml
hex                     1.4.0        Library providing hexadecimal converters
ipaddr                  2.8.0        IP (and MAC) address manipulation
jbuilder                transition   This is a transition package, jbuilder is now named dune. Use the du
js_of_ocaml             3.3.0        Compiler from OCaml bytecode to Javascript
js_of_ocaml-compiler    3.3.0        Compiler from OCaml bytecode to Javascript
js_of_ocaml-lwt         3.3.0        Compiler from OCaml bytecode to Javascript
js_of_ocaml-ppx         3.3.0        Compiler from OCaml bytecode to Javascript
js_of_ocaml-toplevel    3.3.0        Compiler from OCaml bytecode to Javascript
js_of_ocaml-tyxml       3.3.0        Compiler from OCaml bytecode to Javascript
jsonm                   1.0.1        Non-blocking streaming JSON codec for OCaml
logs                    0.6.3        Logging infrastructure for OCaml
lwt                     3.3.0        Promises, concurrency, and parallelized I/O
lwt_react               1.1.1        Helpers for using React with Lwt
lwt_ssl                 1.1.2        OpenSSL binding with concurrent I/O
magic-mime              1.1.1        Map filenames to common MIME types
markup                  0.8.0        Error-recovering functional HTML5 and XML parsers and writers
merlin                  3.1.0        Editor helper, provides completion, typing and source browsing in Vi
num                     0            The Num library for arbitrary-precision integer and rational arithme
ocaml                   4.05.0       The OCaml compiler (virtual package)
ocaml-base-compiler     4.05.0       Official 4.05.0 release
ocaml-compiler-libs     v0.9.0       OCaml compiler libraries repackaged
ocaml-config            1            OCaml Switch Configuration
ocaml-migrate-parsetree 1.3.1        Convert OCaml parsetrees between different versions
ocamlbuild              0.14.0       OCamlbuild is a build system with builtin rules to easily build most
ocamlfind               1.8.0        A library manager for OCaml
ocamlgraph              1.8.8        A generic graph library for OCaml
ocp-build               1.99.20-beta Project builder for OCaml
ocp-indent-nlfork       1.5.3        ocp-indent library, "newline tokens" fork
ocp-ocamlres            0.4          Manipulation, injection and extraction of embedded resources
ocplib-json-typed       0.6          Type-aware JSON and JSON schema utilities
odoc                    1.3.0        OCaml documentation generator
omd                     1.3.1        A Markdown frontend in pure OCaml.
opam-core               2.0.4
opam-file-format        2.0.0        Parser and printer for the opam file syntax
opam-format             2.0.4
opam-installer          2.0.4
optint                  0.0.2        Abstract type on integer between x64 and x86 architecture
pprint                  20180528     A pretty-printing combinator library and rendering engine
ppx_ast                 v0.9.1       OCaml AST used by Jane Street ppx rewriters
ppx_core                v0.9.3       Standard library for ppx rewriters
ppx_cstruct             3.3.0        Access C-like structures directly from OCaml
ppx_derivers            1.2.1        Shared [@@deriving] plugin registry
ppx_driver              v0.9.2       Feature-full driver for OCaml AST transformers
ppx_fields_conv         v0.9.0       Generation of accessor and iteration functions for ocaml records
ppx_metaquot            v0.9.0       Write OCaml AST fragment using OCaml syntax
ppx_optcomp             v0.9.0       Optional compilation for OCaml
ppx_sexp_conv           v0.9.0       Generation of S-expression conversion functions from type definition
ppx_tools               5.0+4.05.0   Tools for authors of ppx rewriters and other syntactic tools
ppx_tools_versioned     5.2.2        A variant of ppx_tools based on ocaml-migrate-parsetree
ppx_traverse_builtins   v0.9.0       Builtins for Ppx_traverse
ppx_type_conv           v0.9.1       Support Library for type-driven code generators
re                      1.9.0        RE is a regular expression library for OCaml
react                   1.2.1        Declarative events and signals for OCaml
reactiveData            0.2.1        Functional reactive programming with incremental changes in data str
result                  1.4          Compatibility Result module
rresult                 0.6.0        Result value combinators for OCaml
seq                     0.1          Compatibility package for OCaml's standard iterator type starting fr
sexplib                 v0.9.3       Library for serializing OCaml values to and from S-expressions
ssl                     0.5.7        Bindings for OpenSSL
stdio                   v0.9.1       Standard IO library for OCaml
stdlib-shims            0.1.0        Backport some of the new stdlib features to older compiler
stringext               1.6.0        Extra string functions for OCaml
topkg                   1.0.0        The transitory OCaml software packager
tyxml                   4.3.0        TyXML is a library for building correct HTML and SVG documents
uchar                   0.0.2        Compatibility library for OCaml's Uchar module
uri                     1.9.7        An RFC3986 URI/URL parsing library
uutf                    1.0.1        Non-blocking streaming Unicode codec for OCaml
yojson                  1.5.0        Yojson is an optimized parsing and printing library for the JSON for

Example session

$ cd .../learn-ocaml && rm -fr www

$ make && make opaminstall && learn-ocaml --repo=../learn-ocaml-corpus build serve
[…]
Learnocaml v.0.11 running.
Updating app at ./www
counting_trees               [OK]
enumerating_trees            [OK]
leftist_heaps                [OK]
merge_sort                   [OK]
persistent_arrays            [OK]
pprint                       [OK]
random_access_lists          [OK]
symbolic_sequences_data      [OK]
symbolic_sequences_objects     [OK]
Cannot process exercises: Unix.Unix_error(Unix.EEXIST, "mkdir", "./www/static/tree_iterators")

$ make && make opaminstall && learn-ocaml --repo=../learn-ocaml-corpus build serve
[…]
Learnocaml v.0.11 running.
Updating app at ./www
counting_trees           (no changes)
enumerating_trees        (no changes)
leftist_heaps            (no changes)
merge_sort               (no changes)
persistent_arrays        (no changes)
pprint                   (no changes)
random_access_lists      (no changes)
symbolic_sequences_data  (no changes)
symbolic_sequences_objects (no changes)
tree_iterators               [OK]
Learnocaml server v.0.11 starting on port 8080
Found the following teacher tokens:
[…]

$ git rev-parse HEAD
718d1f12ff2ef78b1db34d2f70e4b24ddad0ed53

$ cd ../learn-ocaml-corpus && git rev-parse HEAD
8f97c4e831dd04e2659b9122386b96edc90d1a9a

@yurug I suppose the error at stake Cannot process exercises: Unix.Unix_error(Unix.EEXIST, "mkdir", "./www/static/tree_iterators") is triggered in learnocaml_process_exercise_repository.ml, but I did not investigate further.

[erikmd]

By the way, this makes me think it could be worth it to add another Travis job to learn-ocaml's CI, to test it against the corpus…

[agrn]

I think I’m onto something on this one. If I understand the code correctly, here:

learn-ocaml/src/repo/learnocaml_process_exercise_repository.ml

Lines 231 to 238 in a13226b

	Lwt_stream.iter_p (fun base ->
	let d = Filename.concat ex_dir base in
	let dst = String.concat Filename.dir_sep [dest_dir; "static"; id; base] in
	if Sys.is_directory d && base.[0] <> '.' then
	(Lwt_utils.mkdir_p (Filename.dirname dst) >>= fun () ->
	Lwt_utils.copy_tree d dst)
	else Lwt.return_unit)
	(Lwt_unix.files_of_directory ex_dir) >>= fun () ->

All the subdirectories of exercises are copied in parallel, and the parent directory is created if it doesn't exist. If there is only one subdirectory (like in all exercises, but not in tree_iterators), there is no problem: mkdir_p will test if the parent exist. As it does not, it will create it, then create the subdirectory. But I believe that if multiple subdirectories exists, each thread will test if the parent exist and create it in parallel. Only one of them should succeed, all others would fail.

[agrn]

I can confirm that creating the directory of the exercise before copying subdirectories seems to fix the issue.