[test] Integrating spec tests into Web Platform Tests

[titzer]

It's great that the spec test harness matches the Web Platform Test harness. Could we go the extra step and import these tests (exactly how TBD) to be periodically checked into the main Web Platform Test repository (https://github.com/w3c/web-platform-tests)?

[flagxor]

Is there any appetite for moving the source of truth there eventually?
Are you'all running Web Platform Tests against your engines at the right point to catch regressions if that was the primary place they were run?

[lukewagner]

I still need to learn more about how the overall system works and how tests flow in/out, but on its face, yeah, that makes sense.

[titzer]

My personal opinion is that we should keep the source of truth for core WASM tests (i.e. wast tests) here in the spec repo. For JS API tests, I would strongly prefer to keep them here as well. For Web platform integration tests (i.e. the html/ directory here) I would slightly prefer to keep them here, but could be convinced otherwise.

[rossberg]

+1. We should optimise for making the process of adding tests and creating proposals simple and uniform, because that has the best chance of encouraging as many tests as possible. It would be infeasible from my perspective to move the 95% of (web-agnostic) core Wasm tests away from the spec and the reference interpreter that is used to process them. And given that, it seems to make most sense to keep the other tests here as well.

…

On 27 July 2017 at 10:46, titzer ***@***.***> wrote: My personal opinion is that we should keep the source of truth for core WASM tests (i.e. wast tests) here in the spec repo. For JS API tests, I would strongly prefer to keep them here as well. For Web platform integration tests (i.e. the html/ directory here) I would slightly prefer to keep them here, but could be convinced otherwise. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#529 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEDOOw6mIBCJspj2HraULx3_uUdkXpf9ks5sSE5egaJpZM4Oihnu> .

[lukewagner]

Definitely agreed for core and JS (shell) tests staying in the spec repo.

[littledan]

It's great to see so much attention to testing and encouraging the most welcoming environment for adding tests. This makes me really optimistic about the Wasm feature development process.

I think a two-way sync system could help with these goals. This technique has been used successfully in web-platform-tests for Chrome and Mozilla. With two-way sync, you can check in a change to tests either in an implementation repository, or in the upstream repository, and all repos are automatically synchronized with each other very frequently.

It looks like V8 currently maintains Wasm spec tests by pulling down a compressed archive, and in the SpiderMonkey source tree, I see generated tests in JavaScript which are based on the Wasm spec tests with a minimal wrapper template. Both of these processes seem to be "one way"--you add tests upstream, and then periodically pull them down. With either of these processes, I don't see a way that you can easily make a single patch which, e.g., fixes a bug, and in the same patch adds a regression test. It would be great if regression tests, added in a light-weight way, could be shared between implementations.

What would you think about using two-way sync for Wasm tests?

If we have two-way sync, maybe it would make sense to take advantage of web-platform-tests' infrastructure, or, maybe it would make sense to build parallel infrastructure. That's a narrower technical question; I wanted to talk about the project goals first.

cc @mjzffr @jgraham

[jfbastien]

This topic is on the agenda for next week's CG call. @titzer still owes me details on the agenda, but I assume @littledan's ideas are in scope for that discussion.

[foolip]

I just posted a few things in tc39/test262#1258 (comment) that might be of interest to anyone considering the merits of two-way sync. To extract the two links:

• Blink WPT Sync Workflow (WIP documentation by @Hexcles)
• Has two-way sync increased contributions? Yes! (showing a tripling of Chromium contributions)

[jgraham]

We should optimise for making the process of adding tests and creating proposals simple and uniform, because that has the best chance of encouraging as many tests as possible.

I strongly agree with this premise.

It would be infeasible from my perspective to move the 95% of (web-agnostic) core Wasm tests away from the spec and the reference interpreter that is used to process them. And given that, it seems to make most sense to keep the other tests here as well.

From my "generic" (i.e. without being focused on wasm specifically) browser-vendor perspective, I would come to the exact opposite conclusion. Keeping tests with a spec is a local optimum for people directly working on that spec. But there is evidence (for example in @foolip's comment above) that having a central repository for testing that allows vendors to contribute to, run, and sync, tests across the entire platform in a uniform way is more effective in getting vendors to contribute to the tests and to run an up-to-date copy of the testsuite.

Yes, in some ideal world we might be able to create a uniform contribution and sync pipeline that worked across multiple test repositories. But that isn't the world we live in. On the other hand, I expect we could solve technical issues around wasm in wpt specifically is there's some need to e.g. run travis jobs that depend on something in the spec repo. I don't know what the current setup you have is, so it's hard to make specific technical promises, but it certainly seems to me to be worth looking at the detailed requirements and assessing them for feasibility.

[rossberg]

From my "generic" (i.e. without being focused on wasm specifically) browser-vendor perspective, I would come to the exact opposite conclusion. Keeping tests with a spec is a local optimum for people directly working on that spec.

It also is the easier solution for most producers and for all consumers that do not happen to be browser vendors. As misleading as the choice of name may be, keep in mind that WebAssembly is explicitly designed and specified to not be a browser-exclusive technology and that it might potentially have more customers outside the Web (and JavaScript) ecosystem than inside.

[littledan]

Let's separate the two questions of two-way sync or no, and putting in the web-platform-tests repository or no. I definitely share your goal of including non-Web implementations of WebAssembly, including the reference interpreter.

For the two-way sync question, I don't have a clear understanding of how two-way sync would make things harder for non-web users, or less clean or pure, or anything like that. Could you explain more? cc @titzer who expressed similar concerns.

If we decide on two-way sync, for the choice of the main repository which everyone syncs to and from, as long as we make sure that the tests are in a format usable by non-web consumers, I'm not sure what the difference would be between any repository and any other. We could use the spec repository, or web-platform-tests. web-platform-tests might be easier because infrastructure is already there, but I don't see why that technical choice would make things any more tied to the web.

For proposals to include tests, in test262, we upstream tests that are for features that aren't yet at Stage 4, but with metadata that test users can use to switch the tests on or off, or pass runtime flags to turn the experimental features on for that subset of tests. One difficulty is making single patch which both specifies the change and adds tests--maybe we could set up two-way between the proposal repository as well, if that is a requirement.