[reference-types] remove single table restriction in IR #3517

martianboy · 2021-01-26T15:08:23Z

This will remove the single table per module assumption everywhere, and
instead introduces a vector of tables, same as functions, globals, etc.
Support for parsing and writing multiple tables is added and C/JS APIs
have been updated accordingly. A table name argument is also added to
call_indirect.

(Fixes #3512)

martianboy · 2021-01-27T04:29:11Z

Reviews appreciated @kripken @aheejin @sbc100

aheejin

Thank you! Haven't read all the PR yet, but the very initial comments:
I don't think we have a test case that uses multiple tables and call_indirects with a table index yet. Can we have one? Maybe multi-table.wast or something in test/ directory. You can run ./auto_update_tests.py to update other files after adding it.
(Also we would need tests for other added parts such as tests for the added C API)

src/wasm/wasm-binary.cpp

src/binaryen-c.cpp

martianboy · 2021-01-27T17:41:45Z

Thanks, @aheejin.

I don't think we have a test case that uses multiple tables and call_indirects with a table index yet. Can we have one? Maybe multi-table.wast or something in test/ directory.

Right, I started adding tests today. Hopefully I can push some by tomorrow.

this will allow adding further support for reference-types

martianboy · 2021-01-28T21:13:34Z

@aheejin

I don't think we have a test case that uses multiple tables and call_indirects with a table index yet. Can we have one? Maybe multi-table.wast or something in test/ directory. You can run ./auto_update_tests.py to update other files after adding it.

I don't think we can test modules with multiple tables yet, because in this PR I tried to keep it minimal and not implement the new changes from the spec as much as possible. For instance, we cannot parse an elem section in wast files that belong to tables with higher indexes. It does parse elems embedded inside a table declaration, but roundtrip runs fail as the elem section is separated from the table, and is therefore added to table 0. I did add a test/example for the new c apis though.

My plan was to get this PR reviewed and merged, to make sure that everything is working as before and I haven't missed anything. And then, continue with the remaining parts of the reference types spec.

kripken

I didn't review the entire thing yet, but the wasm.h and wasm-delegates-fields.h changes look like the right approach. Thanks for the PR!

Will try to review the rest tomorrow.

src/wasm.h

aheejin

@aheejin

I don't think we have a test case that uses multiple tables and call_indirects with a table index yet. Can we have one? Maybe multi-table.wast or something in test/ directory. You can run ./auto_update_tests.py to update other files after adding it.

I don't think we can test modules with multiple tables yet, because in this PR I tried to keep it minimal and not implement the new changes from the spec as much as possible. For instance, we cannot parse an elem section in wast files that belong to tables with higher indexes. It does parse elems embedded inside a table declaration, but roundtrip runs fail as the elem section is separated from the table, and is therefore added to table 0. I did add a test/example for the new c apis though.

My plan was to get this PR reviewed and merged, to make sure that everything is working as before and I haven't missed anything. And then, continue with the remaining parts of the reference types spec.

Thanks. Sorry I was OOO and I have still read only a fraction of the PR so far; I think this PR is one of the fundamental changes, but the only test additions + changes I see are from metrics, wasm-split (which I don't understand well why they are necessary; I asked questions there), and some table name changes.

I understand it is sometimes hard to add tests for every single part of code changes, especially in this kind of big PRs that change one of the very fundamental assumptions in the code base, so I really appreciate your work! But I still think this kind of big PRs need at least some tests that cover the very backbone of changes, which I think should include wast reading/writing and binary reading/writing. Also I think it is more desirable for a PR to have a smaller change with tests that matches the change, but as I said, I know it is not always easy in this kind of PRs.. 😅

I think maybe we can defer implementations + tests for the new call_indirect for the next PR, but I still would like to see at least some tests that show basic reading/writing of multiple tables working. Can we add those parts to the PR and maybe defer some other parts of the implementation that are not essential to the reading/writing to the next PR? I think C-API also could have been deferred if you wanted to make this minimal but you already added them so that looks good.

Anyway I'll probably take a more close look within a few days but please don't think me as a blocker for LGTM as long as others are happy with it.

test/example/module-splitting.txt

test/passes/remove-unused-module-elements_all-features.txt

tlively

It looks like a lot of code could be simplified by requiring call_indirect to have a real table name rather than defaulting to the first table if the table name is empty.

test/example/c-api-kitchen-sink.c

src/wasm2js.h

src/wasm/wasm-validator.cpp

src/passes/Directize.cpp

src/binaryen-c.h

src/binaryen-c.cpp

src/ir/module-splitting.cpp

test/example/c-api-multiple-tables.txt

martianboy · 2021-01-29T23:16:33Z

@tlively Thanks a lot for the review! I will try to address the points by tomorrow.

martianboy · 2021-02-04T16:57:48Z

@tlively I think I've addressed most of the review points here (I hope I haven't missed any), except for adding support for multiple tables in the fuzzer.

After reading fuzzing.h a little bit, I feel like anything I do in the fuzzer now to support multiple tables will have to be changed very soon when we add data type to tables and element segments, and convert segment data items from function names into (ref) expressions. If you agree, we can put that part for a later PR, but I'm totally open to suggestions.

Other than that, please do let me know if you have other comments about this PR. Hopefully we can finalize it very soon!

tlively · 2021-02-04T22:32:09Z

Neat, the latest changes look good to me. Can you summarize the C/JS API changes in the change log? Beyond that, I want to just do another review pass over everything and then it should be good!

martianboy · 2021-02-05T14:00:02Z

@tlively I added a few missing C/JS APIs and updated the change log. Not sure if the change log update is good, we may want to make it shorter. I also sent PR #3548 to add feature options to wasm-dis.

…ulti-table

src/tools/wasm-as.cpp

tlively · 2021-02-09T04:46:52Z

src/tools/wasm-ctor-eval.cpp

+    if (tableName.is()) {
+      return wasm->getTableOrNull(tableName);
+    } else if (!wasm->tables.empty()) {
+      return wasm->tables.front().get();
+    }


I think this function can be removed and its use replaced with just wasm->getTableOrNull(tableName) now that we are assuming that the table name is always present.

tlively · 2021-02-09T05:20:19Z

src/wasm/wasm-binary.cpp

+        o << U32LEB(0x0);
+      } else {
+        o << U32LEB(0x2);


It would be good to use the enum values here.

Thanks I used constants, although using ?: made it look a bit ugly! I can try several if blocks if you prefer. I've seen both styles in wasm-binary.cpp.

Yes, I think the if blocks would be better in this case, but I'm going to merge this as-is because it is so big and I don't want it to get out of sync with upstream. If you wouldn't mind fixing this in a follow-up PR, that would be great :)

src/wasm/wasm-emscripten.cpp

tlively · 2021-02-09T05:32:52Z

After those final two comments are addressed, I will be happy to merge this :)

martianboy · 2021-02-09T15:33:40Z

@tlively Thanks a lot for catching those! I've already started working on the next PR to add a ref type to Table. Hopefully it will be smaller and easier to review!

tlively · 2021-02-09T21:46:14Z

Thanks for your great work on this, @martianboy!

martianboy · 2021-02-09T21:49:59Z

Yaay! Finally! Thanks again @tlively. :)

tlively · 2021-02-09T21:52:54Z

You're welcome :D

sbc100 · 2021-02-09T22:46:35Z

It looks like this change broke a bunch of emscirpten tests. At first glance it looks like any test related to dyanmic linking:

e.g.:

./tests/runner.py wasm2.test_relocatable_void_function
runner:WARNING: use EMTEST_ALL_ENGINES=1 in the env to run against all JS engines, which is slower but provides more coverage
Test suites:
['test_core']
Running test_core: (1 tests)
(checking sanity from test runner)
shared:INFO: (Emscripten: Running sanity checks)
test_relocatable_void_function (test_core.wasm2) ... wasm-emscripten-finalize: /usr/local/google/home/sbc/dev/wasm/binaryen/src/wasm/wasm-binary.cpp:498: uint32_t wasm::WasmBinaryWriter::getTableIndex(wasm::Name) const: Assertion `it != indexes.tableIndexes.end()' failed.
emcc: error: '/usr/local/google/home/sbc/dev/wasm/binaryen-out/bin/wasm-emscripten-finalize --detect-features --minimize-wasm-changes --dyncalls-i64 --pass-arg=legalize-js-interface-export-originals test_relocatable_void_function.wasm -o test_relocatable_void_function.wasm' failed (-6)
FAIL

martianboy · 2021-02-09T22:50:02Z

@sbc100 I will look into this as soon as I can.

tlively · 2021-02-10T00:27:53Z

I fixed it in #3560

martianboy force-pushed the multi-table branch 2 times, most recently from 16a7a4b to 657e727 Compare January 26, 2021 18:19

aheejin reviewed Jan 27, 2021

View reviewed changes

src/wasm/wasm-binary.cpp Outdated Show resolved Hide resolved

src/binaryen-c.cpp Show resolved Hide resolved

martianboy added 12 commits January 28, 2021 14:46

remove single table constraint from modules

917ac8a

this will allow adding further support for reference-types

a few fixes

45edfa6

fix BinaryenAddTable

def024d

lint

9f99c84

lint

e47c72a

adapt to c74acc0

f74f8a1

fix memory leak in copyTable

9bae447

revert internal table naming with prefix

affa243

add table name getter to c api

7113a62

don't rely on CallIndirect to always have a tableName

ead9806

fix call_indirect validation if target unreachable

80b8f83

lint

c378d6f

martianboy force-pushed the multi-table branch from 46b8b17 to c378d6f Compare January 28, 2021 13:23

martianboy added 3 commits January 28, 2021 23:17

various fixes

24ff11c

lint: clang-format

98e1e42

add an example for c api

73a761b

kripken reviewed Jan 28, 2021

View reviewed changes

src/wasm.h Outdated Show resolved Hide resolved

apply name change suggestion

efb6e9c

aheejin reviewed Jan 29, 2021

View reviewed changes

test/example/module-splitting.txt Outdated Show resolved Hide resolved

test/passes/remove-unused-module-elements_all-features.txt Show resolved Hide resolved

martianboy added 2 commits January 30, 2021 01:27

don't add unnecessary table when splitting

563861d

assert conditions

9c02a5d

tlively reviewed Jan 29, 2021

View reviewed changes

elem header format 0x2 for multiple active tables

5a2991a

martianboy added 4 commits February 4, 2021 16:33

fix js api

8bb5cd6

prevent fuzzer from adding more tables for now

5240f7b

lint

b8f3702

polish

4faff51

martianboy mentioned this pull request Feb 4, 2021

wasm-dis should run validation #3545

Open

format

a7631dc

martianboy mentioned this pull request Feb 5, 2021

Add feature options to wasm-dis #3548

Merged

martianboy added 4 commits February 5, 2021 16:22

add more c & js apis

6c7ed38

add removeTable js api

ac73dbf

update changelog

21f50ce

improved c api multiple tables test

c3f1736

martianboy added 3 commits February 6, 2021 16:37

Merge remote-tracking branch 'upstream/main' into multi-table

ae96436

adapt

aaa12df

Merge branch 'main' of https://github.com/WebAssembly/binaryen into m…

0d0c18e

…ulti-table

tlively reviewed Feb 9, 2021

View reviewed changes

apply suggestions

d902590

tlively approved these changes Feb 9, 2021

View reviewed changes

tlively merged commit 3da8b08 into WebAssembly:main Feb 9, 2021

MrHate mentioned this pull request Mar 21, 2021

Support non-funcref tables #3712

Closed

[reference-types] remove single table restriction in IR #3517

[reference-types] remove single table restriction in IR #3517

Uh oh!

Conversation

martianboy commented Jan 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

martianboy commented Jan 27, 2021

Uh oh!

aheejin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

martianboy commented Jan 27, 2021

Uh oh!

martianboy commented Jan 28, 2021

Uh oh!

kripken left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aheejin left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tlively left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

martianboy commented Jan 29, 2021

Uh oh!

martianboy commented Feb 4, 2021

Uh oh!

tlively commented Feb 4, 2021

Uh oh!

martianboy commented Feb 5, 2021

Uh oh!

Uh oh!

tlively Feb 9, 2021

Choose a reason for hiding this comment

Uh oh!

tlively Feb 9, 2021

Choose a reason for hiding this comment

Uh oh!

martianboy Feb 9, 2021

Choose a reason for hiding this comment

Uh oh!

tlively Feb 9, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tlively commented Feb 9, 2021

Uh oh!

martianboy commented Feb 9, 2021

Uh oh!

tlively commented Feb 9, 2021

Uh oh!

martianboy commented Feb 9, 2021

Uh oh!

tlively commented Feb 9, 2021

Uh oh!

sbc100 commented Feb 9, 2021

Uh oh!

martianboy commented Feb 9, 2021

Uh oh!

tlively commented Feb 10, 2021

Uh oh!

martianboy commented Jan 26, 2021 •

edited

Loading

aheejin left a comment •

edited

Loading

aheejin left a comment •

edited

Loading