Allow Compilation without Collecting Constants

[gcatron]

Description: For the new runtime we need to be able to compile a function but collect the constants slightly later in the initialization process. This PR introduces a compileWithoutConstants method to the backend and updates existing backends to support it.
Testing: passes all unit tests.
Documentation: N/A
Fixes: #2254

[nadavrot]

@gcatron Please make sure to update the docs (see #2226) and communicate the plan. We need to make sure that people who are developing Glow backends are not surprised by these changes. Please make sure to update the PR with the overall plan and send an email with link to the PR.

#2226

[nickgg]

There's new behaviour here (compiling without collecting) that I don't think we're testing anywhere. Can we write a test to cover this?

[bertmaher]

Sorry, but I'm having trouble seeing the desired end-state here. Can you explain in a bit more detail why we need to split out constant collection?

[bertmaher]

Sorry I'm pretty repetitive on the "blank lines separating definitions" theme, but that is a major bugaboo for me ;-).

[bertmaher]

vertical space b/w top level declarations.

[bertmaher]

vertical space b/w top level declarations.

[bertmaher]

I recommend keeping runtimeBundle private; having public data members has a way of bloating the API. It seems like you should be able to just add a collectConstants method to CPUFunction. (Or maybe even to the CompiledFunction interface, since it's used by every backend.)

[rdzhabarov]

Or maybe even to the CompiledFunction interface, since it's used by every backend.

this way we'd also avoid ugly static_casts on pointers to specific CompiledFunctions.

[gcatron]

I think adding it as a method to each compiledFunction makes sense, I think eventually this will go away entirely, the new DeviceManager will have the collection logic. Unfortunately even if I added it to CompiledFunction OpenCL and Interpreter will still require a static_cast to get the IR.

So how about I keep bundle private and add a new method to each CPUFunction, OpenCLFunction, and InterpreterFunction but not to compiledFunction, since ultimately we'd want to move this to the DeviceManager anyway?

[bertmaher]

Blank line between definitions.

[bertmaher]

More blank lines plz :)

[bertmaher]

As above, I think adding a collectConstants would be better than adding getIR just to pass it back to runtime bundle.

[gcatron]

The challenge here is the CPUFunction, it doesn't keep the IR around so it would still need it passed in.

[bertmaher]

That is a good point. Which makes me wonder, how could we even use collectConstants with the CPU backend? Since compileIR takes ownership of the IR and then destroys it, how can we later pass it to collect? (Right now we only collect inside the compile function, but that's going to change soon... right?

[gcatron]

Yes, the device manager will have a reference to the module so that is where it can collect constants from.

[bertmaher]

Looking at collectConstants, does it even need to take an IRFunction? It looks like it pulls all the constants from the Module via F->getGraph()->getParent()->getConstants(). It then uses getWeightForNode to turn those back into WeightVars but I don't see a reason for that, since all we do is call getSizeInBytes, which we can get without getting the WeightVar. Am I missing something?

[rdzhabarov]

nit: just return this right away

[rdzhabarov]

Or maybe even to the CompiledFunction interface, since it's used by every backend.

this way we'd also avoid ugly static_casts on pointers to specific CompiledFunctions.

[rdzhabarov]

where is it invoked?

[gcatron]

Currently nowhere but it will be called from the Provisioner.

[bertmaher]

Did we come to a conclusion about how collectConstants is going to work for the CPU backend yesterday? Since compileWithoutConstants throws away the IR, we don't have anything to pass to collectConstants...

[gcatron]

I think we decided to make expose collect constants as a method on each compiled function, and keep the bundle_ private. We call collectConstants in CompileIR where we still have the IR.

[bertmaher]

Thanks, lgtm!