register-based interpreter

[belm0]

this 3.10 fork is API and ABI compatible with CPython

I'm seeing 5% speedup on my app (long-running, large, heterogeneous, highly concurrent, async workload), which is in line with overall speedup they report across the benchmarks (closer to 9% is reported for ARM).

https://github.com/zq1997/RegCPython
https://dl.acm.org/doi/10.1145/3568973

I'd like to see how this composes with Python 3.11...

[belm0]

I had an exchange with the author. He believes that the RegCPython gains will be orthogonal to Python 3.11 gains, and will compose well. Rebasing to 3.11 would be a large undertaking, however, and he doesn't plan it soon due to finishing his PhD, etc.

[jneb]

I am convinced that a register based interpreter is going to a significant speedup independent of the rest of the optimisations.
However, the rewrite itself is going to interfere a lot with the other activities if we are not careful. Here a short list of what I saw when browsing through the design. I find it hard to see a good approach yet, but I hope this list helps making a good plan.

• Most of the interference is a result of the fact that opcodes get multiple parameters. Many parts of the interpreter implicitly assume opcodes have only one parameter, and
• dis: relatively easy
• opcode pairs (eg LOAD_FAST__LOAD_FAST): looks doable
• collecting stats: doable, maybe it already is independent
• using DSL to generate ceval.c: i think going to registers can only be done AFTER this step, to simplify the separation of work
• specialization: should be not too hard
• code generator: optimisations of the kind pop_and_jump versus jump_and_pop are becoming unnecessary; otherwise these are unaffected in comparison to the expression compilation

I am pretty sure this list is not complete, so feel free to comment.

[iritkatriel]

@vstinner experimented with this on 3.3: https://faster-cpython.readthedocs.io/registervm.html

[iritkatriel]

My main question after reading the paper (but not looking at the code):

How does this not run out of registers? The instruction format is (opcode, oparg1, oparg2, oparg3) where each item is 8 bits. What if we need more than 256 registers? Since the co_consts are "loaded" into registers, and easy way to run out is to have > 256 constants in a function.

[jneb]

I can't stop thinking about this. I would like to have the speed of this without having to have so many dependencies.
So while having a nice bike ride through the dunes, I asked myself the question: WHY is a register based interpreter faster? Maybe thinking about this could lead to ideas giving speedup.
Here are my thoughts so far:

• fewer opcodes per function: we achieve this effect now using combined opcodes. The price is parameter processing, of course
• the most common words are LOAD_CONST and LOAD_FAST; both of these would be obsolete because the values are "already there"
• no stack pointer increment/decrement: part of this is achieved by combining opcodes. The problem are the opcodes (conditional jump, etc) that need to pop to do what they do
• temp values and locals in the same space: you seldom need to store a result, because it is there already
• compile time allocation: in a way, a stack is a run time allocator. That brings me to an idea: The compiler could give every opcode its stack level as a parameter! That way, each opcode can calculate the locations of input and output from that. No increment needed. The output location is a parameter too, which can be a stack level or a local variable.

That last idea could be called a "poor man's register allocator": just make enough locals die the highest stack
Honestly, I don't consider myself qualified to actually build this, but this last idea is so crazy that I feel tempted to try it!

[jneb]

How does this not run out of registers? The instruction format is (opcode, oparg1, oparg2, oparg3) where each item is 8 bits. What if we need more than 256 registers? Since the co_consts are "loaded" into registers, and easy way to run out is to have > 256 constants in a function.

You need some extension coding, a generalisation of EXTENDED_ARG. I bet that functions using so many values wil not be fast anymore.
Another way is to have the compiler fix the problem (as "normal" compilers do: every processors has a limited set of registers, as far as i know :-) )

[jneb]

I figured it out: the crazy idea above is possible. The following conceptual process could change an existing bytecode stream into a register based stream, leaving many parts of the interpreter as they are now.
This implies that many current optimisations (specialisation, for example) will still work:

• associate each bytecode with an equivalent register based bytecode with extra parameters for each input and output. So ADD becomes ADD(inputreg, inputreg -> outputreg)
• at the beginning of a function, fill the registers with the constants, parameters, locals, and stack levels in that order (some copy operations and some allocations)
• replace each opcode by its register based equivalent: e.g. ADD becomes ADD(S-1, S -> S-1) if the stack level is S at that point
• DUP_TOP becomes MOVE S -> S+1
• DROP_TOP disappears
• LOAD_CONST, LOAD_FAST, STORE_CONST (oops, see below), STORE_FAST all become register moves
• after this is all done, a standard register optimization will eliminate a lot of MOVE opcodes

Here a small example:
def square(x): return x ** 2
Is in bytecode (simplified):
LOAD_FAST x; LOAD_CONST 2; POW; RETURN
First transform the opcodes to register operations
Registers 0 ... 3 are filled with 2, x, <space for 2 stack levels>
Code becomes:
MOVE r1 -> r2; MOVE r0 -> r3; POW r2, r3 -> r2; RETURN r2
The optimisation trivially eliminates two registers:
POW r1, r0 -> r1; RETURN r1

There are of course many details, such as free variables, globals, default values, etc but I don't see why this wouldn't work.
This would make it easy to make a register based interpreter without having to rewrite big chunks of code. All you need is:

• a compiler post processor that does the above operation
• a piece of code to set up / allocate the register array at the beginning of the function
• a transformed version of each byte code. With a bit of luck, that could be done partly automatically using the new DSL

(A day later I realized this could work with 3 address as well as 2 address instructions. The latter one is probably going to be faster.)

[gvanrossum]

I'd love to see that STORE_CONST instruction. :-)

[sweeneyde]

How hard would it be to support CPython's current line-tracing behavior (e.g. PEP 626) with register-based bytecode? Looking at that RegCPython repo's test_sys_settrace.py, it looks like they disabled many of the tracing tests.

[gvanrossum]

Do they support settrace at all? If not, that's cheating. It's what everybody looking for a quick win does -- drop tracing support, interpreter gets faster (Cinder does it too IIRC).

But it shouldn't be hard to support it the same way we currently do in 3.11 and main: the tracing flag is either 0 or 255, and or-ed in with the opcode; and case 255 has special handling code that does the tracing and then executes the original opcode.

[carljm]

I think with a register-based interpreter you'd also have fewer opcodes and so might end up needing somewhat more NOPs for locations in some cases in order to cover all the lines?

Cinder does it too IIRC

In the JIT we don't support tracing. Not worth it, if you want to trace just turn off the JIT. We haven't done anything to disturb tracing in the interpreter.

[gvanrossum]

Ah, that makes sense, sorry. FWIW I feel that inserting NOPs just to trace all lines is being hyper-correct. But we're getting ahead of ourselves here.

If we're serious we should at least ask the paper author to sign the PSF CLA, so we can read the code with imputiny (and maybe submit a pro-forma PR, even if we're never going to merge it).

[jneb]

I'd love to see that STORE_CONST instruction. :-)

I admit it. It was in the middle of the night that I typed that. I had to edit it to fix a lot of typos and bad grammar. But I'm happy to make a pull request for you...

[o11c]

Note that Java is an example of using stack-based bytecode but verifying/converting it to register-based before execution. But honestly it's better to just change the bytecode format entirely.

Further details I'll post on the N-address issue just linked, since that's the interesting question.