Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
210 changes: 94 additions & 116 deletions mypyc/doc/introduction.rst
Original file line number Diff line number Diff line change
@@ -1,167 +1,145 @@
Introduction
============

Mypyc is a compiler for a strict, statically typed Python variant that
generates CPython C extension modules. Code compiled with mypyc is
often much faster than CPython. Mypyc uses Python `type hints
Mypyc compiles Python modules to C extensions. It uses standard Python
`type hints
<https://mypy.readthedocs.io/en/stable/cheat_sheet_py3.html>`_ to
generate fast code, and it also restricts the use of some dynamic
Python features to gain performance.
generate fast code.

The compiled language is a strict, statically typed Python variant.
It restricts the use of some dynamic Python features to gain performance,
but it's mostly compatible with standard Python.

Mypyc uses `mypy <http://www.mypy-lang.org/>`_ to perform type
checking and type inference. Most type checking features in the stdlib
checking and type inference. Most type system features in the stdlib
`typing <https://docs.python.org/3/library/typing.html>`_ module are
supported, including generic types, optional and union types, tuple
types, and type variables. Using type hints is not necessary, but type
annotations are the key to impressive performance gains.
supported.

Compiled modules can import arbitrary Python modules, including
third-party libraries, and compiled modules can be freely used from
other Python modules. Often you'd use mypyc to only compile modules
with performance bottlenecks.
Compiled modules can import arbitrary Python modules and third-party
libraries.

You can run the modules you compile also as normal, interpreted Python
modules. Mypyc only compiles valid Python code. This means that all
Python developer tools and debuggers can be used, though some only
fully work in interpreted mode.
modules.

How fast is mypyc
-----------------
You can roughly expect speedups like these (2x improvement means half the
runtime):

The speed improvement from compilation depends on many factors.
Certain operations will be a lot faster, while others will get no
speedup.
* Existing code with type annotations often gets **1.5x to 5x** faster.

These estimates give a rough idea of what to expect (2x improvement
halves the runtime):
* Code tuned for mypyc can be **5x to 15x** faster.

* Typical code with type annotations may get **1.5x to 5x** faster.
There is no simple answer to how fast your code will be when compiled.
You should try it out!

* Typical code with *no* type annotations may get **1.0x to 1.3x**
faster.
Mypyc currently aims to speed up non-numeric code, such as server
applications. We also use mypyc to compile mypyc, of course!

* Code optimized for mypyc may get **5x to 10x** faster.
Motivation
----------

Remember that only performance of compiled modules improves. Time
spent in libraries or on I/O will not change (unless you also compile
libraries).
Though Python has been successful without a good performance story
for non-numeric code, speed still matters:

Why speed matters
-----------------
1. Users prefer more efficient and responsive software and libraries.

Faster code has many benefits, some obvious and others less so:
2. You need less hardware to run your server application and save money.

* Users prefer efficient and responsive applications, tools and
libraries.
3. You'll waste less time waiting for your tests and jobs to finish.

* If your server application is faster, you need less hardware, which
saves money.
4. Faster code correlates with less energy use and is better for the
environment.

* Faster code uses less energy, especially on servers that run 24/7.
This lowers your environmental footprint.
Perks
-----

* If tests or batch jobs run faster, you'll be more productive and
save time.
**Easy to get started.** Compiled code looks and behaves like normal
Python code. You get the benefits of static typing while using the
syntax, libraries and idioms you (and millions of developers) already
know.

How does mypyc work
-------------------
**Expressive types.** Mypyc fully supports standard Python type hints.
Mypyc has local type inference, generics, optional types, tuple types,
union types, and more. Type hints act as machine-checked
documentation, making code not only faster but also easier to
understand and modify.

Mypyc produces fast code via several techniques:
**Fast program startup.** Mypyc uses ahead-of-time compilation, so
compilation does not slow down program startup. Slow program startup
is a common problem with JIT compilers.

* Mypyc uses *ahead-of-time compilation* to native code. This removes
CPython interpreter overhead.

* Mypyc enforces type annotations (and type comments) at runtime,
raising ``TypeError`` if runtime types don't match annotations. This
lets mypyc use operations specialized to specific types.
**Python ecosystem supported.** Mypyc runs on top of CPython, the
standard Python implementation. Code can freely use standard library
features. Use pip to install any third-party libraries you need,
including C extensions.

* Mypyc uses *early binding* to resolve called functions and other
references at compile time. Mypyc avoids many namespace dictionary
lookups.
**Migration path for existing Python code.** Existing Python code
often requires only minor changes to compile using mypyc.

* Mypyc assumes that most compiled functions, compiled classes, and
attributes declared ``Final`` are immutable (and tries to enforce
this).
**Waiting for compilation is optional.** Compiled code also runs as
normal Python code. You can use interpreted Python during development,
with familiar and fast workflows.

* Most classes are compiled to *C extension classes*. They use
`vtables <https://en.wikipedia.org/wiki/Virtual_method_table>`_ for
fast method calls and attribute access.
**Runtime type safety.** Mypyc protects you from segfaults and memory
corruption. Any unexpected runtime type safety violation is a bug in
mypyc.

* Mypyc uses efficient (unboxed) representations for some primitive
types, such as integers and booleans.
**Find errors statically.** Mypyc uses mypy for static type checking
that will catch many bugs. This saves time you'd otherwise spend
debugging.

Why mypyc
Use cases
---------

Here are some mypyc properties and features that can be useful.
**Fix performance bottlenecks.** Often most time is spent in a few
Python modules or functions. Add type annotations and compile these
modules for easy performance gains.

**Powerful Python types.** Mypyc leverages most features of standard
Python type hint syntax, unlike tools such as Cython, which focus on
lower-level types. Our aim is that writing code feels natural and
Pythonic. Mypyc supports a modern type system with powerful features
such as local type inference, generics, optional types, tuple types
and union types. Type hints act as machine-checked documentation,
making code easier to understand and modify.
**Compile it all.** During development you use interpreted mode, for a
quick edit-run cycle. In releases all non-test code is compiled. This
is how mypy got a *4x performance improvement* over interpreted Python.

**Fast program startup.** Python implementations using a JIT compiler,
such as PyPy, slow down program startup, sometimes significantly.
Mypyc uses ahead-of-time compilation, so compilation does not slow
down program startup.
**Take advantage of existing type hints.** If you already use type
annotations in your code, adopting mypyc will be easier. You've already
done most of the work needed to use mypyc.

**Python ecosystem compatibility.** Since mypyc uses the standard
CPython runtime, you can freely use the stdlib and use pip to install
arbitary third-party libraries, including C extensions.
**Alternative to a lower-level language.** Instead of writing
performance-critical code in C, C++, Cython or Rust, you may get good
performance while staying in the comfort of Python.

**Migration path for existing Python code.** Existing Python code
often requires only minor changes to compile using mypyc.

**No need to wait for compilation.** Compiled code also runs as normal
Python code. You can use interpreted Python during development, with
familiar workflows.
**Migrate C extensions.** Maintaining C extensions is not always fun
for a Python developer. With mypyc you may get performance similar to
the original C, with the convenience of Python.

**Runtime type safety.** Mypyc aims to protect you from segfaults and
memory corruption. We consider any unexpected runtime type safety
violation as a bug.

**Find errors statically.** Mypyc uses mypy for powerful static type
checking that will catch many bugs, saving you from a lot of
debugging.
How does it work
----------------

**Easy path to static typing.** Mypyc lets Python developers easily
dip their toes into modern static typing, without having to learn all
new syntax, libraries and idioms.
Mypyc uses several techniques to produce fast code:

Use cases for mypyc
-------------------
* Mypyc uses *ahead-of-time compilation* to native code. This removes
CPython interpreter overhead.

Here are examples of use cases where mypyc can be effective.
* Mypyc enforces type annotations (and type comments) at runtime,
raising ``TypeError`` if runtime values don't match annotations.

**Address a performance bottleneck.** Profiling shows that most time
is spent in a certain Python module. Add type annotations and compile
the module for performance gains.
* Compiled code uses optimized, type-specific primitives.

**Leverage existing type hints.** You already use mypy to type check
your code. Using mypyc will now be easy, since you already use static
typing.
* Mypyc uses *early binding* to resolve called functions and name
references at compile time. Mypyc avoids many dynamic namespace
lookups.

**Compile everything.** You want your whole application to be fast.
During development you use interpreted mode, for a quick edit-run
cycle, but in your releases all (non-test) code is compiled. This is
how mypy achieved a 4x performance improvement using mypyc.
* Classes are compiled to *C extension classes*. They use `vtables
<https://en.wikipedia.org/wiki/Virtual_method_table>`_ for fast
method calls and attribute access.

**Alternative to C.** You are writing a new module that must be fast.
You write the module in Python, and try to use operations that mypyc
can optimize well. The module is much faster when compiled, and you've
saved a lot of effort compared to writing an extension in C (and you
don't need to know C).
* Mypyc treats compiled functions, classes, and attributes declared
``Final`` as immutable.

**Rewriting a C extension.** You've written a C extension, but
maintaining C code is no fun. You might be able to switch to Python
and use mypyc to get performance comparable to the original C.
* Mypyc has memory-efficient, unboxed representions for integers and
booleans.

Development status
------------------

Mypyc is currently *alpha software*. It's only recommended for
production use cases if you are willing to contribute fixes or to work
around issues you will encounter.
production use cases with careful testing, and if you are willing to
contribute fixes or to work around issues you will encounter.