Contents
The object space creates all objects and knows how to perform operations on the objects. You may think of an object space as being a library offering a fixed API, a set of operations, with implementations that correspond to the known semantics of Python objects. An example of an operation is add: add’s implementations are, for example, responsible for performing numeric addition when add works on numbers, concatenation when add works on built-in sequences.
All object-space operations take and return application-level objects. There are only a few, very simple, object-space operations which allow the bytecode interpreter to gain some knowledge about the value of an application-level object. The most important one is is_true(), which returns a boolean interpreter-level value. This is necessary to implement, for example, if-statements (or rather, to be pedantic, to implement the conditional-branching bytecodes into which if-statements get compiled).
We have many working object spaces which can be plugged into the bytecode interpreter:
The present document gives a description of the above object spaces. The sources of PyPy contain the various object spaces in the directory pypy/objspace/.
To choose which object space to use, use the :config:`objspace.name` option.
This is the public API that all Object Spaces implement.
These functions both take and return “wrapped” objects.
The following functions implement operations with a straightforward semantic - they directly correspond to language-level constructs:
id, type, issubtype, iter, next, repr, str, len, hash,
getattr, setattr, delattr, getitem, setitem, delitem,
pos, neg, abs, invert, add, sub, mul, truediv, floordiv, div, mod, divmod, pow, lshift, rshift, and_, or_, xor,
nonzero, hex, oct, int, float, long, ord,
lt, le, eq, ne, gt, ge, cmp, coerce, contains,
inplace_add, inplace_sub, inplace_mul, inplace_truediv, inplace_floordiv, inplace_div, inplace_mod, inplace_pow, inplace_lshift, inplace_rshift, inplace_and, inplace_or, inplace_xor,
get, set, delete, userdel
The following functions are part of the object space interface but would not be strictly necessary because they can be expressed using several other object space methods. However, they are used so often that it seemed worthwhile to introduce them as shortcuts.
space.builtin: The Module containing the builtins
space.sys: The ‘sys’ Module
space.w_None: The ObjSpace’s None
space.w_True: The ObjSpace’s True
space.w_False: The ObjSpace’s False
space.w_Ellipsis: The ObjSpace’s Ellipsis
space.w_NotImplemented: The ObjSpace’s NotImplemented
space.w_int, w_float, w_long, w_tuple, w_str, w_unicode, w_type, w_instance, w_slice: Python’s most common type objects
space.w_XxxError`` for each exception class XxxError (e.g. space.w_KeyError, space.w_IndexError, etc.).
List of tuples (method name, symbol, number of arguments, list of special names) for the regular part of the interface. (Tuples are interpreter level.)
List of names of built-in modules.
List of names of the constants that the object space should define
List of names of exception classes.
List of names of methods that have an irregular API (take and/or return non-wrapped objects).
The Standard Object Space (pypy/objspace/std/) is the direct equivalent of CPython’s object library (the “Objects/” subdirectory in the distribution). It is an implementation of the common Python types in a lower-level language.
The Standard Object Space defines an abstract parent class, W_Object, and a bunch of subclasses like W_IntObject, W_ListObject, and so on. A wrapped object (a “black box” for the bytecode interpreter main loop) is thus an instance of one of these classes. When the main loop invokes an operation, say the addition, between two wrapped objects w1 and w2, the Standard Object Space does some internal dispatching (similar to “Object/abstract.c” in CPython) and invokes a method of the proper W_XyzObject class that can do the operation. The operation itself is done with the primitives allowed by RPython. The result is constructed as a wrapped object again. For example, compare the following implementation of integer addition with the function “int_add()” in “Object/intobject.c”:
def add__Int_Int(space, w_int1, w_int2):
x = w_int1.intval
y = w_int2.intval
try:
z = ovfcheck(x + y)
except OverflowError:
raise FailedToImplementArgs(space.w_OverflowError,
space.wrap("integer addition"))
return W_IntObject(space, z)
Why such a burden just for integer objects? Why did we have to wrap them into W_IntObject instances? For them it seems it would have been sufficient just to use plain Python integers. But this argumentation fails just like it fails for more complex kind of objects. Wrapping them just like everything else is the cleanest solution. You could introduce case testing wherever you use a wrapped object, to know if it is a plain integer or an instance of (a subclass of) W_Object. But that makes the whole program more complicated. The equivalent in CPython would be to use PyObject* pointers all around except when the object is an integer (after all, integers are directly available in C too). You could represent small integers as odd-valuated pointers. But it puts extra burden on the whole C code, so the CPython team avoided it. (In our case it is an optimization that we eventually made, but not hard-coded at this level - see Standard Interpreter Optimizations.)
So in summary: wrapping integers as instances is the simple path, while using plain integers instead is the complex path, not the other way around.
The larger part of the pypy/objspace/std/ package defines and implements the library of Python’s standard built-in object types. Each type (int, float, list, tuple, str, type, etc.) is typically implemented by two modules:
The xxxtype.py module basically defines the type object itself. For example, pypy/objspace/std/listtype.py contains the specification of the object you get when you type list in a PyPy prompt. pypy/objspace/std/listtype.py enumerates the methods specific to lists, like append().
A particular method implemented by all types is the __new__() special method, which in Python’s new-style-classes world is responsible for creating an instance of the type. In PyPy, __new__() locates and imports the module implementing instances of the type, and creates such an instance based on the arguments the user supplied to the constructor. For example, pypy/objspace/std/tupletype.py defines __new__() to import the class W_TupleObject from pypy/objspace/std/tupleobject.py and instantiate it. The pypy/objspace/std/tupleobject.py then contains a “real” implementation of tuples: the way the data is stored in the W_TupleObject class, how the operations work, etc.
The goal of the above module layout is to cleanly separate the Python type object, visible to the user, and the actual implementation of its instances. It is possible to provide several implementations of the instances of the same Python type, by writing several W_XxxObject classes. Every place that instantiates a new object of that Python type can decide which W_XxxObject class to instantiate. For example, the regular string implementation is W_StringObject, but we also have a W_StringSliceObject class whose instances contain a string, a start index, and a stop index; it is used as the result of a string slicing operation to avoid the copy of all the characters in the slice into a new buffer.
From the user’s point of view, the multiple internal W_XxxObject classes are not visible: they are still all instances of exactly the same Python type. PyPy knows that (e.g.) the application-level type of its interpreter-level W_StringObject instances is str because there is a typedef class attribute in W_StringObject which points back to the string type specification from pypy/objspace/std/stringtype.py; all other implementations of strings use the same typedef from pypy/objspace/std/stringtype.py.
For other examples of multiple implementations of the same Python type, see Standard Interpreter Optimizations.
The Standard Object Space allows multiple object implementations per Python type - this is based on multimethods. For a description of the multimethod variant that we implemented and which features it supports, see the comment at the start of pypy/objspace/std/multimethod.py. However, multimethods alone are not enough for the Standard Object Space: the complete picture spans several levels in order to emulate the exact Python semantics.
Consider the example of the space.getitem(w_a, w_b) operation, corresponding to the application-level syntax a[b]. The Standard Object Space contains a corresponding getitem multimethod and a family of functions that implement the multimethod for various combination of argument classes - more precisely, for various combinations of the interpreter-level classes of the arguments. Here are some examples of functions implementing the getitem multimethod:
Note how the multimethod dispatch logic helps writing new object implementations without having to insert hooks into existing code. Note first how we could have defined a regular method-based API that new object implementations must provide, and call these methods from the space operations. The problem with this approach is that some Python operators are naturally binary or N-ary. Consider for example the addition operation: for the basic string implementation it is a simple concatenation-by-copy, but it can have a rather more subtle implementation for strings done as ropes. It is also likely that concatenating a basic string with a rope string could have its own dedicated implementation - and yet another implementation for a rope string with a basic string. With multimethods, we can have an orthogonally-defined implementation for each combination.
The multimethods mechanism also supports delegate functions, which are converters between two object implementations. The dispatch logic knows how to insert calls to delegates if it encounters combinations of interp-level classes which is not directly implemented. For example, we have no specific implementation for the concatenation of a basic string and a StringSlice object; when the user adds two such strings, then the StringSlice object is converted to a basic string (that is, a temporarily copy is built), and the concatenation is performed on the resulting pair of basic strings. This is similar to the C++ method overloading resolution mechanism (but occurs at runtime).
The complete picture is more complicated because the Python object model is based on descriptors: the types int, str, etc. must have methods __add__, __mul__, etc. that take two arguments including the self. These methods must perform the operation or return NotImplemented if the second argument is not of a type that it doesn’t know how to handle.
The Standard Object Space creates these methods by slicing the multimethod tables. Each method is automatically generated from a subset of the registered implementations of the corresponding multimethod. This slicing is performed on the first argument, in order to keep only the implementations whose first argument’s interpreter-level class matches the declared Python-level type.
For example, in a baseline PyPy, int.__add__ is just calling the function add__Int_Int, which is the only registered implementation for add whose first argument is an implementation of the int Python type. On the other hand, if we enable integers implemented as tagged pointers, then there is another matching implementation: add__SmallInt_SmallInt. In this case, the Python-level method int.__add__ is implemented by trying to dispatch between these two functions based on the interp-level type of the two arguments.
Similarly, the reverse methods (__radd__ and others) are obtained by slicing the multimethod tables to keep only the functions whose second argument has the correct Python-level type.
Slicing is actually a good way to reproduce the details of the object model as seen in CPython: slicing is attempted for every Python types for every multimethod, but the __xyz__ Python methods are only put into the Python type when the resulting slices are not empty. This is how our int type has no __getitem__ method, for example. Additionally, slicing ensures that 5 .__add__(6L) correctly returns NotImplemented (because this particular slice does not include add__Long_Long and there is no add__Int_Long), which leads to 6L.__radd__(5) being called, as in CPython.
The Trace Object Space was first written at the Amsterdam sprint. The ease with which the Trace Object Space was implemented in pypy/objspace/trace.py underlines the power of the Object Space abstraction. Effectively it is a simple proxy object space. It has gone through various refactors to reach its original objective, which was to show how bytecode in code objects ultimately performs computation via an object space.
This space will intercept space operations in realtime and as a side effect will memorize them. It also traces frame creation, deletion and bytecode execution. Its implementation delegates to another object space - usually the standard object space - in order to carry out the operations.
The pretty printing aims to be a graphical way of introducing programmers, and especially ones familiar with CPython, to how PyPy works from a bytecode and frames perspective. As a result one can grasp an intuitive idea of how Abstract Interpretation records via tracing all execution paths of the individual operations if one removes the bytecode out of the equation. This is the purpose of the Flow Object Space.
Another educational use of Trace Object Space is that it allows a Python user who has little understanding of how the interpreter works, a rapid way of understanding what bytecodes are and what an object space is. When a statement or expression is typed on the command line, one can see what is happening behind the scenes. This will hopefully give users a better mental framework when they are writing code.
To make use of the tracing facilities you can at runtime switch your interactive session to tracing mode by typing:
>>> __pytrace__ = 1
Note that tracing mode will not show or record all space operations by default to avoid presenting too much information. Only non-helper operations are usually shown.
A quick introduction on how to use the trace object space can be found here. A number of options for configuration is here in pypy/tool/traceconfig.py.
The task of the FlowObjSpace (the source is at pypy/objspace/flow/) is to generate a control-flow graph from a function. This graph will also contain a trace of the individual operations, so that it is actually just an alternate representation for the function.
The FlowObjSpace is an object space, which means that it exports the standard object space interface and it is driven by the bytecode interpreter.
The basic idea is that if the bytecode interpreter is given a function, e.g.:
def f(n):
return 3*n+2
it will do whatever bytecode dispatching and stack-shuffling needed, during which it issues a sequence of calls to the object space. The FlowObjSpace merely records these calls (corresponding to “operations”) in a structure called a basic block. To track which value goes where, the FlowObjSpace invents placeholder “wrapped objects” and give them to the interpreter, so that they appear in some next operation. This technique is an example of Abstract Interpretation.
For example, if the placeholder v1 is given as the argument to the above function, the bytecode interpreter will call v2 = space.mul(space.wrap(3), v1) and then v3 = space.add(v2, space.wrap(2)) and return v3 as the result. During these calls the FlowObjSpace will record a basic block:
Block(v1): # input argument
v2 = mul(Constant(3), v1)
v3 = add(v2, Constant(2))
The data structures built up by the flow object space are described in the translation document.
The FlowObjSpace works by recording all operations issued by the bytecode interpreter into basic blocks. A basic block ends in one of two cases: when the bytecode interpreters calls is_true(), or when a joinpoint is reached.
(This section to be extended...)
We have implemented several proxy object spaces which wrap another space (typically the standard one) and add some capability to all objects. These object spaces are documented in a separate page: What PyPy can do for your objects.