Code Generation stage

Code Generation stage#

The code generation stage is specific to the target language. The files which describe this stage can be found in the folder [pyccel.codegen.printing](https://github.com/pyccel/pyccel/tree/devel/api/pyccel.codegen.printing.rst.

This stage exists to print the Pyccel AST in the syntax of the target language. Each target language has its own file containing a class which inherits from pyccel.codegen.printing.codeprinter.CodePrinter.

The entry point for the class CodePrinter is the function doprint. The key lines of the function doprint are the call to self._print(expr) and the call to self._format_code(lines).

_print#

The _print function internally calls a function named _print_X, where X is the type of the object. The logic of how the _print function chooses the appropriate _print_X function is detailed in the overview. These _print_X functions must have the form:

def _print_ClassName(self, stmt):
    ...
    return Y

where Y must be a string. Each of these _print_X functions should internally call the _print function on each of the PyccelAstNode elements of the object to obtain strings which can be combined to create a string describing the current object in the target language. It can be tempting to skip some of these _print calls, especially for basic types such as Literals. However it is very important to use these functions as much as possible, for several reasons:

  1. It ensures that the same conventions are used throughout the generated code

  2. It ensures that code details are not forgotten

  3. It makes refactoring much simpler

For example, it is sometimes necessary to print a 0. It is very tempting to just use '0' directly, however this would mean that the precision of the integer would not be printed. In some languages (e.g. Fortran) not specifying the precision in literals can have unintended consequences on the final result.

_format_code#

The _format_code function is run on the string generated by the _print function in order to format the code. In this context formatting the code means:

  1. Ensure that the indenting is correct to generate readable (and in the case of the Python printer, valid) code.

  2. Wrap any lines which are too long to be readable. The wrapping aims to split the line in a readable place, not just after a given number of characters (e.g. we aim to not split strings, if this is unavoidable we aim to not split words).

Language-specific semantics#

Although the semantic stage aims to produce code which is low-level enough to be written as-is, some languages have specific requirements which require further changes. The possibility of adding an additional language-specific semantic stage has been discussed, but currently these changes are made in the code generation stage.

This separation has not always been done with as much thought as necessary so improvements could be made.

Adding variables#

Although the semantic layer usually takes care of creating all variables required it is occasionally necessary to create a variable in the code generation stage (e.g. the iteration index used when printing an array). Where possible this should be done using the function Scope.get_temporary_variable. If for some reason it is not possible to use this function (e.g. because a DottedVariable must be created) then it is important to use Scope.insert_variable to insert the variable into the scope. This ensures that it will be correctly declared.

Declarations#

In order to ensure all variables are correctly declared, declarations are always the last object that is printed. The code snippet is then inserted into the final string in the correct position (near the beginning). The declarations are provided by the declarations property of the classes Module and FunctionDef. The only exception to this rule is the case of external variables.

Imports#

While printing imports is relatively simple and can be handled in a language-specific way, there is an additional consideration for imports. When a file imports another it is important that this information be stored somewhere so that the compiler stage is aware that it needs to compile and link the imported files. This additional complexity is handled through an object called _additional_imports which is exposed through the functions pyccel.codegen.printing.XCodePrinter.get_additional_imports. The exact implementation of this object differs from language to language so you should check how it is used in each language when you need it.

Fortran#

In addition to imports mentioned in the original Fortran code, it is also necessary to import objects defined in the standard. In Fortran it is best practice to denote these functions with the intrinsic flag. As they do not need linking they are treated separately from the other additional imports.

A set of function names is stored in the member dictionary FCodePrinter._constantImports. The keys of this dictionary are the names of the modules from which we wish to import. The simplest way to fill this dictionary is to use the setdefault function which adds a key and value pair, only if the key is not already present. For example to print the following:

use, intrinsic :: ISO_FORTRAN_ENV, only : stderr

FCodePrinter._constantImports should be filled with the following call:

self._constantImports.setdefault('ISO_FORTRAN_ENV', set()).add(("stderr"))

It will then be printed with the Module or Program using the function print_constant_imports.

Loop unravelling#

In Python, thanks to NumPy, many vector expressions can be handled natively. This expressiveness makes Python very readable, however not all low-level languages implement vector expressions. As a result it is often necessary to unravel implicit loops into explicit for loops. This is done using the function pyccel.ast.utilities.expand_to_loops. This function takes a pyccel.ast.core.CodeBlock and manipulates it such that the necessary loops appear.

In C this means unravelling all vector expressions to operate on scalars. The Fortran language handles some vector expressions so in this case the unravelling is not always needed. Expressions must just be unrolled so that all expressions have the same number of dimensions.

E.g:

def f(a : 'int[:,:]', b : 'int[:]'):
    c = a + b
    return c

In C with full unravelling this becomes:

array_int64_2d f(array_int64_2d a, array_int64_1d b)
{
    int64_t i;
    int64_t i_0001;
    int64_t* c_ptr;
    array_int64_2d c = {0};
    c_ptr = malloc(sizeof(int64_t) * (a.shape[INT64_C(0)] * a.shape[INT64_C(1)]));
    c = (array_int64_2d)cspan_md_layout(c_ROWMAJOR, c_ptr, a.shape[INT64_C(0)], a.shape[INT64_C(1)]);
    for (i = INT64_C(0); i < c.shape[INT64_C(0)]; i += INT64_C(1))
    {
        for (i_0001 = INT64_C(0); i_0001 < c.shape[INT64_C(1)]; i_0001 += INT64_C(1))
        {
            (*cspan_at(&c, i, i_0001)) = (*cspan_at(&a, i, i_0001)) + (*cspan_at(&b, i_0001));
        }
    }
    return c;
}

While in Fortran it becomes:

  subroutine f(a, b, c)

    implicit none

    integer(i64), allocatable, intent(out) :: c(:,:)
    integer(i64), intent(in) :: a(0:,0:)
    integer(i64), intent(in) :: b(0:)
    integer(i64) :: i

    allocate(c(0:size(a, 1_i64, i64) - 1_i64, 0:size(a, 2_i64, i64) - &
          1_i64))
    do i = 0_i64, size(c, 2_i64, i64) - 1_i64
      c(:, i) = a(:, i) + b
    end do
    return

  end subroutine f

The loop unravelling function must be called every time we want to print a CodeBlock in a language which is less flexible than Python. The algorithm proceeds by inserting indices into the arrays at the dimensions furthest from the contiguous dimension. This is done recursively until the expression is compatible with the language. Where possible the function tries to group expressions in the same for loop.

C-specific problems#

Use of C pointers#

In order to implement certain concepts in C it is important to use pointers. These concepts include returned variables and optional variables. This means that all print functions in the code must be able to correctly handle both pointers and non-pointers. In order to simplify the developer’s job and allow them to focus on the specifics of the AST element they are implementing, rather than the pointers _print_Variable and other similar print functions always print the value of the object. This is true even if the object is accessed via a pointer (i.e. (*var) is printed). If an address is required then the object must be wrapped in the pyccel.ast.c_concepts.ObjectAddress class.

Although this construction ensures that valid code can be written easily, unless care is taken unidiomatic code will be produced (e.g. (*var).shape instead of var->shape). It is therefore very important to use the AST to represent class objects, so that the pointer differentiation can be handled in as few functions as possible. The following objects are very useful for this purpose:

  • DottedVariable : Used to access member variables of a class. It may be necessary to create a DottedVariable to print an object such as a member of the array classes.

  • NumpyArraySize : The size of an array in a given dimension

  • PyccelArraySize : The total size of an array

Finally it is also important to mention the function is_c_pointer, which indicates whether or not its argument is accessed via a pointer in the C code. If code relies on the AST nodes described above the use of this function should be limited to the printing of a few low level objects. However it is occasionally useful (especially when printing the wrapper), so it is important to be aware of this function.

Multiple returns#

C is not designed to allow returning multiple objects from a function. To get around this restriction we treat the results as pointer arguments. This is implemented using the list CCodePrinter._additional_args. This list contains lists of variables which are treated as pointer arguments. It is managed as a list of lists in order to correctly append and pop the arguments in more complicated cases such as nested functions.

Optional arguments#

Optional arguments can take one of two forms depending on the default value. Either the default value is None in which case the argument is truly optional, or the default value is a literal. C does not handle either case explicitly. The latter case is the simplest to handle as we must just print the default value at the function call. This is handled by pyccel.ast.core.FunctionCall. In the former case the optional characteristic is handled using a pointer. The None value is therefore represented by a null pointer. This is an example of where a C pointer may be used unexpectedly.

There is an additional complexity in the case of truly optional arguments as it is common to see code such as:

def f(a : int = None):
    if a is None:
        a = 3
    ...

This does not pose a problem in Fortran as scalar arguments, even optional arguments, can be passed by value allowing them to be modified. However in C this is problematic as we do not wish to change the object pointed to by the pointer. In order to get around this problem, the CCodePrinter contains the dictionary _optional_partners. This dictionary maps optional arguments to local variables. This allows the above code to be translated to:

void f(int64_t *a)
{
    int64_t Dummy_0000;
    if (a == NULL)
    {
        a = &Dummy_0000;
        (*a) = INT64_C(3);
    }
    ...
}

As you can see the local variable Dummy_0000 acts as a “partner” to the optional argument a. When local memory is required for a it is modified to point at its partner. In this way it has access to local memory without modifying the object passed in the argument.

Nested objects#

Currently nested objects are not supported in C. This includes both nested functions and functions in classes, however we plan to support this soon. Several solutions were investigated in discussion #1149. The chosen solution is detailed in issue #1150. It was chosen as the simplest and fastest of the proposed solutions.

When printing nested objects in C we prepend the name of the context to the function name in order to identify it. E.g:

def f():
    def g():
        pass

becomes:

void f() {
}

void f__g() {
}

Care must be taken regarding any variables local to the enclosing function which are used in the nested function. These variables are noted as global variables in the nested function, however as they are not entirely global they must be passed as arguments. If they are not modified then they can be passed as normal arguments, however if they are modified they must be passed as pointers and be saved in CCodePrinter._additional_args (see Multiple returns).

Arrays#

Unlike Fortran, C does not have any native support for arrays. In order to translate array expressions, Pyccel contains a basic array implementation. This implementation can be found in the folder pyccel/stdlib/ndarrays/. It is heavily inspired by the NumPy implementation which makes it easy to collect the array from the NumPy object and pass it to the function.

When adding new array functionalities to Pyccel developers must therefore consider whether it is better/more readable to implement the function in the Pyccel’s standard library (stdlib) or in the generated code. Functions added to the standard library can be more easily optimised, however such implementations can easily lead to large amounts of code duplication, especially if the function involves data manipulation and supports arguments such as axis. On the other hand implementations written directly in the generated code can decrease readability of the code (try translating the print of an array for an example).

In the future we hope to add an additional option to avoid some of these problems where an implementation of common low-level functions (e.g. sum) is written in Python and Pyccel is used to provide the language-specific equivalent of this function with the required argument types. This option requires some discussion amongst the senior developers and some additional functions (e.g. reshape/ravel/flatten).

Fortran-specific problems#

0-based arrays#

Unlike in the majority of coding languages, Fortran array indexes do not have to start from 0. In fact, the default start index is 1. As a result this needs to be taken into consideration when writing code.

There are two possibilities to handle this; either we can modify every index by adding 1, or we can define our arrays in Fortran with syntax which ensures that the indexes start from 0. In Pyccel we have chosen the latter option. It is therefore very important to think about this when allocating arrays or pointers.

Ordering#

Different languages handle multi-dimensional arrays in different ways. This means that optimal indexing is not written in the same way in different languages. Specifically C uses row-major indexing, while Fortran uses column-major indexing. Python uses C ordering for tuples and lists, while NumPy allows both indexing conventions, but defaults to C ordering. As a result this must be taken into consideration when translating to Fortran in order to preserve efficient code. In order to avoid unnecessary copying of array arguments Pyccel handles this difference via the printing. This solution is quite complex so it has its own documentation.

Please see the Order Docs for more details.

Python-specific problems#

Translating Python to Python is theoretically very simple and serves mostly to test our AST (although this functionality is also leveraged by packages such as PSYDAC). However there are occasionally complications as the AST is more complex than necessary to describe a Python function.

Temporary variables#

Pyccel occasionally introduces temporary variables to help express statements in low-level languages. Where the language permits, we aim to not print these variables. As they did not exist in the original Python file it should therefore be possible to completely avoid these variables when printing Python code. The variables can be identified by the is_temp property.

An example of this is return variables. In Pyccel the objects returned by a function are always saved into variables. This is useful as it provides a Variable to help define the function signature. In addition this variable is necessary in Fortran, and in C if the function returns more than one object. However in Python these variables are not necessary. The PythonCodePrinter must therefore take care to avoid printing them.

Contents