.. _build-system-python: =================== Python Build System =================== .. tip:: You can :download:`download the entire example project in one archive `. Required Software ================= In order to use the described build system for |python| you need the following software on your computer: * |python| >= 2.7 * :ref:`RSB ` as an exemplary dependency Brief Overview ============== We will demonstrate how to set up a |python| project using the standard build tool `setuptools`_. `setuptools`_ is configured through a |python| script file in the project root directory, called :file:`setup.py`. In this tutorial we will build a |python| package containing some reusable functions for :term:`downstream` projects. Additionally, the module contains an exemplary function which will be exposed as a script separately installed in the filesystem and callable from the command line. We assume knowledge about the |python| programming language and its structure of modules and packages. Additionally, some rough ideas about how `setuptools`_ can be used from client perspective, which means how to call the :file:`setup.py` to build and install a |python| package are beneficial. .. note:: The structure proposed is mostly based on `this tutorial `_ from the official |python| website. It might be a good idea to quickly browse over this tutorial. Folder Layout ============= The folder layout of our tutorial project will look as follows: * **build-system-essentials-python**: top-level folder of our project * :file:`setup.py`: Build declarations and dependency handling * :file:`setup.cfg`: Configuration options for the build system * :file:`README.txt`, :file:`COPYING.txt`, ...: Files with additional textual information like license, contact information etc. * **src**: Source tree with the actual project code which will also be installed * **test**: Unit tests for the project as separate |python| modules Declaring a setuptools Project and Finding Upstream Dependencies ================================================================ .. seealso:: `setuptools online documentation `_ Online overview of setuptools We start by declaring the basic project in :file:`setup.py`: .. literalinclude:: example-projects/build-system-essentials-python/setup.py :language: python :start-after: mark-start::setup-project :end-before: mark-end::setup-project :linenos: The main work is done through the :py:func:`setup` function provided by `setuptools`_, which is configured through a lengthy list of arguments. The first things to configure are the project name, version and various other metadata. Next, we declare the dependencies of our project: .. literalinclude:: example-projects/build-system-essentials-python/setup.py :language: python :start-after: mark-start::dependencies :end-before: mark-end::dependencies :linenos: ``setup_requires`` declares the |python| packages which are only required to compile the module from source using the provided :file:`setup.py`. ``install_requires`` declares packages that need to be present all the time. For setup time, we require `setuptools-epydoc `_ for API documentation generation and `nose `_ to easily run the unit tests with useful features like XML output. `coverage `_ is used by `nose`_ to generate a code coverage report. At runtime we require the :ref:`RSB ` |python| implementation. If these dependencies are not met, `setuptools`_ will automatically download and build them inside the project's root folder. .. note:: |python| has a central package server called `PyPI - the Python Package Index `_. Package names given in the various ``*_requires`` variables need to match the packages registered at this server. Building Source Code ==================== As no real compilation is required in |python|, the main task we have to fulfill is listing the packages which shall be installed later. Therefore, we have to list each production package in the ``packages`` argument. To avoid the manual listing of many packages in complex projects, `setuptools`_ provides a function called :py:func:`find_packages` to find all available packages in a specific folder. We use this function to list all packages in the source tree. These are the packages that shall be installed later excluding the unit tests. As we have placed the source tree outside of the project's base directory (and instead inside the ``src`` folder) we need to inform the setup script about this. This is done with the second line. The meaning of the map specified here as argument is that all packages (starting from an empty package name with all children) can be found in the ``src`` directory. .. literalinclude:: example-projects/build-system-essentials-python/setup.py :language: python :start-after: mark-start::packages :end-before: mark-end::packages :linenos: Apart from building the source code, `setuptools`_ provides additional support for creating executable console scripts in a portable way. In our case we have defined the function :py:func:`sendMagicNumber` in the module :py:mod:`buildsystemessentials.executable` which shall be installed as an executable script. This fact can be declared in `setuptools`_ as follows: .. literalinclude:: example-projects/build-system-essentials-python/setup.py :language: python :start-after: mark-start::scripts :end-before: mark-end::scripts :linenos: Executing Unit Tests ==================== As we are using `nose`_ to run the unit tests, no actual coding work has to be done for this step apart from registering `nose`_ for executing unit tests. This is done by adding the following fragment to the argument list of the :py:func:`setup` function in :file:`setup.py`: .. literalinclude:: example-projects/build-system-essentials-python/setup.py :language: python :start-after: mark-start::tests :end-before: mark-end::tests :linenos: Moreover, `nose`_ needs to be configured correctly to cope with the folder layout of our project. Configuring `nose`_ means in the `setuptools`_ language to provide some options to the ``nosetests`` command. Such options can be given in two ways. On the one hand, using the command line. E.g. the available options of the ``nosetests`` command can be gathered using ``python setup.py nosetests -h``. On the other hand, a configuration file can be used. As we have to set several of these options and we do not want to specify them on the command line each time we want to execute the unit tests, we will use a configuration file. `setuptools`_ provides a single file called :file:`setup.cfg`, where options for all available commands can be specified. Using this file in the project root, we will first instruct `nose`_ to execute all tests available in the ``test`` folder. Moreover, we instruct it to output test results as an XML file in the XUnit format, which is quite well understood by many tools like `Eclipse `_ and `Jenkins `_. .. literalinclude:: example-projects/build-system-essentials-python/setup.cfg :language: ini :start-after: mark-start::test-layout :end-before: mark-end::test-layout :linenos: `nose`_ can also be used to calculate the `code coverage `_, which means the percentage of production code executed by the unit tests. This is a useful feature to asses the code and test quality which we will also enable: .. literalinclude:: example-projects/build-system-essentials-python/setup.cfg :language: ini :start-after: mark-start::coverage :end-before: mark-end::coverage :linenos: After enabling the coverage feature we instruct `nose`_ use the ``where`` clause to set the project root for searching for modules. This is required for ``cover-inclusive`` to work correctly, which instructs `nose`_ to also include modules in the coverage calculation which are not executed in the tests (so complete lack of coverage). Otherwise, these files do not contribute to the percentage of coverage. With ``cover-branches`` we instruct `nose`_ to also calculate branch metrics, e.g. for every ``if`` statement and ``cover-erase`` ensures that that old coverage data will be deleted before calculating the new coverage. Finally, several output options are set so that an XML report for tools like `Jenkins`_ is generated and an HTML report for easily inspecting the coverage results. .. note:: With `nose`_ unit test modules and classes need to follow a specific naming scheme for being discovered by `nose`_. Please refer to http://nose.readthedocs.org/en/latest/writing_tests.html for the pattern definition. .. note:: Most of the coverage features have been added to `nose`_ in version 1.3. Unfortunately, most Linux distributions still ship an older version. This usually should not be a problem, since `setuptools`_ downloads required dependencies automatically. However, this fails if an older version is already on the ``PYTHONPATH`` and that location is not writable for the current user. So a common situation for normal users without root permissions on a Linux system where the `nose`_ package is installed. `setuptools`_ will die with a ``VersionConflict`` error in this case. There are two sub-optimal solutions to work around this: #. (Delete the `nose`_ package in the operating system.) #. Downgrade your project to the required `nose`_ version and drop the coverage support. #. Prevent the existing `nose`_ package from ending up on the ``PYTHONPATH``, e.g. by using `virtualenv `_. Both solutions have drawbacks and basically mirror reflect a bad design decision in `setuptools`_: Since `nose`_ is only a requirement for the setup phase and is not required for the installed product, there is no reason why `setuptools`_ shouldn't just download the latest version and use it from the local project directory. Exposing the Project to Downstream Projects =========================================== With the aforementioned definitions, `setuptools`_ automatically provides an install command and nothing additional has to be done for this to work. The only thing left to do is generating the API documentation for :term:`downstream` developers. This is performed by calling .. code-block:: sh python setup.py epydoc This command is provided by `setuptools-epydoc`_, which we have previously declared as a build dependency. How to Build and Use the Project ================================ .. code-block:: sh cd path/to/build-system-essentials-python # build python setup.py build # run tests python setup.py nosetests # build documentation python setup.py doc # install python setup.py install --prefix=/your/prefix # clean (some of the) generated files python setup.py clean .. note:: Installation might fail with an error message if the target directory (including the path to the mentioned site-packages location) does not exist or is not listed in the environment variable :envvar:`PYTHONPATH`. In such cases create the directory manually and add it to the :envvar:`PYTHONPATH`. .. note:: In case you are actively developing a Python component you might also want to try ``python setup.py develop --prefix=/your/prefix`` instead of the above mentioned ``install`` command. This will install your project to the prefix in a way that the actual source files are directly taken from your working directory without copying them over to the prefix. Hence, you do not need to reinstall the project after every change for testing.