Python Build System

Tip

You can download the entire example project in one archive.

Required Software

In order to use the described build system for Python you need the following software on your computer:

• Python >= 2.7
• RSB as an exemplary dependency

Brief Overview

We will demonstrate how to set up a Python project using the standard build tool setuptools. setuptools is configured through a Python script file in the project root directory, called setup.py.

In this tutorial we will build a Python package containing some reusable functions for downstream projects. Additionally, the module contains an exemplary function which will be exposed as a script separately installed in the filesystem and callable from the command line.

We assume knowledge about the Python programming language and its structure of modules and packages. Additionally, some rough ideas about how setuptools can be used from client perspective, which means how to call the setup.py to build and install a Python package are beneficial.

Note

The structure proposed is mostly based on this tutorial from the official Python website. It might be a good idea to quickly browse over this tutorial.

Folder Layout

The folder layout of our tutorial project will look as follows:

• build-system-essentials-python: top-level folder of our project
  • setup.py: Build declarations and dependency handling
  • setup.cfg: Configuration options for the build system
  • README.txt, COPYING.txt, …: Files with additional textual information like license, contact information etc.
  • src: Source tree with the actual project code which will also be installed
  • test: Unit tests for the project as separate Python modules

Declaring a setuptools Project and Finding Upstream Dependencies

See also

setuptools online documentation

Online overview of setuptools

We start by declaring the basic project in setup.py:

setup(name='buildsystemessentials',
      version='0.1.0',
      description='''
                  An exemplary python build
                  ''',
      author='Johannes Wienke',
      author_email='jwienke@techfak.uni-bielefeld.de',
      license='LGPLv3+',
      url='http://docs.cor-lab.org',
      keywords=['example'],
      classifiers=['Programming Language :: Python'],

The main work is done through the setup() function provided by setuptools, which is configured through a lengthy list of arguments. The first things to configure are the project name, version and various other metadata.

Next, we declare the dependencies of our project:

      setup_requires=['setuptools-epydoc', 'nose>=1.3', 'coverage'],
      install_requires=['rsb-python'],

setup_requires declares the Python packages which are only required to compile the module from source using the provided setup.py. install_requires declares packages that need to be present all the time.

For setup time, we require setuptools-epydoc for API documentation generation and nose to easily run the unit tests with useful features like XML output. coverage is used by nose to generate a code coverage report. At runtime we require the RSB Python implementation. If these dependencies are not met, setuptools will automatically download and build them inside the project’s root folder.

Note

Python has a central package server called PyPI - the Python Package Index. Package names given in the various *_requires variables need to match the packages registered at this server.

Building Source Code

As no real compilation is required in Python, the main task we have to fulfill is listing the packages which shall be installed later. Therefore, we have to list each production package in the packages argument. To avoid the manual listing of many packages in complex projects, setuptools provides a function called find_packages() to find all available packages in a specific folder. We use this function to list all packages in the source tree. These are the packages that shall be installed later excluding the unit tests. As we have placed the source tree outside of the project’s base directory (and instead inside the src folder) we need to inform the setup script about this. This is done with the second line. The meaning of the map specified here as argument is that all packages (starting from an empty package name with all children) can be found in the src directory.

      packages=find_packages('src'),
      package_dir={'': 'src'},

Apart from building the source code, setuptools provides additional support for creating executable console scripts in a portable way. In our case we have defined the function sendMagicNumber() in the module buildsystemessentials.executable which shall be installed as an executable script. This fact can be declared in setuptools as follows:

      entry_points={
          'console_scripts': [
                 'sendMagicNumber = buildsystemessentials.executable:sendMagicNumber',
              ]
      },

Executing Unit Tests

As we are using nose to run the unit tests, no actual coding work has to be done for this step apart from registering nose for executing unit tests. This is done by adding the following fragment to the argument list of the setup() function in setup.py:

      test_suite='nose.collector',

Moreover, nose needs to be configured correctly to cope with the folder layout of our project. Configuring nose means in the setuptools language to provide some options to the nosetests command. Such options can be given in two ways. On the one hand, using the command line. E.g. the available options of the nosetests command can be gathered using python setup.py nosetests -h. On the other hand, a configuration file can be used. As we have to set several of these options and we do not want to specify them on the command line each time we want to execute the unit tests, we will use a configuration file. setuptools provides a single file called setup.cfg, where options for all available commands can be specified. Using this file in the project root, we will first instruct nose to execute all tests available in the test folder. Moreover, we instruct it to output test results as an XML file in the XUnit format, which is quite well understood by many tools like Eclipse and Jenkins.

[nosetests]
tests = test
with-xunit = 1

nose can also be used to calculate the code coverage, which means the percentage of production code executed by the unit tests. This is a useful feature to asses the code and test quality which we will also enable:

with-coverage = 1
where = .
cover-inclusive = 1
cover-branches = 1
cover-erase = 1
cover-xml = 1
cover-xml-file = ../coverage.xml
cover-html = 1
cover-html-dir = ../coverage

After enabling the coverage feature we instruct nose use the where clause to set the project root for searching for modules. This is required for cover-inclusive to work correctly, which instructs nose to also include modules in the coverage calculation which are not executed in the tests (so complete lack of coverage). Otherwise, these files do not contribute to the percentage of coverage. With cover-branches we instruct nose to also calculate branch metrics, e.g. for every if statement and cover-erase ensures that that old coverage data will be deleted before calculating the new coverage. Finally, several output options are set so that an XML report for tools like Jenkins is generated and an HTML report for easily inspecting the coverage results.

Note

With nose unit test modules and classes need to follow a specific naming scheme for being discovered by nose. Please refer to http://nose.readthedocs.org/en/latest/writing_tests.html for the pattern definition.

Note

Most of the coverage features have been added to nose in version 1.3. Unfortunately, most Linux distributions still ship an older version. This usually should not be a problem, since setuptools downloads required dependencies automatically. However, this fails if an older version is already on the PYTHONPATH and that location is not writable for the current user. So a common situation for normal users without root permissions on a Linux system where the nose package is installed. setuptools will die with a VersionConflict error in this case. There are two sub-optimal solutions to work around this:

1. (Delete the nose package in the operating system.)
2. Downgrade your project to the required nose version and drop the coverage support.
3. Prevent the existing nose package from ending up on the PYTHONPATH, e.g. by using virtualenv.

Both solutions have drawbacks and basically mirror reflect a bad design decision in setuptools: Since nose is only a requirement for the setup phase and is not required for the installed product, there is no reason why setuptools shouldn’t just download the latest version and use it from the local project directory.

Exposing the Project to Downstream Projects

With the aforementioned definitions, setuptools automatically provides an install command and nothing additional has to be done for this to work.

The only thing left to do is generating the API documentation for downstream developers. This is performed by calling

python setup.py epydoc

This command is provided by setuptools-epydoc, which we have previously declared as a build dependency.

How to Build and Use the Project

cd path/to/build-system-essentials-python

# build
python setup.py build

# run tests
python setup.py nosetests

# build documentation
python setup.py doc

# install
python setup.py install --prefix=/your/prefix

# clean (some of the) generated files
python setup.py clean

Note

Installation might fail with an error message if the target directory (including the path to the mentioned site-packages location) does not exist or is not listed in the environment variable PYTHONPATH. In such cases create the directory manually and add it to the PYTHONPATH.

Note

In case you are actively developing a Python component you might also want to try python setup.py develop --prefix=/your/prefix instead of the above mentioned install command. This will install your project to the prefix in a way that the actual source files are directly taken from your working directory without copying them over to the prefix. Hence, you do not need to reinstall the project after every change for testing.