Allowing users to configure and build your app on any platform

Recently, whilst producing distributables for my Python applications I required a method of automatically building the shared C and Fortran libraries. I originally hard coded calls to specific Makefiles in the setup.py file and relied on the user to configure each Makefile correctly. However, I wanted a system independent build process that ran in an automated fashion during python setup.py build.

In years of compiling and running applications on various platforms it would have been hard to avoid:

$ ./configure
$ make
$ make install

I always had a faint idea about what was going on; the system was magically producing Makefiles dependent on what it had found on my computer - but now I realised I should understand how.

GNU Autotools - really autohell?

I will not go into too much depth here (mainly because I have no idea about all of the inner workings) but the GNU Project provides a set of tools to configure source code such that the build process can be carried out easily on different platforms. A brief summary of these tools:

The best way to explain how these tools work is to use an example. If our Python application has a directory structure as follows:

my_app
    __init__.py
    clib
        __init__.py
        interface.py
        src
            clib.c
        lib
            .
    main.py
setup.py

and in interface.py we provide functions to our c library:

my_app/clib/interface.py

import ctypes
import numpy

clib = ctypes.cdll.LoadLibrary('lib/clib.so')

def do_something():
    clib.do_something_in_c()

which is called from main.py:

my_app/main.py

from clib import interface

def main():
        interface.do_something()

if __name__=="__main__":
        main()

The c library simple prints "Hello World!":

my_app/clib/src/clib.c

#include <stdio.h>
#include <string.h>
#include <stdlib.h>

void do_something_in_c()
{
        printf("Hello World from C!\n");
}

We want to be able to build the clib.so library on multiple platforms and put it in clib/lib/. We would want to do this automatically from inside our setup.py script.

Let's concentrate on the clib/ directory.

System checks (autoscan, autoheader, autoconf)

First let's run the command:

$ autoscan

which produces:

configure.scan
autoscan.log

configure.scan is created to avoid conflicts with customised configure.ac files

The important file here is configure.scan, let's take a look:

my_app/clib/configure.scan

# -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

AC_PREREQ([2.69])
AC_INIT([FULL-PACKAGE-NAME], [VERSION], [BUG-REPORT-ADDRESS])
AC_CONFIG_SRCDIR([src/clib.c])
AC_CONFIG_HEADERS([config.h])

# Checks for programs.
AC_PROG_CC

# Checks for libraries.

# Checks for header files.
AC_CHECK_HEADERS([stdlib.h string.h])

# Checks for typedefs, structures, and compiler characteristics.

# Checks for library functions.

AC_OUTPUT

As is clear from the above, autoscan has done its best to populate various fields most of which are self explanatory. The field requiring our input is AC_INIT which we can replace with:

AC_INIT(my_app_lib, 1.0, marc@ifnamemain.com)

An interesting addition to configure.scan is AC_CONFIG_HEADERS which defines a header that contains system constants. These can be checked in your source code to run system specific routines.

AC_CONFIG_HEADERS indicates to configure to look for a config.h.in file which is then used to produce config.h. However, we do not a have a config.h.in file and if we went ahead with the configure (mv configure.scan configure.ac, run autoconf and ./configure) we would get the error:

config.status: error: cannot find input file: `config.h.in`

As is the trend with the GNU autotools, there is a quick method to produce this header file:

$ autoheader

and config.h.in now contains:

my_app/clib/config.h.in

/* config.h.in.  Generated from configure.ac by autoheader.  */

/* Define to 1 if you have the <inttypes.h> header file. */
#undef HAVE_INTTYPES_H

/* Define to 1 if you have the <memory.h> header file. */
#undef HAVE_MEMORY_H

/* Define to 1 if you have the <stdint.h> header file. */
#undef HAVE_STDINT_H

/* Define to 1 if you have the <stdlib.h> header file. */
...

If no platform dependent constants are needed AC_CONFIG_HEADERS can be removed from the configure.scan file.

We are now in a position to run autoconf and produce the configure script. To do this we first need to rename the configure.scan file to configure.ac.

$ mv configure.scan configure.ac
$ autoconf

which produces the monster shell script configure.

WARNING! - Looking directly at the > 4000 lines of the configure script can cause seizures

We will not pay any attention to the contents of the configure script and if you want to examine its contents then you are braver than me. With the script in place we can run the magical command:

$ ./configure

which produces the output familiar to many:

checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking how to run the C preprocessor... gcc -E
checking for grep that handles long lines and -e... /usr/bin/grep
checking for egrep... /usr/bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
configure: creating ./config.status
config.status: creating config.h

The above shows all the checks required to compile our very simple C library.

What to build? (automake)

Up until this point we have only produced a means of checking the current system. We haven't connected what was found to rules on what to build. This is where automake comes in. automake uses template makefiles (Makefile.am) and works in conjunction with 'autoconf' to produce the fully-fledged Makefiles.

First, we shall produce a Makefile.am file in our lib directory:

my_app/clib/Makefile.am

AUTOMAKE_OPTIONS = foreign
SUBDIRS = src

The first command removes restrictions on organising the code in a way GNU expects. The second command is the important one and directs automake to our source. automake will enter this subdirectory and expect another Makefile.am, so let's create one:

my_app/clib/src/Makefile.am

lib_LTLIBRARIES = clib.la

CFLAGS = -Wall -O2
LD_FLAGS = -all-static

libdir= ${abs_top_builddir}/lib
clib_la_SOURCES = clib.c
clib_la_LDFLAGS = -module -avoid-version -shared

Here is where we define targets, flags and rules regarding the building of our shared library. Firstly we define our library name clib.la. Although this is given the extension .la, a .so library will also be built. Next are the compile and link flags. The link flags '-all-static' forces the compilation of a statically linked library. We don't really want the library dynamically linked. The libdir parameter points to the destination of the library once the make install command is issued.

With the Makefile.am files in place, we need to modify configure.ac to tell it about the new files. We introduce a new command, involving a new prefix AM directly after AC_INIT:

AM_INIT_AUTOMAKE(my_app_lib, 1.0)

We also have to initialise libtool as we will use it to create our library:

LT_INIT

Now our configure.ac looks like this:

my_app/lib/configure.ac

# -*- Autoconf -*-
# Process this file with autoconf to produce a configure script.

AC_PREREQ([2.69])
AC_INIT(my_app_lib, 1.0, marc@ifnamemain.com)
AM_INIT_AUTOMAKE
LT_INIT

AC_CONFIG_SRCDIR([src/clib.c])
AC_CONFIG_HEADERS([config.h])

# Checks for programs.
AC_PROG_CC

# Checks for libraries.

# Checks for header files.
AC_CHECK_HEADERS([stdlib.h string.h])

# Checks for typedefs, structures, and compiler characteristics.

# Checks for library functions.

AC_OUTPUT(Makefile src/Makefile)

Before running automake we need to run the command

$ libtoolize

or on OSX, typically:

$ glibtoolize

This pulls across various scripts associated with libtool such that it can be used to build the shared libraries.

We can now run automake which will read the new configure.ac file and Makefile.am files and produce Makefile.in files:

$ aclocal
$ automake --add-missing

The additional command aclocal creates a file containing macros required for automake. The --add-missing flag tells automake to pull across additional utility scripts.

Now we can finally run:

$ autoconf

which produces the all powerful configure script.

Summary

The the two files you need to produce the configure script are:

configure.ac = Point to makefile.am files, define checks makefile.am (in parent and each subdirectory) = What to build, how and where to put it

All tools are based on creating or using the above files.

Testing

With the configure script generated, we can test the build as if we have just downloaded our Python app my_app. If we cd into our clib directory, we should be able to configure, make and make install our library:

$ cd my_app/clib
$ ./configure
$ make
$ make install

The final make install command moves the built libraries into the location defined by the libdir variable in my_app/lib/src/Makefile.am (i.e. my_app/clib/lib).

We should now be able to run our Python app, along with anyone else who downloads it!:

$ python main.py
Hello World from C!

Final Notes

Things can get a bit trickier if you have multiple libraries with common libraries to link against etc. To link 'clib' against a library named common and the following to the Makefile.am:

clib_la_LIBADD = common.o

Autotools can also be used to make Fortran extensions for Python. This requires only minor modifications to the steps above, such as adding :

AC_PROG_FC

to configure.ac to detect Fortran compilers.

Good luck...