You must log in to use subscribtions.

Clear message

Table of Contents

Introduction

ctypes is an advanced Foreign Function Interface package for Python 2.3 and higher. It is included in the standard library for Python 2.5.

ctypes allows to call functions exposed from DLLs/shared libraries and has extensive facilities to create, access and manipulate simple and complicated C data types in Python - in other words: wrap libraries in pure Python. It is even possible to implement C callback functions in pure Python.

ctypes also includes a code generator tool chain which allows automatic creation of library wrappers from C header files. ctypes works on Windows, Mac OS X, Linux, Solaris, FreeBSD, OpenBSD and other systems.

Ensure that you have at least ctypes version 1.0.1 or later.

Other possibilities to call or run C code in python include: SWIG, Cython, Weave, etc

Getting Started with ctypes

The ctypes tutorial and the ctypes documentation for Python provide extensive information on getting started with ctypes.

Assuming you've built a library called foo.dll or libfoo.so containing a function called bar that takes a pointer to a buffer of doubles and an int as arguments and returns an int, the following code should get you up and running. The following sections cover some possible build scripts, C code and Python code.

If you would like to build your DLL/shared library with distutils, take a look at the SharedLibrary distutils extension included with OOF2. This should probably be included in numpy.distutils at some point.

Nmake Makefile (Windows)

Run nmake inside the Visual Studio Command Prompt to build with the following file.

You should be able to build the DLL with any version of the Visual Studio compiler regardless of the compiler used to compile Python. Keep in mind that you shouldn't allocate/deallocate memory across different debug/release and single-threaded/multi-threaded runtimes or operate on FILE*s from different runtimes.

CXX = cl.exe
LINK = link.exe

CPPFLAGS = -D_WIN32 -D_USRDLL -DFOO_DLL -DFOO_EXPORTS
CXXFLAGSALL = -nologo -EHsc -GS -W3 -Wp64 $(CPPFLAGS)
CXXFLAGSDBG = -MDd -Od -Z7 -RTCcsu
CXXFLAGSOPT = -MD -O2
#CXXFLAGS = $(CXXFLAGSALL) $(CXXFLAGSDBG)
CXXFLAGS = $(CXXFLAGSALL) $(CXXFLAGSOPT)

LINKFLAGSALL = /nologo /DLL
LINKFLAGSDBG = /DEBUG
LINKFLAGSOPT =
#LINKFLAGS = $(LINKFLAGSALL) $(LINKFLAGSDBG)
LINKFLAGS = $(LINKFLAGSALL) $(LINKFLAGSOPT)

all: foo.dll

foo.dll: foo.obj
    $(LINK) $(LINKFLAGS) foo.obj /OUT:foo.dll

svm.obj: svm.cpp svm.h
    $(CXX) $(CXXFLAGS) -c foo.cpp

clean:
    -erase /Q *.obj *.dll *.exp *.lib

SConstruct (GCC)

You can use the following file with SCons to build a shared library.

   1 env = Environment()
   2 env.Replace(CFLAGS=['-O2','-Wall','-ansi','-pedantic'])
   3 env.SharedLibrary('foo', ['foo.cpp'])

foo.cpp

   1 #include <stdio.h>
   2 
   3 #ifdef FOO_DLL
   4 #ifdef FOO_EXPORTS
   5 #define FOO_API __declspec(dllexport)
   6 #else
   7 #define FOO_API __declspec(dllimport)
   8 #endif /* FOO_EXPORTS */
   9 #else
  10 #define FOO_API extern /* XXX confirm this */
  11 #endif /* FOO_DLL */
  12 
  13 #ifdef __cplusplus
  14 extern "C" {
  15 #endif
  16 
  17 extern FOO_API int bar(double* data, int len) {
  18    int i;
  19    printf("data = %p\n", (void*) data);
  20    for (i = 0; i < len; i++) {
  21       printf("data[%d] = %f\n", i, data[i]);
  22    }
  23    printf("len = %d\n", len);
  24    return len + 1;
  25 }
  26 
  27 #ifdef __cplusplus
  28 }
  29 #endif

When building the DLL for foo on Windows, define FOO_DLL and FOO_EXPORTS (this is what you want to do when building a DLL for use with ctypes). When linking against the DLL, define FOO_DLL. When linking against a static library that contains foo, or when including foo in an executable, don't define anything.

If you're unclear about what extern "C" is for, read section 3 of the C++ dlopen mini HOWTO. This allows you to write function wrappers with C linkage on top of a bunch of C++ classes so that you can use them with ctypes. Alternatively, you might prefer to write C code.

foo.py

   1 import numpy as N
   2 import ctypes as C
   3 _foo = N.ctypeslib.load_library('libfoo', '.')
   4 _foo.bar.restype = C.c_int
   5 _foo.bar.argtypes = [C.POINTER(C.c_double), C.c_int]
   6 def bar(x):
   7     return _foo.bar(x.ctypes.data_as(C.POINTER(C.c_double)), len(x))
   8 x = N.random.randn(10)
   9 n = bar(x)

NumPy arrays' ctypes property

A ctypes property was recently added to NumPy arrays:

In [18]: x = N.random.randn(2,3,4)

In [19]: x.ctypes.data
Out[19]: c_void_p(14394256)

In [21]: x.ctypes.data_as(ctypes.POINTER(c_double))

In [24]: x.ctypes.shape
Out[24]: <ctypes._endian.c_long_Array_3 object at 0x00DEF2B0>

In [25]: x.ctypes.shape[:3]
Out[25]: [2, 3, 4]

In [26]: x.ctypes.strides
Out[26]: <ctypes._endian.c_long_Array_3 object at 0x00DEF300>

In [27]: x.ctypes.strides[:3]
Out[27]: [96, 32, 8]

In general, a C function might take a pointer to the array's data, an integer indicating the number of array dimensions, (pass the value of the ndim property here) and two int pointers to the shapes and stride information.

If your C function assumes contiguous storage, you might want to wrap it with a Python function that calls NumPy's ascontiguousarray function on all the input arrays.

NumPy's ndpointer with ctypes argtypes

Starting with ctypes 0.9.9.9, any class implementing the from_param method can be used in the argtypes list of a function. Before ctypes calls a C function, it uses the argtypes list to check each parameter.

Using NumPy's ndpointer function, some very useful argtypes classes can be constructed, for example:

   1 from numpy.ctypeslib import ndpointer
   2 arg1 = ndpointer(dtype='<f4')
   3 arg2 = ndpointer(ndim=2)
   4 arg3 = ndpointer(shape=(10,10))
   5 arg4 = ndpointer(flags='CONTIGUOUS,ALIGNED')
   6 # or any combination of the above
   7 arg5 = ndpointer(dtype='>i4', flags='CONTIGUOUS')
   8 func.argtypes = [arg1,arg2,arg3,arg4,arg5]

Now, if an argument doesn't meet the requirements, a TypeError is raised. This allows one to make sure that arrays passed to the C function is in a form that the function can handle.

See also the mailing list thread on ctypes and ndpointer.

Dynamic allocation through callbacks

ctypes supports the idea of callbacks, allowing C code to call back into Python through a function pointer. This is possible because ctypes releases the Python Global Interpreter Lock (GIL) before calling the C function.

We can use this feature to allocate NumPy arrays if and when we need a buffer for C code to operate on. This could avoid having to copy data in certain cases. You also don't have to worry about freeing the C data after you're done with it. By allocating your buffers as NumPy arrays, the Python garbage collector can take care of this.

Python code:

   1 from ctypes import *
   2 ALLOCATOR = CFUNCTYPE(c_long, c_int, POINTER(c_int))
   3 # load your library as lib
   4 lib.baz.restype = None
   5 lib.baz.argtypes = [c_float, c_int, ALLOCATOR]

This isn't the prettiest way to define the allocator (I'm also not sure if c_long is the right return type), but there are a few bugs in ctypes that seem to make this the only way at present. Eventually, we'd like to write the allocator like this (but it doesn't work yet):

   1 from numpy.ctypeslib import ndpointer
   2 ALLOCATOR = CFUNCTYPE(ndpointer('f4'), c_int, POINTER(c_int))

The following also seems to cause problems:

   1 ALLOCATOR = CFUNCTYPE(POINTER(c_float), c_int, POINTER(c_int))
   2 ALLOCATOR = CFUNCTYPE(c_void_p, c_int, POINTER(c_int))
   3 ALLOCATOR = CFUNCTYPE(None, c_int, POINTER(c_int), POINTER(c_void_p))

Possible failures include a SystemError exception being raised, the interpreter crashing or the interpreter hanging. Check these mailing list threads for more details:

Time for an example. The C code for the example:

   1 #ifndef CSPKREC_H
   2 #define CSPKREC_H
   3 #ifdef FOO_DLL
   4 #ifdef FOO_EXPORTS
   5 #define FOO_API __declspec(dllexport)
   6 #else
   7 #define FOO_API __declspec(dllimport)
   8 #endif
   9 #else
  10 #define FOO_API
  11 #endif
  12 #endif
  13 #include <stdio.h>
  14 #ifdef __cplusplus
  15 extern "C" {
  16 #endif
  17 
  18 typedef void*(*allocator_t)(int, int*);
  19 
  20 extern FOO_API void foo(allocator_t allocator) {
  21    int dim = 2;
  22    int shape[] = {2, 3};
  23    float* data = NULL;
  24    int i, j;
  25    printf("foo calling allocator\n");
  26    data = (float*) allocator(dim, shape);
  27    printf("allocator returned in foo\n");
  28    printf("data = 0x%p\n", data);
  29    for (i = 0; i < shape[0]; i++) {
  30       for (j = 0; j < shape[1]; j++) {
  31          *data++ = (i + 1) * (j + 1);
  32       }
  33    }
  34 }
  35 
  36 #ifdef __cplusplus
  37 }
  38 #endif

Check the The Function Pointer Tutorials if you're new to function pointers in C or C++. And the Python code:

   1 from ctypes import *
   2 import numpy as N
   3 
   4 allocated_arrays = []
   5 def allocate(dim, shape):
   6     print 'allocate called'
   7     x = N.zeros(shape[:dim], 'f4')
   8     allocated_arrays.append(x)
   9     ptr = x.ctypes.data_as(c_void_p).value
  10     print hex(ptr)
  11     print 'allocate returning'
  12     return ptr
  13 
  14 lib = cdll['callback.dll']
  15 lib.foo.restype = None
  16 ALLOCATOR = CFUNCTYPE(c_long, c_int, POINTER(c_int))
  17 lib.foo.argtypes = [ALLOCATOR]
  18 
  19 print 'calling foo'
  20 lib.foo(ALLOCATOR(allocate))
  21 print 'foo returned'
  22 
  23 print allocated_arrays[0]

The allocate function creates a new NumPy array and puts it in a list so that we keep a reference to it after the callback function returns. Expected output:

calling foo
foo calling allocator
allocate called
0xaf5778
allocate returning
allocator returned in foo
data = 0x00AF5778
foo returned
[[ 1.  2.  3.]
 [ 2.  4.  6.]]

Here's another idea for an Allocator class to manage this kind of thing. In addition to dimension and shape, this allocator function takes a char indicating what type of array to allocate. You can get these typecodes from the ndarrayobject.h header, in the NPY_TYPECHAR enum.

   1 from ctypes import *
   2 import numpy as N
   3 
   4 class Allocator:
   5     CFUNCTYPE = CFUNCTYPE(c_long, c_int, POINTER(c_int), c_char)
   6 
   7     def __init__(self):
   8         self.allocated_arrays = []
   9 
  10     def __call__(self, dims, shape, dtype):
  11         x = N.empty(shape[:dims], N.dtype(dtype))
  12         self.allocated_arrays.append(x)
  13         return x.ctypes.data_as(c_void_p).value
  14 
  15     def getcfunc(self):
  16         return self.CFUNCTYPE(self)
  17     cfunc = property(getcfunc)

Use it like this in Python:

   1 lib.func.argtypes = [..., Allocator.CFUNCTYPE]
   2 def func():
   3     alloc = Allocator()
   4     lib.func(..., alloc.cfunc)
   5     return tuple(alloc.allocated_arrays[:3])

Corresponding C code:

   1 typedef void*(*allocator_t)(int, int*, char);
   2 
   3 void func(..., allocator_t allocator) {
   4    /* ... */
   5    int dims[] = {2, 3, 4};
   6    double* data = (double*) allocator(3, dims, 'd');
   7    /* allocate more arrays here */
   8 }

None of the allocators presented above are thread safe. If you have multiple Python threads calling the C code that invokes your callbacks, you will have to do something a bit smarter.

More useful code frags

Suppose you have a C function like the following, which operates on a pointer-to-pointers data structure.

   1 void foo(float** data, int len) {
   2     float** x = data;
   3     for (int i = 0; i < len; i++, x++) {
   4         /* do something with *x */
   5     }
   6 }

You can create the necessary structure from an existing 2-D NumPy array using the following code:

   1 x = N.array([[10,20,30], [40,50,60], [80,90,100]], 'f4')
   2 f4ptr = POINTER(c_float)
   3 data = (f4ptr*len(x))(*[row.ctypes.data_as(f4ptr) for row in x])

f4ptr*len(x) creates a ctypes array type that is just large enough to contain a pointer to every row of the array.

Heterogeneous Types Example

Here's a simple example when using heterogeneous dtypes (record arrays).

But, be warned that NumPy recarrays and corresponding structs in C may not be congruent.

Also structs are not standardized across platforms ...In other words, be aware of padding issues!

sample.c

   1 #include <stdio.h>
   2 
   3 typedef struct Weather_t {
   4     int timestamp;
   5     char desc[12];
   6 } Weather;
   7 
   8 void print_weather(Weather* w, int nelems)
   9 {
  10     int i;
  11     for (i=0;i<nelems;++i) {
  12         printf("timestamp: %d\ndescription: %s\n\n", w[i].timestamp, w[i].desc);
  13     }
  14 }

SConstruct

   1 env = Environment()
   2 env.Replace(CFLAGS=['-O2','-Wall','-ansi','-pedantic'])
   3 env.SharedLibrary('sample', ['sample.c'])

sample.py

   1 import numpy as N
   2 import ctypes as C
   3 
   4 dat = [[1126877361,'sunny'], [1126877371,'rain'], [1126877385,'damn nasty'], [1126877387,'sunny']]
   5 
   6 dat_dtype = N.dtype([('timestamp','i4'),('desc','|S12')])
   7 arr = N.rec.fromrecords(dat,dtype=dat_dtype)
   8 
   9 _sample = N.ctypeslib.load_library('libsample','.')
  10 _sample.print_weather.restype = None
  11 _sample.print_weather.argtypes = [N.ctypeslib.ndpointer(dat_dtype, flags='aligned, contiguous'), C.c_int]
  12 
  13 
  14 def print_weather(x):
  15     _sample.print_weather(x, x.size)
  16 
  17 
  18 
  19 if __name__=='__main__':
  20     print_weather(arr)

Fibonacci example (using NumPy arrays, C and Scons)

The following was tested and works on Windows (using MinGW) and GNU/Linux 32-bit OSs (last tested 13-08-2009). Copy all three files to the same directory.

The C code (this calculates the Fibonacci number recursively):

   1 /*
   2     Filename: fibonacci.c
   3     To be used with fibonacci.py, as an imported library. Use Scons to compile,
   4     simply type 'scons' in the same directory as this file (see www.scons.org).
   5 */
   6 
   7 /* Function prototypes */
   8 int fib(int a);
   9 void fibseries(int *a, int elements, int *series);
  10 void fibmatrix(int *a, int rows, int columns, int *matrix);
  11 
  12 int fib(int a)
  13 {
  14     if (a <= 0) /*  Error -- wrong input will return -1. */
  15         return -1;
  16     else if (a==1)
  17         return 0;
  18     else if ((a==2)||(a==3))
  19         return 1;
  20     else
  21         return fib(a - 2) + fib(a - 1);
  22 }
  23 
  24 void fibseries(int *a, int elements, int *series)
  25 {
  26     int i;
  27     for (i=0; i < elements; i++)
  28     {
  29     series[i] = fib(a[i]);
  30     }
  31 }
  32 
  33 void fibmatrix(int *a, int rows, int columns, int *matrix)
  34 {
  35     int i, j;
  36     for (i=0; i<rows; i++)
  37         for (j=0; j<columns; j++)
  38         {
  39             matrix[i * columns + j] = fib(a[i * columns + j]);
  40         }
  41 }

The Python code:

   1 """
   2 Filename: fibonacci.py
   3 Demonstrates the use of ctypes with three functions:
   4 
   5     (1) fib(a)
   6     (2) fibseries(b)
   7     (3) fibmatrix(c)
   8 """
   9 
  10 import numpy as nm
  11 import ctypes as ct
  12 
  13 # Load the library as _libfibonacci.
  14 # Why the underscore (_) in front of _libfibonacci below?
  15 # To mimimise namespace pollution -- see PEP 8 (www.python.org).
  16 _libfibonacci = nm.ctypeslib.load_library('libfibonacci', '.')
  17 
  18 _libfibonacci.fib.argtypes = [ct.c_int] #  Declare arg type, same below.
  19 _libfibonacci.fib.restype  =  ct.c_int  #  Declare result type, same below.
  20 
  21 _libfibonacci.fibseries.argtypes = [nm.ctypeslib.ndpointer(dtype = nm.int),\
                                     ct.c_int,\
                                     nm.ctypeslib.ndpointer(dtype = nm.int)]
  22 _libfibonacci.fibseries.restype  = ct.c_void_p
  23 
  24 _libfibonacci.fibmatrix.argtypes = [nm.ctypeslib.ndpointer(dtype = nm.int),\
                                     ct.c_int, ct.c_int,\
                                    nm.ctypeslib.ndpointer(dtype = nm.int)]
  25 _libfibonacci.fibmatrix.restype  = ct.c_void_p
  26 
  27 def fib(a):
  28     """Compute the n'th Fibonacci number.
  29 
  30     ARGUMENT(S):
  31         An integer.
  32 
  33     RESULT(S):
  34         The n'th Fibonacci number.
  35 
  36     EXAMPLE(S):
  37     >>> fib(8)
  38     13
  39     >>> fib(23)
  40     17711
  41     >>> fib(0)
  42     -1
  43     """
  44     return _libfibonacci.fib(int(a))
  45 
  46 def fibseries(b):
  47     """Compute an array containing the n'th Fibonacci number of each entry.
  48 
  49     ARGUMENT(S):
  50         A list or NumPy array (dim = 1) of integers.
  51 
  52     RESULT(S):
  53         NumPy array containing the n'th Fibonacci number of each entry.
  54 
  55     EXAMPLE(S):
  56     >>> fibseries([1,2,3,4,5,6,7,8])
  57     array([ 0,  1,  1,  2,  3,  5,  8, 13])
  58     >>> fibseries(range(1,12))
  59     array([ 0,  1,  1,  2,  3,  5,  8, 13, 21, 34, 55])
  60     """
  61     b = nm.asarray(b, dtype=nm.intc)
  62     result = nm.empty(len(b), dtype=nm.intc)
  63     _libfibonacci.fibseries(b, len(b), result)
  64     return result
  65 
  66 def fibmatrix(c):
  67     """Compute a matrix containing the n'th Fibonacci number of each entry.
  68 
  69     ARGUMENT(S):
  70         A nested list or NumPy array (dim = 2) of integers.
  71 
  72     RESULT(S):
  73         NumPy array containing the n'th Fibonacci number of each entry.
  74 
  75     EXAMPLE(S):
  76     >>> from numpy import array
  77     >>> fibmatrix([[3,4],[5,6]])
  78     array([[1, 2],
  79            [3, 5]])
  80     >>> fibmatrix(array([[1,2,3],[4,5,6],[7,8,9]]))
  81     array([[ 0,  1,  1],
  82            [ 2,  3,  5],
  83            [ 8, 13, 21]])
  84     """
  85     tmp = nm.asarray(c)
  86     rows, cols = tmp.shape
  87     c = tmp.astype(nm.intc)
  88     result = nm.empty(c.shape, dtype=nm.intc)
  89     _libfibonacci.fibmatrix(c, rows, cols, result)
  90     return result

Here's the SConstruct file contents (filename: SConstruct):

   1 env = Environment()
   2 env.Replace(CFLAGS=['-O2', '-Wall', '-ansi', '-pedantic'])
   3 env.SharedLibrary('libfibonacci', ['fibonacci.c'])

In Python interpreter (or whatever you use), do:

>>> import fibonacci as fb
>>> fb.fib(8)
13
>>> fb.fibseries([5,13,2,6]
array([  3, 144,   1,   5])

etc.

Pertinent Mailing List Threads

Some useful threads on the ctypes-users mailing list:

Thomas Heller's answers are particularly insightful.

Documentation


CategoryCookbook CategoryCookbook

Cookbook/Ctypes (last edited 2011-11-14 22:56:29 by mauro)