Entries‎ > ‎

Python project from scratch with distutils and friends

Python has several more or less standard tools to properly create and/or distribute python code:

  • virtualenv allows to create a self contained python environment, with any dependencies you want, without impacting the system instance. Extremely useful for developing, testing or even running any python code.
  • distutils provide the basics of python package management. It's now quite old, so to do something you must use some of the extensions. Distutils2 is around the corner.
  • setuptools are extensions to distutils; it's one of the most wide spread library for python packages currently. It includes easy_install which allows to easily install python packages.
  • distribute is a fork of setuptools. It evolves regularly (which is not the case of setuptools) and is maintained by the creators of distutils2.
  • PIP is a replacement for easy_install.

We're going to use Distribute and PIP here it looks like they become the standard.

Distribute guide has more complete and accurate documentation. What I'm describing below is really the minimal set to get started with a basic python project.

Let's assume that your project is called $PROJECT. We'll store it in $BASEDIR, typically something like hello-0.1$BASEDIR is the current directory.

  • First, let's prepare a python environment to play safely. Get virtualenv; that's probably the easiest way:
  • wget 'http://bitbucket.org/ianb/virtualenv/raw/tip/virtualenv.py'
  • On my machine, there's a small issue with multiarch mode, so an extra symlink is needed. Most of the time it should not be needed:
  • mkdir -p env/include/multiarch-x86_64-linux ln -s ../python2.6 \   env/include/multiarch-x86_64-linux/python2.6
  • And then create the python environment:
  • python virtualenv.py --no-site-packages --distribute env

    It will create a subdirectory env containing the virtual environment.

  • Activate the python environment:
  • source env/bin/activate

    From now on, everytime you invoke python or similar commands, it will be using the self-contained environment. That includes when you call a setup.py which is pretty useful.

  • Let's now create our basic source. Edit file $BASEDIR/$PROJECT/main.py. It will contain the main function of your application. We put the file in a subdirectory called with your project name; this way, all your code is part of the python module $PROJECT, which is way cleaner. The content of the file is something like that: 
  • def main():     print 'Hello, World!'

    This is not a completely usual self contained script. The packaging system will take care of creating the binary, which will call your function.

  • To make sure that Python knows that your subdirectory is an actual module you need to add an empty file named __init__.py next to your main.py.
  • Now, let's create setup.py, which will create the package description.
  • from setuptools import setup, find_packages setup(   name='hello',   version='0.1',   entry_points = {     'console_scripts': ['hello = hello.main:main']   },   packages=find_packages(exclude=['ez_setup']),   install_requires=[     'nose'   ], )

    Many other things are possible to add here (including all details such as author name and so on), but I've kept the minimum here. The install_requires list describe the list of package your software need. Here, I've requested only nose, which is a test framework. You can list whaterever you want here, including things like Twisted. You can found all Python package on Pypi.

    The line console_scripts describe the list of binaries that you package provides. In this case, we provide one binary called hello, which will call function main from modulehello.main.

From here, we're pretty much set. You can run python setup.py to have more information on how to manipulate your package. Remember to always do that while being in your virtual python environment.
  • To easily work on your project, you can do the following command:
  • python setup.py develop

    It will prepare your package exactly like if you were installing it, but will reference directly your python code with symlink. This way, you can easily test your code while working on it, without having to reinstall everytime.

  • To distribute your package, simply do the following command:
  • python setup.py sdist

    It will create a .tar.gz in the dist/ subdirectory.

There are many other possibilities. Executing python setup.py can give you a lot more information. The default sdist tarball doesn't provide a virtual env script which is not practical. It's possible to create a custom version of virtualenv do distribute along your package, allowing extremely easy installtion.