... or buildout for pip folks.
In this article I'm going to talk about how to manage software (Python) projects with
buildout or
pip.
What do you mean for
project?
A package that contains all the
application-specific settings, database configuration, which packages
your project will need and where they lives.
Projects should be managed like a software if you want to assure the needed
quality:
This blog post is not:
- intended to be a complete guide to pip or buildout. If you want to know more about pip or buildout
- talking about how to deploy remotely your projects
Buildout
I've been using buildout for many years and we are still good friends.
Buildout definition (from
http://www.buildout.org):
"""
Buildout
is a Python-based build system for creating, assembling and deploying
applications from multiple parts, some of which may be non-Python-based.
It lets you create a buildout configuration and reproduce the same
software later.
"""
With buildout you can build and share reproducible environments,
not only for Python based components.
Before
buildout (if I remember well the first time I get started to use
buildout was in 2007, probably during the very first Plone Sorrento sprint) it
was a real pain sharing a complete and working developing environment
pointing to the right version of several repositories, etc. With
buildout it was questions of minutes.
 |
From https://pypi.python.org/pypi/mr.developer.
Probably with pip there is less fun because there isn't a funny picture that celebrates it?! |
Buildout configuration files are modular and
extensible (not only on per-section basis). There are a lot of buildout
recipes, probably the one I prefer is
mr.developer
(
https://pypi.python.org/pypi/mr.developer). It allowed me to fetch
different versions of the repositories depending on the buildout profile
in use, for example:
- production -> each developed private egg point to a tag version
- devel -> the same eggs point to the develop/master
You can accomplish this thing creating different configurations for different profiles, like that:
[buildout]
...
[sources]
your_plugin = git git@github.com:username/your_plugin.git
...
I don't like calling
./bin/buildout -c [production|devel].cfg
with the
-c syntax because it is too much error prone. I prefer to create a symbolic link
to the right buildout profile (called buildout.cfg) and you'll perform the same command both
in production or during development always typing:
$ ./bin/buildout
This way you'll avoid nasty errors
like launching a wrong profile in producion. So use just the plain
./bin/buildout command and live happy.
With buildout you can show and freeze all the installed versions of your packages providing a
versions.cfg file.
Here you can see my preferred buildout recipes:
Buildout or not buildout, one of the of the most common needs it is the ability to switch from
develop to
tags depending on you are in development or production mode and
reproduce the same software later. I can't figure out to manage software installations without this quality assurance.
More info:
http://www.buildout.org
Pip
Let's see how to create reproducible environments with develop or tags dependencies for production environments with pip (
https://pip.pypa.io/en/latest/).
Basically you specify your devel requirements on a
devel-requirements.txt file (the name doesn't matter) pointing to the develop/master/trunk on your repository.
There is another file that I call
production-requirements (the file name doesn't matter) that it is equivalent to the previous one but:
- without devel dependencies you don't want to install in production mode
- tagging your private applications (instead of master -> 0.1.1)
This way it is quite simple seeing which releases are installed in production mode, with no cryptic hash codes.
You can use now the
production-requirements.txt as a template for generating an easy to read
requirements.txt. You'll use this file when installing in production.
You can create a regular
Makefile if you don't want to repeat yourself or make scripts if you prefer:
- compile Sphinx documentation
- provide virtualenv initialization
- launch tests against all developed eggs
- update the final requirements.txt file
For example if you are particular lazy you can create a script that will create your
requirements.txt file using the
production-requirements.txt like a template.
This is a simple script, it is just an example, that shows how to build your requirements.txt omitting lines with grep, sed, etc:
#!/bin/bash
pip install -r production-requirements.txt
pip freeze -r production-requirements.txt | grep -v mip_project | sed '1,2d' > requirements.txt
When running this script, you should activate another Python environment in order to not pollute the production requirements list with development stuff.
If you want to make your software reusable and as flexible as possible, you can add a regular
setup.py module with optional dependencies, that you can activate depending on what you need. For example in devel-mode you might want to activate an entry point called
docs (see
-e .[docs] in
devel-requirements.txt) with optional Sphinx dependencies. Or in production you can install MySQL specific dependencies (
-e .[mysql]).
In the examples below I'll also show how to refer to external requirements file (url or a file).
setup.py
You can define optional extra requirements in your
setup.py module.
mysql_requires = [
'MySQL-python',
]
docs_requires = [
'Sphinx',
'docutils',
'repoze.sphinx.autointerface',
]
...
setup(
name='mip_project',
version=version,
...
extras_require={
'mysql': mysql_requires,
'docs': docs_requires,
...
},
devel-requirements.txt
Optional extra requirement can be activated using the [] syntax (see
-e .[docs]).
You can also include external requirement files or urls (see
-r) and tell pip how to fetch some concrete dependencies (see
-e git+...#egg=your_egg).
-r https://github.com/.../.../blob/VERSION/requirements.txt
# Kotti
Kotti[development,testing]==VERSION
# devel (to no be added in production)
zest.releaser
# Third party's eggs
kotti_newsitem==0.2
kotti_calendar==0.8.2
kotti_link==0.1
kotti_navigation==0.3.1
# Develop eggs
-e git+https://github.com/truelab/kotti_actions.git#egg=kotti_actions
-e git+https://github.com/truelab/kotti_boxes.git#egg=kotti_boxes
...
-e .[docs]
production_requirements.txt
The production requirements should point to tags (see
@VERSION).
-r https://github.com/Kotti/Kotti/blob/VERSION/requirements.txt
Kotti[development,testing]==VERSION
# Third party's eggs
kotti_newsitem==0.2
kotti_calendar==0.8.2
kotti_link==0.1
kotti_navigation==0.3.1
# Develop eggs
-e git+https://github.com/truelab/kotti_actions.git@0.1.1#egg=kotti_actions
-e git+https://github.com/truelab/kotti_boxes.git@0.1.3#egg=kotti_boxes
...
-e .[mysql]
requirements.txt
The
requirements.txt is autogenerated based on the
production-requirements.txt model file. All the installed versions are appended in alphabetical at the end of the file, it can be a very long list.
All the tag versions provided in the production-requirements.txt are automatically converted to hash values (
@VERSION ->
@3c1a191...).
Kotti==1.0.0a4
# Third party's eggs
kotti-newsitem==0.2
kotti-calendar==0.8.2
kotti-link==0.1
kotti-navigation==0.3.1
# Develop eggs
-e git+https://github.com/truelab/kotti_actions.git@3c1a1914901cb33fcedc9801764f2749b4e1df5b#egg=kotti_actions-dev
-e git+https://github.com/truelab/kotti_boxes.git@3730705703ef4e523c566c063171478902645658#egg=kotti_boxes-dev
...
## The following requirements were added by pip freeze:
alembic==0.6.7
appdirs==1.4.0
Babel==1.3
Beaker==1.6.4
...
Final consideration
Use pip to install Python packages from
Pypi.
If you’re looking for management of fully integrated cross-platform software
stacks, buildout is for you.
With buildout no Python code needed unless you are going to write new
recipes (the plugin mechanism provided by buildout to add new
functionalities to your software building, see
http://buildout.readthedocs.org/en/latest/docs/recipe.html).
Instead with pip you can manage also cross-platform stacks but you loose the flexibility of buildout recipes and inheritable configuration files.
Anyway if you consider buildout too magic or you just need a way to switch from production vs development mode you can use pip as well.
Links
If you need more info have a look at the following urls:
Other useful links:
Update 20150629
If you want an example I've created a pip-based project for
Kotti CMS (
http://kotti.pylonsproject.org):
