pip: script afterwards

Let's consider a common workflow with pip:

# Suppose we are using virtualenv.
virtualenv --distribute --no-site-packages demo
source demo/bin/activate

# Install some dependencies.
pip install django
pip install gunicorn supervisord
pip install sphinx
# If your "myproject" Django project is an egg:
python myproject/setup.py develop

# Deployment is not only a matter of dependencies.
mkdir -p docs/_static
mkdir -p var/log
mkdir -p var/run
mkdir -p var/upload
mkdir -p etc/
touch etc/supervisord.conf

# Check deployment success.
python myproject/manage.py test  # Run some tests.

# If it doesn't work, go back and perform some manual tasks to fix the
# deployment.

# Else, it works. Let's share our work!
pip freeze > requirements.txt
# Did I performed other actions?
history > docs/INSTALL.txt
# Ok, let's actually share our work now!
hg commit requirements.txt docs/INSTALL.txt  # or git ;)

So we:

• Work
• Remember what we did
• Commit and share what we remembered (not what we did actually)

And say "it just works". Fine!

In fact, we'd rather say "it just worked for me"...

Buildout: start with configuration

Let's consider a common workflow with buildout:

# Configure deployment.
vim buildout.cfg

# Deploy.
bin/buildout

# Test.
bin/django test

# If it doesn't work, edit buildout.cfg and retry.

# Else, it worked for me. It will work for others too. Let's share our work.
echo "python bootstrap.py -d && bin/buildout" > docs/INSTALL.txt
hg commit buildout.cfg docs/INSTALL.txt

Your work is de facto repeatable: your work is the configuration file, just share it.

Welcome TDD!

Remember what is test driven development?

• Write tests
• Run tests and see them fail
• Develop
• Tests pass.
• Be happy!

When you write tests after implementation, you may forget some checks. In fact, in most cases, you don't write tests after implementation...

TDD is a best practice. Let's make a parallel between tests (in a development scope) and procedures (in a deployment scope). In short, it's a matter of repeatability: tests assert that product functionalities or behaviour are reproduced, and procedures are used to reproduce the product...

With pip, we can write procedures first: start with the edition of requirements.txt file and then run "pip install -r requirements.txt". Who does it? Most pip tutorials start with pip install and then talk about pip freeze.

With buildout, we have to write procedures first. Everybody does it.

Fabric?

Fabric [4] is really handy when you come to script deployment. So some developers could say: let's start with a fabfile!

Yes, you'd better use "fabric + pip" than only pip.

But don't reinvent the wheel! I mean buildout is here, it works, it is extensible. It already has many useful recipes and extensions.

So I would recommend using "fabric + buildout" instead of "fabric + pip":

• buildout is really good at deploying Python stuff. Use it!
• buildout can deploy most non-Python stuff as normal user via recipes. Use it or not depending on your team's preferences. Here is where interactions between sysadmins and developers are required.
• fabric or any remote control utility can run buildout to manage, at least, python stuff.
• fabric or any remote control utility are better than buildout to run tasks as power user (sudoer).
• fabric or any remote control utility are not limited to deployment. They are handy to trigger many other tasks, such as managing services (via tools like supervisord), performing upgrades (via tools like south)...

Conclusion

Tests matter. Deployment procedures too.

Yes, pip "just works". But it's like writing tests after implementation. You usually don't write deployment procedures or forget some steps. So repeatability is not guaranteed.

Buildout's design makes you produce repeatable deployment from the start. Every team should appreciate it.

Fabric and other "remote control utilities" are great, they can run many tasks, including buildout.

References