Tidy up your buildout directory structure

How to arrange buildout directories

As explained in directory structure of a Buildout [1] in the buildout's documentation, zc.buildout uses several files and directories, such as:

  • bin/ ;
  • parts/ ;
  • develop-eggs/ ;
  • eggs/ ;
  • downloads/ ;
  • .installed.cfg ;
  • and maybe other ones, depending on recipes or packages.

Some people get confused when they see buildout directories at the deployment root:

Ooops! What are these folders? Buildout? Man, this is (over)complicated.

This article will demonstrate how to tidy your deployment root and put buildout files in subdirectories.

So that people say:

Great job! How did you do that? Buildout? Man, this is powerful!

Why?

Here are some motivations:

  • Buildout is not the only deployment tool.
  • You want to focus on the deployed project, not on buildout.
  • When you deploy a project, you take care to place every file in a smart place. There is no reason for buildout files to be exceptions.
  • You could use the same workflow and directory structure with other tools than buildout. So the less your workflow is buildout dependent, the more you could change the tool. You may also share the directory structure with other teams, which may use other technologies.

I mean, even if buildout is a great tool, you should integrate buildout in your workflow. Not the opposite.

And because buildout is a great tool, you can do it.

So, let's do it!

What?

Here is a directory structure I'd like for a deployment:

  • bin/ => scripts and binaries
  • etc/ => local configuration
  • lib/ => libraries
  • src/ => code you develop, things that are under version control
  • var/ => data
  • README => Minimal information about product

Sounds good, but what about buildout?

  • Buildout installs binaries in bin/
  • Buildout is a library, so let's put it in lib/buildout/
  • If Buildout installs some packages for development (via mr.developer extension), let's put them in src/

Let's explain how to achieve this.

Demo project

Read

Browse https://github.com/benoitbryon/python-buildout-directories:

  • .gitignore: no need to explain

  • README.txt: self-explained

  • bootstrap.py: a copy of the zc.buildout bootstrap script, as provided at zc.buildout code repository [2]

  • buildout.cfg: a buildout configuration file where:

    • we customize directories, which is the subject of this article:

      [buildout]
bin-directory = bin
develop-eggs-directory = lib/buildout/develop-eggs
downloads-directory = lib/buildout/downloads
eggs-directory = lib/buildout/eggs
installed = lib/buildout/.installed.cfg
parts-directory = lib/buildout/parts

    • we tell buildout to deploy something, just for the example. Since we are dealing with directories, let's tell buildout to create additional some additional directories, via the z3c.recipe.mkdir [3] recipe:

      Note

      This is a bit outside the primary scope of this article. Anyway, I wanted a buildout that actually does something. I presume that creating directories is a useful complement to arranging buildout directories.

      [buildout]
parts = project-directories

[project-directories]
recipe = z3c.recipe.mkdir
paths =
    ${buildout:directory}/etc
    ${buildout:directory}/var/log
    ${buildout:directory}/var/run
    ${buildout:directory}/var/tmp

Run

Ok. Let's run the demo, as explained in the README:

# Get the code.
git clone https://github.com/benoitbryon/python-buildout-directories.git
cd python-buildout-directories/

# Bootstrap zc.buildout, i.e. install it.
mkdir -p lib/buildout
python bootstrap.py --distribute

# Run zc.buildout.
bin/buildout -N

# Look at the result.
ls -al ./

You got the following files:

  • originally in the repository:
    • bootstrap.py
    • buildout.cfg
    • README.txt
    • .git
    • .gitignore
  • created by you:
    • lib/buildout/ (you created this as an empty directory)
  • created by buildout:
    • bin/
    • lib/buildout/* (buildout populated lib/buildout/):
      • develop-eggs
      • eggs
      • .installed.cfg
      • parts
  • created by zc.recipe.mkdir:
    • etc
    • var
      • log
      • run
      • tmp

Conclusion

Good points

  • It is easy to customize buildout directories.
  • Most "obscure" buildout stuff has been put in lib/buildout.
  • Valuable buildout stuff, such as bin/ folder, can be put in foreground.
  • The directory structure is clean enough.

buildout.cfg remains in root folder

In the example, buildout.cfg remains at the root of the deployment. Since it's a configuration file, why didn't I put it in some etc/ folder?

By default, zc.buildout assumes ${buildout:directory} is the directory where the main configuration file lives (see buildout's "-c" option at "bin/buildout -help").

Here, we want ${buildout:directory} to be the deployment root. So we can:

  • Put main configuration file at deployment root:

    bin/buildout

  • Put main configuration file in some etc/buildout/ folder. We have to run buildout with the "-c" and "buildout:directory" options:

    bin/buildout -c etc/buildout/buildout.cfg buildout:directory=`pwd`

I suggest to keep the first solution, because:

  • the command to run buildout is the simplest;
  • having one buildout file at the root is not a blocker issue.

Note

We cannot (or I failed to) declare the "directory" directive in [buildout] section of the configuration file. Bug, feature or personal fail, I don't know by now.

bootstrap.py remains in root folder

We can move bootstrap.py in some subdirectory. It is easy.

In fact, I suggest to encapsulate it in some "standard bootstrap script" instead:

  • call some bin/bootstrap which initializes the deployment process.
  • your team(s) can use the same command on several projects. Whatever the workflow is, whatever the deployment tool is. You don't have to think, just get the code and run the bootstrap script.
  • you can do more than "python bootstrap.py" in bootstrap, like running "mkdir lib/buildout". I personally like to bootstrap buildout and actually run a light buildout process to install some code generators. Then I can use code generators to generate local configuration files, and more.

I may write another article about bootstrap scripts.

Non standard "mkdir -p lib/buildout"

One thing has been added to a classic buildout workflow: the "mkdir -p lib/buildout" command.

In the demo, we cannot remove it, because buildout doesn't create parent directories automatically, and would return an error. Is it a bug or a feature? I don't know by now.

IMHO, it is not a big issue, and it is not a big overhead. So I suggest we keep it.

Anyway, you could replace the "mkdir" command:

  • by a bootstrap script, which encapsulates "bootstrap.py", "mkdir", and maybe other actions, as suggested in the previous section.
  • by editing zc.buildout's original bootstrap.py: add the mkdir command at the beginning. Maybe not a good idea.
  • by contributing to zc.buildout code, so that it creates parent directories when it is necessary.
  • by versionning bootstrap.py in lib/buildout directory. This solutions confuses me, because:
    • on one hand, it is a good practice to keep files that are generated apart from files under version control. In the example, bin/ and lib/ are completely generated, so I simply "gitignore" them. So I wouldn't move bootstrap.py to lib/buildout.
    • on the other hand, it would be a smart solution. So I would move bootstrap.py to lib/buildout and add a "!/lib/buildout/bootstrap.py" exception in .gitignore. In fact, this solution may remain acceptable while it is the only exception to the rule.

lib/ only contains buildout!

Yes, it does in the demo.

If your project is not only about Python and buildout, you may put other things in lib. In some projects, I saw some PHP libraries in lib/.

Additional notes

Reusing directories configuration

If you know the buildout's "extends" directive, you may move directories directives to a separate configuration file, so that you can easily reuse it.

Convention over configuration?

Convention is usually default configuration. With buildout it would mean:

  • buildout.cfg at the root of deployment directory;
  • buildout directories at the root of the deployment directory.

So, convention would be: "don't hide buildout".

One project <=> one buildout

In this article I assumed we install buildout in a "project-centric" way: everything, including buildout itself, is installed in a folder.

Another usage is to install zc.buildout somewhere, and use it to build various projects elsewhere. Several projects can share the same buildout installation. With this scheme, directory layout may be different... but I can't tell you about it, since I personally don't use it ;)