We have been working hard in the past few months with my Plone friends to make it happen.

It’s now available : Plone.org is acting like PyPI in its Products section, which means that you can use Distutils to register and upload your packages in both places now.

I have written a small tutorial on plone.org to explain how to use it: http://plone.org/documentation/tutorial/how-to-upload-your-package-to-plone.org

I will also explain during my Pycon presentation (http://us.pycon.org/2009/conference/schedule/event/44) how this new feature at Plone.org will help people that are working with Plone and Python packages.

Here’s a proposal I will send to the python-dev. What do you think ?

(Disclaimer : this proposal is highly inspired from the work done by people in various tools, it does not reinvent anything)

The problem

Python developers distribute and deploy their packages using myriads of dependencies. Some of them are not yet available as official OS python packages.  Even sometimes one package conflicts with the official version of a package installed in a given OS.

In any case, the cycle of development of most Python applications is shorter than the release cycle of Linux distributions, so it is impossible for application Foo to wait that Bar 5.6 is officialy available in Debian 4.x.

Therefore, there’s a need to provide or describe a specific list of dependencies for their application to work.

And this list of dependency might conflict with the existing list of packages installed in Python. In other words, even if this is not a wanted behavior from an os packager point of view, an application might need to provide its own execution context.

Right now, when Python is loaded, it uses the site module to browse the site-packages directory to populate the path with packages it find there.  .pth files are also parsed to provide extra paths.

Python 2.6 has introduced per-user site-packages directory, where you can define an extra directory, which is added in the path like the central one.

But both will append new paths to the environment without any rule of exclusion or version checking.

The workarounds

A few workarounds exist to be able to express what packages (and version) an application needs to run, or to set up an isolated environment for it:

• setuptools provides the install_requires mechanism where you can define dependencies directly inside the package, as a new metadata. It also provides a way to install two different versions of one package and let you pick by code or when the program starts, which one you want to activate.
• virtualenv will let you create an isolated Python environment, where you can define your own site-packages. This allows you to make sure you are not conflicting with a incompatible version of a given package.
• zc.buildout relies on setuptools and provides an isolated environment a bit similar in some aspects to virtualenv.
• pip provides a way to describe requirements in a file, which can be used to define bundles, which are very similar to what zc.buildout provides.

But they all aim at the same goal : define a specific execution context for a specific application, and declare dependencies with no respect to other applications or to the OS environment.

This proposal describes a solution that can be added to Python to provide that feature.

The solution

A isolated environment file that describes dependencies is added. This file can be tweaked by the application packager, or later by the OS packager if something goes wrong.

The isolated environment file

A new file called a  Python Isolated Environment file (PIE file) can be provided by any  application to define the list of dependencies and their versions.

It is a simple text file with a first line that provides :

• a list of paths, separated by ‘:’, on line 1
• then one package per line, starting at line 2. each package can be prefixed by a `!`

For example:

/var/myapp/myenv
lxml
sqlite
sqlalchemy
!sqlobject

This list of packages might or might not be installed in Python.

Versions can be provided as well in this file :

/var/myapp/myenv:/var/myapp/myenv2
lxml >= 0.9
sqlite > 1.8
sqlalchemy == 0.7
!sqlobject == 0.6

The file is saved with the pie extension,

Loading an isolated environment file

A new function called load_isolated_environment is added in site.py, that let you load a PIE file.

Loading a PIE file means:

• for each package defined, starting at line 2, load_isolated_environment will look into the environment if the package with the particular version exists. The version is given by the package.__version__ value or the PKG-INFO one when available. If the package exists but the version is not available, the version 0.0 is used.
• for packages without the ! prefix:
  • if the  package is not found, it will scan each path provided on line 1 of the file, using the site-packages method, looking for that package.
  • if the package is found, it is added in the path.
  • if the package is not found, a PackageMissing error is raised.
• for packages starting with the ! prefix:
  • if the  package is found, it is removed from the path

This function can be called by code like this:

>>> from site import load_isolated_environment
>>> load_isolated_environment('/path/to/context.pie')

From there, sys.path meets the requirements and the code that is executed after this call will benefit from this context.
Another context can be loaded in the same process :

>>> load_isolated_environment('/path/to/another_context.pie')

Limitations:

• if the new context brakes other programs in the process. It’s up to the application packager to fix the context file.
• it’s not the job of load_isolated_environment to resolve dependencies issues : if the foo package needs the bar package, it won’t complain.
• it is not the job of load_isolated_environment to get missing dependencies.

Using an isolated environment file

Typically, an isolated environment file can be used into high-level Python scripts. For example, any script an application provides to be launched :

# this script runs zope
from site import load_isolated_environment
load_isolated_environment('zope-3.4.pie')

import zope

if __name__ == '__main__':
    zope.run()

The Problem

I need to distribute pre-compiled buildouts because some projects don’t allow us to have gcc installed on the production system for security reasons.

Fair enough, we need to provide a pre-compiled buildout.

If you want to distribute your buildout-based Plone application in a binary form, so it can be installed without requiring any compiler on the platform, you need to compile all .c modules before you provide a tarball of your buildout folder.

This is easy : just run your buildout and all .so files will be created in the zope 2 installation. (.pyd under windows)

But this will work only if you compile in a directory that is located within the same path on the target machine, because zc.buildout uses absolute paths when it builds scripts.

Furthermore, if the python interpreter is not located in the same place, your buildout script itself is screwed.

Last but not least, plone.recipe.zope2install is not clever enough. It will remove your zope2 installation when it detects that the path has changed. This is pretty annoying even if you have gcc : what is the point of compiling the c extension again since they
are statically compiled in-place ?

The solution

I have changed plone.recipe.zope2install and added a new option called `smart-recompile` (in trunk right now, not released).

If you use it, the recipe will check for .so or .pyd files before trying to ditch your zope 2 installation and recompile it. Even if you don’t use it to build binary distributions, it will make your buildout build faster if you already have zope compiled in there.

Next, I have created a special bootstrap.py, who is clever enough to rebuild the buildout script with the right path to the used interpreter, and with offline-mode capabilities. To make it short : boostrap.py works no matter if you have an internet connection or not. Grab it here : http://ziade.org/bootstrap.py

So now, basically you can compile your buildout and deploy it on any system, on any path, without any internet connection, like this:

$ python bootstrap.py    # will rebuild the buildout script   
$ bin/buildout

Of course this doesn’t work if you have dynamically compiled extensions like python-ldap. For theses, the best pick is to rely on the system ones.

I found a bit of time to work on distribution matters. Here’s a status of what I am doing there.

There are two topics I am focusing on right now.

• clean up and enhance Python’s distutils package
• implement the mirroring infrastructure at PyPI

distutils work

Nathan Van Gheem proposed a cool patch in collective.dist, (this package is a port of the new features I have added in distutils so they are available in 2.4 and 2.5).

Nathan proposed a patch to be able to avoid the storage of the password in the .pypirc file. The prompt is used in that case. This is something that was in my pile for a long time.

I have added a few things to Nathan’s patch, and a test, and proposed it to Python. I am now waiting for its integration in 2.7 trunk: http://bugs.python.org/issue4394. If it’s accepted, I will backport it to collective.dist.

There are some other tickets I am waiting to be accepted:

• http://bugs.python.org/issue4400
• http://bugs.python.org/issue2461
• http://bugs.python.org/issue3992
• http://bugs.python.org/issue3985
• http://bugs.python.org/issue3986

I am not sure when those will be integrated. The average time for the integration of tickets in distutils in Python is between 6 months and 8 months. hihihi.

PyPI mirroring

The job I am doing in PyPI will be in three phase :

• Phase 1: implement the mirroring infrastructure in PyPI
• Phase 2: promote it, and propose patches for the mirroring tools out there so they use the protocol
• Phase 3: promote and propose patches for pip so it can use the mirrors efficiently (fail-over and nearest mirror infrastructure).

Phase 1: so far, so good.

With some insights from Richard Jones and Martin von Löwis, I am currently implementing the mirroring infrastructure for PyPI we have defined during the D.C. sprint (I still owe a blog entry about this sprint). The code lives in a branch on the python svn folder dedicated to PyPI.

The idea of the mirroring infrastructure is to be able to get a list of official mirrors for PyPI, that can be used as alternatives sources . (It is described here: http://wiki.python.org/moin/PEP_374). A great behavior could be that the client application interacts with the nearest mirror location automatically, and switch to another if it goes down.

So, a list of mirrors will be made available at /mirrors, and the client applications will be able from there to use an alternative location for every package. The hardest part concerns the stats : we want to display in PyPI the download counts for each package by summing downloads from every mirror.

So every mirror will have to provide its “local stats” that can be visited by PyPI. That’s the biggest part of the work I am doing. It will build the stats for PyPI by parsing its Apache log file. And hopefully, this code should be reusable by the mirrors themselve so they can build their stats the same way.

Of course this infrastructure could be used for any PyPI-compatible server even if is not a mirror of PyPI (like a private PyPI server)

Phase 2 will consist in promoting the infrastructure to the mirroring softwares out there. Maybe Pycon will be a good place for that.

Phase 3 is the most interesting one : make sure the client applications use the mirrors ! I think Ian Bicking’s pip project could be the right place for these innovations.

Next topics in the pile:

• index-merging: describe in a PEP-like document the index-merging feature that would allow clients to merge several indexes with a content that differe. For example: PyPI + a private PyPI server. I have written a first draft of such a patch in setuptools in the past (http://bugs.python.org/setuptools/issue32) but I have lost all my hopes to see this project moving forward lately.
• Brainstorming: try to understand the Python Packaging Paradox. That is = how come the community, which is composed of many briliant people, is unable to move forward in packaging matters.
• Distribute the return

I was totally drowned into some customer projects since I came back from the Plone Conference. But things are looking better now, so I can take a bit of time to start blogging about the conference. I’ll probably do three blog posts: this one about my tutorial, the next one about the conference itself and last, an entry about the sprints.

So I gave a tutorial about zc.buildout. The length was a bit challenging, since I had 90 minutes. Enough time to explain more things than in a regular talk, but not enough time to get into great details, as a tutorial should be.

The other thing was about the topic: two talks were covering zc.buildout, Clayton Parker’s one and mine. So my goal was to make sure they were not overlapping too much.

I had the chance to meet Clayton before we both gave our talk, since the Six Feet Up crew gave me shelter in the house they rented (nicest guys in the block). Even if we didn’t exchange a lot on the slides themselves, I could figure what Clayton was going to present. So I…. started my slides from scratch two days before my tutorial and worked carefully on their scope

Alex Clark came and backed me up during the talk, since we are working together one plone.org for months now and since the talk presented the new plone.org that is coming up.

I think I did quite well during the talk, because we had a pause half way, and when we started back, the room stayed full

This is the second time I record all the console work in small screencasts, to avoid live problems, and I think this is the best way to go if you need to do some demos, so I’ll keep on doing it. Plus, it’s nice to provide them to people after the talk.

Anyway, the talk was videotaped so you can to judge by yourself:

• The video
• The slides.

And everytime you see a “get the screencast at http://ziade.org/ploneconf” in there, well get them

In detail:

1. Distutils demo
2. setuptools demo
3. collective.eggproxy demo
4. PloneSoftwareCenter installation
5. collective.dist demo
6. new.plone.org demo
7. collective.releaser demo
8. plone 3 buildout demo
9. Multiple target releasing demo

What’s next ?

• We need to finish the work with Alex on plone.org. It’s not hard, it just takes time, and we both are quit busy in our jobs
• I need to polish collective.releaser and collective.eggproxy. They are brut de fonderie, and the code suck a bit. If you are using them and want to help, or have some feedback/issues, please, pretty please, let me know.

I just wanted to announce the release of collective.eggproxy (http://pypi.python.org/pypi/collective.eggproxy/0.2.0)

collective.eggproxy is a smart mirror for PyPI, thaught and coded by my colleague Bertrand Mathieu.

It will collect packages on PyPI only when a program like easy_install or zc.buildout asks for it. In other words, unlike some mirrors that act like rsync and get the whole PyPI base (more than 5 gigas) collective.eggproxy will only get what you need.

At first run collective.eggproxy downloads pypi index and builds a page of links. When a software asks for a specific package, version, etc. collective.eggproxy downloads it if needed and stores it locally.

Want to give a try ? try it in two lines with easy_install:

easy_install collective.eggproxy
eggproxy_run

That’s it !

Mirroring PyPI becomes a recurrent need for Zope development, because zc.buildout makes a lot of package downloads to build one application.

This is useful when you are working in an intranet with limited web access or when you want to speed up download times. It also makes things safer: if PyPI is down and if developers computers don’t have caches, having a mirror will save your day.

While PyPI has proven its robustness (it is 100% up for months now as far as I can see), having mirrors makes a lot of sense.

We have created a small mirror application here at Ingeniweb, that we use for our buildouts needs. This work was thaught and created by my colleague Bertrand Mathieu.

It is a smart proxy that will download packages at PyPI everytime they have been asked by a buildout or an easy_install client. When the package is downloaded, it is kept in the proxy side for any new requests. This means that after a while, the proxy has its own collection of packages that corresponds to the real needs and will not query PyPI anymore.

This approach avoids having to download and synchronize PyPI with crons, which is a heavy process since PyPI weight several gigas. The caveat of course, is that it won’t be able to get a new package if PyPI is down.

Take a look ! http://pypi.python.org/pypi/iw.eggproxy

By the way there is an interesting sprint coming up on all these topics, in Germany :

http://www.openplans.org/projects/black-forest-sprint/project-home

I am leaving tomorrow, heading to Portland, OR, to OSCON. My talk on zc.buildout and Plone will be thurdsay the 24th, and I’ll be there the whole week.

If you are going there and want to meet leave me a note, I am looking forward to meet other geeks there.

Just a quick post about the work going on in zc.buildout.

Work published lately:

• Malthe and Mustapha have added a nice option to be able to add or substract values in variables that are inherited from an extended cfg file (already available, see the doc here)
• I have added an allow-hosts option, that behaves like easy_install one, which can be used to restrict some accesses (see here)
• Sidnei fixed some annoying bugs (missing quotes in some process calls)

On-going work:

• I am working on a timeout config option, to be able to set the socket timeout (see my previous post on this). Andreas added a command-line option a bit ago as well, but we need to refactor it a bit to have it as a config file option.
• I work on a python API so the buildout can be driven from the code. This useful for instance to create tests that are not using a os.popen or os.system call to build a buildout: a separated process is hard to debug.
• We are having some thaughts on having a multiple-index enabled buildout. Still brainstorming.

Disclaimer: this post is Subversion-centric

When releasing a setuptools based-package to the world, developers will eventually tag it as a stable version, then upload it to PyPI or to their own website. So typically they create a branch out of the trunk, fix the version, then create a tag for the release.

Now for continuous development releases, where people get for example a daily snapshot from the trunk, having a dev suffix to the egg version makes it possible to distinguish it from a stable release, so ‘my.package-2.4dev’ will not superseed ‘my.package-2.4′. So a given package will need a dev prefix in its trunk, that gets removed in the branch where the next release occurs.

From there, easy_install will be able to distinguish them.

The question is: how should we do this ?

The Zope 3 way

The Zope 3 egg collection has a simple way to manage it, describe here: http://svn.zope.org/*checkout*/Sandbox/philikon/foundation/releasing-software.txt. The packages have a dev tag suffix in the version metadata, that is removed in the subversion branch used to tag and release

This is simple and straight forward. Although, releasing the trunk has to be done manually. The problem is that any release of the trunk will have the same name for a given version (my.package-2.4dev) , and in some case people won’t upgrade their environment because the archive keeps the same name.

The setuptools way

Setuptools comes with a continuous integration feature that allows people to push the dev tag in setup.cfg. Then it adds it automatically to the archive name when it is generated.

see: http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion

Building a release will then consist of using the same process than Zope 3 one, except that the tag is removed from the setup.cfg file this time.

Now if you try to release the trunk using disutils sdist or bdist_egg command you will automatically get a dev suffix and the Subversion revision number sticked to it. This means that each new revision can generate a new version that will have a distinct name:

• my.package-2.4dev-r1245
• my.package-2.4dev-r1246
• …

easy_install will grab the latest trunk revision when “my.package-2.4dev” is required, and handle upgrades the right way. This is better that a manual dev tag because when you re-release a new version of the trunk, it will superseeds and therefore upgrades previous revisions.

Another nice feature is to be able to provide to easy_install a subversion link directly, as long as you append the egg full name to the link:

easy_install http://my.svn/my.package/trunk#egg=my.package-dev

collective.releaser

The problem with setup.cfg is that if you forget to remove the dev tag from it when releasing a stable version (you bad boy), you will get the dev-r4565 suffix in the egg name.

collective.releaser takes care of this with a new setuptools command called release, by creating a release branch and removing the dev tag automatically. It upgrades the CHANGES.txt file and version.txt for Plone products.

It also registers and uploads the package to any PyPI-like server. To decide where the package should be sent, it looks at the release-packages variable into the .pypirc, to see if the package name matches it. To release all packages to PyPI a default file would be:

[distutils]
index-servers =
  pypi

[pypi]
username:tarek
password:secret
# regular expression-based variable
release-packages =
  .*

From there, making a release is as simple as:

$ python2.4 setup.py release
running release
This package is version 0.1.1
Do you want to run tests before releasing ? [y/N]: n
Do you want to create the release ? If no, you will just be able to deploy again the current release [y/N]: y
Enter a version [0.1.1]: 0.1.2
Commiting changes...
Creating branches...
...
Running "mregister sdist bdist_egg mupload -r pypi"
running mregister
Using PyPI login from /home/tarek/.pypirc
Registering iw.resourcetraverser to http://pypi.python.org/pypi
...
running mupload
Using PyPI login from /home/tarek/.pypirc
Submitting dist/iw.resourcetraverser-0.1.2.tar.gz to http://pypi.python.org/pypi
Submitting dist/iw.resourcetraverser-0.1.2-py2.4.egg to http://pypi.python.org/pypi
...
0.1.2 released

As described on its PyPI page, it has a plugin system to perform extra steps when a release is made. For instance, it is provided with a mail hook to be able to send mails everytime a release is made.

The tool is pretty new and needs to be smoothed up. For instance, we need to add a few extra controls like making sure long_description is reST-valid. But It works, and is being used by a few fellows in the Plone community already :).

Last, for continuous releases from the trunk, you can set up an alias:

$ python setup.py alias devrelease mregister sdist mupload

You will then be able to upload trunk releases with a single call:

$ python setup.py devrelease

The idea is to provide it as the standard tool to release add-on Products when Plone.org will be upgraded and able to interact with distutils commands. I will promote it through the work I am doing for the PSPS task I am championning (Improve release procedures for add-on products), but it has to live a bit.

Help and opinions welcome !

The main task I worked on during the sprint with Alex and Matthew was making PloneSoftwareCenter ready for the new version of Plone.org. These guys rock. We did tons of things and the new plone.org website is coming up. Alex worked for quite a while on migrating plone.org to plone 3, but let me focus on the software center part.

First of all, let me explain what is the final goal of the work done in the software center.

The future of Plonistas: a 100 % egg-based world and what is means for production

Since a few years, Zope code base was moved into a set of namespaced packages. Plone is following closely. From there zc.buildout is providing a simple way to pick up the right set of packages to build an application. This set is automatically chosen by recipes at Zope and Plone level. Then developers add their own packages and dependencies to provide custom features in their applications.

This mechanism means that each team has to:

• make distributions of packages (tarball, eggs, ..)
• build the application with buildout, then release a source distribution
• and provide a way for other developers to build the application on their own, by:
  • publishing the buildout configuration files
  • pushing the packages the buildout uses into to a server that is reachable by developers
  • make sure Plone and Zope packages are also reachable

This means that each team is responsible at least to release packages.

Distutils and Cheeseshop

Distutils provides two commands to publish packages to the world: register and upload. These commands were intended to provide to the developer a way to push packages to any server that supports the protocol, by using the –repository option.

But in reality, the only server that is publicly available is the Cheeseshop (pypi.python.org). Furthermore Distutils is not providing everything needed to work with another server than the Cheeseshop, like I will explain later in this post. So all Plone and Zope packages are uploaded there.

Therefore, pypi.python.org became a single point of failure when you need to build a Plone or Zope application. With the growth of the community, this means that the repository will get bigger and have to deal with more and more request. PyPI weight around 4 gigas at this time, of zip files and tarballs.

While the actual server is really fast, it makes it a bit hard to work when it is down. It didn’t happen often. Twice as far as I remember. But when it happens, the crowd that uses zc.buildout is frozen because they run buildouts several time per day. They can use cache of course, as long as those are up-to-date.

Some mirrors emerged, like http://release.ingeniweb.com/pypi.python.org-mirror. Some enhanced indexes were created as well, like http://download.zope.org/simple, which is still referring to pypi.python.org but performs a bit of crawling to speed things up when a buildout uses it through easy_install.

But all of these are just enhancement to a cheeseshop-centric model.

Furthermore, in case of a private application, you do not want to see packages in the Cheeseshop but you might also want to provide developers a similar way to manage and use them.

Plone.org software center is dying !

For public applications, given the fact that Distutils provides a simple way to push a package to a public location (one shell command), people have started to stop updating the Products section of Plone.org. This happens because updating it means doing a whole lot more than the simple register+upload call. You need to login into the website, then manually upload the packages, then change the front page of the product if needed etc..

So being able to push packages to plone.org with the same set of commands, is the right solution for developers.

Pushing a package means uploading a public archive but it also means updating the front page for the given package, with the register command.

That what I worked on, at PloneSoftwareCenter level, continuing Sidnei’s work: making it act like Cheeseshop. Now the feature is ready and being alpha-tested at new.plone.org

Toward a distributed model

What we will be able to do from there, is to distribute packages (in blue) to the Cheeseshop (2) as usual, but also to Plone.org (1). Having such a tool also allows people to run other cheeseshop-like servers, wether they are public (3) or private (4). This is useful when you are working on customer projects and do not wish to make their packages available to the world, or even if it is a public project, you do not want to push them to PyPi.

Furthermore, it allows having a bit of redundancy : your packages become available in several places, which is better. Think about the mirrors at Sourceforge, same thing here…

This distributed model is used at Ingeniweb, and in other companies I am starting to list (If you do please let me know, this is important to promote the tool).

The new Plone Software Center (PSC)

If you are extensively working with packages and buildouts, you should consider trying the new PSC, read http://tarekziade.wordpress.com/2008/03/20/how-to-run-your-own-private-pypi-cheeseshop-server/ for this. Each project now in the software center can hold several eggs, and it makes it possible to provide a nice deployment model for your teams: developer push packages releases in such a server with the distutils standards, while customers or deployers can build their applications by picking them up through their buildouts.

And it is in Plone, so you can provide extensive features, like a bug tracker, and all the sweet things Plonecan provide.

The work left to be done in PSC is :

• making the storage for archives (tgz, egg) pluggable so new storing strategies can be provided in separate packages, under the collective.psc namespace, I started this work in a branch.
• finish the collective.psc.mirroring package, that will be used in plone.org to fill a system folder, in order to provide an Apache direct view over the archives stored. This will make zc.buildout / easy_install use this stream rather than hitting the Plone instance. Although the final goal will be to transform it into a storage strategy, but time is running and this is useful now for plone.org.
• extensive tests on new.plone.org

collective.dist

To be able to deal with several servers, I have released collective.dist after the sprint (previously iw.dist). This package provides two new distutils commands: mupload and mregister, together with a new pypirc format.

Follow the documentation, and you will be set in a matter of minutes. This package is already used by many developers to make their life easier, and will help you when the new plone.org goes live.

disutils evolution

collective.dist is an evolution of distutils I have been working at Python level for months. It is available as a Python patch here : http://bugs.python.org/issue1858. mregister and mupload are just modified register and upload commands that makes it possible to interact with several PyPi-like servers, that’s it. PSC uses Distutils standards in any case, so it is possible to use the regular register and upload commands with it of course. It is just not convenient because you will need to have the same username and password on both CheeseShop and the third-party egg server (or change your pypirc file everytime :D)

So the patch, basically, just makes the -r option of distutils commands a 100% operational.

Now my job is to convince Python core developers it should be integrated into Python 2.6, and I am working on this. It is a logical evolution, but it sounds overkill to some people that does not have this kind of need: “Why changing that ? we have only one package server, which is the cheeseshop”.

It can also sound to some of them like I am (together with the team of 20 developers we have ) the only person on earth that wishes it, but as soon as the new Plone.org will be online, a lot of people will start needing such a feature. But having it in 2.6 will make it available as a standard in Plone/Zope world in… a few years.

In any case, most of them agree in the fact that this change is logical and that the current pypirc format is not to be kept. So I guess it is just a matter of time and patience.

It is interesting to look at the usage statistics at PyPI: http://pypi.python.org/webstats/usage_200805.html#DAYSTATS

Besides the front page, the most visited pages are in order (same thing in April):

• zc.buildout
• setuptools
• zc.recipe.egg
• plone.recipe.zope2instance
• plone.recipe.distros
• python-openid
• elementtree
• plone.recipe.zope2install

Then the packages in the top 60 are mainly zope eggs. This corresponds to the sequence done by a Plone or Zope being build in a buildout.

If you take a closer look at http://pypi.python.org/webstats/url_200804.html, the  next top hits  corresponds to zope.* then plone.* packages.

This is the first post of the wrapups I want to do about the sprint that we had in Paris last week-end. We had a Bird of Feather about how people use eggs and zc.buildout in their projects, how they release and deploy them.

There were some people from Headnet (Anton, Mustapha) and Infrae (Kit, Sylvain) and Ingeniweb (Me), and we compared a bit how we are working with eggs, zc.buildout etc.

That is what I love in our community: companies can share their knowledge and grow up all together, technically speaking.

We all have internal recipes, command-line scripts, and are all relatively happy with zc.buildout. This discussion was very instructive.

From there, I thaught it would be a good idea to launch a new project in the community, on the top of zc.buildout (and maybe zc.sourcerelease), that would provide a common set of tools and deploying best practices, for people that works with the buildout, no matter which framework they use (Silva, Plone, etc.)

The first step for this project is to find the common needs and see if we can join forces to build common tools. To start it up, I decided to wrap up and release our internal set of tools into a single package called iw.releaser and publish it. This is the work I have done during the last months with the help of Gael, to help our team to work with zc.buildout in Plone Projects. It is Subversion dependent at this time.

I am expecting some feedback from Anton and Sylvain to see if we can make it a collective tool.

This package provides:

• a skeleton to build a project structure (buildout, packages; docs, etc.) so all projects have a standardized structure
• a ‘release’ distutils command that releases a package, upload it to the Cheeseshop or other servers, and send a shout out mail
• a set of command line tools, that can be used to deploy a buildout. These commands are doing many things besides launching a buildout building (which is a bit different from zc.sourcerelease)

This package is used at Ingeniweb for a few months now, and I tried to summarize how it is used in the docs. I bet a lot of bugs will be found if you try it, so consider this package as a non-mature package yet.

Join us all ! So you will be able to release and deploy your buildout-based apps with a few command-line calls,

When a site used by your buildout is not responding, you can stare at it for … ever

Add these two lines in your bin/buildout script:

import socket
socket.setdefaulttimeout(10)

With this, the buildout will go to the next link after ten seconds. This trick made my day

Ben Bangert was reacting about zc.buildout doctests, saying that they are hard to read from the PyPI page, and the examples hard to use and follow.

I agree with Ben as these doctests are very hard to read when you are not familiar with zc.buildout testing modules, which provides a set of API the doctests relies on.

But from a developer point of view, adding a feature to such a package is best done through doctests, using zc.buildout.testing goodies. And a developer that is familiar with this package, will find this doctest very useful.

zc.buildout in any case, is trying to structurize its PyPI front page, and push a maximum amount of doc for users, so … kudos !

I think the problem is more about specifying the target readership.

I would like to point another example that comes in zc.buildout: dependency-links.

The main doctest that appears at PyPI as a light, human-readable section: http://pypi.python.org/pypi/zc.buildout#dependency-links

And the very same section is continued in a specific doctest that does not appear on the main page: http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/dependencylinks.txt?rev=81182&view=markup

So what happened here is that the developer specified two kind of readers:

• people that will reach the package through its PyPI page
• people that will go deeper in how the package works, through recipes or tutorials

From there I think there’s a simple guideline that could be applied to enhance the package documentation when adding a feature:

• resume the feature in the main page (long_description) with examples that does not rely on specific testing API. In other words it should be made of plain english and plain python when needed;
• leave doctest that relies on internal testing API as complementary documentation.
• define for each doctest its nature (recipe, tutorial, etc)

How could we help people doing such structuration ?

The distutils metadata could be a good place to do it, by adding an extra_doctests list for example, that would contain a list of doctests. From there, PyPI could display the long_description text as usual, and add a “more information section”.

Let’s take an example:

def _(f)
return open(f).read()

setup('my.package', ...
long_decription=_('README.txt'),
extra_description=
{'recipe': [_('create_this.txt'),
_('do_that.txt')],
‘tutorial’: [_('how_to_use.txt')
_('how_to_2.txt')]}

From there, PyPI could provide a Table of content, with a structurized documentation, and additional pages for the package, grouped by types (recipe, tutorial) etc. -> Maybe a Sphinx-powered PyPI ?

By the way, I have another post related, which tries to summarized good pratices in technical writing : http://tarekziade.wordpress.com/2007/02/23/technical-writing-the-seven-laws

A new recipe has been added to the iw.recipe.* family: iw.recipe.backup.

If you use zc.buildout to deploy Python applications in production, this one can be used to backup a buildout folder, before any upgrade is attempted, or for a daily backup. It creates an archive with a timestamp on its name, in a defined folder, with the whole folder content. Some directories can be excluded if needed.

This script is not doing fine-grained backups of the Data.fs like repozo.py does, but can be combined with it if this finesse is required. The recipe creates two scripts in the bin folder: backup and restore.

backup creates an archive and takes no parameters, whereas restore takes an archive name and decompress it into the buildout folder. The latter does a backup before it decompresses the archive for more safety.

Last, it will raise an error if the backup folder is located within the buildout folder ;), and logs all commands into a log file. This is a tiny recipe, but provides a simple way to backup your buildouts in a single call, no matter the platform you are in.

Other recipes we provide at this time are:

• iw.recipe.cmd: provides a way to execute shell calls, or inline Python calls. This is useful when you need to set a few things when a buildout is built, that does not worth creating a new recipe;
• iw.recipe.fetcher: similar to wget. We use it to build python-win32.zip archive, to get all installers on the web;
• iw.recipe.fss: will let you configure FileSystemStorage, so no extra step is required;
• iw.recipe.pound: compiles Pound load balancer, and creates its configuration file, so it can be run directly;
• iw.recipe.sendmail: set zope.sendmail as the default mail sender in a Zope 2-based application. This is useful if you wish to have a fast and robust mailhost service in your Plone sites;
• iw.recipe.squid: This recipe installs all parts needed to run a Squid server dedicated to serve a Zope application, staying friendly with Apache and setting things nicely for Plone;
• iw.recipe.subversion: recipe to checkout a svn location into a part. It is different from infrae.subversion because it creates a tarball out of the subversion checkout, and put it in the downloads folder. In other words, it works offline as well as long as the tarball was built once. It doesn’t have the nice feature infrae.subversion provides which is a code checker that will raise an error when buildout tries to remove a changed file. But as we use the develop feature of buildout to create our code, we didn’t need that feature. Subversion checkouts are useful to us when we want to use a bleeding-edge version of a third-party package that has not been released yet;
• iw.recipe.template: this one provides a way to write files based on Cheetah templates, like Python Paste does, and is useful for micro-needs, like iw.recipe.cmd.

If you use one of those recipes and whish to make a feature request or a bug report, I have created a new space on plone.org to have a bug tracker for all recipes: http://plone.org/products/iw-recipes/issues