Mirroring PyPI becomes a recurrent need for Zope development, because zc.buildout makes a lot of package downloads to build one application.

This is useful when you are working in an intranet with limited web access or when you want to speed up download times. It also makes things safer: if PyPI is down and if developers computers don’t have caches, having a mirror will save your day.

While PyPI has proven its robustness (it is 100% up for months now as far as I can see), having mirrors makes a lot of sense.

We have created a small mirror application here at Ingeniweb, that we use for our buildouts needs. This work was thaught and created by my colleague Bertrand Mathieu.

It is a smart proxy that will download packages at PyPI everytime they have been asked by a buildout or an easy_install client. When the package is downloaded, it is kept in the proxy side for any new requests. This means that after a while, the proxy has its own collection of packages that corresponds to the real needs and will not query PyPI anymore.

This approach avoids having to download and synchronize PyPI with crons, which is a heavy process since PyPI weight several gigas. The caveat of course, is that it won’t be able to get a new package if PyPI is down.

Take a look ! http://pypi.python.org/pypi/iw.eggproxy

By the way there is an interesting sprint coming up on all these topics, in Germany :

http://www.openplans.org/projects/black-forest-sprint/project-home

I am leaving tomorrow, heading to Portland, OR, to OSCON. My talk on zc.buildout and Plone will be thurdsay the 24th, and I’ll be there the whole week.

If you are going there and want to meet leave me a note, I am looking forward to meet other geeks there.

Just a quick post about the work going on in zc.buildout.

Work published lately:

• Malthe and Mustapha have added a nice option to be able to add or substract values in variables that are inherited from an extended cfg file (already available, see the doc here)
• I have added an allow-hosts option, that behaves like easy_install one, which can be used to restrict some accesses (see here)
• Sidnei fixed some annoying bugs (missing quotes in some process calls)

On-going work:

• I am working on a timeout config option, to be able to set the socket timeout (see my previous post on this). Andreas added a command-line option a bit ago as well, but we need to refactor it a bit to have it as a config file option.
• I work on a python API so the buildout can be driven from the code. This useful for instance to create tests that are not using a os.popen or os.system call to build a buildout: a separated process is hard to debug.
• We are having some thaughts on having a multiple-index enabled buildout. Still brainstorming.

Disclaimer: this post is Subversion-centric

When releasing a setuptools based-package to the world, developers will eventually tag it as a stable version, then upload it to PyPI or to their own website. So typically they create a branch out of the trunk, fix the version, then create a tag for the release.

Now for continuous development releases, where people get for example a daily snapshot from the trunk, having a dev suffix to the egg version makes it possible to distinguish it from a stable release, so ‘my.package-2.4dev’ will not superseed ‘my.package-2.4′. So a given package will need a dev prefix in its trunk, that gets removed in the branch where the next release occurs.

From there, easy_install will be able to distinguish them.

The question is: how should we do this ?

The Zope 3 way

The Zope 3 egg collection has a simple way to manage it, describe here: http://svn.zope.org/*checkout*/Sandbox/philikon/foundation/releasing-software.txt. The packages have a dev tag suffix in the version metadata, that is removed in the subversion branch used to tag and release

This is simple and straight forward. Although, releasing the trunk has to be done manually. The problem is that any release of the trunk will have the same name for a given version (my.package-2.4dev) , and in some case people won’t upgrade their environment because the archive keeps the same name.

The setuptools way

Setuptools comes with a continuous integration feature that allows people to push the dev tag in setup.cfg. Then it adds it automatically to the archive name when it is generated.

see: http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion

Building a release will then consist of using the same process than Zope 3 one, except that the tag is removed from the setup.cfg file this time.

Now if you try to release the trunk using disutils sdist or bdist_egg command you will automatically get a dev suffix and the Subversion revision number sticked to it. This means that each new revision can generate a new version that will have a distinct name:

• my.package-2.4dev-r1245
• my.package-2.4dev-r1246
• …

easy_install will grab the latest trunk revision when “my.package-2.4dev” is required, and handle upgrades the right way. This is better that a manual dev tag because when you re-release a new version of the trunk, it will superseeds and therefore upgrades previous revisions.

Another nice feature is to be able to provide to easy_install a subversion link directly, as long as you append the egg full name to the link:

easy_install http://my.svn/my.package/trunk#egg=my.package-dev

collective.releaser

The problem with setup.cfg is that if you forget to remove the dev tag from it when releasing a stable version (you bad boy), you will get the dev-r4565 suffix in the egg name.

collective.releaser takes care of this with a new setuptools command called release, by creating a release branch and removing the dev tag automatically. It upgrades the CHANGES.txt file and version.txt for Plone products.

It also registers and uploads the package to any PyPI-like server. To decide where the package should be sent, it looks at the release-packages variable into the .pypirc, to see if the package name matches it. To release all packages to PyPI a default file would be:

[distutils]
index-servers =
  pypi

[pypi]
username:tarek
password:secret
# regular expression-based variable
release-packages =
  .*

From there, making a release is as simple as:

$ python2.4 setup.py release
running release
This package is version 0.1.1
Do you want to run tests before releasing ? [y/N]: n
Do you want to create the release ? If no, you will just be able to deploy again the current release [y/N]: y
Enter a version [0.1.1]: 0.1.2
Commiting changes...
Creating branches...
...
Running "mregister sdist bdist_egg mupload -r pypi"
running mregister
Using PyPI login from /home/tarek/.pypirc
Registering iw.resourcetraverser to http://pypi.python.org/pypi
...
running mupload
Using PyPI login from /home/tarek/.pypirc
Submitting dist/iw.resourcetraverser-0.1.2.tar.gz to http://pypi.python.org/pypi
Submitting dist/iw.resourcetraverser-0.1.2-py2.4.egg to http://pypi.python.org/pypi
...
0.1.2 released

As described on its PyPI page, it has a plugin system to perform extra steps when a release is made. For instance, it is provided with a mail hook to be able to send mails everytime a release is made.

The tool is pretty new and needs to be smoothed up. For instance, we need to add a few extra controls like making sure long_description is reST-valid. But It works, and is being used by a few fellows in the Plone community already :).

Last, for continuous releases from the trunk, you can set up an alias:

$ python setup.py alias devrelease mregister sdist mupload

You will then be able to upload trunk releases with a single call:

$ python setup.py devrelease

The idea is to provide it as the standard tool to release add-on Products when Plone.org will be upgraded and able to interact with distutils commands. I will promote it through the work I am doing for the PSPS task I am championning (Improve release procedures for add-on products), but it has to live a bit.

Help and opinions welcome !

The main task I worked on during the sprint with Alex and Matthew was making PloneSoftwareCenter ready for the new version of Plone.org. These guys rock. We did tons of things and the new plone.org website is coming up. Alex worked for quite a while on migrating plone.org to plone 3, but let me focus on the software center part.

First of all, let me explain what is the final goal of the work done in the software center.

The future of Plonistas: a 100 % egg-based world and what is means for production

Since a few years, Zope code base was moved into a set of namespaced packages. Plone is following closely. From there zc.buildout is providing a simple way to pick up the right set of packages to build an application. This set is automatically chosen by recipes at Zope and Plone level. Then developers add their own packages and dependencies to provide custom features in their applications.

This mechanism means that each team has to:

• make distributions of packages (tarball, eggs, ..)
• build the application with buildout, then release a source distribution
• and provide a way for other developers to build the application on their own, by:
  • publishing the buildout configuration files
  • pushing the packages the buildout uses into to a server that is reachable by developers
  • make sure Plone and Zope packages are also reachable

This means that each team is responsible at least to release packages.

Distutils and Cheeseshop

Distutils provides two commands to publish packages to the world: register and upload. These commands were intended to provide to the developer a way to push packages to any server that supports the protocol, by using the –repository option.

But in reality, the only server that is publicly available is the Cheeseshop (pypi.python.org). Furthermore Distutils is not providing everything needed to work with another server than the Cheeseshop, like I will explain later in this post. So all Plone and Zope packages are uploaded there.

Therefore, pypi.python.org became a single point of failure when you need to build a Plone or Zope application. With the growth of the community, this means that the repository will get bigger and have to deal with more and more request. PyPI weight around 4 gigas at this time, of zip files and tarballs.

While the actual server is really fast, it makes it a bit hard to work when it is down. It didn’t happen often. Twice as far as I remember. But when it happens, the crowd that uses zc.buildout is frozen because they run buildouts several time per day. They can use cache of course, as long as those are up-to-date.

Some mirrors emerged, like http://release.ingeniweb.com/pypi.python.org-mirror. Some enhanced indexes were created as well, like http://download.zope.org/simple, which is still referring to pypi.python.org but performs a bit of crawling to speed things up when a buildout uses it through easy_install.

But all of these are just enhancement to a cheeseshop-centric model.

Furthermore, in case of a private application, you do not want to see packages in the Cheeseshop but you might also want to provide developers a similar way to manage and use them.

Plone.org software center is dying !

For public applications, given the fact that Distutils provides a simple way to push a package to a public location (one shell command), people have started to stop updating the Products section of Plone.org. This happens because updating it means doing a whole lot more than the simple register+upload call. You need to login into the website, then manually upload the packages, then change the front page of the product if needed etc..

So being able to push packages to plone.org with the same set of commands, is the right solution for developers.

Pushing a package means uploading a public archive but it also means updating the front page for the given package, with the register command.

That what I worked on, at PloneSoftwareCenter level, continuing Sidnei’s work: making it act like Cheeseshop. Now the feature is ready and being alpha-tested at new.plone.org

Toward a distributed model

What we will be able to do from there, is to distribute packages (in blue) to the Cheeseshop (2) as usual, but also to Plone.org (1). Having such a tool also allows people to run other cheeseshop-like servers, wether they are public (3) or private (4). This is useful when you are working on customer projects and do not wish to make their packages available to the world, or even if it is a public project, you do not want to push them to PyPi.

Furthermore, it allows having a bit of redundancy : your packages become available in several places, which is better. Think about the mirrors at Sourceforge, same thing here…

This distributed model is used at Ingeniweb, and in other companies I am starting to list (If you do please let me know, this is important to promote the tool).

The new Plone Software Center (PSC)

If you are extensively working with packages and buildouts, you should consider trying the new PSC, read http://tarekziade.wordpress.com/2008/03/20/how-to-run-your-own-private-pypi-cheeseshop-server/ for this. Each project now in the software center can hold several eggs, and it makes it possible to provide a nice deployment model for your teams: developer push packages releases in such a server with the distutils standards, while customers or deployers can build their applications by picking them up through their buildouts.

And it is in Plone, so you can provide extensive features, like a bug tracker, and all the sweet things Plonecan provide.

The work left to be done in PSC is :

• making the storage for archives (tgz, egg) pluggable so new storing strategies can be provided in separate packages, under the collective.psc namespace, I started this work in a branch.
• finish the collective.psc.mirroring package, that will be used in plone.org to fill a system folder, in order to provide an Apache direct view over the archives stored. This will make zc.buildout / easy_install use this stream rather than hitting the Plone instance. Although the final goal will be to transform it into a storage strategy, but time is running and this is useful now for plone.org.
• extensive tests on new.plone.org

collective.dist

To be able to deal with several servers, I have released collective.dist after the sprint (previously iw.dist). This package provides two new distutils commands: mupload and mregister, together with a new pypirc format.

Follow the documentation, and you will be set in a matter of minutes. This package is already used by many developers to make their life easier, and will help you when the new plone.org goes live.

disutils evolution

collective.dist is an evolution of distutils I have been working at Python level for months. It is available as a Python patch here : http://bugs.python.org/issue1858. mregister and mupload are just modified register and upload commands that makes it possible to interact with several PyPi-like servers, that’s it. PSC uses Distutils standards in any case, so it is possible to use the regular register and upload commands with it of course. It is just not convenient because you will need to have the same username and password on both CheeseShop and the third-party egg server (or change your pypirc file everytime :D)

So the patch, basically, just makes the -r option of distutils commands a 100% operational.

Now my job is to convince Python core developers it should be integrated into Python 2.6, and I am working on this. It is a logical evolution, but it sounds overkill to some people that does not have this kind of need: “Why changing that ? we have only one package server, which is the cheeseshop”.

It can also sound to some of them like I am (together with the team of 20 developers we have ) the only person on earth that wishes it, but as soon as the new Plone.org will be online, a lot of people will start needing such a feature. But having it in 2.6 will make it available as a standard in Plone/Zope world in… a few years.

In any case, most of them agree in the fact that this change is logical and that the current pypirc format is not to be kept. So I guess it is just a matter of time and patience.

It is interesting to look at the usage statistics at PyPI: http://pypi.python.org/webstats/usage_200805.html#DAYSTATS

Besides the front page, the most visited pages are in order (same thing in April):

• zc.buildout
• setuptools
• zc.recipe.egg
• plone.recipe.zope2instance
• plone.recipe.distros
• python-openid
• elementtree
• plone.recipe.zope2install

Then the packages in the top 60 are mainly zope eggs. This corresponds to the sequence done by a Plone or Zope being build in a buildout.

If you take a closer look at http://pypi.python.org/webstats/url_200804.html, the  next top hits  corresponds to zope.* then plone.* packages.

This is the first post of the wrapups I want to do about the sprint that we had in Paris last week-end. We had a Bird of Feather about how people use eggs and zc.buildout in their projects, how they release and deploy them.

There were some people from Headnet (Anton, Mustapha) and Infrae (Kit, Sylvain) and Ingeniweb (Me), and we compared a bit how we are working with eggs, zc.buildout etc.

That is what I love in our community: companies can share their knowledge and grow up all together, technically speaking.

We all have internal recipes, command-line scripts, and are all relatively happy with zc.buildout. This discussion was very instructive.

From there, I thaught it would be a good idea to launch a new project in the community, on the top of zc.buildout (and maybe zc.sourcerelease), that would provide a common set of tools and deploying best practices, for people that works with the buildout, no matter which framework they use (Silva, Plone, etc.)

The first step for this project is to find the common needs and see if we can join forces to build common tools. To start it up, I decided to wrap up and release our internal set of tools into a single package called iw.releaser and publish it. This is the work I have done during the last months with the help of Gael, to help our team to work with zc.buildout in Plone Projects. It is Subversion dependent at this time.

I am expecting some feedback from Anton and Sylvain to see if we can make it a collective tool.

This package provides:

• a skeleton to build a project structure (buildout, packages; docs, etc.) so all projects have a standardized structure
• a ‘release’ distutils command that releases a package, upload it to the Cheeseshop or other servers, and send a shout out mail
• a set of command line tools, that can be used to deploy a buildout. These commands are doing many things besides launching a buildout building (which is a bit different from zc.sourcerelease)

This package is used at Ingeniweb for a few months now, and I tried to summarize how it is used in the docs. I bet a lot of bugs will be found if you try it, so consider this package as a non-mature package yet.

Join us all ! So you will be able to release and deploy your buildout-based apps with a few command-line calls,