Wow.

I have been really involved in Plone for about one year now. This community just rocks. I met incredible guys, and did tons of work and experiments with them.

I have asked the community for help to go to the Plone Conf with a fund raising. I raised the funds in 3 days… I am astonished, this is a marvelous gift from you guys. I alsmot cried when I saw all those people donating.

I am so lucky, and so proud to be part of the Plone community and to know all of you. I will work hard to produce the best tutorial I can for the Plone Conf, and continue of course my work in the community tasks I am currently involved in.

I would like to say a big, warm thank to (the order is chronological, from the latest donation to the first one):

• Andreas Jung
• Veda Williams
• David Glick
• Jeff Kowalczyk
• Youenn Boussard
• Christian Klinger
• Jesse Snyder
• Alec Mitchell
• John Habermann
• Maurits van Rees
• Jean-François Roche
• Martin Aspeli
• Alain Meurant
• Aleksandr Vladimirskiy
• Jon Stahl
• Alexander Limi
• Stephen McMahon

And a very special thanks to Calvin Hendryx-Parker.

This conference will be a blast for sure, if you can go there, do it !

Now I better find a good idea for those T-Shirts I have promised ! …

Python needs a better testing tool, hopefully this will happen some days.

Until then we, Plone developers use zope.testing for the best and the worst.

• the best because zope.testing provides layers and the Zope/Plone stack provides really nice things to work with them.
• the worst because sometimes it just gives me some headaches to figure out how to set my test fixtures correctly.
• the worst because I cannot easy-install zope.testing and just call it from the prompt to run some tests (this should get better sometimes see here)

That said, when you work with a zc.buildout based environment, it is easy to use zope.testing, thanks to zc.recipe.testrunner.

But since the eggification of Zope, we write more and more code that would benefit from a lighter test framework. Nose has this lighter approach, I love it to write simple tests without the Java-like heavy UnitTest framework. I can just write:

def test_something():

    assert 1 == 1

And call nosetests. Test fixtures are easy to set at all levels as well, using the with_setup decorator.

I tried to write some tests that could be launchable from both frameworks, by making some bridges, but I came up to the conclusion that a package should fully use either zope.testing, either Nose.

So now, I use two test scripts in my zc.buildout environments, and decide depending on the package. I have written another recipe for buildout to bind Nose: pbp.recipe.noserunner.

From there, I create two sections in my buildout.cfg file:

[buildout]
parts =
    ...
    nose
    ztest

[nose]
recipe = pbp.recipe.noserunner
eggs =
    egg1
    egg2

[ztest]
recipe = zc.recipe.testrunner
eggs =
    egg3
    egg4

These will generate two test scripts

• nose to run nosetests with egg1 and egg2 in sys.path
• ztest to run zope.testing with egg3 and egg4 in sys.path

Here’s my grid of choices when I am coding:

• zc.buildout recipes : zope.testing (doctests using zc.buildout testing framework)
• Plone packages: zope.testing
• Python packages and ‘everything’ else: Nose

I have been asked to be a “champion” last winter at the Plone summit.

My task was :

Improve release procedures for add-ons on plone.org: document a release process, and create release tools for packaging and uploading products from the command line.

Since then, the major steps that I have done for it were :

• change Python’s .pypirc format so it can be used to interact with PyPI and Plone. DONE
• provide a package to make the new .pypirc format available under Python 2.4 and 2.5. DONE
• make PloneSoftwareCenter PyPI-compatible. DONE

The steps left are:

• continue the help on Plone.org Plone 3 migration, so it becomes reality
• document the release process, based on PyPI, Plone.org and collective.dist

I am planning to go at the Plone Conference and to give a tutorial on all this work to deliver to the community the releasing process documentation. So these two remaining steps have to happen before the conference. I am also planning to sprint with people there.

My company contributes a lot to the Plone community. For instance we have organized the Paris Plone Sprint and we try to contribute as often as we can into the collective. But it has its limits and my airfare cannot be covered at this time.

This is the first time I am asking something to the community : please, help me go to the Plone conference !

I have created a page at ChipIn, and I am trying to raise the money to go to D.C.

What you get if you help me:

• You will contribute to plone.org and release procedures enhancements
• Since I can create t-shirts, I will create a unique Plone’08 DC logo T-Shirt that I will ship to the 5 biggest donators.

EDIT : It seems that Paypal won’t take your browser language settings into account. If you can’t read french you can try this hack I have found to switch to english :

when the paypal page is loaded add “locale.x=en_US&” into the url after “websrc?=”

so it looks like this:
https://www.paypal.com/fr/cgi-bin/webscr?locale.x=en_US&cmd=…

Reload that link and it should switch in english.

EDIT 2: don’t worry if Paypal tells you Amina El Kamel received the payment. This is my girlfriend Paypal account .

Click on the cat :

Nose comes with a handy option to generate profiling stats.

To profile your code, create a test dedicated to this purpose and run it with the right options:

$ nosetests --with-profile --profile-stats-file stats.pf test_performance

This will run the tests that corresponds to the test_performance name and generate a stats.pf file.

Nose uses hotshot, so if you want to generate a file that can be read directly by the pstats module and all the statistics tools out there, you need to convert it using the hotshot.stats module.

From there, there is plenty of tools that can transform such a file into a visual graph. Most of the time, they use Graphviz to render a graph, by generating a file dot can read. This software is most of the time easy to install through a binary distribution on your system. If you need to compile it… good luck..

Anyway, from there, I use gprof2dot, which renders a nice graph with meaningful colors.

From the author:

The color of the nodes and edges varies according to the total time % value. In the default temperature-like color-map, functions where most time is spent (hot-spots) are marked as saturated red, and functions where little time is spent are marked as dark blue.

If you want to use it, I have created some console scripts for conveniency, you can install using easy_install :

$ easy_install pbp.scripts

It creates a gprof2dot script you can use, following the author documentation, but also a hotshot2dot script that will convert automatically a statistics file and pass it to gprof2dot:

$ hotshot2dot /path/to/my/hotshot/file

This will print in the output a dot file, you can send to the dot program, using a pipe:

$ hotshot2dot /path/to/my/hotshot/file | dot -Tpng -o output.png

You will get the visual result in output.png.

We are all overwhelmed by the amount of data in our feed readers. While this problem is unavoidable if you keep on adding new feeds in it, they could be automatically filtered and categorized to reduce the flow of data.

I wanted for a long time to try out some custom filters over my feeds to find for example related entries, by trying to understand the meaning of the posts, using tools like NLTK.

So I needed a playground for this, where I could play with feeds.

I think the closest tool for this is to use Yahoo Pipes but as far as I know, the only way to create custom filters is to run a web service and call it from Yahoo Pipes.

Anyways, I started to code a framework (at first it was an example for my latest book) that looks a lot like Yahoo Pipes in its principles. I don’t have any User Interface at this time of course, but a simple plugin-based tool that will let me combine my code snippets with feeds.

It is called Atomisator (see http://atomisator.ziade.org).

The big picture

The process is quite simple:

1. Readers are plugins that know how to read a source and provide entries out of it.
2. Filters are plugins that know how to remove unwanted entries, or enhance them (change their title, summary, etc.). They can be combined.
3. the entries are then pushed in a database. This is useful to avoid doublons, and to keep track of past entries.
4. to create the feed, the entries are read from the database
5. Enhancers are plugins that will add to entries extra info. Typically info that can’t be stored, like Digg comments if the entry is detected on Digg, or Google related searches, and so on
6. The feed is then generated.

Right now I am focusing on making it fast, which is not simple because the plugins can play with all entries in the database.

It is in early stage and undertested, but it kinda works. I pushed it at PyPI to see of it meets interest. If it does, I will document the process of writing plugins.

Make sure you have SQlite installed, and give it a try :

$ easy_install atomisator.main

$ atomisator -c atomisator.cfg

$ atomisator

You will have an atomisator.xml feed created. You can add other feeds in atomisator.cfg as well and try them.

Now with this environment, I can start to try out custom algorithms over my feeds.

I’ve been told the name doesn’t sound right in Ehglish, but it does in French so I keep it

Mirroring PyPI becomes a recurrent need for Zope development, because zc.buildout makes a lot of package downloads to build one application.

This is useful when you are working in an intranet with limited web access or when you want to speed up download times. It also makes things safer: if PyPI is down and if developers computers don’t have caches, having a mirror will save your day.

While PyPI has proven its robustness (it is 100% up for months now as far as I can see), having mirrors makes a lot of sense.

We have created a small mirror application here at Ingeniweb, that we use for our buildouts needs. This work was thaught and created by my colleague Bertrand Mathieu.

It is a smart proxy that will download packages at PyPI everytime they have been asked by a buildout or an easy_install client. When the package is downloaded, it is kept in the proxy side for any new requests. This means that after a while, the proxy has its own collection of packages that corresponds to the real needs and will not query PyPI anymore.

This approach avoids having to download and synchronize PyPI with crons, which is a heavy process since PyPI weight several gigas. The caveat of course, is that it won’t be able to get a new package if PyPI is down.

Take a look ! http://pypi.python.org/pypi/iw.eggproxy

By the way there is an interesting sprint coming up on all these topics, in Germany :

http://www.openplans.org/projects/black-forest-sprint/project-home

The Packt team added a page about my new book, which will be out sometimes in September, so I guess it is the right time for me to announce it here !

So here comes “Expert Python Programming”, where I explain how we work with Python every day to create software.

This is my first book in English, and writing in another language was quite challenging

Anyway, this book is intended for developers that already have a background in Python and covers only advanced topics (see the editor details). But as I said, it explains how we develop our applications in Python so topics like continuous integration, documentation, testing, releasing, refactoring, etc. are covered.
Managers will also have a good overview of how a Python project can be run and managed, using modern tools like Distributed Version Control Systems (Mercurial for instance) or Buildbot.

Even if I am working on Zope and Plone these days, I have focused on writing a book that is only about Python, to make it useful to any developer. The fact that my friend Shannon did the technical reviewing helped on this : he doesn’t use Zope too much so he was close to the target readership.

I will take more time when it comes out to announce it in the various mailing lists, and to get into greater details about the content that will be available online besides the book (some Python packages, etc)

I can’t wait to see it out and hold it

Covering OSCON when you are a speaker is quite hard. Until my talk was done, I was more in a mood of reviewing my slides and not really thinking about other talks or blogging.

Convention center from the Bridge (the two twin towers)

I will try to write a few reports now that everything is over. This first report is about Portland, the city and is not technical at all.

This is the first non-technical entry in my blog, but let’s this it for once. So if you are looking for technical content, skip this entry

Portland is an amazing place. The weather was nice throughout the whole week: sunny and not too hot. Some evenings were a bit chilly but that felt nice to me. The convention center where OSCON took place was located on the other side of the Willamette river, in front of the city center. It is a bit far from the center, but finding places were you can put so many people must be hard I guess.

The top six things I have noticed about Portland:

Local beers are great. You can find many places where they brew their own beers, that are comparable to Belgian beers in quality and taste, like the Lucky Lab (Where Jon, a member of the local Plone user group took us, thanks!).  Some place have samples so you can taste several kinds of beers. I also went to the Oregon Brewers Festival right after OSCON was over, where you could taste beers from many places.

Oregon Brewers Festival

Parking a car is a pain. I think the best way to travel in Portland is a mix of Tram, Bus and Bike. You can put your bike on the nose of the buses or inside the trams. Smart thing. This would be useful in Paris. We ended up parking our car far from the center and used the tram (called Max, and free in the center). Portland should set up a bike rental system like in Paris (Velib’), I am sure this would rock.

Organics: people here seem to be really concerned about sustainability and organics. I’ve been told there are a lot of organics farm around Portland. Supermarkets have a nice amount of organics stuff as well. This is nice.

Starbucks Coffee owns the streets: this is scary. there’s a Starbucks almost on each block here. Anyway, local coffee shops have better coffees and they provide nice places to chill out, read a book or talk with people. I think this is where you can feel the real Portland way of living.

The food is good : I have to admit, when we, french people, travel, we feel a bit superior to some countries on food matters. We tend to show off about it :). But Portland has great places to eat. If you like donuts, the place you must go to is called Voodoo Doughnut. They are creative !

Voodoo Doughnut

Big open spaces: I went to Mount Hood where people are skying in… July. I wanted to visit this place because it is where Stanley Kubrick did the outside shots of the famous motel in The Shining. I also had a walk at the Hood river sandbar, where people do kite surfing. One tip: don’t walk there with shorts on, the wind is so strong that you get slapped by the sand. So if you like big open spaces and sports, Portland is the place to be: all those place are one or two hours drive from the city center.

Mt Hood

End of the aparté ! The next entry will focus on OSCON

After a few attempts to make my screencast look nice under Google Video, I decided I would just upload the original ones (.mov and .m4v files) and a PDF export of my presentation.

So everything is available here: http://ziade.org/oscon

Slides: http://ziade.org/oscon/oscon.pdf

• Screencast #1, playing with distutils: http://ziade.org/oscon/demo1.m4v
• Screencast #2, playing with setuptools: http://ziade.org/oscon/demo2.m4v
• Screencast #3, playing with zc.buildout: http://ziade.org/oscon/demo3.mov
• Screencast #4, creating a recipe: http://ziade.org/oscon/demo4.mov
• Screencast #5, creating a plone buildout and adding a develop package in it: http://ziade.org/oscon/demo5.mov
• Screencast #6, collective.buildbot demo (no coding, just showing buildbot.ingeniweb.com): http://ziade.org/oscon/demo6.mov

The .mov files are streamed automatically in your browser I believe, not the m4v ones.

Thanks to Jim Fulton for the quick feedback.

I am leaving tomorrow, heading to Portland, OR, to OSCON. My talk on zc.buildout and Plone will be thurdsay the 24th, and I’ll be there the whole week.

If you are going there and want to meet leave me a note, I am looking forward to meet other geeks there.

When you have to work with directories and files, there’s a very common pattern: copying a directory into another, but filtering out a few files and directories.

For instance, if you want to copy a directory that contains source code, you will probably remove .pyc files and .svn directories if you work with Subversion. In that case, shutil.copytree cannot be used, so os.walk is the usual way to go (rough example):

import os
from os.path import join, splitext, split, exists
from shutil import copyfile

def copy_directory(source, target):
    if not os.path.exists(target):
        os.mkdir(target)
    for root, dirs, files in os.walk(source):
        if '.svn' in dirs:
            dirs.remove('.svn')  # don't visit .svn directories           
        for file in files:
            if splitext(file)[-1] in ('.pyc', '.pyo', '.fs'):
                continue
            from_ = join(root, file)           
            to_ = from_.replace(source, target, 1)
            to_directory = split(to_)[0]
            if not exists(to_directory):
                os.makedirs(to_directory)
            copyfile(from_, to_)

This is a lot of boiler-plate code, so I usually create a small function that accepts more arguments to filter out files and directory. But Python should provide this pattern in the standard library.

I have proposed a patch for shutil.copytree, to integrate filtering capability. It has been reviewed and commited in the trunk this week, so we will have it in Python 2.6.  Now copytree comes with an ignore argument that has to be a callable. If given it will be called on each visited directory to decide what is copied and what is not.

There’s a default callable in shutil called ignore_patterns, that can be used to filter out files with glob-style patterns. I have added this example to Python doc:

from shutil import copytree, ignore_patterns
copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*'))

Pretty straight forward !
More info and examples here : http://docs.python.org/dev/library/shutil.html#shutil.copytree

Plone.org migration to Plone 3 is taking a bit longer than expected, but it should turn into reality soon.

There will be many improvements on the set of packages the website uses (I am thinking in particular about Maurits’s work on POI that will speed up the trackers, but it is just an example), and blobs should be used for the products section (more than 700 projects are registered there).

I worked last week-end on the products section, by finishing collective.psc.mirroring, which will copy all packages that are uploaded at plone.org into a file system directory. This directory will be published directly by Apache so the website will become a new package location for zc.buildout (find-links section) and easy_install calls without invoking the Plone instance.

Now I am focusing on PloneSoftwareCenter (PSC) migration. It is a pretty interesting topic: for every project located in the products folder with releases, I am going to extract its “distutils ids”. These ids are the name set in the setup.py file for each release.

I will then look at PyPI through XML-RPC if the package is also released, using the id. In that case, and if the author email is the same on both side, I will validate that the project on plone.org “owns” the given distutils id. From there PSC will act like PyPI and will reject uploads of packages if the user is not the owner of the project that owns the package id.

Of course there will be errors and some people might feel like their package has been hijacked if they cannot upload their packages. But this should be minor and should be OK after a while. A mail will soon be send to the community to ask people to check that they are synced between PyPI and plone.org.

I am really excited about this work because plone.org will then be compatible with distutils register and upload commands, which means that people will be able to update plone.org products section like they do with PyPI : through a single commande line.

Hey Sidnei, what you thought of several years ago is about to turn true.

Edit: My apologies goes to Maurits, who did the work on POI, not Reinout, his brother

I have encoutered really bad bugs in some Python 2.4 applications (Zope based) using the tarfile module. For instance, the TarFile.getmembers method that returns the files a tar file contains will just fail to return all files …

Hopefully Zope will work under Python 2.5 sometimes. This is a work in progress in GSOC if I recall it correclty, but I am not sure of the current state.

Until then I just stick Python 2.5 module in my packages. It seems to work fine (tests pass :D) after a few changes as it is isolated from the rest of Python.

If you want to do it as well, there are only two changes to make it work under Python 2.4, to introduce 3 new constants in os module and to get rid of a syntax that does not work under 2.4:

$ diff /opt/local/lib/python2.5/tarfile.py tarfile.py

46a47,49
>
> os.SEEK_SET, os.SEEK_CUR, os.SEEK_END = range(3)
>
1070c1073,1076
<         self.name = os.path.abspath(name) if name else None
---
>         if name:
>             self.name = os.path.abspath(name)
>         else:
>             self.name = None

Interested in collective.buildbot usage or development ?

Join us in the dedicated google group : http://groups.google.com/group/collectivebuildbo

Reminders:

• collective.buildbot is a set of zc.buildout recipes and support for declarative configuration for Buildbot.
• BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes, to set up a continuous integration system.

Just a quick post about the work going on in zc.buildout.

Work published lately:

• Malthe and Mustapha have added a nice option to be able to add or substract values in variables that are inherited from an extended cfg file (already available, see the doc here)
• I have added an allow-hosts option, that behaves like easy_install one, which can be used to restrict some accesses (see here)
• Sidnei fixed some annoying bugs (missing quotes in some process calls)

On-going work:

• I am working on a timeout config option, to be able to set the socket timeout (see my previous post on this). Andreas added a command-line option a bit ago as well, but we need to refactor it a bit to have it as a config file option.
• I work on a python API so the buildout can be driven from the code. This useful for instance to create tests that are not using a os.popen or os.system call to build a buildout: a separated process is hard to debug.
• We are having some thaughts on having a multiple-index enabled buildout. Still brainstorming.

Disclaimer: this post is Subversion-centric

When releasing a setuptools based-package to the world, developers will eventually tag it as a stable version, then upload it to PyPI or to their own website. So typically they create a branch out of the trunk, fix the version, then create a tag for the release.

Now for continuous development releases, where people get for example a daily snapshot from the trunk, having a dev suffix to the egg version makes it possible to distinguish it from a stable release, so ‘my.package-2.4dev’ will not superseed ‘my.package-2.4′. So a given package will need a dev prefix in its trunk, that gets removed in the branch where the next release occurs.

From there, easy_install will be able to distinguish them.

The question is: how should we do this ?

The Zope 3 way

The Zope 3 egg collection has a simple way to manage it, describe here: http://svn.zope.org/*checkout*/Sandbox/philikon/foundation/releasing-software.txt. The packages have a dev tag suffix in the version metadata, that is removed in the subversion branch used to tag and release

This is simple and straight forward. Although, releasing the trunk has to be done manually. The problem is that any release of the trunk will have the same name for a given version (my.package-2.4dev) , and in some case people won’t upgrade their environment because the archive keeps the same name.

The setuptools way

Setuptools comes with a continuous integration feature that allows people to push the dev tag in setup.cfg. Then it adds it automatically to the archive name when it is generated.

see: http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion

Building a release will then consist of using the same process than Zope 3 one, except that the tag is removed from the setup.cfg file this time.

Now if you try to release the trunk using disutils sdist or bdist_egg command you will automatically get a dev suffix and the Subversion revision number sticked to it. This means that each new revision can generate a new version that will have a distinct name:

• my.package-2.4dev-r1245
• my.package-2.4dev-r1246
• …

easy_install will grab the latest trunk revision when “my.package-2.4dev” is required, and handle upgrades the right way. This is better that a manual dev tag because when you re-release a new version of the trunk, it will superseeds and therefore upgrades previous revisions.

Another nice feature is to be able to provide to easy_install a subversion link directly, as long as you append the egg full name to the link:

easy_install http://my.svn/my.package/trunk#egg=my.package-dev

collective.releaser

The problem with setup.cfg is that if you forget to remove the dev tag from it when releasing a stable version (you bad boy), you will get the dev-r4565 suffix in the egg name.

collective.releaser takes care of this with a new setuptools command called release, by creating a release branch and removing the dev tag automatically. It upgrades the CHANGES.txt file and version.txt for Plone products.

It also registers and uploads the package to any PyPI-like server. To decide where the package should be sent, it looks at the release-packages variable into the .pypirc, to see if the package name matches it. To release all packages to PyPI a default file would be:

[distutils]
index-servers =
  pypi

[pypi]
username:tarek
password:secret
# regular expression-based variable
release-packages =
  .*

From there, making a release is as simple as:

$ python2.4 setup.py release
running release
This package is version 0.1.1
Do you want to run tests before releasing ? [y/N]: n
Do you want to create the release ? If no, you will just be able to deploy again the current release [y/N]: y
Enter a version [0.1.1]: 0.1.2
Commiting changes...
Creating branches...
...
Running "mregister sdist bdist_egg mupload -r pypi"
running mregister
Using PyPI login from /home/tarek/.pypirc
Registering iw.resourcetraverser to http://pypi.python.org/pypi
...
running mupload
Using PyPI login from /home/tarek/.pypirc
Submitting dist/iw.resourcetraverser-0.1.2.tar.gz to http://pypi.python.org/pypi
Submitting dist/iw.resourcetraverser-0.1.2-py2.4.egg to http://pypi.python.org/pypi
...
0.1.2 released

As described on its PyPI page, it has a plugin system to perform extra steps when a release is made. For instance, it is provided with a mail hook to be able to send mails everytime a release is made.

The tool is pretty new and needs to be smoothed up. For instance, we need to add a few extra controls like making sure long_description is reST-valid. But It works, and is being used by a few fellows in the Plone community already :).

Last, for continuous releases from the trunk, you can set up an alias:

$ python setup.py alias devrelease mregister sdist mupload

You will then be able to upload trunk releases with a single call:

$ python setup.py devrelease

The idea is to provide it as the standard tool to release add-on Products when Plone.org will be upgraded and able to interact with distutils commands. I will promote it through the work I am doing for the PSPS task I am championning (Improve release procedures for add-on products), but it has to live a bit.

Help and opinions welcome !

We made the shirts at home for PyCon FR and that was a LOT of fun, even if we had to work hard to make sure all shirt were ready by the time the event started. We used a silk-screen technique and the output looks really professional. The ink used makes the shirt design last longer than shirts that uses transfer. Although the number of colors is limited.

Have a look at the pictures : http://www.flickr.com/photos/82007723@N00/sets/72157605181699082/

Guess what: some people ordered us some t-shirts for some other events, so some more fun ahead !

Late (but exciting) news: a sprint room will be held at Pycon FR, by the Logilab team, that published a GPL framework on the top of Google AppEngine, called Logilab Appengine eXtension (lax).

Participant will be able to learn how to play with AppEngine technologies.

All infos (in french): http://fr.pycon.org/programme/sprint-appengine

The Pimp my Buildbot project that was started here at Ingeniweb some time ago, to be able to set up a buildbot in a matter of minutes with zc.buildout, was continued during the sprint, and the guys did a great job on it.

It will be used here in customer projects with a Paster that adds buildbot support when a project starts, because it is a waste of time for the developers to set everything everytime.

Jean-Francois Roche, Kai Lautaportti and Gael Pasgrimaud added extensive configuration options (mail, scheduling), and made the SVN Poller works. This feature allows for instance to make the buildbot watch a SVN repository without having to add a hook in the server (post-commit hook for instance), when you don’t own it (SourceForge, collective, etc)

The tool is released in the collective, and available at the Cheeseshop in one single package !

If you want to set a buildbot

• provide for each one of your project a buildout that has a test script
• make sure the test script returns exit code (–with-exit-status with zope.testing)
• create a buildout cfg file using collective.buildbot
• run buildout, that’s it !
• run the master, slaves scripts, and go to the /waterfall page

Just try out our own buildbot by running this sequence:

$ cd /tmp/
$ mkdir my_bot
$ cd my_bot/
$ svn co https://ingeniweb.svn.sourceforge.net/svnroot/ingeniweb/buildbot/trunk .
$ python bootstrap.py
$ bin/buildout
$ bin/master start
$ bin/linux_debian start   (that's our slave)

You should have a buildbot running then at http://localhost:9000/waterfall

The tool, without the polling stuff, also works with Mercurial and Bzr, but probably needs more tests with these repositories. We also need to make sure the slaves works fine under Windows, and add a nice front page to the buildbot.

If you use it let us know !

The main task I worked on during the sprint with Alex and Matthew was making PloneSoftwareCenter ready for the new version of Plone.org. These guys rock. We did tons of things and the new plone.org website is coming up. Alex worked for quite a while on migrating plone.org to plone 3, but let me focus on the software center part.

First of all, let me explain what is the final goal of the work done in the software center.

The future of Plonistas: a 100 % egg-based world and what is means for production

Since a few years, Zope code base was moved into a set of namespaced packages. Plone is following closely. From there zc.buildout is providing a simple way to pick up the right set of packages to build an application. This set is automatically chosen by recipes at Zope and Plone level. Then developers add their own packages and dependencies to provide custom features in their applications.

This mechanism means that each team has to:

• make distributions of packages (tarball, eggs, ..)
• build the application with buildout, then release a source distribution
• and provide a way for other developers to build the application on their own, by:
  • publishing the buildout configuration files
  • pushing the packages the buildout uses into to a server that is reachable by developers
  • make sure Plone and Zope packages are also reachable

This means that each team is responsible at least to release packages.

Distutils and Cheeseshop

Distutils provides two commands to publish packages to the world: register and upload. These commands were intended to provide to the developer a way to push packages to any server that supports the protocol, by using the –repository option.

But in reality, the only server that is publicly available is the Cheeseshop (pypi.python.org). Furthermore Distutils is not providing everything needed to work with another server than the Cheeseshop, like I will explain later in this post. So all Plone and Zope packages are uploaded there.

Therefore, pypi.python.org became a single point of failure when you need to build a Plone or Zope application. With the growth of the community, this means that the repository will get bigger and have to deal with more and more request. PyPI weight around 4 gigas at this time, of zip files and tarballs.

While the actual server is really fast, it makes it a bit hard to work when it is down. It didn’t happen often. Twice as far as I remember. But when it happens, the crowd that uses zc.buildout is frozen because they run buildouts several time per day. They can use cache of course, as long as those are up-to-date.

Some mirrors emerged, like http://release.ingeniweb.com/pypi.python.org-mirror. Some enhanced indexes were created as well, like http://download.zope.org/simple, which is still referring to pypi.python.org but performs a bit of crawling to speed things up when a buildout uses it through easy_install.

But all of these are just enhancement to a cheeseshop-centric model.

Furthermore, in case of a private application, you do not want to see packages in the Cheeseshop but you might also want to provide developers a similar way to manage and use them.

Plone.org software center is dying !

For public applications, given the fact that Distutils provides a simple way to push a package to a public location (one shell command), people have started to stop updating the Products section of Plone.org. This happens because updating it means doing a whole lot more than the simple register+upload call. You need to login into the website, then manually upload the packages, then change the front page of the product if needed etc..

So being able to push packages to plone.org with the same set of commands, is the right solution for developers.

Pushing a package means uploading a public archive but it also means updating the front page for the given package, with the register command.

That what I worked on, at PloneSoftwareCenter level, continuing Sidnei’s work: making it act like Cheeseshop. Now the feature is ready and being alpha-tested at new.plone.org

Toward a distributed model

What we will be able to do from there, is to distribute packages (in blue) to the Cheeseshop (2) as usual, but also to Plone.org (1). Having such a tool also allows people to run other cheeseshop-like servers, wether they are public (3) or private (4). This is useful when you are working on customer projects and do not wish to make their packages available to the world, or even if it is a public project, you do not want to push them to PyPi.

Furthermore, it allows having a bit of redundancy : your packages become available in several places, which is better. Think about the mirrors at Sourceforge, same thing here…

This distributed model is used at Ingeniweb, and in other companies I am starting to list (If you do please let me know, this is important to promote the tool).

The new Plone Software Center (PSC)

If you are extensively working with packages and buildouts, you should consider trying the new PSC, read http://tarekziade.wordpress.com/2008/03/20/how-to-run-your-own-private-pypi-cheeseshop-server/ for this. Each project now in the software center can hold several eggs, and it makes it possible to provide a nice deployment model for your teams: developer push packages releases in such a server with the distutils standards, while customers or deployers can build their applications by picking them up through their buildouts.

And it is in Plone, so you can provide extensive features, like a bug tracker, and all the sweet things Plonecan provide.

The work left to be done in PSC is :

• making the storage for archives (tgz, egg) pluggable so new storing strategies can be provided in separate packages, under the collective.psc namespace, I started this work in a branch.
• finish the collective.psc.mirroring package, that will be used in plone.org to fill a system folder, in order to provide an Apache direct view over the archives stored. This will make zc.buildout / easy_install use this stream rather than hitting the Plone instance. Although the final goal will be to transform it into a storage strategy, but time is running and this is useful now for plone.org.
• extensive tests on new.plone.org

collective.dist

To be able to deal with several servers, I have released collective.dist after the sprint (previously iw.dist). This package provides two new distutils commands: mupload and mregister, together with a new pypirc format.

Follow the documentation, and you will be set in a matter of minutes. This package is already used by many developers to make their life easier, and will help you when the new plone.org goes live.

disutils evolution

collective.dist is an evolution of distutils I have been working at Python level for months. It is available as a Python patch here : http://bugs.python.org/issue1858. mregister and mupload are just modified register and upload commands that makes it possible to interact with several PyPi-like servers, that’s it. PSC uses Distutils standards in any case, so it is possible to use the regular register and upload commands with it of course. It is just not convenient because you will need to have the same username and password on both CheeseShop and the third-party egg server (or change your pypirc file everytime :D)

So the patch, basically, just makes the -r option of distutils commands a 100% operational.

Now my job is to convince Python core developers it should be integrated into Python 2.6, and I am working on this. It is a logical evolution, but it sounds overkill to some people that does not have this kind of need: “Why changing that ? we have only one package server, which is the cheeseshop”.

It can also sound to some of them like I am (together with the team of 20 developers we have ) the only person on earth that wishes it, but as soon as the new Plone.org will be online, a lot of people will start needing such a feature. But having it in 2.6 will make it available as a standard in Plone/Zope world in… a few years.

In any case, most of them agree in the fact that this change is logical and that the current pypirc format is not to be kept. So I guess it is just a matter of time and patience.