I feel really bad about this comment on my book : How To Be Dissappointed in Something You Recommend.

Just a quick word about the try, return finally code pattern, since I had some feedback about it. I would like to mention that this code pattern is perfectly right:

def function():
    try:
      return something
    finally:
      do something

I should have explained it better, because this pattern is not used a lot by people, so you can think that “do something” is called after the return of the function, which is not the case.

For the typos now:

The first thing I did wrong: when I started the book, I wanted, as I did in my previous book, to run unit tests on the book itself to avoid those mistakes. That said, the previous one was in Latex, which is quite simple to interact with, and this one is in OpenOffice, because that is how the editor works. I had to write a script to extract the Python code from the Ooo file, to unit test it. I didn’t. I simply ran out of time, as usual when you have deadlines on books.

The second thing I did wrong: I should have told the editor to wait a bit, I didn’t.

But Packt does Print On Demand, so I know that the Errata page I am maintaining here : http://atomisator.ziade.org/wiki/Errata, is being processed by the editor, and that the typos will be removed from the book at some point, without having to wait for a second edition.

I’ll update this blog entry as soon as I know the status on this.

I am really sorry Calvin, and all the people that are suffering from these typos.

Nose comes with a handy option to generate profiling stats.

To profile your code, create a test dedicated to this purpose and run it with the right options:

$ nosetests --with-profile --profile-stats-file stats.pf test_performance

This will run the tests that corresponds to the test_performance name and generate a stats.pf file.

Nose uses hotshot, so if you want to generate a file that can be read directly by the pstats module and all the statistics tools out there, you need to convert it using the hotshot.stats module.

From there, there is plenty of tools that can transform such a file into a visual graph. Most of the time, they use Graphviz to render a graph, by generating a file dot can read. This software is most of the time easy to install through a binary distribution on your system. If you need to compile it… good luck..

Anyway, from there, I use gprof2dot, which renders a nice graph with meaningful colors.

From the author:

The color of the nodes and edges varies according to the total time % value. In the default temperature-like color-map, functions where most time is spent (hot-spots) are marked as saturated red, and functions where little time is spent are marked as dark blue.

If you want to use it, I have created some console scripts for conveniency, you can install using easy_install :

$ easy_install pbp.scripts

It creates a gprof2dot script you can use, following the author documentation, but also a hotshot2dot script that will convert automatically a statistics file and pass it to gprof2dot:

$ hotshot2dot /path/to/my/hotshot/file

This will print in the output a dot file, you can send to the dot program, using a pipe:

$ hotshot2dot /path/to/my/hotshot/file | dot -Tpng -o output.png

You will get the visual result in output.png.

Interested in collective.buildbot usage or development ?

Join us in the dedicated google group : http://groups.google.com/group/collectivebuildbo

Reminders:

• collective.buildbot is a set of zc.buildout recipes and support for declarative configuration for Buildbot.
• BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes, to set up a continuous integration system.

The Pimp my Buildbot project that was started here at Ingeniweb some time ago, to be able to set up a buildbot in a matter of minutes with zc.buildout, was continued during the sprint, and the guys did a great job on it.

It will be used here in customer projects with a Paster that adds buildbot support when a project starts, because it is a waste of time for the developers to set everything everytime.

Jean-Francois Roche, Kai Lautaportti and Gael Pasgrimaud added extensive configuration options (mail, scheduling), and made the SVN Poller works. This feature allows for instance to make the buildbot watch a SVN repository without having to add a hook in the server (post-commit hook for instance), when you don’t own it (SourceForge, collective, etc)

The tool is released in the collective, and available at the Cheeseshop in one single package !

If you want to set a buildbot

• provide for each one of your project a buildout that has a test script
• make sure the test script returns exit code (–with-exit-status with zope.testing)
• create a buildout cfg file using collective.buildbot
• run buildout, that’s it !
• run the master, slaves scripts, and go to the /waterfall page

Just try out our own buildbot by running this sequence:

$ cd /tmp/
$ mkdir my_bot
$ cd my_bot/
$ svn co https://ingeniweb.svn.sourceforge.net/svnroot/ingeniweb/buildbot/trunk .
$ python bootstrap.py
$ bin/buildout
$ bin/master start
$ bin/linux_debian start   (that's our slave)

You should have a buildbot running then at http://localhost:9000/waterfall

The tool, without the polling stuff, also works with Mercurial and Bzr, but probably needs more tests with these repositories. We also need to make sure the slaves works fine under Windows, and add a nice front page to the buildbot.

If you use it let us know !

This is the first post of the wrapups I want to do about the sprint that we had in Paris last week-end. We had a Bird of Feather about how people use eggs and zc.buildout in their projects, how they release and deploy them.

There were some people from Headnet (Anton, Mustapha) and Infrae (Kit, Sylvain) and Ingeniweb (Me), and we compared a bit how we are working with eggs, zc.buildout etc.

That is what I love in our community: companies can share their knowledge and grow up all together, technically speaking.

We all have internal recipes, command-line scripts, and are all relatively happy with zc.buildout. This discussion was very instructive.

From there, I thaught it would be a good idea to launch a new project in the community, on the top of zc.buildout (and maybe zc.sourcerelease), that would provide a common set of tools and deploying best practices, for people that works with the buildout, no matter which framework they use (Silva, Plone, etc.)

The first step for this project is to find the common needs and see if we can join forces to build common tools. To start it up, I decided to wrap up and release our internal set of tools into a single package called iw.releaser and publish it. This is the work I have done during the last months with the help of Gael, to help our team to work with zc.buildout in Plone Projects. It is Subversion dependent at this time.

I am expecting some feedback from Anton and Sylvain to see if we can make it a collective tool.

This package provides:

• a skeleton to build a project structure (buildout, packages; docs, etc.) so all projects have a standardized structure
• a ‘release’ distutils command that releases a package, upload it to the Cheeseshop or other servers, and send a shout out mail
• a set of command line tools, that can be used to deploy a buildout. These commands are doing many things besides launching a buildout building (which is a bit different from zc.sourcerelease)

This package is used at Ingeniweb for a few months now, and I tried to summarize how it is used in the docs. I bet a lot of bugs will be found if you try it, so consider this package as a non-mature package yet.

Join us all ! So you will be able to release and deploy your buildout-based apps with a few command-line calls,

When a project team use Test-Driven Development to build the code (everyone should), the next step is to set up automate builds, as explained in continous integration.

This is where Buildbot is great. It is easy and flexible to install, even more since Twisted has been eggified. A few easy_install steps are now sufficient to create a buildbot waterfall. Configuring a buildbot requires writing a few Python scripts though, and this has to be done everytime a project starts.

In my work, I need to be able to set buildbots in a matter of minutes, and they are always similar. They just need a buildmaster, a buildslave, and a few rules.

A first attempt to make things easier is to write a Python Paste script that generates default files. This is not very flexible though, as upgrades are still a bit tedious.

A more interesting solution is to provide zc.buildout recipes that take care of buildmaster and buildslave generation, through a very simple configuration file.

We have started this project, in order to be able to launch buildbot within a buildout.

The project has three parts:

• iw.buildbot: a thin layer on the top of Buildbot that allows to drive it with configuration files, instead of Python code. In other words, it makes buildbot configuration based on declarative configuration file and dynamic Python code, instead of declarative Python code.
• iw.recipe.buildmaster: a zc.buildout recipe that creates a buildbot instance together with configuration files.

• iw.recipe.buildslave: a zc.buildout recipe that creates a buildslave

The result is that creating a buildbot is done in a few lines in the buildout.cfg file:

[buildout]
parts =
  buildmaster
  linux_debian

[buildmaster]
recipe = iw.recipe.buildmaster
project-name = Ingeniweb Public buildbot
project-url = http://ingeniweb.com
port = 8999
wport = 9000
url = http://buildbot.ingeniweb.com
slaves =
  linux_debian    xxxxx
projects =
  iw.recipes 

[linux_debian]
recipe = iw.recipe.buildslave
host = localhost
port = ${buildmaster:port}
password = xxxx 

[iw.recipes]
slave-name=linux_debian
base-url=http://ingeniweb.svn.sourceforge.net
repository=/svnroot/ingeniweb/projects/iw.recipes/
branch=buildout
build-sequence =
    python bootstrap.py
    bin/buildout 

test-sequence =
    bin/test -v –exit-with-status

• buildmaster defines the project name and url, the buildbot web port, slave port and url, and a list of slaves and projects.
• each project has his own section, where the Subversion path is defined, as well as the build sequence and the test sequence.
• each slave is defined in its section

From there other buildouts can be created to group packages to be tested, and to define a script that can be used for tests. For Zope applications, that would be zopectl of course, as long as it is used with –exit-with-status.

This is the case for iw.recipes: it grabs all iw.recipes.* eggs to hook them to a testrunner.

The result can be see here: http://buildbot.ingeniweb.com

The next steps are to make sure everything works fine under Windows, and see how things goes in various projects, then make it work with all kinds of VCS and schedulers.

I will probably propose a sprint task on this in Paris sprint and see if the package meets interest.

I love Pylint.

Correctly configured, it is really useful to raise your code quality. But it can be really painful to install if you are not under a Debian-like or OpenSuse-like system, because of its dependencies that are not namespaced packages (logilab-common and logilab-astng).

In other words, don’t try to easy_install the packages that are on PyPI, it will not work.

That’s why I have created an egg, called logilab.pylintinstaller, that will let you install it as easy as:

$  easy_install logilab.pylintinstaller

It bundles all dependencies and grabs pylint 0.14, then installs everything.

Thanks to Sylvain Thénault from Logilab for his help on this.