Interested in collective.buildbot usage or development ?

Join us in the dedicated google group : http://groups.google.com/group/collectivebuildbo

Reminders:

• collective.buildbot is a set of zc.buildout recipes and support for declarative configuration for Buildbot.
• BuildBot is a system to automate the compile/test cycle required by most software projects to validate code changes, to set up a continuous integration system.

The Pimp my Buildbot project that was started here at Ingeniweb some time ago, to be able to set up a buildbot in a matter of minutes with zc.buildout, was continued during the sprint, and the guys did a great job on it.

It will be used here in customer projects with a Paster that adds buildbot support when a project starts, because it is a waste of time for the developers to set everything everytime.

Jean-Francois Roche, Kai Lautaportti and Gael Pasgrimaud added extensive configuration options (mail, scheduling), and made the SVN Poller works. This feature allows for instance to make the buildbot watch a SVN repository without having to add a hook in the server (post-commit hook for instance), when you don’t own it (SourceForge, collective, etc)

The tool is released in the collective, and available at the Cheeseshop in one single package !

If you want to set a buildbot

• provide for each one of your project a buildout that has a test script
• make sure the test script returns exit code (–with-exit-status with zope.testing)
• create a buildout cfg file using collective.buildbot
• run buildout, that’s it !
• run the master, slaves scripts, and go to the /waterfall page

Just try out our own buildbot by running this sequence:

$ cd /tmp/
$ mkdir my_bot
$ cd my_bot/
$ svn co https://ingeniweb.svn.sourceforge.net/svnroot/ingeniweb/buildbot/trunk .
$ python bootstrap.py
$ bin/buildout
$ bin/master start
$ bin/linux_debian start   (that's our slave)

You should have a buildbot running then at http://localhost:9000/waterfall

The tool, without the polling stuff, also works with Mercurial and Bzr, but probably needs more tests with these repositories. We also need to make sure the slaves works fine under Windows, and add a nice front page to the buildbot.

If you use it let us know !

This is the first post of the wrapups I want to do about the sprint that we had in Paris last week-end. We had a Bird of Feather about how people use eggs and zc.buildout in their projects, how they release and deploy them.

There were some people from Headnet (Anton, Mustapha) and Infrae (Kit, Sylvain) and Ingeniweb (Me), and we compared a bit how we are working with eggs, zc.buildout etc.

That is what I love in our community: companies can share their knowledge and grow up all together, technically speaking.

We all have internal recipes, command-line scripts, and are all relatively happy with zc.buildout. This discussion was very instructive.

From there, I thaught it would be a good idea to launch a new project in the community, on the top of zc.buildout (and maybe zc.sourcerelease), that would provide a common set of tools and deploying best practices, for people that works with the buildout, no matter which framework they use (Silva, Plone, etc.)

The first step for this project is to find the common needs and see if we can join forces to build common tools. To start it up, I decided to wrap up and release our internal set of tools into a single package called iw.releaser and publish it. This is the work I have done during the last months with the help of Gael, to help our team to work with zc.buildout in Plone Projects. It is Subversion dependent at this time.

I am expecting some feedback from Anton and Sylvain to see if we can make it a collective tool.

This package provides:

• a skeleton to build a project structure (buildout, packages; docs, etc.) so all projects have a standardized structure
• a ‘release’ distutils command that releases a package, upload it to the Cheeseshop or other servers, and send a shout out mail
• a set of command line tools, that can be used to deploy a buildout. These commands are doing many things besides launching a buildout building (which is a bit different from zc.sourcerelease)

This package is used at Ingeniweb for a few months now, and I tried to summarize how it is used in the docs. I bet a lot of bugs will be found if you try it, so consider this package as a non-mature package yet.

Join us all ! So you will be able to release and deploy your buildout-based apps with a few command-line calls,

When a project team use Test-Driven Development to build the code (everyone should), the next step is to set up automate builds, as explained in continous integration.

This is where Buildbot is great. It is easy and flexible to install, even more since Twisted has been eggified. A few easy_install steps are now sufficient to create a buildbot waterfall. Configuring a buildbot requires writing a few Python scripts though, and this has to be done everytime a project starts.

In my work, I need to be able to set buildbots in a matter of minutes, and they are always similar. They just need a buildmaster, a buildslave, and a few rules.

A first attempt to make things easier is to write a Python Paste script that generates default files. This is not very flexible though, as upgrades are still a bit tedious.

A more interesting solution is to provide zc.buildout recipes that take care of buildmaster and buildslave generation, through a very simple configuration file.

We have started this project, in order to be able to launch buildbot within a buildout.

The project has three parts:

• iw.buildbot: a thin layer on the top of Buildbot that allows to drive it with configuration files, instead of Python code. In other words, it makes buildbot configuration based on declarative configuration file and dynamic Python code, instead of declarative Python code.
• iw.recipe.buildmaster: a zc.buildout recipe that creates a buildbot instance together with configuration files.

• iw.recipe.buildslave: a zc.buildout recipe that creates a buildslave

The result is that creating a buildbot is done in a few lines in the buildout.cfg file:

[buildout]
parts =
  buildmaster
  linux_debian

[buildmaster]
recipe = iw.recipe.buildmaster
project-name = Ingeniweb Public buildbot
project-url = http://ingeniweb.com
port = 8999
wport = 9000
url = http://buildbot.ingeniweb.com
slaves =
  linux_debian    xxxxx
projects =
  iw.recipes 

[linux_debian]
recipe = iw.recipe.buildslave
host = localhost
port = ${buildmaster:port}
password = xxxx 

[iw.recipes]
slave-name=linux_debian
base-url=http://ingeniweb.svn.sourceforge.net
repository=/svnroot/ingeniweb/projects/iw.recipes/
branch=buildout
build-sequence =
    python bootstrap.py
    bin/buildout 

test-sequence =
    bin/test -v –exit-with-status

• buildmaster defines the project name and url, the buildbot web port, slave port and url, and a list of slaves and projects.
• each project has his own section, where the Subversion path is defined, as well as the build sequence and the test sequence.
• each slave is defined in its section

From there other buildouts can be created to group packages to be tested, and to define a script that can be used for tests. For Zope applications, that would be zopectl of course, as long as it is used with –exit-with-status.

This is the case for iw.recipes: it grabs all iw.recipes.* eggs to hook them to a testrunner.

The result can be see here: http://buildbot.ingeniweb.com

The next steps are to make sure everything works fine under Windows, and see how things goes in various projects, then make it work with all kinds of VCS and schedulers.

I will probably propose a sprint task on this in Paris sprint and see if the package meets interest.

I love Pylint.

Correctly configured, it is really useful to raise your code quality. But it can be really painful to install if you are not under a Debian-like or OpenSuse-like system, because of its dependencies that are not namespaced packages (logilab-common and logilab-astng).

In other words, don’t try to easy_install the packages that are on PyPI, it will not work.

That’s why I have created an egg, called logilab.pylintinstaller, that will let you install it as easy as:

$  easy_install logilab.pylintinstaller

It bundles all dependencies and grabs pylint 0.14, then installs everything.

Thanks to Sylvain Thénault from Logilab for his help on this.

I have recently found some UnicodeDecodeError bugs on some products, that some people couldn’t reproduced. The bug was due to a call to a CMF API that was doing a str() over the object, right before using it.

This is perfectly fine in that case, because the object is supposed to be a ZODB id, so it has to be full ASCII.

So the bug looks like this :

>>> id = u'éou'
>>> str(id)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in
position 0: ordinal not in range(128)

The people that couldn’t reproduced it because they use that ugly hack which consists of setting Python’s default encoding to utf8:

>>> import sys
>>> sys.setdefaultencoding('utf8')
>>> id = u'éou'
>>> str(id)
'\xc3\xa9ou'

This will be applied to the whole process, and Python itself dynamically removes the method from the module at it first use. From the official doc:

setdefaultencoding(name)

Set the current default string encoding used by the Unicode implementation.
If name does not match any available encoding, LookupError is raised.
This function is only intended to be used by the site module implementation and,
where needed, by sitecustomize. Once used by the site module, it is removed from
the sys module's namespace. New in version 2.0.

I can’t find the link back, but I have read once that this built-in was to be removed because it should not be used outside site.py

The problem is that people tend to add a sitecustomize.py in their environment, then work with str() and unicode() calls and forget about doing it right. The result is a major
misused of strings and unicodes and the code created will be buggy on other computers.

So never ever use this in your code. If you have a UnicodeDecodeError it probably means the function is waiting for a string. If you have a UnicodeEncodeError, it should be unicode. In the same way, do not guess the encoding in your code. You should work with one type (str or unicode) and know exactly what is its encoding.

I think this misued is partly due to a lack of warning here: http://www.diveintopython.org/xml_processing/unicode.html

Because that’s one of the first page a developer finds when he tries to understand why

he has such bugs.

See a similar entry on the topic 2 years ago here: http://faassen.n–tree.net/blog/view/weblog/2005/08/02/0

EDIT: Justus provided in the comments a great link where Jim Fulton argues that __ should be marked deprecated, folllowed by Neal Norwiz and Tim Peters answers. This helps a lot understanding what __ should be used for: http://mail.python.org/pipermail/python-dev/2005-December/058555.html 

I am writing on Python OOP best practices and I was wondering what are the best ways to name attributes in classes. My main concern is about the distinction between private and protected attributes.

• private attributes are attributes that cannot be seen or used outside of the class, even buy subclasses. The Python parser calls the name mangling algorithm when it finds them to prevent name collision;
• protected attributes are attributes that can be used and seen in subclasses and should not be used outside. Nothing is done on them and they can be used like public attributes.

When should we use them ?

If you read PEP8, it’s clearly said that name mangling (using a ‘__’ prefix) is the best way to protect an attribute from beeing accessed or overriden. So it should be used for all class internals that is not intended to be overriden.

But in the biggest open source code bases like Zope or Plone, ‘__’ usage is very uncommon. The simple ‘_’ prefix is often used instead, to mark attributes that are private to the class or to the module. So there are no real distinction between private and protected attributes. It seems that the ‘private’ concept is not even used, and people often cut their class code in two parts: public and protected.

In other languages (like Delphi) that define protected and private levels though, protected attributes are not used a lot, and people tend to cut their code in private and public parts and make the protected layer as slim as possible.

Practical rules

Based on these remarks, here’s a tentative of ‘__’ and ‘_’ prefixes best usages in Python, for the use cases I know :

• use __ with property. since properties cannot use overriden methods and are tied to the class, the methods used with it should always be private;
• use __ for methods that works with private attributes. If your methods works for private attributes, make them private too;
• use _ on methods when they are clearly intended to be overriden;
• use __ for all module functions and variables that are private. A protected level is not needed since a module cannot be overriden.

Following these rules would probably make 90% of class attributes private instead of protected, and change all base code conventions. So I am wondering: am I a bit unpythonic if I try to follow this standard in attribute naming ? My guess is that most base code are not clean enough in that matter. For instance, many of them use both new-style and old-style classes under Python 2.5, which lead to a MRO algorithm that differs depending on the classes !

I would love to hear how you people deal with these conventions.