I am taking a break for a week starting tomorrow morning. My wife made it clear : my laptop and internet are totally forbidden during 7 days. So I thaught it was a good opportunity to make a status of all the projects that are going on in the packaging world.

Distribute

The setuptools fork is doing good. We’ve discussed the roadmap strategy with some people at Distutils-SIG and came up with that plan:

0.6 will be released the first week of August. It will support Python 2.x only and will provide a bootstrap to replace an existing version of setuptools, as described in my previous entry.

0.7 will support Python 2.x and 3.x and will refactor the code in three distributions. It will also rename all parts under new names, so they will not compete anymore with setuptools. Its development will start right after 0.6 is out. If you have ideas or feature requests, please add them in our issue tracker at bitbucket.

Distutils

They are some patches waiting for my attention, I need to work on when I am back. But globally, if you open Distutils code, you will notice that it has been PEP8-fied (besides a few places I need to finishe) and test covered.

Some small refactorings have been done, like removing duplicate code and using the tar module of the stdlib instead of the tar command on the system, thus allowing building tar archive under win32 without having to install the tar program for example. Another nice change was done on the upload command: it uses urllib2 now, meaning that you can use an http proxy with a environment variable transparently. Other changes are on their way, like fixing the mess with “compiler” option, and cleaning up some remaining duplicate behaviors.

But nothing is deeply changing yet on how Distutils works. It waits for the PEPs we are working on to be accepted…

PEP 376

After I thaught this PEP was ready to be accepted, I sent it from Distutils-SIG to Python-dev. Then we had very long threads on Python-dev about it, showing that it was not ready at all…

Man, I don’t think I’ve picked the easiest topic

Anyways, people gave a lot of thaught on the topic, and Paul Moore helped building a PEP 302-compatible version of the prototype, you can see at bickbucket. From there, we still need to fix the problem of the absolute/relative paths in the RECORD file, and try to have real world uses cases, with various package managers applications. But we will eventually have an acceptable PEP, I hope, within a few months.

This pep will add query APIs in pkgutils in the stdlib, so you know what’s installed in your Python. It will also provide tools to uninstall installed ditributions.

It’s a hot topic because we are dealing with the fuzzy boundary between Python the extendable language, where you can install distributions using distutils-based tools, and Python the interpreter, that gets extended with packages managed by an OS-based package manager like APT under Debian for example. So for the latter, all our tool are getting too much in the way and should not make it hard for system packagers to extract the metadata they need to re-package distutils-based distributions.

PEP 386

Man, this PEP would make our life easier. It’s all the work we started during Pycon, to find an acceptable way to compare versions. By acceptable I mean that could be used in our community and workable by OS packagers. I said I would drop it, because there were too much controversy, but quite a few people would like to see it accepted.

I doubt Guido will accept a PEP that would enforce a version scheme though.

I had created this PEP, so we could move forward on PEP 345, where we want to introduce “install_requires”, the field used in setuptools to list dependencies.

The prototype for PEP 386 is in bitbucket too : verlib

Whatever happens, I’ll probably publish it. Its useful to build a package system, and it has a function to translate most distutils/setuptools version schemes.

Now about my addiction

I know that sounds stupid, but I don’t know what is going to happen next week. It’s been years since I’ve not been fully offline for a week. I already have right now so many threads to catch up on various mailing lists that I am scared of how it’s going to be in 7 days…

But it’s probably a good thing for me to do this

I am off, see you in 7 days !

Again, like a year ago, people had enough of the fact that the setuptools project is not maintained since 9 months.

Phillip Eby explained that he doesn’t have time to do it unless someone would pay him for that. But in the meantime, he doesn’t bless anyone to do it. Well, he has blessed some people to do it (Ian Bicking and Jim Fulton), but unfortunately these people are not willing to do it because they have a lot of other projects going on. Other people that could maintain it, including me, fail in his “unqualified people” category

So we are all locked in a strange situation where tons of patches are ready to be commited in the setuptools tracker but are not making it. Several non-public forks have started to appear around of course.

So again, I decided with some other people to create a fork called “Distribute”. It’s a real fork located here : http://bitbucket.org/tarek/distribute.

By real I mean that this fork was not created with the purpose of forcing Phillip to do a release like we did last year for the 0.6c9 release, but with the intention to free us from that strange situation where we all depend on his wills and (lack of) time.

The plan is to release a first version next week, that corresponds to the setuptools 0.6 branch, with some patches applied.

Next, we are planning to start a 0.7 version where the code will be splitted in several distributions:

• a distribution for pkg_resources
• a distribution for the setuptools package itself
• a distribution for easy_install

A little bit of bikeshedding is going on to pick a name for that fork, and we ended up running a poll. (vote!)

Now, right after I have announced this plan on Distutils-SIG, Phillip reacted by annoucing a similar plan, e.g. splitting the setuptools project in several distributions. But since he previously said that he didn’t have the time to do it, I doubt that it’ll work out unless he’s opening its development to a wider range of developers and maintainers.

That’s the strange world of packaging…

A quick tip useful to keep the coherence between the setup.py file of your egg and the profiles/default/metadata.xml file. The simpliest way to do this is to compute version value in setup.py from the metadata.xml. In the setup.py file remove the version = ‘x.y.z’ line by the following code:

from xml.dom import minidom

metadata_file = os.path.join(os.path.dirname(__file__),
                             'collective', 'bobo',
                             'profiles', 'default', 'metadata.xml')
metadata = minidom.parse(metadata_file)
version = metadata.getElementsByTagName("version")[0].childNodes[0].nodeValue
version = str(version).strip()

You can check it works with the command:

python setup.py --version

As you might know, we are working hard on Distutils side for Python 2.7 and 3.2 upcoming releases. The biggest work is on PEP 376, that will introduce among other things a uninstaller function and functions to query installed distributions.

The other “big” work is on PEP 345. We want to introduce a new metadata field called “install_requires” to be able to express requirements. That’s from the setuptools project and is quite used by the community. Notice that there were several attempts to define requirements in the past in Distutils, but none of them really made it through.

For instance, if you want to define docutils as a dependency for your distribution but a version less or equal to 0.4, you can say :

docutils <= 0.4

But as long as you want to work with such dependencies and provide a way to express them with operators, you have to be able to compare versions. For instance if you want to compare an installed docutils distribution to see of it is compatible with 0.4.

That’s another big topic we have been working on for the last few months, with people from various communities (Fedora, Ubuntu, etc). And I have started to write down PEP 386.

But comparing version appears to be a topic that cannot be generic. It seems that Distutils, therefore Python, shouldn’t enforce any rule on this.

Furthermore, since we have said that Distutils should be a lighter package, it will not implement a complete package managment system, like setuptools or zc.buildout does.

So I’ve decided to propose to drop PEP 386, and stick on a very simple rule in PEP 345, saying that requirements can be defined, with :

distribution_name OPERATOR version

where OPERATOR is in >, <, ==, !=, >= or <=.

Last, so the work done at Pycon and in Distutils-SIG is not lost, I will publish the library we wrote. This could be a very good basis for packaging managment systems out there.

If you want to work (or try to find some bugs in it to help out) with the latest bleeding-edge Distutils trunk version that will be shipped with Python 2.7 and Python 3.2, you can do it now !

I am creating a nighlty build every day here now : http://nightly.ziade.org/

This distribution can be installed in your existing Python 2.6 installation, and will probably work with 2.5.

I’m not planning to publish standalone versions yet, except nighlty build, at least until 2.7 and 3.2 are starting to be released as alphas.

We had to install mod_ntlm under a FreeBSD 6.4 server with a colleague, but the port for this package seems broken at least for FreeBSD 6.2 and 6.4. It just doesn’t work when it initiates an NTLM session through SMB, and it seems to be compiled without the log support so the last log you get doesn’t give any useful hint on what’s wrong.

So we had to recompile it to activate the log and try understand what the problem was. But Apache doesn’t not support threaded mode under FreeBSD 6.x (or I couldn’t manage to make it work howsoever), so mod_ntlm failed to compile since it uses Apache’s thread mutexes to perform some locks on some operations.

We’ve deactivated them and recompiled the module, and now it works now like a charm. It’s weird because it means that the binary package for this module is completely broken. I couldn’t find any place mentioning this.

Anyways, I know this is an old software combination, but since Google doesn’t give any hint on this problem, I wanted to blog about it and join the diff I made out of mod_ntlm2 at https://modntlm.svn.sourceforge.net/svnroot/modntlm/trunk here: diff file

So if you bump into this problem, you will hopefully reach this page and save a few hours.

Since few weeks we have problems to use collective.releaser when we use release-packages: packages are not uploaded to our private pypi or are published in pypi when they should be private and public packages are not published at all.
collective.releaser seems be a victim of this bug as the 0.6.2 release was never published on pypi even if the tag was done 2 months ago.

Today ready this:
collective/releaser/commands.py:L107

Line 110, when the expression is matching something the loop continues, then in the line 117 an empty list is in found sometimes.

If you add these two lines after the line 112 you can again upload your releases in cheeseshops:

if founded != []:
    break

Since few weeks plone.recipe.pound 0.5.4 was broken with an output like this:

An internal error occured due to a bug in either zc.buildout or in a recipe being used:
Traceback (most recent call last):
File "/tmp/tmpVcq-b1/zc.buildout-1.2.1-py2.4.egg/zc/buildout/buildout.py", line 1509, in main
File "/tmp/tmpVcq-b1/zc.buildout-1.2.1-py2.4.egg/zc/buildout/buildout.py", line 473, in install
File "/tmp/tmpVcq-b1/zc.buildout-1.2.1-py2.4.egg/zc/buildout/buildout.py", line 1091, in _call
File "/home/encolpe/.virtualenvs/internal/preprod/plone.recipe.pound/plone/recipe/pound/build.py", line 74, in install
installed = CMMIRecipe.install(self)
File "/home/encolpe/.virtualenvs/internal/preprod/downloads/dist/zc.recipe.cmmi-1.2.0/src/zc/recipe/cmmi/__init__.py", line 155, in install
system("make install")
File "/home/encolpe/.virtualenvs/internal/preprod/downloads/dist/zc.recipe.cmmi-1.2.0/src/zc/recipe/cmmi/__init__.py", line 27, in system
raise SystemError("Failed", c)
SystemError: ('Failed', 'make install')

This recipe directly unherit from zc.recipe.cmmi and act as a wrapper around the CMMIRecipe.install method. Most arguments used are transmitted to the recipe are self attributes or keys in self.options. The version 1.2.0 of this recipe publish the May, 18 moved the way that extra arguments are transmitted:

• zc.recipe.cmmi 1.1.x:  self.options['extra_options']
• zc.recipe.cmmi 1.2.0: self.extra_options (an empty string by default)

There’s no regression tests, neither an entry in the release history. Youenn did a new release of plone.recipe.pound to fix this new behavior.

Morality:

• if you are using zc.recipe.cmmi 1.1.x you can still use plone.recipe.pound 0.5.4
• if you are using zc.recipe.cmmi 1.2.0 or above use plone.recipe.pound 0.5.5
• other recipes using zc.recipe.cmmi may be broken

I am not really sure smoke testing is the best name to describe what I am trying to set up.

Wikipedia says:

In computer programming and software testing, smoke testing is a preliminary to further testing, which should reveal simple failures severe enough to reject a prospective software release. In this case, the smoke is metaphorical.

But I think it describes well the need I have : being able to test the output of several releases of Distutils, without relying on unit tests for that. Simply because they did not exist on early versions of Distutils.

Why do I need “smoke tests”

Lately, I was working on Distutils, removing some old bugs and adding some tests. Working on an under-tested package is like opening a can of worms : when you change something you can introduce some regressions. It’s hard to avoid for sure, because even if you add some tests subsequently for some features, you are not really doing proper TDD, where the tests and the code grow together. So there might be some uncovered cases left even if the coverage is improving slowly.

Regression also happens when you correct a behavior that was broken for years : if third party tools used a broken behavior without knowing it, fixing the behavior at some point becomes a regression from them.

But these pitfalls are unavoidable. Hopefully they don’t last for too long in a big project like Python. They are quickly reported, either by buildbots, either by someone from the community.

The last behavior problem I had was on the build_ext command. There’s a string option called “compiler”, were you can force the compiler type. You can put for instance ‘linux‘ and build_ext will use the UnixCompiler compiler class.

Although this option name is misleading. It should have been called “compiler_type” in the first place.  And the build_ext command adds another problem : it turns the compiler attribute of the class where the option value is stored, into a compiler instance. No need to say a class attribute value should not have several types.

The first application to suffer from this is Python itself. It uses build_ext.compiler, as a compiler instance to build its modules.

This means that “compiler” is now doomed to be both an option and a compiler attribute. It’s not easy to fix and will require a deprecation step.

So the idea of the smoke test is to make sure Distutils still knows how to produce releases of existing projects.

The Smoke testing buildbot

I have built a buildbot instance using collective.buildbot. The script the slaves run is quite simple: it gets some projects source and run their setup.py script, using various versions of the Python interpreter, then the trunk itself. The commands run so far are sdist and bdist.

Then it compares every file contained in the archive created to make sure produced archives are similar.

I will encouter some exceptions, when the package I am testing includes different files in its distribution depending on the Python version, but those exception will be managed.

Last, I cannot run this test on every package out there, for security reasons. Until I have the proper virtual environment to do that, I’ll work on a white list of packages I “trust”. So far, there’s just a package of mine, and NumPy. But this list will grow.

If you are curious to know if your package builds under Python trunk, get in touch with me, I’ll probably add it if I know I can trust it.

The last problem I have with this system yet is the fact that setuptools is not working anymore with the Distutils trunk, so I am unable to test setuptools-based packages.

The buildbot is located here : http://buildbot.ziade.org/waterfall

Since Pycon, a lot of discussion and work has been going on.

During the summit, we made a few decisions (see http://mail.python.org/pipermail/python-dev/2009-March/087834.html) but this topic is so wide and so complex that a lot of discussion still needs to be done to have a clear and complete picture of where we want to go and how we are going to do it

But we have made significant progress and reached consensus on some key points. This entry is a short list of these key points.

Being able to compare project versions

If you take a look in Distutils code, there’s a version and a versionpredicate module that provides a way to compare versions. This feature is barely used by Distutils itself, and a very few number of projects out there are using it (If you do so, please let me know).

This is probably because Distutils provides two different ways to compare versions: a “strict” one and a “loose” one. In other words Distutils clearly states that it is unable to provide a unique version comparison algorithm that can be used by anyone out there. Anyone means here : Python developers, OS packagers, etc.

Setuptools did a better job by providing an algorithm that covers most cases, but suffers from this universality: it’s too heuristic.

So one of the main topic during Pycon was to try to find a version comparison algorithm that would just work for everyone and in the meantime would be strict enough to be useful. That’s a pretty tough task, but I think we have reached something that is “good enough” for everybody. We had the chance to have people from Ubuntu, RedHat during Pycon to work on this task, and Trent Mick took the lead during the sprints to come up with a prototype.

It’s described here : http://wiki.python.org/moin/Distutils/VersionComparison and I have put the code here http://bitbucket.org/tarek/distutilsversion.

Two more things to take care of:

• Philip Eby came up with an edge-case : being able to do a development version of a post release.
• Jean-Paul Calderone proposed to add a constructor that would take explicit arguments to describe the version, rather than a string representation

There’s a branch (tarek-postdev) on bitbucket to work on these two cases. But basically, it seems that we have a consensus on a unique way to compare versions in Python ! This is a great step forward.

Another point that Toshio Kuratomi raised during a hall discussion was the fact that some Python developers are not having good practices when versioning their releases. So we agreed that a good “How to properly use version numbers for your projects releases” document will have to be delivered in Distutils documentation, besides the new algorithm. I am pretty confident that people will eventually follow it.

APIs to access installed project metadata, and an egg-info standard

Once a project is installed in Python, you don’t have a way to get its metadata and to answer to basic questions like:

• what are the installed packages ?
• what is the version ?
• how is the author ?

Of course you can dig in your installation and get back most of these, but depending how you installed your package (manual, easy_install, pip) it might be located in different places.

So we agreed on a standard for this, described in PEP 376 : http://www.python.org/dev/peps/pep-0376

An API will be introduced to be able to get the info for a installed package, as long as it was installed using PEP 376 standard.

An uninstall feature !

A feature that is claimed for a long time now should be introduced in Distutils : uninstallation !

Distutils will provide a basic, reference uninstall feature that will remove all files that where previously installed for a project. This will be doable as long as this list of files is recorded at installation time, and not used by another project.

See the details in PEP 376.

Standardize PKG-INFO

New fields will be added in the metadata of a project. The most important one is install_requires from setuptools, which let you list the projects your project depend on and their versions.

This is for informational purpose, and Distutils will not provide an install feature that will fetch and install dependencies. This can be done by third-party tools like easy_install  as long as they use the installation standard described earlier.

We started this work with Jim Fulton at Pycon, and Tres Seaver took the lead on this task since then. The plan is to update PEP 345 . The work is done in a branch here : http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?view=markup

But the remaining work is focusing on practical details. Just remember that PKG-INFO will evolve and install_requires will be integrated amongst other changes.

Distutils code cleanup, test coverage

The test coverage is starting to look good and most modules are covered around 80%. I guess this is the average in most Test-Driven Development projects out there. So Distutils is becoming a good citizen of the standard library 

The remaining work for a good test coverage is mostly on compilers side and os specific commands.

My black list of untested (or nearly untested) modules :

• bcppcompiler
• cygwinccompiler
• emxccompiler
• msvc9compiler
• msvccompiler
• command.bdist_msi
• command.bdist_rpm

I need to figure out how to properly test them. Last, I need to set up a buildbot that will try out Distutils trunk on a list of projects out there, like numpy for example.

Other topics

Check this ! http://wiki.python.org/moin/Distutils

I am very proud to be a Gsoc mentor this year on a very interesting topic : an universal keyring library for Python.

About the topic

In Distutils, if you want to interact with PyPI, you have to register in the website and you get a login/password so you can register and/or upload your packages.

Before Python 2.6, the only way you could interact with PyPI was by storing these info into your .pypirc file in clear text. This was not the best solution. For example we have in my company some staging servers we share, and from whom we upload packages to various PyPI-like servers. So we have to store PyPI login/password info in them. This means that if Bob wants to push his package from that server, he has to put his password into a clear text file which is most of the time readable by everyone. It’s not such a big deal in our company since we can trust each other, but it’s a very bad practice.

So I ended up changing this in Python 2.6 so people could type their password in a prompt when working with packages, using getpass.getpass. So they wouldn’t have to store them anymore.

But this is not enough : we need to provide something better. We need getpass.getpass to be able to interact with keyring libraries like KeyChain, Gnome Keyring, etc. So the login/password info are safely stored and can be reused.

This service will be useful for Distutils, but also for any Python application.

The idea of the GSOC task is to provide an unified keyring library for Python, and it’s harder that it sounds. For instance, we need to find a way to provide something that works under Windows. So the whole work is quite challenging and interesting, and the goal is to end up with a keyring library we can use into Distutils and propose for inclusion in getpass.

About Kang

Kang Zhang is the student that was selected for this work. Congrats ! He has started to work on it. You can follow his work in his blog. I have a strong feeling that he will succeed in this work and come up with something good.

Take a look at the Python Soc planet too, where all students involved in Python GSOC are blogging about their ongoing work.

I need a very simple plugin system for one of my projects. The project is a small WSGI application called mysysadmin that allows you to launch some commands on your system to manage some applications. It also allows you to view log files in your web browser.

It’s similar in some ways to WebMin,

So in my application, every tab is a plugin that manages one application. I have a plugin for Apache, another one for MySQL, and so one.

Back to my plugin system. So every plugin that is registered becomes a tab in the WSGI application, as long as it provides all the methods my web application needs to interact with it. So I want to check that each plugin strictly provides the API needed by the main program.

The first tools that came in mind were :

• zope.interface, to be able to provide that each plugin meets the requirements.
• setuptools entry points, so it’s easy for a third party code to implement a plugin.

But I find both projects a bit complex to implement such a simple plugin system.I could use the standalone Plugins package Phillip provides instead of setuptools, but it still does too much things imho. That’s someting I am currently learning by working on packaging matters : one library should not provide too many features.

Extensions : a simple plugin system

So I have started to implement a light-weight plugin system called extensions, which reuses setuptools entry points principles but is more simple to use. The goal of this project is to provide very simple APIs to handle plugins, and to make it work without introducing a new argument into the setup.py setup method, like setuptools does.

For instance, if you want to define an apache function in your modules module, in your myapp package, you just call the register function :

from extensions import register

register('mysysadmin.modules', 'apache', 'myapp.modules:apache')

That’s it !

And to use it, the mysysadmin application can use a simple API called get, that iterates over all plugins defined for “mysysadmin.modules” :

from extensions import get

for plugin in get(group='mysysadmin.modules'):
    # do something with the plugin

The magic is done by writing in the .egg-info directory installed for the package that contains each plugin, a PLUGIN file that contains the list of registered elements. It’s an idea borrowed from setuptoools entry points. So get iterates over all .egg-info directories in your path and load the PLUGIN files it finds.  Nothing new here. That’s how setuptools does, and that’s perfect.

If you have any feedback on extensions, let me know !

Strict plugins

The other need is to strictly check that every plugin provides the API needed, e.g. fulfill the requirements. This is what we could call Design by contract .

You can provide a base class for this, and ask the plugins to inherit from it. Or you can ask the Plugin to provide a marker to specify it implements a given behavior. zope.interface can do a nice job for the latter,and let you check that a given object implements an interface.

But I wanted to give a shot to the brand new Python ABCs and make sure anyone can write a plugin in plain Python, without having to rely on any kind of marker system. ABCs will let you check that a class implements some methods without requiring it to inherit from a specific class, to implement a specific interface or provide a custom marker. Pure duck typing !

So let’s define for our application a Plugin class, that gives the signature every plugin will need to provide. It uses ABCMeta as its meta class, and the abstractmethod for every method that should be implemented by every plugin.

Here’s an extract :

from abc import ABCMeta, abstractmethod

class Plugin(object):

    __metaclass__ = ABCMeta

    @abstractmethod
    def get_command_list(self):
        return []

    @abstractmethod
    def run_command(self, name):
        pass

    @classmethod
    def __subclasshook__(cls, klass):
        if cls is Plugin:
            for method in cls.__abstractmethods__:
                if any(method in base.__dict__ for base in klass.__mro__):
                    continue
                return NotImplemented
            return True
        return NotImplemented

The __subclasshook__ method is a class method that will be called everytime a class is tested using issubclass(klass, Plugin). In that case, it will check that every method marked with the abstractmethod decorator is provided by the class.

So basically, the application can discover and use the plugins, with:

from extensions import get

for plugin in get(group='mysysadmin.modules'):
    klass = plugin.load()
    if not issubclass(klass, Plugin):
        logging.info('sorry, not a suitable plugin')
        continue
    # do something with the plugin
    xxx

Abstract Base Classes are one honking great idea — let’s do more of those!