I am taking a break for a week starting tomorrow morning. My wife made it clear : my laptop and internet are totally forbidden during 7 days. So I thaught it was a good opportunity to make a status of all the projects that are going on in the packaging world.

Distribute

The setuptools fork is doing good. We’ve discussed the roadmap strategy with some people at Distutils-SIG and came up with that plan:

0.6 will be released the first week of August. It will support Python 2.x only and will provide a bootstrap to replace an existing version of setuptools, as described in my previous entry.

0.7 will support Python 2.x and 3.x and will refactor the code in three distributions. It will also rename all parts under new names, so they will not compete anymore with setuptools. Its development will start right after 0.6 is out. If you have ideas or feature requests, please add them in our issue tracker at bitbucket.

Distutils

They are some patches waiting for my attention, I need to work on when I am back. But globally, if you open Distutils code, you will notice that it has been PEP8-fied (besides a few places I need to finishe) and test covered.

Some small refactorings have been done, like removing duplicate code and using the tar module of the stdlib instead of the tar command on the system, thus allowing building tar archive under win32 without having to install the tar program for example. Another nice change was done on the upload command: it uses urllib2 now, meaning that you can use an http proxy with a environment variable transparently. Other changes are on their way, like fixing the mess with “compiler” option, and cleaning up some remaining duplicate behaviors.

But nothing is deeply changing yet on how Distutils works. It waits for the PEPs we are working on to be accepted…

PEP 376

After I thaught this PEP was ready to be accepted, I sent it from Distutils-SIG to Python-dev. Then we had very long threads on Python-dev about it, showing that it was not ready at all…

Man, I don’t think I’ve picked the easiest topic

Anyways, people gave a lot of thaught on the topic, and Paul Moore helped building a PEP 302-compatible version of the prototype, you can see at bickbucket. From there, we still need to fix the problem of the absolute/relative paths in the RECORD file, and try to have real world uses cases, with various package managers applications. But we will eventually have an acceptable PEP, I hope, within a few months.

This pep will add query APIs in pkgutils in the stdlib, so you know what’s installed in your Python. It will also provide tools to uninstall installed ditributions.

It’s a hot topic because we are dealing with the fuzzy boundary between Python the extendable language, where you can install distributions using distutils-based tools, and Python the interpreter, that gets extended with packages managed by an OS-based package manager like APT under Debian for example. So for the latter, all our tool are getting too much in the way and should not make it hard for system packagers to extract the metadata they need to re-package distutils-based distributions.

PEP 386

Man, this PEP would make our life easier. It’s all the work we started during Pycon, to find an acceptable way to compare versions. By acceptable I mean that could be used in our community and workable by OS packagers. I said I would drop it, because there were too much controversy, but quite a few people would like to see it accepted.

I doubt Guido will accept a PEP that would enforce a version scheme though.

I had created this PEP, so we could move forward on PEP 345, where we want to introduce “install_requires”, the field used in setuptools to list dependencies.

The prototype for PEP 386 is in bitbucket too : verlib

Whatever happens, I’ll probably publish it. Its useful to build a package system, and it has a function to translate most distutils/setuptools version schemes.

Now about my addiction

I know that sounds stupid, but I don’t know what is going to happen next week. It’s been years since I’ve not been fully offline for a week. I already have right now so many threads to catch up on various mailing lists that I am scared of how it’s going to be in 7 days…

But it’s probably a good thing for me to do this

I am off, see you in 7 days !

According to the poll, The name of the fork will be Distribute !

The code should not be changed anymore at this point, and I am working on the bootstraping so installing Distribute will work with an existing Setuptools installation and will replace it for the applications that requires it.

This is done by detecting an installation of Setuptools, and replacing it with a fake installation. This means that once you’ve installed Distribute, applications and especially installers will think that setuptools 0.6c9 is installed.

That’s pretty strong and intrusive, but required for a simple switch : even if the programs you are using have a setuptools dependency, they will work without requiring any change on the code or in their setup.py files. Same goes for zc.buildout apps.

I still have a lot of work for this part :

• I don’t detect and patch properly single-version-externally-managed setuptools installation yet (required for pip)
• I need to fix a bug on a first run under jython
• I need to fix a bug when virtualenv is not used with –no-site-packages
• I need to do extensive tests on zc.buildout to see if it behaves correctly

If you want to help:

1. download http://nightly.ziade.org/install_test.py
2. run it with the Python interpreter of your choice (possibly a virtualenv-ed one)

To uninstall, follow the Uninstallation instructions here : http://bitbucket.org/tarek/distribute/src/tip/README.txt

*Disclaimer: it might break your installation*

If the test end up with this line: **** Test is OK. It worked. Otherwise, please let me know !

The next “big” step will happen with Distribute 0.7 because the plan is to split the code in several distributions with renamed modules:

• Distribute : the core (setuptools package but renamed)
• DistributeInstall : easy_install on its own, and renamed.
• DistributeResource : will contain the pkg_resources.py module, renamed

I’ll post more details on it in the upcoming days.

Again, like a year ago, people had enough of the fact that the setuptools project is not maintained since 9 months.

Phillip Eby explained that he doesn’t have time to do it unless someone would pay him for that. But in the meantime, he doesn’t bless anyone to do it. Well, he has blessed some people to do it (Ian Bicking and Jim Fulton), but unfortunately these people are not willing to do it because they have a lot of other projects going on. Other people that could maintain it, including me, fail in his “unqualified people” category

So we are all locked in a strange situation where tons of patches are ready to be commited in the setuptools tracker but are not making it. Several non-public forks have started to appear around of course.

So again, I decided with some other people to create a fork called “Distribute”. It’s a real fork located here : http://bitbucket.org/tarek/distribute.

By real I mean that this fork was not created with the purpose of forcing Phillip to do a release like we did last year for the 0.6c9 release, but with the intention to free us from that strange situation where we all depend on his wills and (lack of) time.

The plan is to release a first version next week, that corresponds to the setuptools 0.6 branch, with some patches applied.

Next, we are planning to start a 0.7 version where the code will be splitted in several distributions:

• a distribution for pkg_resources
• a distribution for the setuptools package itself
• a distribution for easy_install

A little bit of bikeshedding is going on to pick a name for that fork, and we ended up running a poll. (vote!)

Now, right after I have announced this plan on Distutils-SIG, Phillip reacted by annoucing a similar plan, e.g. splitting the setuptools project in several distributions. But since he previously said that he didn’t have the time to do it, I doubt that it’ll work out unless he’s opening its development to a wider range of developers and maintainers.

That’s the strange world of packaging…

As you might know, we are working hard on Distutils side for Python 2.7 and 3.2 upcoming releases. The biggest work is on PEP 376, that will introduce among other things a uninstaller function and functions to query installed distributions.

The other “big” work is on PEP 345. We want to introduce a new metadata field called “install_requires” to be able to express requirements. That’s from the setuptools project and is quite used by the community. Notice that there were several attempts to define requirements in the past in Distutils, but none of them really made it through.

For instance, if you want to define docutils as a dependency for your distribution but a version less or equal to 0.4, you can say :

docutils <= 0.4

But as long as you want to work with such dependencies and provide a way to express them with operators, you have to be able to compare versions. For instance if you want to compare an installed docutils distribution to see of it is compatible with 0.4.

That’s another big topic we have been working on for the last few months, with people from various communities (Fedora, Ubuntu, etc). And I have started to write down PEP 386.

But comparing version appears to be a topic that cannot be generic. It seems that Distutils, therefore Python, shouldn’t enforce any rule on this.

Furthermore, since we have said that Distutils should be a lighter package, it will not implement a complete package managment system, like setuptools or zc.buildout does.

So I’ve decided to propose to drop PEP 386, and stick on a very simple rule in PEP 345, saying that requirements can be defined, with :

distribution_name OPERATOR version

where OPERATOR is in >, <, ==, !=, >= or <=.

Last, so the work done at Pycon and in Distutils-SIG is not lost, I will publish the library we wrote. This could be a very good basis for packaging managment systems out there.

If you want to work (or try to find some bugs in it to help out) with the latest bleeding-edge Distutils trunk version that will be shipped with Python 2.7 and Python 3.2, you can do it now !

I am creating a nighlty build every day here now : http://nightly.ziade.org/

This distribution can be installed in your existing Python 2.6 installation, and will probably work with 2.5.

I’m not planning to publish standalone versions yet, except nighlty build, at least until 2.7 and 3.2 are starting to be released as alphas.

We had to install mod_ntlm under a FreeBSD 6.4 server with a colleague, but the port for this package seems broken at least for FreeBSD 6.2 and 6.4. It just doesn’t work when it initiates an NTLM session through SMB, and it seems to be compiled without the log support so the last log you get doesn’t give any useful hint on what’s wrong.

So we had to recompile it to activate the log and try understand what the problem was. But Apache doesn’t not support threaded mode under FreeBSD 6.x (or I couldn’t manage to make it work howsoever), so mod_ntlm failed to compile since it uses Apache’s thread mutexes to perform some locks on some operations.

We’ve deactivated them and recompiled the module, and now it works now like a charm. It’s weird because it means that the binary package for this module is completely broken. I couldn’t find any place mentioning this.

Anyways, I know this is an old software combination, but since Google doesn’t give any hint on this problem, I wanted to blog about it and join the diff I made out of mod_ntlm2 at https://modntlm.svn.sourceforge.net/svnroot/modntlm/trunk here: diff file

So if you bump into this problem, you will hopefully reach this page and save a few hours.

I am not really sure smoke testing is the best name to describe what I am trying to set up.

Wikipedia says:

In computer programming and software testing, smoke testing is a preliminary to further testing, which should reveal simple failures severe enough to reject a prospective software release. In this case, the smoke is metaphorical.

But I think it describes well the need I have : being able to test the output of several releases of Distutils, without relying on unit tests for that. Simply because they did not exist on early versions of Distutils.

Why do I need “smoke tests”

Lately, I was working on Distutils, removing some old bugs and adding some tests. Working on an under-tested package is like opening a can of worms : when you change something you can introduce some regressions. It’s hard to avoid for sure, because even if you add some tests subsequently for some features, you are not really doing proper TDD, where the tests and the code grow together. So there might be some uncovered cases left even if the coverage is improving slowly.

Regression also happens when you correct a behavior that was broken for years : if third party tools used a broken behavior without knowing it, fixing the behavior at some point becomes a regression from them.

But these pitfalls are unavoidable. Hopefully they don’t last for too long in a big project like Python. They are quickly reported, either by buildbots, either by someone from the community.

The last behavior problem I had was on the build_ext command. There’s a string option called “compiler”, were you can force the compiler type. You can put for instance ‘linux‘ and build_ext will use the UnixCompiler compiler class.

Although this option name is misleading. It should have been called “compiler_type” in the first place.  And the build_ext command adds another problem : it turns the compiler attribute of the class where the option value is stored, into a compiler instance. No need to say a class attribute value should not have several types.

The first application to suffer from this is Python itself. It uses build_ext.compiler, as a compiler instance to build its modules.

This means that “compiler” is now doomed to be both an option and a compiler attribute. It’s not easy to fix and will require a deprecation step.

So the idea of the smoke test is to make sure Distutils still knows how to produce releases of existing projects.

The Smoke testing buildbot

I have built a buildbot instance using collective.buildbot. The script the slaves run is quite simple: it gets some projects source and run their setup.py script, using various versions of the Python interpreter, then the trunk itself. The commands run so far are sdist and bdist.

Then it compares every file contained in the archive created to make sure produced archives are similar.

I will encouter some exceptions, when the package I am testing includes different files in its distribution depending on the Python version, but those exception will be managed.

Last, I cannot run this test on every package out there, for security reasons. Until I have the proper virtual environment to do that, I’ll work on a white list of packages I “trust”. So far, there’s just a package of mine, and NumPy. But this list will grow.

If you are curious to know if your package builds under Python trunk, get in touch with me, I’ll probably add it if I know I can trust it.

The last problem I have with this system yet is the fact that setuptools is not working anymore with the Distutils trunk, so I am unable to test setuptools-based packages.

The buildbot is located here : http://buildbot.ziade.org/waterfall

We have a simple need for a customer project that runs a Zeo server and a few Zeo clients : being able to check the status of every Zeo client, and monitor what they are doing.

DeadlockDebugger almost provides this feature since it is able to produce a dump of the execution stack for every thread a Zope instance is running.

Based on this tool, I have developed ZopeHealthWatcher, that provides a console script to query a Zope instance, and get back a status for every running thread. It tells you if the thread is idling or if it’s running some code. The script also returns an exit code depending on the number of busy threads, so it can be used in tools like Nagios.

When there are 4 or more busy threads, the script will return the execution stacks for every busy thread and some extra info like the system load and memory info. The returned info will be extendable through plug-ins in the next version, but right now the provided info are enough for our needs.

I have also created an HTML version, so when the dump is requested from another tool than the console script (e.g. a browser), it displays a nice human-readable interface (check the PyPI page for more info and a screenshot).

Notice that DeadLockDebugger is hackish since it patches the Zope publisher at startup. But we won’t change this part: we need this tool to run from the oldest to the newest Zope 2 version. And the patch just works fine, so…

The provided version should run out of the box in a buildout-based Plone 3 application, but requires manual installation steps on older Plone or CPS versions.

I didn’t mention these manual steps in the documentation. I think I am the only person in the world interested in running this tool on the dead-but-still-in-production-in-many-places Nuxeo CPS.

By the way: kudos goes to Marc-Aurèle Darche, who is maintaining CPS for years now, making it one of the most bug-free and stable CMS solution out there.  Ok it’s probably easier to reach this level of quality since the platform is very stable and only evolves very slowly thanks to Georges Racinet.

Since Pycon, a lot of discussion and work has been going on.

During the summit, we made a few decisions (see http://mail.python.org/pipermail/python-dev/2009-March/087834.html) but this topic is so wide and so complex that a lot of discussion still needs to be done to have a clear and complete picture of where we want to go and how we are going to do it

But we have made significant progress and reached consensus on some key points. This entry is a short list of these key points.

Being able to compare project versions

If you take a look in Distutils code, there’s a version and a versionpredicate module that provides a way to compare versions. This feature is barely used by Distutils itself, and a very few number of projects out there are using it (If you do so, please let me know).

This is probably because Distutils provides two different ways to compare versions: a “strict” one and a “loose” one. In other words Distutils clearly states that it is unable to provide a unique version comparison algorithm that can be used by anyone out there. Anyone means here : Python developers, OS packagers, etc.

Setuptools did a better job by providing an algorithm that covers most cases, but suffers from this universality: it’s too heuristic.

So one of the main topic during Pycon was to try to find a version comparison algorithm that would just work for everyone and in the meantime would be strict enough to be useful. That’s a pretty tough task, but I think we have reached something that is “good enough” for everybody. We had the chance to have people from Ubuntu, RedHat during Pycon to work on this task, and Trent Mick took the lead during the sprints to come up with a prototype.

It’s described here : http://wiki.python.org/moin/Distutils/VersionComparison and I have put the code here http://bitbucket.org/tarek/distutilsversion.

Two more things to take care of:

• Philip Eby came up with an edge-case : being able to do a development version of a post release.
• Jean-Paul Calderone proposed to add a constructor that would take explicit arguments to describe the version, rather than a string representation

There’s a branch (tarek-postdev) on bitbucket to work on these two cases. But basically, it seems that we have a consensus on a unique way to compare versions in Python ! This is a great step forward.

Another point that Toshio Kuratomi raised during a hall discussion was the fact that some Python developers are not having good practices when versioning their releases. So we agreed that a good “How to properly use version numbers for your projects releases” document will have to be delivered in Distutils documentation, besides the new algorithm. I am pretty confident that people will eventually follow it.

APIs to access installed project metadata, and an egg-info standard

Once a project is installed in Python, you don’t have a way to get its metadata and to answer to basic questions like:

• what are the installed packages ?
• what is the version ?
• how is the author ?

Of course you can dig in your installation and get back most of these, but depending how you installed your package (manual, easy_install, pip) it might be located in different places.

So we agreed on a standard for this, described in PEP 376 : http://www.python.org/dev/peps/pep-0376

An API will be introduced to be able to get the info for a installed package, as long as it was installed using PEP 376 standard.

An uninstall feature !

A feature that is claimed for a long time now should be introduced in Distutils : uninstallation !

Distutils will provide a basic, reference uninstall feature that will remove all files that where previously installed for a project. This will be doable as long as this list of files is recorded at installation time, and not used by another project.

See the details in PEP 376.

Standardize PKG-INFO

New fields will be added in the metadata of a project. The most important one is install_requires from setuptools, which let you list the projects your project depend on and their versions.

This is for informational purpose, and Distutils will not provide an install feature that will fetch and install dependencies. This can be done by third-party tools like easy_install  as long as they use the installation standard described earlier.

We started this work with Jim Fulton at Pycon, and Tres Seaver took the lead on this task since then. The plan is to update PEP 345 . The work is done in a branch here : http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?view=markup

But the remaining work is focusing on practical details. Just remember that PKG-INFO will evolve and install_requires will be integrated amongst other changes.

Distutils code cleanup, test coverage

The test coverage is starting to look good and most modules are covered around 80%. I guess this is the average in most Test-Driven Development projects out there. So Distutils is becoming a good citizen of the standard library 

The remaining work for a good test coverage is mostly on compilers side and os specific commands.

My black list of untested (or nearly untested) modules :

• bcppcompiler
• cygwinccompiler
• emxccompiler
• msvc9compiler
• msvccompiler
• command.bdist_msi
• command.bdist_rpm

I need to figure out how to properly test them. Last, I need to set up a buildbot that will try out Distutils trunk on a list of projects out there, like numpy for example.

Other topics

Check this ! http://wiki.python.org/moin/Distutils

I am very proud to be a Gsoc mentor this year on a very interesting topic : an universal keyring library for Python.

About the topic

In Distutils, if you want to interact with PyPI, you have to register in the website and you get a login/password so you can register and/or upload your packages.

Before Python 2.6, the only way you could interact with PyPI was by storing these info into your .pypirc file in clear text. This was not the best solution. For example we have in my company some staging servers we share, and from whom we upload packages to various PyPI-like servers. So we have to store PyPI login/password info in them. This means that if Bob wants to push his package from that server, he has to put his password into a clear text file which is most of the time readable by everyone. It’s not such a big deal in our company since we can trust each other, but it’s a very bad practice.

So I ended up changing this in Python 2.6 so people could type their password in a prompt when working with packages, using getpass.getpass. So they wouldn’t have to store them anymore.

But this is not enough : we need to provide something better. We need getpass.getpass to be able to interact with keyring libraries like KeyChain, Gnome Keyring, etc. So the login/password info are safely stored and can be reused.

This service will be useful for Distutils, but also for any Python application.

The idea of the GSOC task is to provide an unified keyring library for Python, and it’s harder that it sounds. For instance, we need to find a way to provide something that works under Windows. So the whole work is quite challenging and interesting, and the goal is to end up with a keyring library we can use into Distutils and propose for inclusion in getpass.

About Kang

Kang Zhang is the student that was selected for this work. Congrats ! He has started to work on it. You can follow his work in his blog. I have a strong feeling that he will succeed in this work and come up with something good.

Take a look at the Python Soc planet too, where all students involved in Python GSOC are blogging about their ongoing work.

I need a very simple plugin system for one of my projects. The project is a small WSGI application called mysysadmin that allows you to launch some commands on your system to manage some applications. It also allows you to view log files in your web browser.

It’s similar in some ways to WebMin,

So in my application, every tab is a plugin that manages one application. I have a plugin for Apache, another one for MySQL, and so one.

Back to my plugin system. So every plugin that is registered becomes a tab in the WSGI application, as long as it provides all the methods my web application needs to interact with it. So I want to check that each plugin strictly provides the API needed by the main program.

The first tools that came in mind were :

• zope.interface, to be able to provide that each plugin meets the requirements.
• setuptools entry points, so it’s easy for a third party code to implement a plugin.

But I find both projects a bit complex to implement such a simple plugin system.I could use the standalone Plugins package Phillip provides instead of setuptools, but it still does too much things imho. That’s someting I am currently learning by working on packaging matters : one library should not provide too many features.

Extensions : a simple plugin system

So I have started to implement a light-weight plugin system called extensions, which reuses setuptools entry points principles but is more simple to use. The goal of this project is to provide very simple APIs to handle plugins, and to make it work without introducing a new argument into the setup.py setup method, like setuptools does.

For instance, if you want to define an apache function in your modules module, in your myapp package, you just call the register function :

from extensions import register

register('mysysadmin.modules', 'apache', 'myapp.modules:apache')

That’s it !

And to use it, the mysysadmin application can use a simple API called get, that iterates over all plugins defined for “mysysadmin.modules” :

from extensions import get

for plugin in get(group='mysysadmin.modules'):
    # do something with the plugin

The magic is done by writing in the .egg-info directory installed for the package that contains each plugin, a PLUGIN file that contains the list of registered elements. It’s an idea borrowed from setuptoools entry points. So get iterates over all .egg-info directories in your path and load the PLUGIN files it finds.  Nothing new here. That’s how setuptools does, and that’s perfect.

If you have any feedback on extensions, let me know !

Strict plugins

The other need is to strictly check that every plugin provides the API needed, e.g. fulfill the requirements. This is what we could call Design by contract .

You can provide a base class for this, and ask the plugins to inherit from it. Or you can ask the Plugin to provide a marker to specify it implements a given behavior. zope.interface can do a nice job for the latter,and let you check that a given object implements an interface.

But I wanted to give a shot to the brand new Python ABCs and make sure anyone can write a plugin in plain Python, without having to rely on any kind of marker system. ABCs will let you check that a class implements some methods without requiring it to inherit from a specific class, to implement a specific interface or provide a custom marker. Pure duck typing !

So let’s define for our application a Plugin class, that gives the signature every plugin will need to provide. It uses ABCMeta as its meta class, and the abstractmethod for every method that should be implemented by every plugin.

Here’s an extract :

from abc import ABCMeta, abstractmethod

class Plugin(object):

    __metaclass__ = ABCMeta

    @abstractmethod
    def get_command_list(self):
        return []

    @abstractmethod
    def run_command(self, name):
        pass

    @classmethod
    def __subclasshook__(cls, klass):
        if cls is Plugin:
            for method in cls.__abstractmethods__:
                if any(method in base.__dict__ for base in klass.__mro__):
                    continue
                return NotImplemented
            return True
        return NotImplemented

The __subclasshook__ method is a class method that will be called everytime a class is tested using issubclass(klass, Plugin). In that case, it will check that every method marked with the abstractmethod decorator is provided by the class.

So basically, the application can discover and use the plugins, with:

from extensions import get

for plugin in get(group='mysysadmin.modules'):
    klass = plugin.load()
    if not issubclass(klass, Plugin):
        logging.info('sorry, not a suitable plugin')
        continue
    # do something with the plugin
    xxx

Abstract Base Classes are one honking great idea — let’s do more of those!

I received some complaints about the fact that some links in my books were dead by the time they were printed.

For the next book I am working on, I have proposed to my editor to set up a website to keep track of all references mentioned.

By using unique short ascii references throughout the book, it’s easy to provide a simple redirect service to the target URL, and to fix it when it changes (just by setting up a mail alert if your redirect reaches a 404).

For example, if I am referring to mod_wsgi in my book, I can write this reference: #mod_wsgi, and provide a redirection to http://code.google.com/p/modwsgi into my website, through a unique, mnemotechnic permanent URL : http://ziade.org/urls/mod_wsgi.

This small service, à la Tiny URL is not a burden for the reader imho : he is using his computer anyway when he visits an URL mentioned in a book.

It’s a simple idea I am sure a lot of people have thaught about before, but I fail to see it applied in the books I am buying these days. Is there any good reason I fail to see ?

I was thinking about doing this change for a while, and I took the time to do it last week-end:

My personal website at http://ziade.org is now powered by Pylons and Atomisator. It’s both in French and English (the urls are translated too).

It contains:

• a home page with my latest twitter entries
• a blog (this one)
• some other pages like a résumé, a list of the books I wrote, a projects page, etc.

I am still using WordPress to write my blog entries but they are now processed with Atomisator and displayed at http://ziade.org/blog. The feed is now located at http://ziade.org/blog/xml and I will make sure all Planets use this URL for now on.

Next I’ll probably :

• move to another blogging tool, and my readers will not suffer from the change as long as they use the ziade.org feed url.
• add some portlets into the new site, like the ones I have on WordPress.

How do you like this new look ? Any comment / advice ?

I am introducing the check command in Distutils. This command will check your package metadata, like the sdist and the register command already do (they display warnings).

But the new thing is that it will also allow you to check if long_description is reStructuredText compliant.

Its usage will be:

$ python setup.py check --restructuredtext
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)

And there’s also a strict mode, that raises an error in case something is wrong

$ python setup.py check --restructuredtext --strict
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)
error: Please correct your package.

Last, this command will be used by register, and sdist, so you can stop the process in case the metadata are not correct. This is useful to make sure your PyPI home page is not broken for example, since it parses long_description to build it. So a good practice will be to use strict when registering a package:

$ python setup.py register --strict
running register
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)
error: Please correct your package.

This feature will land in Python 2.7 (I am working on it and it should be commited this week end). Of course, it will not introduce a hard dependency on docutils in Python, neither it will change the current default behavior.

Until then, you can use collective.dist 0.2.3, that provides this feature for Python 2.4 to 2.6

We had an excellent brainstorming session today in the hall, with Toshio, Georg, Martin, Thomas, etc.. (sorry we were so many I don’t have the full list) with some insights from Guido and Brett. We tried to think about a way to handle multiple versions of a same package.

Here’s the two most important concepts :

• Unicity: There should be one and only one instance of a Python package at a given version on a system
• Combination: One Python application combines several packages to run

About unicity

A Python package is a component that can be installed on a system. If you use the standard Distutils approach, it will end up in the Python site-packages directory and be importable by the interpreter. This package comes with a version number and is unique.

This unicity is important for security and maintainability. For instance, if there’s a security hole in a package, the fix is applied in one place and the system maintainer knows it can’t be present elsewhere on the system.

This is the system administrator point of view

About Combination

What defines a Python application is the fact that is selects a list of packages it needs to run. And this varies for every application. So two Python applications might need a different version of a given package and that is normal.

What’s important is to have the right list of packages when an application loads. Tools like zc.buildout or virtualenv are perfect for this need : they create an isolated environment for your application to run with the right set of packages.

So the simplest way to release an application is to ship it with everything required, regardless the unicity.

This is the application developer point of view

The idea

zc.buildout and virtualenv are a blast for developers, and another thing system packagers might dislike. This is because they break the unicity by allowing developers to ship their applications as black boxes. Of course one may say that this is perfectly fine since what’s inside an application is not the problem of the system packager. But since this application is made of packages that may be shared on the system by other applications, that is redondant.

Forget site-packages for a moment. And let’s think about a new loading system for packages. This approach is similar in some ways to setuptools’ multiple version system.

Storing multiple versions

First of all, let’s store the packages in a directory, and for each version of the package, under a sub-directory which name is the version.

For example:

• SQLAlchemy
  • 0.4
    • package code is here
    • package egg-info here
  • 0.5
    • package code is here
    • package egg-info here
• jsonlib
  • 1.2.6
    • package code is here
    • package egg-info here
  • 1.3.10
    • package code is here
    • package egg-info here

Given this structure, some mechanism can provide to the interpreter the latest available version of a package, as long as Distutils knows how to handle version comparisons correctly (which will be the case in a near future)

Choosing specific versions

Back to our application. Let’s call it MyApp, version 1.0.

It needs specific versions for some packages. Forget zc.buildout and pip for a moment. And let’s think about a different way to express the packages (and versions) its needs.

Let’s make a Python package for this application and let’s make a few assumptions on some features in Distutils:

• setuptools’ install_requires has made it into Distutils, as part of metadata.
• metadata are defined statically in a package, apart from setup.py.

So basically, the application is mainly a static list of dependencies defined into install_requires. For example:

• SQLAlchemy > 0.4
• jsonlib == 1.2.6

When MyApp will get installed by Distutils, it will be added in the packages tree.

When it is used, it will need to load the versions of SQLAlchemy and jsonlib it needs. The ones that are inside its metadata.

To make it possible, the script that launches the application calls a built-in function called read_deps, that takes the metadata and reads them to know which versions fit:

# the script
read_deps('PKG-INFO')

import SQLAlchemy
import jsonlib
...

This call will load the right versions in the packages tree:

• SQLAlchemy
  • 0.4
  • 0.5
• jsonlib
  • 1.2.6
  • 1.3.10
• MyApp
  • 1.0

What’s next

I just dropped here the rough idea, and a lot of details are missing. But I think it’s a good thing to share this idea here in its early stage.

We have decided we would try to write a prototype using importlib and sys.meta_path. Maybe Georg will start it during the Pycon sprint, and start to digg into the details. He was working on this when I left the open session at Pycon.

Stay tuned

Before I sit down and clean up my summit notes to send them to python-dev, I wanted to post an entry about a small project which I think could be a great task for a student at the Summer of Code (I doubt it can fill 4 months of work but it could be done amongst other tasks).

Yesterday, we did a late session with Martin von Loewis, Jim Fulton and Georg Brandl about PyPI and the fact that it needed a better way to handle passwords on client side. That is, the fact that you have to store your password in the .pypirc file if you want to upload your package to PyPI.

This is unsafe and unwanted. A few months ago, I have made a small change in Distutils so it would prompt for the password using the getpass module if it doesn’t find it in the .pypirc file. (This was a contrbution of Nathan Van Gheem).

Anyways, this is not enough. Jim suggested to set up a SSH server on PyPI using Paramiko, so we could use a standard ssh connection and benefit from ssh-agent. But this is unfortunately not universal.

So let me get back to the idea I sent some time ago : http://mail.python.org/pipermail/python-ideas/2009-January/002465.html

What about having an option in getpass to store and reuse passwords in
system keyrings ?

    getpass(prompt[, stream])

would become:

    getpass(prompt[, stream, keyring])

where keyring would be a callable that can be use to retrieve the
password from a keyring system
and store it the first time.

The getpass module could provide some keyring support for:

- ssh-agent under Linux
- keychain under Mac OS X
- ...
ss

And let the developers use their own keyring system by providing a callable.

As Greg Smith said in the thread, the first task is to create a library that supports all standard keyring systems out there, including things like KWallet, Internet Explorer, Fireforx and so on…

I’ll mentor this project if any student would like to do it.

Tomorrow is the Language Summit, yeepee.

The package and distribution part of the Summit is going to be great since we have key people coming up.

zc.buildout and pip leaders (Jim Fulton and Ian Bicking) will be present, and many others. I’ll be representing Distutils, since I am its current maintainer. Unfortunately Philip Eby (setuptools) can’t make it, but he should be reachable via IRC (I am trying to set something up for tomorrow).

Anyway, one of the goal of the Summit is to validate the new features and enhancements we want to introduce in Distutils and PyPI. It’s important to make sure they play well with third-party tools like zc.buildout, setuptools and pip. We also need to make sure these tools will evolve in the same direction in the future.

We have reached a point in Python where we need to concentrate all the packaging effort to build a common standard in the standard library, because it is badly needed.

Here’s a draft of the slides I will present tomorrow, during the first 5/10 minutes, as a session leader:

• Packaging Survey results overview
• Topics to discuss
  • setting up an organized network of mirrors (see PEP 381)
  • discuss about other PyPI enhancements
  • improve the package installation / uninstallation  (see PEP 376)
  • discuss the package dependencies problem and see if we can come up with a PEP
  • discuss the multiple version problem and see if we can come up with a PEP
  • discuss the isolated environment vs the OS vendor approach and see what can be done to improve their coexistence.

Summit schedule:

• 13:20 -> 13:30 : presentation
• 13:30 -> 14:30 : discussions/work in small groups
• 14:30 -> 14:50 : “tour de table”
• 14:50 -> 15:10 : break
• 15:10 -> onwards: sprint !

I am not sure about the ‘work in small group’ part yet, because I don’t know how many people will show up, and what people will want to focus on.

Around 570 people answered the survey, which is a great number I didn’t expect. Thanks again to Massimo for his help on this.

I have a lot of work to read all the open question answers, and all the comments that goes with the “other” answer, but I wanted to publish the results of the closed questions before the summit.

I don’t want to comment the results yet. I will after I have studied all answers, so it’ll be a little while

Who are you ?

Where are you located ?

If you are a web programmer, what is the framework you use the most ?

How do you organize your application code most of the time ?

For libraries you don’t distribute publicly, do you you create a setup.py script ?

What is the main tool or combination of tools you are using to package and distribute your Python application ?

How do you install a package that does not provide a standalone installer (but provides a standard setup.py script) most of the time ?

How do you remove a package ?

How do you manage using more than one version of a library on a system ?

Do you work with setuptools’ namespace packages ?

Has PyPI become mandatory in your everyday work (if you use zc.buildout for example) ?

If you previously answered Yes, did you set up an alternative solution (mirror, cache..) in case PyPI is down ?

Do you register your packages on PyPI ?

Do you upload your package on PyPI ?

If you previously answered No, how do you distribute your packages ?

We have been working hard in the past few months with my Plone friends to make it happen.

It’s now available : Plone.org is acting like PyPI in its Products section, which means that you can use Distutils to register and upload your packages in both places now.

I have written a small tutorial on plone.org to explain how to use it: http://plone.org/documentation/tutorial/how-to-upload-your-package-to-plone.org

I will also explain during my Pycon presentation (http://us.pycon.org/2009/conference/schedule/event/44) how this new feature at Plone.org will help people that are working with Plone and Python packages.

The Python Langage Summit is coming up. To prepare this event, I have put online a survey you can take to tell us a bit more about you and how you package your Python applications.

• Who should take the survey : any Python developer that packages and distributes his code, no matter how.
• Take the survey: http://tinyurl.com/packaging-survey

Thanks to all the people that helped building the survey, and a special thanks to Massimo Di Pierro who created the application that runs the Survey and helped me set up the survey. It runs under web2py.