I am taking a break for a week starting tomorrow morning. My wife made it clear : my laptop and internet are totally forbidden during 7 days. So I thaught it was a good opportunity to make a status of all the projects that are going on in the packaging world.

Distribute

The setuptools fork is doing good. We’ve discussed the roadmap strategy with some people at Distutils-SIG and came up with that plan:

0.6 will be released the first week of August. It will support Python 2.x only and will provide a bootstrap to replace an existing version of setuptools, as described in my previous entry.

0.7 will support Python 2.x and 3.x and will refactor the code in three distributions. It will also rename all parts under new names, so they will not compete anymore with setuptools. Its development will start right after 0.6 is out. If you have ideas or feature requests, please add them in our issue tracker at bitbucket.

Distutils

They are some patches waiting for my attention, I need to work on when I am back. But globally, if you open Distutils code, you will notice that it has been PEP8-fied (besides a few places I need to finishe) and test covered.

Some small refactorings have been done, like removing duplicate code and using the tar module of the stdlib instead of the tar command on the system, thus allowing building tar archive under win32 without having to install the tar program for example. Another nice change was done on the upload command: it uses urllib2 now, meaning that you can use an http proxy with a environment variable transparently. Other changes are on their way, like fixing the mess with “compiler” option, and cleaning up some remaining duplicate behaviors.

But nothing is deeply changing yet on how Distutils works. It waits for the PEPs we are working on to be accepted…

PEP 376

After I thaught this PEP was ready to be accepted, I sent it from Distutils-SIG to Python-dev. Then we had very long threads on Python-dev about it, showing that it was not ready at all…

Man, I don’t think I’ve picked the easiest topic

Anyways, people gave a lot of thaught on the topic, and Paul Moore helped building a PEP 302-compatible version of the prototype, you can see at bickbucket. From there, we still need to fix the problem of the absolute/relative paths in the RECORD file, and try to have real world uses cases, with various package managers applications. But we will eventually have an acceptable PEP, I hope, within a few months.

This pep will add query APIs in pkgutils in the stdlib, so you know what’s installed in your Python. It will also provide tools to uninstall installed ditributions.

It’s a hot topic because we are dealing with the fuzzy boundary between Python the extendable language, where you can install distributions using distutils-based tools, and Python the interpreter, that gets extended with packages managed by an OS-based package manager like APT under Debian for example. So for the latter, all our tool are getting too much in the way and should not make it hard for system packagers to extract the metadata they need to re-package distutils-based distributions.

PEP 386

Man, this PEP would make our life easier. It’s all the work we started during Pycon, to find an acceptable way to compare versions. By acceptable I mean that could be used in our community and workable by OS packagers. I said I would drop it, because there were too much controversy, but quite a few people would like to see it accepted.

I doubt Guido will accept a PEP that would enforce a version scheme though.

I had created this PEP, so we could move forward on PEP 345, where we want to introduce “install_requires”, the field used in setuptools to list dependencies.

The prototype for PEP 386 is in bitbucket too : verlib

Whatever happens, I’ll probably publish it. Its useful to build a package system, and it has a function to translate most distutils/setuptools version schemes.

Now about my addiction

I know that sounds stupid, but I don’t know what is going to happen next week. It’s been years since I’ve not been fully offline for a week. I already have right now so many threads to catch up on various mailing lists that I am scared of how it’s going to be in 7 days…

But it’s probably a good thing for me to do this

I am off, see you in 7 days !

Again, like a year ago, people had enough of the fact that the setuptools project is not maintained since 9 months.

Phillip Eby explained that he doesn’t have time to do it unless someone would pay him for that. But in the meantime, he doesn’t bless anyone to do it. Well, he has blessed some people to do it (Ian Bicking and Jim Fulton), but unfortunately these people are not willing to do it because they have a lot of other projects going on. Other people that could maintain it, including me, fail in his “unqualified people” category

So we are all locked in a strange situation where tons of patches are ready to be commited in the setuptools tracker but are not making it. Several non-public forks have started to appear around of course.

So again, I decided with some other people to create a fork called “Distribute”. It’s a real fork located here : http://bitbucket.org/tarek/distribute.

By real I mean that this fork was not created with the purpose of forcing Phillip to do a release like we did last year for the 0.6c9 release, but with the intention to free us from that strange situation where we all depend on his wills and (lack of) time.

The plan is to release a first version next week, that corresponds to the setuptools 0.6 branch, with some patches applied.

Next, we are planning to start a 0.7 version where the code will be splitted in several distributions:

• a distribution for pkg_resources
• a distribution for the setuptools package itself
• a distribution for easy_install

A little bit of bikeshedding is going on to pick a name for that fork, and we ended up running a poll. (vote!)

Now, right after I have announced this plan on Distutils-SIG, Phillip reacted by annoucing a similar plan, e.g. splitting the setuptools project in several distributions. But since he previously said that he didn’t have the time to do it, I doubt that it’ll work out unless he’s opening its development to a wider range of developers and maintainers.

That’s the strange world of packaging…

As you might know, we are working hard on Distutils side for Python 2.7 and 3.2 upcoming releases. The biggest work is on PEP 376, that will introduce among other things a uninstaller function and functions to query installed distributions.

The other “big” work is on PEP 345. We want to introduce a new metadata field called “install_requires” to be able to express requirements. That’s from the setuptools project and is quite used by the community. Notice that there were several attempts to define requirements in the past in Distutils, but none of them really made it through.

For instance, if you want to define docutils as a dependency for your distribution but a version less or equal to 0.4, you can say :

docutils <= 0.4

But as long as you want to work with such dependencies and provide a way to express them with operators, you have to be able to compare versions. For instance if you want to compare an installed docutils distribution to see of it is compatible with 0.4.

That’s another big topic we have been working on for the last few months, with people from various communities (Fedora, Ubuntu, etc). And I have started to write down PEP 386.

But comparing version appears to be a topic that cannot be generic. It seems that Distutils, therefore Python, shouldn’t enforce any rule on this.

Furthermore, since we have said that Distutils should be a lighter package, it will not implement a complete package managment system, like setuptools or zc.buildout does.

So I’ve decided to propose to drop PEP 386, and stick on a very simple rule in PEP 345, saying that requirements can be defined, with :

distribution_name OPERATOR version

where OPERATOR is in >, <, ==, !=, >= or <=.

Last, so the work done at Pycon and in Distutils-SIG is not lost, I will publish the library we wrote. This could be a very good basis for packaging managment systems out there.

If you want to work (or try to find some bugs in it to help out) with the latest bleeding-edge Distutils trunk version that will be shipped with Python 2.7 and Python 3.2, you can do it now !

I am creating a nighlty build every day here now : http://nightly.ziade.org/

This distribution can be installed in your existing Python 2.6 installation, and will probably work with 2.5.

I’m not planning to publish standalone versions yet, except nighlty build, at least until 2.7 and 3.2 are starting to be released as alphas.

We had to install mod_ntlm under a FreeBSD 6.4 server with a colleague, but the port for this package seems broken at least for FreeBSD 6.2 and 6.4. It just doesn’t work when it initiates an NTLM session through SMB, and it seems to be compiled without the log support so the last log you get doesn’t give any useful hint on what’s wrong.

So we had to recompile it to activate the log and try understand what the problem was. But Apache doesn’t not support threaded mode under FreeBSD 6.x (or I couldn’t manage to make it work howsoever), so mod_ntlm failed to compile since it uses Apache’s thread mutexes to perform some locks on some operations.

We’ve deactivated them and recompiled the module, and now it works now like a charm. It’s weird because it means that the binary package for this module is completely broken. I couldn’t find any place mentioning this.

Anyways, I know this is an old software combination, but since Google doesn’t give any hint on this problem, I wanted to blog about it and join the diff I made out of mod_ntlm2 at https://modntlm.svn.sourceforge.net/svnroot/modntlm/trunk here: diff file

So if you bump into this problem, you will hopefully reach this page and save a few hours.

I am not really sure smoke testing is the best name to describe what I am trying to set up.

Wikipedia says:

In computer programming and software testing, smoke testing is a preliminary to further testing, which should reveal simple failures severe enough to reject a prospective software release. In this case, the smoke is metaphorical.

But I think it describes well the need I have : being able to test the output of several releases of Distutils, without relying on unit tests for that. Simply because they did not exist on early versions of Distutils.

Why do I need “smoke tests”

Lately, I was working on Distutils, removing some old bugs and adding some tests. Working on an under-tested package is like opening a can of worms : when you change something you can introduce some regressions. It’s hard to avoid for sure, because even if you add some tests subsequently for some features, you are not really doing proper TDD, where the tests and the code grow together. So there might be some uncovered cases left even if the coverage is improving slowly.

Regression also happens when you correct a behavior that was broken for years : if third party tools used a broken behavior without knowing it, fixing the behavior at some point becomes a regression from them.

But these pitfalls are unavoidable. Hopefully they don’t last for too long in a big project like Python. They are quickly reported, either by buildbots, either by someone from the community.

The last behavior problem I had was on the build_ext command. There’s a string option called “compiler”, were you can force the compiler type. You can put for instance ‘linux‘ and build_ext will use the UnixCompiler compiler class.

Although this option name is misleading. It should have been called “compiler_type” in the first place.  And the build_ext command adds another problem : it turns the compiler attribute of the class where the option value is stored, into a compiler instance. No need to say a class attribute value should not have several types.

The first application to suffer from this is Python itself. It uses build_ext.compiler, as a compiler instance to build its modules.

This means that “compiler” is now doomed to be both an option and a compiler attribute. It’s not easy to fix and will require a deprecation step.

So the idea of the smoke test is to make sure Distutils still knows how to produce releases of existing projects.

The Smoke testing buildbot

I have built a buildbot instance using collective.buildbot. The script the slaves run is quite simple: it gets some projects source and run their setup.py script, using various versions of the Python interpreter, then the trunk itself. The commands run so far are sdist and bdist.

Then it compares every file contained in the archive created to make sure produced archives are similar.

I will encouter some exceptions, when the package I am testing includes different files in its distribution depending on the Python version, but those exception will be managed.

Last, I cannot run this test on every package out there, for security reasons. Until I have the proper virtual environment to do that, I’ll work on a white list of packages I “trust”. So far, there’s just a package of mine, and NumPy. But this list will grow.

If you are curious to know if your package builds under Python trunk, get in touch with me, I’ll probably add it if I know I can trust it.

The last problem I have with this system yet is the fact that setuptools is not working anymore with the Distutils trunk, so I am unable to test setuptools-based packages.

The buildbot is located here : http://buildbot.ziade.org/waterfall

Since Pycon, a lot of discussion and work has been going on.

During the summit, we made a few decisions (see http://mail.python.org/pipermail/python-dev/2009-March/087834.html) but this topic is so wide and so complex that a lot of discussion still needs to be done to have a clear and complete picture of where we want to go and how we are going to do it

But we have made significant progress and reached consensus on some key points. This entry is a short list of these key points.

Being able to compare project versions

If you take a look in Distutils code, there’s a version and a versionpredicate module that provides a way to compare versions. This feature is barely used by Distutils itself, and a very few number of projects out there are using it (If you do so, please let me know).

This is probably because Distutils provides two different ways to compare versions: a “strict” one and a “loose” one. In other words Distutils clearly states that it is unable to provide a unique version comparison algorithm that can be used by anyone out there. Anyone means here : Python developers, OS packagers, etc.

Setuptools did a better job by providing an algorithm that covers most cases, but suffers from this universality: it’s too heuristic.

So one of the main topic during Pycon was to try to find a version comparison algorithm that would just work for everyone and in the meantime would be strict enough to be useful. That’s a pretty tough task, but I think we have reached something that is “good enough” for everybody. We had the chance to have people from Ubuntu, RedHat during Pycon to work on this task, and Trent Mick took the lead during the sprints to come up with a prototype.

It’s described here : http://wiki.python.org/moin/Distutils/VersionComparison and I have put the code here http://bitbucket.org/tarek/distutilsversion.

Two more things to take care of:

• Philip Eby came up with an edge-case : being able to do a development version of a post release.
• Jean-Paul Calderone proposed to add a constructor that would take explicit arguments to describe the version, rather than a string representation

There’s a branch (tarek-postdev) on bitbucket to work on these two cases. But basically, it seems that we have a consensus on a unique way to compare versions in Python ! This is a great step forward.

Another point that Toshio Kuratomi raised during a hall discussion was the fact that some Python developers are not having good practices when versioning their releases. So we agreed that a good “How to properly use version numbers for your projects releases” document will have to be delivered in Distutils documentation, besides the new algorithm. I am pretty confident that people will eventually follow it.

APIs to access installed project metadata, and an egg-info standard

Once a project is installed in Python, you don’t have a way to get its metadata and to answer to basic questions like:

• what are the installed packages ?
• what is the version ?
• how is the author ?

Of course you can dig in your installation and get back most of these, but depending how you installed your package (manual, easy_install, pip) it might be located in different places.

So we agreed on a standard for this, described in PEP 376 : http://www.python.org/dev/peps/pep-0376

An API will be introduced to be able to get the info for a installed package, as long as it was installed using PEP 376 standard.

An uninstall feature !

A feature that is claimed for a long time now should be introduced in Distutils : uninstallation !

Distutils will provide a basic, reference uninstall feature that will remove all files that where previously installed for a project. This will be doable as long as this list of files is recorded at installation time, and not used by another project.

See the details in PEP 376.

Standardize PKG-INFO

New fields will be added in the metadata of a project. The most important one is install_requires from setuptools, which let you list the projects your project depend on and their versions.

This is for informational purpose, and Distutils will not provide an install feature that will fetch and install dependencies. This can be done by third-party tools like easy_install  as long as they use the installation standard described earlier.

We started this work with Jim Fulton at Pycon, and Tres Seaver took the lead on this task since then. The plan is to update PEP 345 . The work is done in a branch here : http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?view=markup

But the remaining work is focusing on practical details. Just remember that PKG-INFO will evolve and install_requires will be integrated amongst other changes.

Distutils code cleanup, test coverage

The test coverage is starting to look good and most modules are covered around 80%. I guess this is the average in most Test-Driven Development projects out there. So Distutils is becoming a good citizen of the standard library 

The remaining work for a good test coverage is mostly on compilers side and os specific commands.

My black list of untested (or nearly untested) modules :

• bcppcompiler
• cygwinccompiler
• emxccompiler
• msvc9compiler
• msvccompiler
• command.bdist_msi
• command.bdist_rpm

I need to figure out how to properly test them. Last, I need to set up a buildbot that will try out Distutils trunk on a list of projects out there, like numpy for example.

Other topics

Check this ! http://wiki.python.org/moin/Distutils

I am very proud to be a Gsoc mentor this year on a very interesting topic : an universal keyring library for Python.

About the topic

In Distutils, if you want to interact with PyPI, you have to register in the website and you get a login/password so you can register and/or upload your packages.

Before Python 2.6, the only way you could interact with PyPI was by storing these info into your .pypirc file in clear text. This was not the best solution. For example we have in my company some staging servers we share, and from whom we upload packages to various PyPI-like servers. So we have to store PyPI login/password info in them. This means that if Bob wants to push his package from that server, he has to put his password into a clear text file which is most of the time readable by everyone. It’s not such a big deal in our company since we can trust each other, but it’s a very bad practice.

So I ended up changing this in Python 2.6 so people could type their password in a prompt when working with packages, using getpass.getpass. So they wouldn’t have to store them anymore.

But this is not enough : we need to provide something better. We need getpass.getpass to be able to interact with keyring libraries like KeyChain, Gnome Keyring, etc. So the login/password info are safely stored and can be reused.

This service will be useful for Distutils, but also for any Python application.

The idea of the GSOC task is to provide an unified keyring library for Python, and it’s harder that it sounds. For instance, we need to find a way to provide something that works under Windows. So the whole work is quite challenging and interesting, and the goal is to end up with a keyring library we can use into Distutils and propose for inclusion in getpass.

About Kang

Kang Zhang is the student that was selected for this work. Congrats ! He has started to work on it. You can follow his work in his blog. I have a strong feeling that he will succeed in this work and come up with something good.

Take a look at the Python Soc planet too, where all students involved in Python GSOC are blogging about their ongoing work.

I need a very simple plugin system for one of my projects. The project is a small WSGI application called mysysadmin that allows you to launch some commands on your system to manage some applications. It also allows you to view log files in your web browser.

It’s similar in some ways to WebMin,

So in my application, every tab is a plugin that manages one application. I have a plugin for Apache, another one for MySQL, and so one.

Back to my plugin system. So every plugin that is registered becomes a tab in the WSGI application, as long as it provides all the methods my web application needs to interact with it. So I want to check that each plugin strictly provides the API needed by the main program.

The first tools that came in mind were :

• zope.interface, to be able to provide that each plugin meets the requirements.
• setuptools entry points, so it’s easy for a third party code to implement a plugin.

But I find both projects a bit complex to implement such a simple plugin system.I could use the standalone Plugins package Phillip provides instead of setuptools, but it still does too much things imho. That’s someting I am currently learning by working on packaging matters : one library should not provide too many features.

Extensions : a simple plugin system

So I have started to implement a light-weight plugin system called extensions, which reuses setuptools entry points principles but is more simple to use. The goal of this project is to provide very simple APIs to handle plugins, and to make it work without introducing a new argument into the setup.py setup method, like setuptools does.

For instance, if you want to define an apache function in your modules module, in your myapp package, you just call the register function :

from extensions import register

register('mysysadmin.modules', 'apache', 'myapp.modules:apache')

That’s it !

And to use it, the mysysadmin application can use a simple API called get, that iterates over all plugins defined for “mysysadmin.modules” :

from extensions import get

for plugin in get(group='mysysadmin.modules'):
    # do something with the plugin

The magic is done by writing in the .egg-info directory installed for the package that contains each plugin, a PLUGIN file that contains the list of registered elements. It’s an idea borrowed from setuptoools entry points. So get iterates over all .egg-info directories in your path and load the PLUGIN files it finds.  Nothing new here. That’s how setuptools does, and that’s perfect.

If you have any feedback on extensions, let me know !

Strict plugins

The other need is to strictly check that every plugin provides the API needed, e.g. fulfill the requirements. This is what we could call Design by contract .

You can provide a base class for this, and ask the plugins to inherit from it. Or you can ask the Plugin to provide a marker to specify it implements a given behavior. zope.interface can do a nice job for the latter,and let you check that a given object implements an interface.

But I wanted to give a shot to the brand new Python ABCs and make sure anyone can write a plugin in plain Python, without having to rely on any kind of marker system. ABCs will let you check that a class implements some methods without requiring it to inherit from a specific class, to implement a specific interface or provide a custom marker. Pure duck typing !

So let’s define for our application a Plugin class, that gives the signature every plugin will need to provide. It uses ABCMeta as its meta class, and the abstractmethod for every method that should be implemented by every plugin.

Here’s an extract :

from abc import ABCMeta, abstractmethod

class Plugin(object):

    __metaclass__ = ABCMeta

    @abstractmethod
    def get_command_list(self):
        return []

    @abstractmethod
    def run_command(self, name):
        pass

    @classmethod
    def __subclasshook__(cls, klass):
        if cls is Plugin:
            for method in cls.__abstractmethods__:
                if any(method in base.__dict__ for base in klass.__mro__):
                    continue
                return NotImplemented
            return True
        return NotImplemented

The __subclasshook__ method is a class method that will be called everytime a class is tested using issubclass(klass, Plugin). In that case, it will check that every method marked with the abstractmethod decorator is provided by the class.

So basically, the application can discover and use the plugins, with:

from extensions import get

for plugin in get(group='mysysadmin.modules'):
    klass = plugin.load()
    if not issubclass(klass, Plugin):
        logging.info('sorry, not a suitable plugin')
        continue
    # do something with the plugin
    xxx

Abstract Base Classes are one honking great idea — let’s do more of those!

I was thinking about doing this change for a while, and I took the time to do it last week-end:

My personal website at http://ziade.org is now powered by Pylons and Atomisator. It’s both in French and English (the urls are translated too).

It contains:

• a home page with my latest twitter entries
• a blog (this one)
• some other pages like a résumé, a list of the books I wrote, a projects page, etc.

I am still using WordPress to write my blog entries but they are now processed with Atomisator and displayed at http://ziade.org/blog. The feed is now located at http://ziade.org/blog/xml and I will make sure all Planets use this URL for now on.

Next I’ll probably :

• move to another blogging tool, and my readers will not suffer from the change as long as they use the ziade.org feed url.
• add some portlets into the new site, like the ones I have on WordPress.

How do you like this new look ? Any comment / advice ?

I am introducing the check command in Distutils. This command will check your package metadata, like the sdist and the register command already do (they display warnings).

But the new thing is that it will also allow you to check if long_description is reStructuredText compliant.

Its usage will be:

$ python setup.py check --restructuredtext
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)

And there’s also a strict mode, that raises an error in case something is wrong

$ python setup.py check --restructuredtext --strict
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)
error: Please correct your package.

Last, this command will be used by register, and sdist, so you can stop the process in case the metadata are not correct. This is useful to make sure your PyPI home page is not broken for example, since it parses long_description to build it. So a good practice will be to use strict when registering a package:

$ python setup.py register --strict
running register
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)
error: Please correct your package.

This feature will land in Python 2.7 (I am working on it and it should be commited this week end). Of course, it will not introduce a hard dependency on docutils in Python, neither it will change the current default behavior.

Until then, you can use collective.dist 0.2.3, that provides this feature for Python 2.4 to 2.6

We had an excellent brainstorming session today in the hall, with Toshio, Georg, Martin, Thomas, etc.. (sorry we were so many I don’t have the full list) with some insights from Guido and Brett. We tried to think about a way to handle multiple versions of a same package.

Here’s the two most important concepts :

• Unicity: There should be one and only one instance of a Python package at a given version on a system
• Combination: One Python application combines several packages to run

About unicity

A Python package is a component that can be installed on a system. If you use the standard Distutils approach, it will end up in the Python site-packages directory and be importable by the interpreter. This package comes with a version number and is unique.

This unicity is important for security and maintainability. For instance, if there’s a security hole in a package, the fix is applied in one place and the system maintainer knows it can’t be present elsewhere on the system.

This is the system administrator point of view

About Combination

What defines a Python application is the fact that is selects a list of packages it needs to run. And this varies for every application. So two Python applications might need a different version of a given package and that is normal.

What’s important is to have the right list of packages when an application loads. Tools like zc.buildout or virtualenv are perfect for this need : they create an isolated environment for your application to run with the right set of packages.

So the simplest way to release an application is to ship it with everything required, regardless the unicity.

This is the application developer point of view

The idea

zc.buildout and virtualenv are a blast for developers, and another thing system packagers might dislike. This is because they break the unicity by allowing developers to ship their applications as black boxes. Of course one may say that this is perfectly fine since what’s inside an application is not the problem of the system packager. But since this application is made of packages that may be shared on the system by other applications, that is redondant.

Forget site-packages for a moment. And let’s think about a new loading system for packages. This approach is similar in some ways to setuptools’ multiple version system.

Storing multiple versions

First of all, let’s store the packages in a directory, and for each version of the package, under a sub-directory which name is the version.

For example:

• SQLAlchemy
  • 0.4
    • package code is here
    • package egg-info here
  • 0.5
    • package code is here
    • package egg-info here
• jsonlib
  • 1.2.6
    • package code is here
    • package egg-info here
  • 1.3.10
    • package code is here
    • package egg-info here

Given this structure, some mechanism can provide to the interpreter the latest available version of a package, as long as Distutils knows how to handle version comparisons correctly (which will be the case in a near future)

Choosing specific versions

Back to our application. Let’s call it MyApp, version 1.0.

It needs specific versions for some packages. Forget zc.buildout and pip for a moment. And let’s think about a different way to express the packages (and versions) its needs.

Let’s make a Python package for this application and let’s make a few assumptions on some features in Distutils:

• setuptools’ install_requires has made it into Distutils, as part of metadata.
• metadata are defined statically in a package, apart from setup.py.

So basically, the application is mainly a static list of dependencies defined into install_requires. For example:

• SQLAlchemy > 0.4
• jsonlib == 1.2.6

When MyApp will get installed by Distutils, it will be added in the packages tree.

When it is used, it will need to load the versions of SQLAlchemy and jsonlib it needs. The ones that are inside its metadata.

To make it possible, the script that launches the application calls a built-in function called read_deps, that takes the metadata and reads them to know which versions fit:

# the script
read_deps('PKG-INFO')

import SQLAlchemy
import jsonlib
...

This call will load the right versions in the packages tree:

• SQLAlchemy
  • 0.4
  • 0.5
• jsonlib
  • 1.2.6
  • 1.3.10
• MyApp
  • 1.0

What’s next

I just dropped here the rough idea, and a lot of details are missing. But I think it’s a good thing to share this idea here in its early stage.

We have decided we would try to write a prototype using importlib and sys.meta_path. Maybe Georg will start it during the Pycon sprint, and start to digg into the details. He was working on this when I left the open session at Pycon.

Stay tuned

Before I sit down and clean up my summit notes to send them to python-dev, I wanted to post an entry about a small project which I think could be a great task for a student at the Summer of Code (I doubt it can fill 4 months of work but it could be done amongst other tasks).

Yesterday, we did a late session with Martin von Loewis, Jim Fulton and Georg Brandl about PyPI and the fact that it needed a better way to handle passwords on client side. That is, the fact that you have to store your password in the .pypirc file if you want to upload your package to PyPI.

This is unsafe and unwanted. A few months ago, I have made a small change in Distutils so it would prompt for the password using the getpass module if it doesn’t find it in the .pypirc file. (This was a contrbution of Nathan Van Gheem).

Anyways, this is not enough. Jim suggested to set up a SSH server on PyPI using Paramiko, so we could use a standard ssh connection and benefit from ssh-agent. But this is unfortunately not universal.

So let me get back to the idea I sent some time ago : http://mail.python.org/pipermail/python-ideas/2009-January/002465.html

What about having an option in getpass to store and reuse passwords in
system keyrings ?

    getpass(prompt[, stream])

would become:

    getpass(prompt[, stream, keyring])

where keyring would be a callable that can be use to retrieve the
password from a keyring system
and store it the first time.

The getpass module could provide some keyring support for:

- ssh-agent under Linux
- keychain under Mac OS X
- ...
ss

And let the developers use their own keyring system by providing a callable.

As Greg Smith said in the thread, the first task is to create a library that supports all standard keyring systems out there, including things like KWallet, Internet Explorer, Fireforx and so on…

I’ll mentor this project if any student would like to do it.

Tomorrow is the Language Summit, yeepee.

The package and distribution part of the Summit is going to be great since we have key people coming up.

zc.buildout and pip leaders (Jim Fulton and Ian Bicking) will be present, and many others. I’ll be representing Distutils, since I am its current maintainer. Unfortunately Philip Eby (setuptools) can’t make it, but he should be reachable via IRC (I am trying to set something up for tomorrow).

Anyway, one of the goal of the Summit is to validate the new features and enhancements we want to introduce in Distutils and PyPI. It’s important to make sure they play well with third-party tools like zc.buildout, setuptools and pip. We also need to make sure these tools will evolve in the same direction in the future.

We have reached a point in Python where we need to concentrate all the packaging effort to build a common standard in the standard library, because it is badly needed.

Here’s a draft of the slides I will present tomorrow, during the first 5/10 minutes, as a session leader:

• Packaging Survey results overview
• Topics to discuss
  • setting up an organized network of mirrors (see PEP 381)
  • discuss about other PyPI enhancements
  • improve the package installation / uninstallation  (see PEP 376)
  • discuss the package dependencies problem and see if we can come up with a PEP
  • discuss the multiple version problem and see if we can come up with a PEP
  • discuss the isolated environment vs the OS vendor approach and see what can be done to improve their coexistence.

Summit schedule:

• 13:20 -> 13:30 : presentation
• 13:30 -> 14:30 : discussions/work in small groups
• 14:30 -> 14:50 : “tour de table”
• 14:50 -> 15:10 : break
• 15:10 -> onwards: sprint !

I am not sure about the ‘work in small group’ part yet, because I don’t know how many people will show up, and what people will want to focus on.

Around 570 people answered the survey, which is a great number I didn’t expect. Thanks again to Massimo for his help on this.

I have a lot of work to read all the open question answers, and all the comments that goes with the “other” answer, but I wanted to publish the results of the closed questions before the summit.

I don’t want to comment the results yet. I will after I have studied all answers, so it’ll be a little while

Who are you ?

Where are you located ?

If you are a web programmer, what is the framework you use the most ?

How do you organize your application code most of the time ?

For libraries you don’t distribute publicly, do you you create a setup.py script ?

What is the main tool or combination of tools you are using to package and distribute your Python application ?

How do you install a package that does not provide a standalone installer (but provides a standard setup.py script) most of the time ?

How do you remove a package ?

How do you manage using more than one version of a library on a system ?

Do you work with setuptools’ namespace packages ?

Has PyPI become mandatory in your everyday work (if you use zc.buildout for example) ?

If you previously answered Yes, did you set up an alternative solution (mirror, cache..) in case PyPI is down ?

Do you register your packages on PyPI ?

Do you upload your package on PyPI ?

If you previously answered No, how do you distribute your packages ?

We have been working hard in the past few months with my Plone friends to make it happen.

It’s now available : Plone.org is acting like PyPI in its Products section, which means that you can use Distutils to register and upload your packages in both places now.

I have written a small tutorial on plone.org to explain how to use it: http://plone.org/documentation/tutorial/how-to-upload-your-package-to-plone.org

I will also explain during my Pycon presentation (http://us.pycon.org/2009/conference/schedule/event/44) how this new feature at Plone.org will help people that are working with Plone and Python packages.

The Python Langage Summit is coming up. To prepare this event, I have put online a survey you can take to tell us a bit more about you and how you package your Python applications.

• Who should take the survey : any Python developer that packages and distributes his code, no matter how.
• Take the survey: http://tinyurl.com/packaging-survey

Thanks to all the people that helped building the survey, and a special thanks to Massimo Di Pierro who created the application that runs the Survey and helped me set up the survey. It runs under web2py.

After the next commit I will make in Distutils (that adds tests for bdist_rpm), the test coverage of this Python standard library  package will be at 41%. This means that I have doubled the test coverage over the past few months, from 18% to 41%.

My goal is to double it again, and reach 80% in the next 6 months.

This also means I am just half an idiot now ! (since people who don’t have 100% code coverage are idiots ).

So does it make Distutils more robust ?

It would have probably made the latest Python 3 release looks better for this package, since we had a uncovered cmp() call left in Distutils by the time the release was made. In the meantime, as I said before, the “real” Distutils regression test suite is held by all the packages out there in the community, that are built and installed everyday.

Python trunk Distutils test coverage : 41%

Name               Stmts   Exec  Cover
--------------------------------------
__init__               3      0     0%
archive_util          77     61    79%
bcppcompiler         185      0     0%
ccompiler            453    211    46%
cmd                  180    134    74%
config                73     59    80%
core                  93     50    53%
cygwinccompiler      161      0     0%
debug                  3      3   100%
dep_util              43     11    25%
dir_util             109     76    69%
dist                 581    386    66%
emxccompiler         118      0     0%
errors                49      0     0%
extension             97     28    28%
fancy_getopt         233    126    54%
file_util            124     77    62%
filelist             161    102    63%
log                   46     21    45%
msvc9compiler        408      0     0%
msvccompiler         370      0     0%
spawn                 93     28    30%
sysconfig            323     51    15%
text_file            112     61    54%
unixccompiler        160     64    40%
util                 255    157    61%
version               68     62    91%
versionpredicate      61     51    83%
__init__               3      3   100%
bdist                 61     35    57%
bdist_dumb            57     47    82%
bdist_msi            322      0     0%
bdist_rpm            252    198    78%
bdist_wininst        170      0     0%
build                 60     54    90%
build_clib            90      0     0%
build_ext            334    160    47%
build_py             213    178    83%
build_scripts         78     65    83%
clean                 35      0     0%
config               185      0     0%
install              251    156    62%
install_data          44      0     0%
install_egg_info      40     32    80%
install_headers       25      0     0%
install_lib           97     50    51%
install_scripts       33     29    87%
register             173     82    47%
sdist                228    180    78%
upload               112     38    33%
--------------------------------------
TOTAL               7502   3126    41%

Python 2.5.4 Distutils test coverage : 18%

Name               Stmts   Exec  Cover
--------------------------------------
__init__               3      0     0%
archive_util          78     11    14%
bcppcompiler         185      0     0%
ccompiler            453      0     0%
cmd                  180     79    43%
core                  93     15    16%
cygwinccompiler      160      0     0%
debug                  3      3   100%
dep_util              43      4     9%
dir_util             106     50    47%
dist                 578    342    59%
emxccompiler         118      0     0%
errors                49      0     0%
extension             97      9     9%
fancy_getopt         233    121    51%
file_util            121     50    41%
filelist             162      0     0%
log                   46     15    32%
msvccompiler         365      0     0%
mwerkscompiler       140      0     0%
spawn                 93      0     0%
sysconfig            296     10     3%
text_file            146      0     0%
unixccompiler        159      0     0%
util                 235     69    29%
version               68     48    70%
versionpredicate      61     51    83%
__init__               3      3   100%
bdist                 59      0     0%
bdist_dumb            57      0     0%
bdist_msi            320      0     0%
bdist_rpm            248      0     0%
bdist_wininst        159      0     0%
build                 52     47    90%
build_clib            90      0     0%
build_ext            304      0     0%
build_py             213    143    67%
build_scripts         78     64    82%
clean                 35      0     0%
config               185      0     0%
install              220    120    54%
install_data          44      0     0%
install_egg_info      40      0     0%
install_headers       26      0     0%
install_lib           96      0     0%
install_scripts       33     29    87%
register             171      0     0%
sdist                204      0     0%
upload               118      0     0%
--------------------------------------
TOTAL               7026   1283    18%

Since Python 3.0.1 was released this week, here’s a quick wrapup of what is going on in Distutils.

Code work (since one month)

New features

• Issue 2563 : now the manifest is embed in windows extensions
• Issue 4394 : the storage of the password in .pypirc file is optional now

Fixed bugs

• Issue 4524: distutils was failing to build scripts with the ‘–with-suffix=3′
• Issue 5132 : build_ext command was failing under Solaris with ‘–enabled-shared’
• Issue 5075 : bdist_wininst was depending on the vc runtime

Refactoring

• Issue 2461 : added test coverage for util.py
• Issue 3986 : removed  string and type usage from distutils.cmd
• Issue 3987 : removed type usage from distutils.core

Documentation

• Issue 5158 : added documentation for depends option for extensions
• Issue 4987: updated README info
• Issue 4137 : SIG web pages were updated

Design work

The main topics that are being discussed are:

• Improving the console script story. thread starts here.
• Publishing a Survey on Distutils before the Language Summit. thread starts here.
• Adding an uninstall command to unistall packages in the sdtlib : thread starts here
• Adding a get_metadata API in pkgutil to get the metadata of an arbritary package The same code can also be used for various lookups (uninstall, console script) : thread starts here
• Having a new PEP to make egg.info a directory and clearly define PKG-INFO and its match with metdata. thread starts here

I am making some progress in Distutils. I closed something like 10 bugs last week, and I am reaching issues that were added 8 months ago. Not that everything is entirely cleaned up in the newest issues, but they’re almost all being processed. Every commit comes with at least a test, to get the code base back into a state were it is easier to make things evolve without the risk of breaking it up.

It comes through tiny little changes, with tests and an eye on the coverage.

Now I am facing an unpleasant situation : since the test coverage is still low, I am always scared of breaking something in Distutils when I am fixing a bug or making a change.Buildbots are watching, and I run some of my own packaging work with the current trunk.

But still, this is an unpleasant situation, and I don’t want to cause the package to be broken in the next Python version…

But the regression tests exists ! They are there, hidden, in the community. It’s everyone package.

1. Joe adds an issue in the Python bug tracker, because Distutils didn’t work as expected on his package because of a bug
2. At some point the bug is (was) fixed.
3. The test to make sure the bug is fixed is “Joe is running Distutils over his package again, and makes sure it is properly installed, compiled, etc”.
4. The bug is closed.

So how can I get back this test to make sure Joe’s package is still working properly, so he doesn’t hate us at the next major Python release ?

A Distutils Regression Test Server

If Joe’s package is on PyPI, we can set something up. A dedicated server that watches the PyPI changelog and triggers a buildbot when:

• a new release of Joe’s Package comes out
• we change something in Distutils code

The precise test to be run is still unclear to me but, I am thinking about some generic strategies and I think it’s possible. Let’s call this test a distutils regression test. (If you have a better name, I’ll buy it)

Of course it doesn’t have to be on all the packages that are uploaded out there at PyPI. Just Joe’s one, because he came up with a problem we fixed. And we would be ashamed if the bug comes back on Joe’s package.

This requires of course a server, and probably a vmware-like system if Joe runs Windows or Solaris, to make buildbot slaves etc. It also requires that Joe uses the right metadata in his package so we know if it works under Python 2, Python 3, etc. MvL added enough classifiers lately for this.

A Distributed Distutils Regression Test

But some package are not on PyPI, for privacy or conveniency in the packaging process of the person in charge. So, what if the distutils regression test is provided in a Distutils command ? It can run the same test the server runs, and come up with a report that is sendable or sent by mail to a special mailing list or so.

This supposes that the developer is cooperative. So maybe it can even be automatically triggered in case of any failure on any Distutils command, and ask the user if he would like to send a report ?

The good thing here is that it doesn’t require CPU power on the test server, and that anyone can run that test.

So what ?

Well I am just throwing an idea here, because I am really concerned about the potential regression problems. Even if Distutils is 100% covered with tests, it’s not possible to test all combinations. The real world environment is the only test that can be trusted at the end in the packaging area.

I’ll throw this idea at the Language Summit in March, and if it catches people interest, maybe a Google Summer of Code task could be done for that topic ?  Can’t implement it myself, I am overwhelmed already in Distutils maintenance

Just out of curiosity, how do *you* test your packages to make sure they get installed correctly ?