According to the poll, The name of the fork will be Distribute !

The code should not be changed anymore at this point, and I am working on the bootstraping so installing Distribute will work with an existing Setuptools installation and will replace it for the applications that requires it.

This is done by detecting an installation of Setuptools, and replacing it with a fake installation. This means that once you’ve installed Distribute, applications and especially installers will think that setuptools 0.6c9 is installed.

That’s pretty strong and intrusive, but required for a simple switch : even if the programs you are using have a setuptools dependency, they will work without requiring any change on the code or in their setup.py files. Same goes for zc.buildout apps.

I still have a lot of work for this part :

• I don’t detect and patch properly single-version-externally-managed setuptools installation yet (required for pip)
• I need to fix a bug on a first run under jython
• I need to fix a bug when virtualenv is not used with –no-site-packages
• I need to do extensive tests on zc.buildout to see if it behaves correctly

If you want to help:

1. download http://nightly.ziade.org/install_test.py
2. run it with the Python interpreter of your choice (possibly a virtualenv-ed one)

To uninstall, follow the Uninstallation instructions here : http://bitbucket.org/tarek/distribute/src/tip/README.txt

*Disclaimer: it might break your installation*

If the test end up with this line: **** Test is OK. It worked. Otherwise, please let me know !

The next “big” step will happen with Distribute 0.7 because the plan is to split the code in several distributions with renamed modules:

• Distribute : the core (setuptools package but renamed)
• DistributeInstall : easy_install on its own, and renamed.
• DistributeResource : will contain the pkg_resources.py module, renamed

I’ll post more details on it in the upcoming days.

As you might know, we are working hard on Distutils side for Python 2.7 and 3.2 upcoming releases. The biggest work is on PEP 376, that will introduce among other things a uninstaller function and functions to query installed distributions.

The other “big” work is on PEP 345. We want to introduce a new metadata field called “install_requires” to be able to express requirements. That’s from the setuptools project and is quite used by the community. Notice that there were several attempts to define requirements in the past in Distutils, but none of them really made it through.

For instance, if you want to define docutils as a dependency for your distribution but a version less or equal to 0.4, you can say :

docutils <= 0.4

But as long as you want to work with such dependencies and provide a way to express them with operators, you have to be able to compare versions. For instance if you want to compare an installed docutils distribution to see of it is compatible with 0.4.

That’s another big topic we have been working on for the last few months, with people from various communities (Fedora, Ubuntu, etc). And I have started to write down PEP 386.

But comparing version appears to be a topic that cannot be generic. It seems that Distutils, therefore Python, shouldn’t enforce any rule on this.

Furthermore, since we have said that Distutils should be a lighter package, it will not implement a complete package managment system, like setuptools or zc.buildout does.

So I’ve decided to propose to drop PEP 386, and stick on a very simple rule in PEP 345, saying that requirements can be defined, with :

distribution_name OPERATOR version

where OPERATOR is in >, <, ==, !=, >= or <=.

Last, so the work done at Pycon and in Distutils-SIG is not lost, I will publish the library we wrote. This could be a very good basis for packaging managment systems out there.

If you want to work (or try to find some bugs in it to help out) with the latest bleeding-edge Distutils trunk version that will be shipped with Python 2.7 and Python 3.2, you can do it now !

I am creating a nighlty build every day here now : http://nightly.ziade.org/

This distribution can be installed in your existing Python 2.6 installation, and will probably work with 2.5.

I’m not planning to publish standalone versions yet, except nighlty build, at least until 2.7 and 3.2 are starting to be released as alphas.

I am not really sure smoke testing is the best name to describe what I am trying to set up.

Wikipedia says:

In computer programming and software testing, smoke testing is a preliminary to further testing, which should reveal simple failures severe enough to reject a prospective software release. In this case, the smoke is metaphorical.

But I think it describes well the need I have : being able to test the output of several releases of Distutils, without relying on unit tests for that. Simply because they did not exist on early versions of Distutils.

Why do I need “smoke tests”

Lately, I was working on Distutils, removing some old bugs and adding some tests. Working on an under-tested package is like opening a can of worms : when you change something you can introduce some regressions. It’s hard to avoid for sure, because even if you add some tests subsequently for some features, you are not really doing proper TDD, where the tests and the code grow together. So there might be some uncovered cases left even if the coverage is improving slowly.

Regression also happens when you correct a behavior that was broken for years : if third party tools used a broken behavior without knowing it, fixing the behavior at some point becomes a regression from them.

But these pitfalls are unavoidable. Hopefully they don’t last for too long in a big project like Python. They are quickly reported, either by buildbots, either by someone from the community.

The last behavior problem I had was on the build_ext command. There’s a string option called “compiler”, were you can force the compiler type. You can put for instance ‘linux‘ and build_ext will use the UnixCompiler compiler class.

Although this option name is misleading. It should have been called “compiler_type” in the first place.  And the build_ext command adds another problem : it turns the compiler attribute of the class where the option value is stored, into a compiler instance. No need to say a class attribute value should not have several types.

The first application to suffer from this is Python itself. It uses build_ext.compiler, as a compiler instance to build its modules.

This means that “compiler” is now doomed to be both an option and a compiler attribute. It’s not easy to fix and will require a deprecation step.

So the idea of the smoke test is to make sure Distutils still knows how to produce releases of existing projects.

The Smoke testing buildbot

I have built a buildbot instance using collective.buildbot. The script the slaves run is quite simple: it gets some projects source and run their setup.py script, using various versions of the Python interpreter, then the trunk itself. The commands run so far are sdist and bdist.

Then it compares every file contained in the archive created to make sure produced archives are similar.

I will encouter some exceptions, when the package I am testing includes different files in its distribution depending on the Python version, but those exception will be managed.

Last, I cannot run this test on every package out there, for security reasons. Until I have the proper virtual environment to do that, I’ll work on a white list of packages I “trust”. So far, there’s just a package of mine, and NumPy. But this list will grow.

If you are curious to know if your package builds under Python trunk, get in touch with me, I’ll probably add it if I know I can trust it.

The last problem I have with this system yet is the fact that setuptools is not working anymore with the Distutils trunk, so I am unable to test setuptools-based packages.

The buildbot is located here : http://buildbot.ziade.org/waterfall

Since Pycon, a lot of discussion and work has been going on.

During the summit, we made a few decisions (see http://mail.python.org/pipermail/python-dev/2009-March/087834.html) but this topic is so wide and so complex that a lot of discussion still needs to be done to have a clear and complete picture of where we want to go and how we are going to do it

But we have made significant progress and reached consensus on some key points. This entry is a short list of these key points.

Being able to compare project versions

If you take a look in Distutils code, there’s a version and a versionpredicate module that provides a way to compare versions. This feature is barely used by Distutils itself, and a very few number of projects out there are using it (If you do so, please let me know).

This is probably because Distutils provides two different ways to compare versions: a “strict” one and a “loose” one. In other words Distutils clearly states that it is unable to provide a unique version comparison algorithm that can be used by anyone out there. Anyone means here : Python developers, OS packagers, etc.

Setuptools did a better job by providing an algorithm that covers most cases, but suffers from this universality: it’s too heuristic.

So one of the main topic during Pycon was to try to find a version comparison algorithm that would just work for everyone and in the meantime would be strict enough to be useful. That’s a pretty tough task, but I think we have reached something that is “good enough” for everybody. We had the chance to have people from Ubuntu, RedHat during Pycon to work on this task, and Trent Mick took the lead during the sprints to come up with a prototype.

It’s described here : http://wiki.python.org/moin/Distutils/VersionComparison and I have put the code here http://bitbucket.org/tarek/distutilsversion.

Two more things to take care of:

• Philip Eby came up with an edge-case : being able to do a development version of a post release.
• Jean-Paul Calderone proposed to add a constructor that would take explicit arguments to describe the version, rather than a string representation

There’s a branch (tarek-postdev) on bitbucket to work on these two cases. But basically, it seems that we have a consensus on a unique way to compare versions in Python ! This is a great step forward.

Another point that Toshio Kuratomi raised during a hall discussion was the fact that some Python developers are not having good practices when versioning their releases. So we agreed that a good “How to properly use version numbers for your projects releases” document will have to be delivered in Distutils documentation, besides the new algorithm. I am pretty confident that people will eventually follow it.

APIs to access installed project metadata, and an egg-info standard

Once a project is installed in Python, you don’t have a way to get its metadata and to answer to basic questions like:

• what are the installed packages ?
• what is the version ?
• how is the author ?

Of course you can dig in your installation and get back most of these, but depending how you installed your package (manual, easy_install, pip) it might be located in different places.

So we agreed on a standard for this, described in PEP 376 : http://www.python.org/dev/peps/pep-0376

An API will be introduced to be able to get the info for a installed package, as long as it was installed using PEP 376 standard.

An uninstall feature !

A feature that is claimed for a long time now should be introduced in Distutils : uninstallation !

Distutils will provide a basic, reference uninstall feature that will remove all files that where previously installed for a project. This will be doable as long as this list of files is recorded at installation time, and not used by another project.

See the details in PEP 376.

Standardize PKG-INFO

New fields will be added in the metadata of a project. The most important one is install_requires from setuptools, which let you list the projects your project depend on and their versions.

This is for informational purpose, and Distutils will not provide an install feature that will fetch and install dependencies. This can be done by third-party tools like easy_install  as long as they use the installation standard described earlier.

We started this work with Jim Fulton at Pycon, and Tres Seaver took the lead on this task since then. The plan is to update PEP 345 . The work is done in a branch here : http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?view=markup

But the remaining work is focusing on practical details. Just remember that PKG-INFO will evolve and install_requires will be integrated amongst other changes.

Distutils code cleanup, test coverage

The test coverage is starting to look good and most modules are covered around 80%. I guess this is the average in most Test-Driven Development projects out there. So Distutils is becoming a good citizen of the standard library 

The remaining work for a good test coverage is mostly on compilers side and os specific commands.

My black list of untested (or nearly untested) modules :

• bcppcompiler
• cygwinccompiler
• emxccompiler
• msvc9compiler
• msvccompiler
• command.bdist_msi
• command.bdist_rpm

I need to figure out how to properly test them. Last, I need to set up a buildbot that will try out Distutils trunk on a list of projects out there, like numpy for example.

Other topics

Check this ! http://wiki.python.org/moin/Distutils

I am introducing the check command in Distutils. This command will check your package metadata, like the sdist and the register command already do (they display warnings).

But the new thing is that it will also allow you to check if long_description is reStructuredText compliant.

Its usage will be:

$ python setup.py check --restructuredtext
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)

And there’s also a strict mode, that raises an error in case something is wrong

$ python setup.py check --restructuredtext --strict
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)
error: Please correct your package.

Last, this command will be used by register, and sdist, so you can stop the process in case the metadata are not correct. This is useful to make sure your PyPI home page is not broken for example, since it parses long_description to build it. So a good practice will be to use strict when registering a package:

$ python setup.py register --strict
running register
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)
error: Please correct your package.

This feature will land in Python 2.7 (I am working on it and it should be commited this week end). Of course, it will not introduce a hard dependency on docutils in Python, neither it will change the current default behavior.

Until then, you can use collective.dist 0.2.3, that provides this feature for Python 2.4 to 2.6

We had an excellent brainstorming session today in the hall, with Toshio, Georg, Martin, Thomas, etc.. (sorry we were so many I don’t have the full list) with some insights from Guido and Brett. We tried to think about a way to handle multiple versions of a same package.

Here’s the two most important concepts :

• Unicity: There should be one and only one instance of a Python package at a given version on a system
• Combination: One Python application combines several packages to run

About unicity

A Python package is a component that can be installed on a system. If you use the standard Distutils approach, it will end up in the Python site-packages directory and be importable by the interpreter. This package comes with a version number and is unique.

This unicity is important for security and maintainability. For instance, if there’s a security hole in a package, the fix is applied in one place and the system maintainer knows it can’t be present elsewhere on the system.

This is the system administrator point of view

About Combination

What defines a Python application is the fact that is selects a list of packages it needs to run. And this varies for every application. So two Python applications might need a different version of a given package and that is normal.

What’s important is to have the right list of packages when an application loads. Tools like zc.buildout or virtualenv are perfect for this need : they create an isolated environment for your application to run with the right set of packages.

So the simplest way to release an application is to ship it with everything required, regardless the unicity.

This is the application developer point of view

The idea

zc.buildout and virtualenv are a blast for developers, and another thing system packagers might dislike. This is because they break the unicity by allowing developers to ship their applications as black boxes. Of course one may say that this is perfectly fine since what’s inside an application is not the problem of the system packager. But since this application is made of packages that may be shared on the system by other applications, that is redondant.

Forget site-packages for a moment. And let’s think about a new loading system for packages. This approach is similar in some ways to setuptools’ multiple version system.

Storing multiple versions

First of all, let’s store the packages in a directory, and for each version of the package, under a sub-directory which name is the version.

For example:

• SQLAlchemy
  • 0.4
    • package code is here
    • package egg-info here
  • 0.5
    • package code is here
    • package egg-info here
• jsonlib
  • 1.2.6
    • package code is here
    • package egg-info here
  • 1.3.10
    • package code is here
    • package egg-info here

Given this structure, some mechanism can provide to the interpreter the latest available version of a package, as long as Distutils knows how to handle version comparisons correctly (which will be the case in a near future)

Choosing specific versions

Back to our application. Let’s call it MyApp, version 1.0.

It needs specific versions for some packages. Forget zc.buildout and pip for a moment. And let’s think about a different way to express the packages (and versions) its needs.

Let’s make a Python package for this application and let’s make a few assumptions on some features in Distutils:

• setuptools’ install_requires has made it into Distutils, as part of metadata.
• metadata are defined statically in a package, apart from setup.py.

So basically, the application is mainly a static list of dependencies defined into install_requires. For example:

• SQLAlchemy > 0.4
• jsonlib == 1.2.6

When MyApp will get installed by Distutils, it will be added in the packages tree.

When it is used, it will need to load the versions of SQLAlchemy and jsonlib it needs. The ones that are inside its metadata.

To make it possible, the script that launches the application calls a built-in function called read_deps, that takes the metadata and reads them to know which versions fit:

# the script
read_deps('PKG-INFO')

import SQLAlchemy
import jsonlib
...

This call will load the right versions in the packages tree:

• SQLAlchemy
  • 0.4
  • 0.5
• jsonlib
  • 1.2.6
  • 1.3.10
• MyApp
  • 1.0

What’s next

I just dropped here the rough idea, and a lot of details are missing. But I think it’s a good thing to share this idea here in its early stage.

We have decided we would try to write a prototype using importlib and sys.meta_path. Maybe Georg will start it during the Pycon sprint, and start to digg into the details. He was working on this when I left the open session at Pycon.

Stay tuned

Tomorrow is the Language Summit, yeepee.

The package and distribution part of the Summit is going to be great since we have key people coming up.

zc.buildout and pip leaders (Jim Fulton and Ian Bicking) will be present, and many others. I’ll be representing Distutils, since I am its current maintainer. Unfortunately Philip Eby (setuptools) can’t make it, but he should be reachable via IRC (I am trying to set something up for tomorrow).

Anyway, one of the goal of the Summit is to validate the new features and enhancements we want to introduce in Distutils and PyPI. It’s important to make sure they play well with third-party tools like zc.buildout, setuptools and pip. We also need to make sure these tools will evolve in the same direction in the future.

We have reached a point in Python where we need to concentrate all the packaging effort to build a common standard in the standard library, because it is badly needed.

Here’s a draft of the slides I will present tomorrow, during the first 5/10 minutes, as a session leader:

• Packaging Survey results overview
• Topics to discuss
  • setting up an organized network of mirrors (see PEP 381)
  • discuss about other PyPI enhancements
  • improve the package installation / uninstallation  (see PEP 376)
  • discuss the package dependencies problem and see if we can come up with a PEP
  • discuss the multiple version problem and see if we can come up with a PEP
  • discuss the isolated environment vs the OS vendor approach and see what can be done to improve their coexistence.

Summit schedule:

• 13:20 -> 13:30 : presentation
• 13:30 -> 14:30 : discussions/work in small groups
• 14:30 -> 14:50 : “tour de table”
• 14:50 -> 15:10 : break
• 15:10 -> onwards: sprint !

I am not sure about the ‘work in small group’ part yet, because I don’t know how many people will show up, and what people will want to focus on.

Around 570 people answered the survey, which is a great number I didn’t expect. Thanks again to Massimo for his help on this.

I have a lot of work to read all the open question answers, and all the comments that goes with the “other” answer, but I wanted to publish the results of the closed questions before the summit.

I don’t want to comment the results yet. I will after I have studied all answers, so it’ll be a little while

Who are you ?

Where are you located ?

If you are a web programmer, what is the framework you use the most ?

How do you organize your application code most of the time ?

For libraries you don’t distribute publicly, do you you create a setup.py script ?

What is the main tool or combination of tools you are using to package and distribute your Python application ?

How do you install a package that does not provide a standalone installer (but provides a standard setup.py script) most of the time ?

How do you remove a package ?

How do you manage using more than one version of a library on a system ?

Do you work with setuptools’ namespace packages ?

Has PyPI become mandatory in your everyday work (if you use zc.buildout for example) ?

If you previously answered Yes, did you set up an alternative solution (mirror, cache..) in case PyPI is down ?

Do you register your packages on PyPI ?

Do you upload your package on PyPI ?

If you previously answered No, how do you distribute your packages ?

We have been working hard in the past few months with my Plone friends to make it happen.

It’s now available : Plone.org is acting like PyPI in its Products section, which means that you can use Distutils to register and upload your packages in both places now.

I have written a small tutorial on plone.org to explain how to use it: http://plone.org/documentation/tutorial/how-to-upload-your-package-to-plone.org

I will also explain during my Pycon presentation (http://us.pycon.org/2009/conference/schedule/event/44) how this new feature at Plone.org will help people that are working with Plone and Python packages.

After the next commit I will make in Distutils (that adds tests for bdist_rpm), the test coverage of this Python standard library  package will be at 41%. This means that I have doubled the test coverage over the past few months, from 18% to 41%.

My goal is to double it again, and reach 80% in the next 6 months.

This also means I am just half an idiot now ! (since people who don’t have 100% code coverage are idiots ).

So does it make Distutils more robust ?

It would have probably made the latest Python 3 release looks better for this package, since we had a uncovered cmp() call left in Distutils by the time the release was made. In the meantime, as I said before, the “real” Distutils regression test suite is held by all the packages out there in the community, that are built and installed everyday.

Python trunk Distutils test coverage : 41%

Name               Stmts   Exec  Cover
--------------------------------------
__init__               3      0     0%
archive_util          77     61    79%
bcppcompiler         185      0     0%
ccompiler            453    211    46%
cmd                  180    134    74%
config                73     59    80%
core                  93     50    53%
cygwinccompiler      161      0     0%
debug                  3      3   100%
dep_util              43     11    25%
dir_util             109     76    69%
dist                 581    386    66%
emxccompiler         118      0     0%
errors                49      0     0%
extension             97     28    28%
fancy_getopt         233    126    54%
file_util            124     77    62%
filelist             161    102    63%
log                   46     21    45%
msvc9compiler        408      0     0%
msvccompiler         370      0     0%
spawn                 93     28    30%
sysconfig            323     51    15%
text_file            112     61    54%
unixccompiler        160     64    40%
util                 255    157    61%
version               68     62    91%
versionpredicate      61     51    83%
__init__               3      3   100%
bdist                 61     35    57%
bdist_dumb            57     47    82%
bdist_msi            322      0     0%
bdist_rpm            252    198    78%
bdist_wininst        170      0     0%
build                 60     54    90%
build_clib            90      0     0%
build_ext            334    160    47%
build_py             213    178    83%
build_scripts         78     65    83%
clean                 35      0     0%
config               185      0     0%
install              251    156    62%
install_data          44      0     0%
install_egg_info      40     32    80%
install_headers       25      0     0%
install_lib           97     50    51%
install_scripts       33     29    87%
register             173     82    47%
sdist                228    180    78%
upload               112     38    33%
--------------------------------------
TOTAL               7502   3126    41%

Python 2.5.4 Distutils test coverage : 18%

Name               Stmts   Exec  Cover
--------------------------------------
__init__               3      0     0%
archive_util          78     11    14%
bcppcompiler         185      0     0%
ccompiler            453      0     0%
cmd                  180     79    43%
core                  93     15    16%
cygwinccompiler      160      0     0%
debug                  3      3   100%
dep_util              43      4     9%
dir_util             106     50    47%
dist                 578    342    59%
emxccompiler         118      0     0%
errors                49      0     0%
extension             97      9     9%
fancy_getopt         233    121    51%
file_util            121     50    41%
filelist             162      0     0%
log                   46     15    32%
msvccompiler         365      0     0%
mwerkscompiler       140      0     0%
spawn                 93      0     0%
sysconfig            296     10     3%
text_file            146      0     0%
unixccompiler        159      0     0%
util                 235     69    29%
version               68     48    70%
versionpredicate      61     51    83%
__init__               3      3   100%
bdist                 59      0     0%
bdist_dumb            57      0     0%
bdist_msi            320      0     0%
bdist_rpm            248      0     0%
bdist_wininst        159      0     0%
build                 52     47    90%
build_clib            90      0     0%
build_ext            304      0     0%
build_py             213    143    67%
build_scripts         78     64    82%
clean                 35      0     0%
config               185      0     0%
install              220    120    54%
install_data          44      0     0%
install_egg_info      40      0     0%
install_headers       26      0     0%
install_lib           96      0     0%
install_scripts       33     29    87%
register             171      0     0%
sdist                204      0     0%
upload               118      0     0%
--------------------------------------
TOTAL               7026   1283    18%

Since Python 3.0.1 was released this week, here’s a quick wrapup of what is going on in Distutils.

Code work (since one month)

New features

• Issue 2563 : now the manifest is embed in windows extensions
• Issue 4394 : the storage of the password in .pypirc file is optional now

Fixed bugs

• Issue 4524: distutils was failing to build scripts with the ‘–with-suffix=3′
• Issue 5132 : build_ext command was failing under Solaris with ‘–enabled-shared’
• Issue 5075 : bdist_wininst was depending on the vc runtime

Refactoring

• Issue 2461 : added test coverage for util.py
• Issue 3986 : removed  string and type usage from distutils.cmd
• Issue 3987 : removed type usage from distutils.core

Documentation

• Issue 5158 : added documentation for depends option for extensions
• Issue 4987: updated README info
• Issue 4137 : SIG web pages were updated

Design work

The main topics that are being discussed are:

• Improving the console script story. thread starts here.
• Publishing a Survey on Distutils before the Language Summit. thread starts here.
• Adding an uninstall command to unistall packages in the sdtlib : thread starts here
• Adding a get_metadata API in pkgutil to get the metadata of an arbritary package The same code can also be used for various lookups (uninstall, console script) : thread starts here
• Having a new PEP to make egg.info a directory and clearly define PKG-INFO and its match with metdata. thread starts here

I am making some progress in Distutils. I closed something like 10 bugs last week, and I am reaching issues that were added 8 months ago. Not that everything is entirely cleaned up in the newest issues, but they’re almost all being processed. Every commit comes with at least a test, to get the code base back into a state were it is easier to make things evolve without the risk of breaking it up.

It comes through tiny little changes, with tests and an eye on the coverage.

Now I am facing an unpleasant situation : since the test coverage is still low, I am always scared of breaking something in Distutils when I am fixing a bug or making a change.Buildbots are watching, and I run some of my own packaging work with the current trunk.

But still, this is an unpleasant situation, and I don’t want to cause the package to be broken in the next Python version…

But the regression tests exists ! They are there, hidden, in the community. It’s everyone package.

1. Joe adds an issue in the Python bug tracker, because Distutils didn’t work as expected on his package because of a bug
2. At some point the bug is (was) fixed.
3. The test to make sure the bug is fixed is “Joe is running Distutils over his package again, and makes sure it is properly installed, compiled, etc”.
4. The bug is closed.

So how can I get back this test to make sure Joe’s package is still working properly, so he doesn’t hate us at the next major Python release ?

A Distutils Regression Test Server

If Joe’s package is on PyPI, we can set something up. A dedicated server that watches the PyPI changelog and triggers a buildbot when:

• a new release of Joe’s Package comes out
• we change something in Distutils code

The precise test to be run is still unclear to me but, I am thinking about some generic strategies and I think it’s possible. Let’s call this test a distutils regression test. (If you have a better name, I’ll buy it)

Of course it doesn’t have to be on all the packages that are uploaded out there at PyPI. Just Joe’s one, because he came up with a problem we fixed. And we would be ashamed if the bug comes back on Joe’s package.

This requires of course a server, and probably a vmware-like system if Joe runs Windows or Solaris, to make buildbot slaves etc. It also requires that Joe uses the right metadata in his package so we know if it works under Python 2, Python 3, etc. MvL added enough classifiers lately for this.

A Distributed Distutils Regression Test

But some package are not on PyPI, for privacy or conveniency in the packaging process of the person in charge. So, what if the distutils regression test is provided in a Distutils command ? It can run the same test the server runs, and come up with a report that is sendable or sent by mail to a special mailing list or so.

This supposes that the developer is cooperative. So maybe it can even be automatically triggered in case of any failure on any Distutils command, and ask the user if he would like to send a report ?

The good thing here is that it doesn’t require CPU power on the test server, and that anyone can run that test.

So what ?

Well I am just throwing an idea here, because I am really concerned about the potential regression problems. Even if Distutils is 100% covered with tests, it’s not possible to test all combinations. The real world environment is the only test that can be trusted at the end in the packaging area.

I’ll throw this idea at the Language Summit in March, and if it catches people interest, maybe a Google Summer of Code task could be done for that topic ?  Can’t implement it myself, I am overwhelmed already in Distutils maintenance

Just out of curiosity, how do *you* test your packages to make sure they get installed correctly ?

I have the opportunity to lead a session at the Python Language Summit at Pycon.

In order to prepare this event, I am currently building a survey because we need to know how people use Distutils, what is wrong with it, and so on…

The draft is here : http://wiki.python.org/moin/Packaging%20Survey

When it’s ready (hopefully soon), I’ll post it in a SurveyMonkey-like system online for people to take it, and synthetize the results, so what I am saying at the summit reflects hopefully the reality.

If you are a Python developer, an OS packager, come to the Distutils-SIG mailing list and help us build the best survey possible !

(while the draft is in a wiki, it’s better to discuss the changes in the mailing list before it is applied)

When you launch such a command:

$ python setup.py register sdist upload

There’s no way to give to Distutils your PyPI password in the prompt, so you distribution is uploaded to the server. You have to store your password in the .pypirc file:

    [distutils]
    index-servers =
        pypi

    [pypi]
    username: <username>
    password: <password>

The password is stored in clear text, so it can be used by Distutils to authenticate. This is rather unsecure, since anyone who has a read access to your home can get your password.

I have detected this problem this summer while listing the possible enhancements in Distutils. Nathan Van Gheem sent me a mail a month ago to ask for that same feature in collective.dist; which is a port of the latest Distutils features into Python 2.4 so Zope can use them. So before having it into collective.dist, the first step was to introduce it into Python itself.

The idea is to be able to remove from .pypirc the password so it’s asked at the prompt. Nothing fancy here : the Distribution object that is created before you launch any command is the place where you can share a context between commands.

So when you launch:

$ python setup.py register sdist upload

Here’s what is happening:

1. register looks into .pypirc, if no password is found, it asks it to the user using getpass
2. register use it then store it in the Distribution instance
3. upload look into the Distribution instance to see if the password was stored, and use it

This is now available in Python 2.7 and 3.1, and heavily tested.

I’d like to go further and to think about a ssh-agent like system, so there’s no need to enter the pasword everytime you work with PyPI in the same session.

Does anyone knows what would be the way to do it properly ? I think a ssh-agent like mechanism in Python’s getpass would be a great feature itself.

Happy New year all !

I am going to make a few posts on the things I would like to achieve in 2009. Each entry will focus on a topic. This one is about Distutils.

I was granted a commit privilege in Python, specifically to work on Distutils maintenance. This is a huge privilege, and I try will do my best in this job. I have worked on a few tickets already and closed some. I learnt the Python development process, which requires to backport and to forward-port changesets in various Python versions. While this can be taken care of automatically by someone else if you don’t do it, it’s better that every commiter takes the time to merge his own work.

So what’s next ?

• There are 132 tickets that are open in the Python tracker, that match the word distutils, and some of them are 5 years old !
• There’s a Python language summit to be held in Chicago right before Pycon, and I volunteered to champion the task about Distutils, PyPI and packaging matters.

I am planning to :

• review and classify all the tickets in the tracker;
• fix the maximum amount of them before the summit;
• make Distutils a first class citizen in test coverage;
• make Distutils code more modern.

Besides, I will try to build a roadmap for Distutils I will present in Chicago.

To build this roadmap, I will ask for input in the distutils-SIG mailing list in the coming days, and see what people will come up with. There’s no crowd in this list these days but sometimes some threads are hot when it comes to the future of packaging in Python.

The roadmap I am planning to build will not address all the issues people have when it comes to distribute a Python application, since there is no consensus yet on the best practices. It will rather try to see if the current version of Distutils can be enhanced to adress some problems, and at least be the bridge to something new in the future. Maybe by including some best practices from third-party tools (the pre-condition for all of this imho is to make the Distutils code base healthier).

Anyway, I hope that the lead developers of: zc.buildout, pip, setuptools, paver (and those projects I forget about right now) will  participate in this discussion, and that we will be able to find pragmatic enhancements.

I found a bit of time to work on distribution matters. Here’s a status of what I am doing there.

There are two topics I am focusing on right now.

• clean up and enhance Python’s distutils package
• implement the mirroring infrastructure at PyPI

distutils work

Nathan Van Gheem proposed a cool patch in collective.dist, (this package is a port of the new features I have added in distutils so they are available in 2.4 and 2.5).

Nathan proposed a patch to be able to avoid the storage of the password in the .pypirc file. The prompt is used in that case. This is something that was in my pile for a long time.

I have added a few things to Nathan’s patch, and a test, and proposed it to Python. I am now waiting for its integration in 2.7 trunk: http://bugs.python.org/issue4394. If it’s accepted, I will backport it to collective.dist.

There are some other tickets I am waiting to be accepted:

• http://bugs.python.org/issue4400
• http://bugs.python.org/issue2461
• http://bugs.python.org/issue3992
• http://bugs.python.org/issue3985
• http://bugs.python.org/issue3986

I am not sure when those will be integrated. The average time for the integration of tickets in distutils in Python is between 6 months and 8 months. hihihi.

PyPI mirroring

The job I am doing in PyPI will be in three phase :

• Phase 1: implement the mirroring infrastructure in PyPI
• Phase 2: promote it, and propose patches for the mirroring tools out there so they use the protocol
• Phase 3: promote and propose patches for pip so it can use the mirrors efficiently (fail-over and nearest mirror infrastructure).

Phase 1: so far, so good.

With some insights from Richard Jones and Martin von Löwis, I am currently implementing the mirroring infrastructure for PyPI we have defined during the D.C. sprint (I still owe a blog entry about this sprint). The code lives in a branch on the python svn folder dedicated to PyPI.

The idea of the mirroring infrastructure is to be able to get a list of official mirrors for PyPI, that can be used as alternatives sources . (It is described here: http://wiki.python.org/moin/PEP_374). A great behavior could be that the client application interacts with the nearest mirror location automatically, and switch to another if it goes down.

So, a list of mirrors will be made available at /mirrors, and the client applications will be able from there to use an alternative location for every package. The hardest part concerns the stats : we want to display in PyPI the download counts for each package by summing downloads from every mirror.

So every mirror will have to provide its “local stats” that can be visited by PyPI. That’s the biggest part of the work I am doing. It will build the stats for PyPI by parsing its Apache log file. And hopefully, this code should be reusable by the mirrors themselve so they can build their stats the same way.

Of course this infrastructure could be used for any PyPI-compatible server even if is not a mirror of PyPI (like a private PyPI server)

Phase 2 will consist in promoting the infrastructure to the mirroring softwares out there. Maybe Pycon will be a good place for that.

Phase 3 is the most interesting one : make sure the client applications use the mirrors ! I think Ian Bicking’s pip project could be the right place for these innovations.

Next topics in the pile:

• index-merging: describe in a PEP-like document the index-merging feature that would allow clients to merge several indexes with a content that differe. For example: PyPI + a private PyPI server. I have written a first draft of such a patch in setuptools in the past (http://bugs.python.org/setuptools/issue32) but I have lost all my hopes to see this project moving forward lately.
• Brainstorming: try to understand the Python Packaging Paradox. That is = how come the community, which is composed of many briliant people, is unable to move forward in packaging matters.
• Distribute the return

Just a quick note:

We have started the sprint in D.C. on packaging matters. Our tasks for the moment:

• re-read all the threads in distutils-ML since mid-september.
• write a short list of actions that can be done NOW in distutils, setuptools and PyPI
• write a short list of writing that can be done to define a “new” tool

Please, join at any time in #distutils in freenode, and ping us, if you want to get involved.

Disclaimer: this post is Subversion-centric

When releasing a setuptools based-package to the world, developers will eventually tag it as a stable version, then upload it to PyPI or to their own website. So typically they create a branch out of the trunk, fix the version, then create a tag for the release.

Now for continuous development releases, where people get for example a daily snapshot from the trunk, having a dev suffix to the egg version makes it possible to distinguish it from a stable release, so ‘my.package-2.4dev’ will not superseed ‘my.package-2.4′. So a given package will need a dev prefix in its trunk, that gets removed in the branch where the next release occurs.

From there, easy_install will be able to distinguish them.

The question is: how should we do this ?

The Zope 3 way

The Zope 3 egg collection has a simple way to manage it, describe here: http://svn.zope.org/*checkout*/Sandbox/philikon/foundation/releasing-software.txt. The packages have a dev tag suffix in the version metadata, that is removed in the subversion branch used to tag and release

This is simple and straight forward. Although, releasing the trunk has to be done manually. The problem is that any release of the trunk will have the same name for a given version (my.package-2.4dev) , and in some case people won’t upgrade their environment because the archive keeps the same name.

The setuptools way

Setuptools comes with a continuous integration feature that allows people to push the dev tag in setup.cfg. Then it adds it automatically to the archive name when it is generated.

see: http://peak.telecommunity.com/DevCenter/setuptools#managing-continuous-releases-using-subversion

Building a release will then consist of using the same process than Zope 3 one, except that the tag is removed from the setup.cfg file this time.

Now if you try to release the trunk using disutils sdist or bdist_egg command you will automatically get a dev suffix and the Subversion revision number sticked to it. This means that each new revision can generate a new version that will have a distinct name:

• my.package-2.4dev-r1245
• my.package-2.4dev-r1246
• …

easy_install will grab the latest trunk revision when “my.package-2.4dev” is required, and handle upgrades the right way. This is better that a manual dev tag because when you re-release a new version of the trunk, it will superseeds and therefore upgrades previous revisions.

Another nice feature is to be able to provide to easy_install a subversion link directly, as long as you append the egg full name to the link:

easy_install http://my.svn/my.package/trunk#egg=my.package-dev

collective.releaser

The problem with setup.cfg is that if you forget to remove the dev tag from it when releasing a stable version (you bad boy), you will get the dev-r4565 suffix in the egg name.

collective.releaser takes care of this with a new setuptools command called release, by creating a release branch and removing the dev tag automatically. It upgrades the CHANGES.txt file and version.txt for Plone products.

It also registers and uploads the package to any PyPI-like server. To decide where the package should be sent, it looks at the release-packages variable into the .pypirc, to see if the package name matches it. To release all packages to PyPI a default file would be:

[distutils]
index-servers =
  pypi

[pypi]
username:tarek
password:secret
# regular expression-based variable
release-packages =
  .*

From there, making a release is as simple as:

$ python2.4 setup.py release
running release
This package is version 0.1.1
Do you want to run tests before releasing ? [y/N]: n
Do you want to create the release ? If no, you will just be able to deploy again the current release [y/N]: y
Enter a version [0.1.1]: 0.1.2
Commiting changes...
Creating branches...
...
Running "mregister sdist bdist_egg mupload -r pypi"
running mregister
Using PyPI login from /home/tarek/.pypirc
Registering iw.resourcetraverser to http://pypi.python.org/pypi
...
running mupload
Using PyPI login from /home/tarek/.pypirc
Submitting dist/iw.resourcetraverser-0.1.2.tar.gz to http://pypi.python.org/pypi
Submitting dist/iw.resourcetraverser-0.1.2-py2.4.egg to http://pypi.python.org/pypi
...
0.1.2 released

As described on its PyPI page, it has a plugin system to perform extra steps when a release is made. For instance, it is provided with a mail hook to be able to send mails everytime a release is made.

The tool is pretty new and needs to be smoothed up. For instance, we need to add a few extra controls like making sure long_description is reST-valid. But It works, and is being used by a few fellows in the Plone community already :).

Last, for continuous releases from the trunk, you can set up an alias:

$ python setup.py alias devrelease mregister sdist mupload

You will then be able to upload trunk releases with a single call:

$ python setup.py devrelease

The idea is to provide it as the standard tool to release add-on Products when Plone.org will be upgraded and able to interact with distutils commands. I will promote it through the work I am doing for the PSPS task I am championning (Improve release procedures for add-on products), but it has to live a bit.

Help and opinions welcome !

The main task I worked on during the sprint with Alex and Matthew was making PloneSoftwareCenter ready for the new version of Plone.org. These guys rock. We did tons of things and the new plone.org website is coming up. Alex worked for quite a while on migrating plone.org to plone 3, but let me focus on the software center part.

First of all, let me explain what is the final goal of the work done in the software center.

The future of Plonistas: a 100 % egg-based world and what is means for production

Since a few years, Zope code base was moved into a set of namespaced packages. Plone is following closely. From there zc.buildout is providing a simple way to pick up the right set of packages to build an application. This set is automatically chosen by recipes at Zope and Plone level. Then developers add their own packages and dependencies to provide custom features in their applications.

This mechanism means that each team has to:

• make distributions of packages (tarball, eggs, ..)
• build the application with buildout, then release a source distribution
• and provide a way for other developers to build the application on their own, by:
  • publishing the buildout configuration files
  • pushing the packages the buildout uses into to a server that is reachable by developers
  • make sure Plone and Zope packages are also reachable

This means that each team is responsible at least to release packages.

Distutils and Cheeseshop

Distutils provides two commands to publish packages to the world: register and upload. These commands were intended to provide to the developer a way to push packages to any server that supports the protocol, by using the –repository option.

But in reality, the only server that is publicly available is the Cheeseshop (pypi.python.org). Furthermore Distutils is not providing everything needed to work with another server than the Cheeseshop, like I will explain later in this post. So all Plone and Zope packages are uploaded there.

Therefore, pypi.python.org became a single point of failure when you need to build a Plone or Zope application. With the growth of the community, this means that the repository will get bigger and have to deal with more and more request. PyPI weight around 4 gigas at this time, of zip files and tarballs.

While the actual server is really fast, it makes it a bit hard to work when it is down. It didn’t happen often. Twice as far as I remember. But when it happens, the crowd that uses zc.buildout is frozen because they run buildouts several time per day. They can use cache of course, as long as those are up-to-date.

Some mirrors emerged, like http://release.ingeniweb.com/pypi.python.org-mirror. Some enhanced indexes were created as well, like http://download.zope.org/simple, which is still referring to pypi.python.org but performs a bit of crawling to speed things up when a buildout uses it through easy_install.

But all of these are just enhancement to a cheeseshop-centric model.

Furthermore, in case of a private application, you do not want to see packages in the Cheeseshop but you might also want to provide developers a similar way to manage and use them.

Plone.org software center is dying !

For public applications, given the fact that Distutils provides a simple way to push a package to a public location (one shell command), people have started to stop updating the Products section of Plone.org. This happens because updating it means doing a whole lot more than the simple register+upload call. You need to login into the website, then manually upload the packages, then change the front page of the product if needed etc..

So being able to push packages to plone.org with the same set of commands, is the right solution for developers.

Pushing a package means uploading a public archive but it also means updating the front page for the given package, with the register command.

That what I worked on, at PloneSoftwareCenter level, continuing Sidnei’s work: making it act like Cheeseshop. Now the feature is ready and being alpha-tested at new.plone.org

Toward a distributed model

What we will be able to do from there, is to distribute packages (in blue) to the Cheeseshop (2) as usual, but also to Plone.org (1). Having such a tool also allows people to run other cheeseshop-like servers, wether they are public (3) or private (4). This is useful when you are working on customer projects and do not wish to make their packages available to the world, or even if it is a public project, you do not want to push them to PyPi.

Furthermore, it allows having a bit of redundancy : your packages become available in several places, which is better. Think about the mirrors at Sourceforge, same thing here…

This distributed model is used at Ingeniweb, and in other companies I am starting to list (If you do please let me know, this is important to promote the tool).

The new Plone Software Center (PSC)

If you are extensively working with packages and buildouts, you should consider trying the new PSC, read http://tarekziade.wordpress.com/2008/03/20/how-to-run-your-own-private-pypi-cheeseshop-server/ for this. Each project now in the software center can hold several eggs, and it makes it possible to provide a nice deployment model for your teams: developer push packages releases in such a server with the distutils standards, while customers or deployers can build their applications by picking them up through their buildouts.

And it is in Plone, so you can provide extensive features, like a bug tracker, and all the sweet things Plonecan provide.

The work left to be done in PSC is :

• making the storage for archives (tgz, egg) pluggable so new storing strategies can be provided in separate packages, under the collective.psc namespace, I started this work in a branch.
• finish the collective.psc.mirroring package, that will be used in plone.org to fill a system folder, in order to provide an Apache direct view over the archives stored. This will make zc.buildout / easy_install use this stream rather than hitting the Plone instance. Although the final goal will be to transform it into a storage strategy, but time is running and this is useful now for plone.org.
• extensive tests on new.plone.org

collective.dist

To be able to deal with several servers, I have released collective.dist after the sprint (previously iw.dist). This package provides two new distutils commands: mupload and mregister, together with a new pypirc format.

Follow the documentation, and you will be set in a matter of minutes. This package is already used by many developers to make their life easier, and will help you when the new plone.org goes live.

disutils evolution

collective.dist is an evolution of distutils I have been working at Python level for months. It is available as a Python patch here : http://bugs.python.org/issue1858. mregister and mupload are just modified register and upload commands that makes it possible to interact with several PyPi-like servers, that’s it. PSC uses Distutils standards in any case, so it is possible to use the regular register and upload commands with it of course. It is just not convenient because you will need to have the same username and password on both CheeseShop and the third-party egg server (or change your pypirc file everytime :D)

So the patch, basically, just makes the -r option of distutils commands a 100% operational.

Now my job is to convince Python core developers it should be integrated into Python 2.6, and I am working on this. It is a logical evolution, but it sounds overkill to some people that does not have this kind of need: “Why changing that ? we have only one package server, which is the cheeseshop”.

It can also sound to some of them like I am (together with the team of 20 developers we have ) the only person on earth that wishes it, but as soon as the new Plone.org will be online, a lot of people will start needing such a feature. But having it in 2.6 will make it available as a standard in Plone/Zope world in… a few years.

In any case, most of them agree in the fact that this change is logical and that the current pypirc format is not to be kept. So I guess it is just a matter of time and patience.