We had an excellent brainstorming session today in the hall, with Toshio, Georg, Martin, Thomas, etc.. (sorry we were so many I don’t have the full list) with some insights from Guido and Brett. We tried to think about a way to handle multiple versions of a same package.

Here’s the two most important concepts :

• Unicity: There should be one and only one instance of a Python package at a given version on a system
• Combination: One Python application combines several packages to run

About unicity

A Python package is a component that can be installed on a system. If you use the standard Distutils approach, it will end up in the Python site-packages directory and be importable by the interpreter. This package comes with a version number and is unique.

This unicity is important for security and maintainability. For instance, if there’s a security hole in a package, the fix is applied in one place and the system maintainer knows it can’t be present elsewhere on the system.

This is the system administrator point of view

About Combination

What defines a Python application is the fact that is selects a list of packages it needs to run. And this varies for every application. So two Python applications might need a different version of a given package and that is normal.

What’s important is to have the right list of packages when an application loads. Tools like zc.buildout or virtualenv are perfect for this need : they create an isolated environment for your application to run with the right set of packages.

So the simplest way to release an application is to ship it with everything required, regardless the unicity.

This is the application developer point of view

The idea

zc.buildout and virtualenv are a blast for developers, and another thing system packagers might dislike. This is because they break the unicity by allowing developers to ship their applications as black boxes. Of course one may say that this is perfectly fine since what’s inside an application is not the problem of the system packager. But since this application is made of packages that may be shared on the system by other applications, that is redondant.

Forget site-packages for a moment. And let’s think about a new loading system for packages. This approach is similar in some ways to setuptools’ multiple version system.

Storing multiple versions

First of all, let’s store the packages in a directory, and for each version of the package, under a sub-directory which name is the version.

For example:

• SQLAlchemy
  • 0.4
    • package code is here
    • package egg-info here
  • 0.5
    • package code is here
    • package egg-info here
• jsonlib
  • 1.2.6
    • package code is here
    • package egg-info here
  • 1.3.10
    • package code is here
    • package egg-info here

Given this structure, some mechanism can provide to the interpreter the latest available version of a package, as long as Distutils knows how to handle version comparisons correctly (which will be the case in a near future)

Choosing specific versions

Back to our application. Let’s call it MyApp, version 1.0.

It needs specific versions for some packages. Forget zc.buildout and pip for a moment. And let’s think about a different way to express the packages (and versions) its needs.

Let’s make a Python package for this application and let’s make a few assumptions on some features in Distutils:

• setuptools’ install_requires has made it into Distutils, as part of metadata.
• metadata are defined statically in a package, apart from setup.py.

So basically, the application is mainly a static list of dependencies defined into install_requires. For example:

• SQLAlchemy > 0.4
• jsonlib == 1.2.6

When MyApp will get installed by Distutils, it will be added in the packages tree.

When it is used, it will need to load the versions of SQLAlchemy and jsonlib it needs. The ones that are inside its metadata.

To make it possible, the script that launches the application calls a built-in function called read_deps, that takes the metadata and reads them to know which versions fit:

# the script
read_deps('PKG-INFO')

import SQLAlchemy
import jsonlib
...

This call will load the right versions in the packages tree:

• SQLAlchemy
  • 0.4
  • 0.5
• jsonlib
  • 1.2.6
  • 1.3.10
• MyApp
  • 1.0

What’s next

I just dropped here the rough idea, and a lot of details are missing. But I think it’s a good thing to share this idea here in its early stage.

We have decided we would try to write a prototype using importlib and sys.meta_path. Maybe Georg will start it during the Pycon sprint, and start to digg into the details. He was working on this when I left the open session at Pycon.

Stay tuned

Before I sit down and clean up my summit notes to send them to python-dev, I wanted to post an entry about a small project which I think could be a great task for a student at the Summer of Code (I doubt it can fill 4 months of work but it could be done amongst other tasks).

Yesterday, we did a late session with Martin von Loewis, Jim Fulton and Georg Brandl about PyPI and the fact that it needed a better way to handle passwords on client side. That is, the fact that you have to store your password in the .pypirc file if you want to upload your package to PyPI.

This is unsafe and unwanted. A few months ago, I have made a small change in Distutils so it would prompt for the password using the getpass module if it doesn’t find it in the .pypirc file. (This was a contrbution of Nathan Van Gheem).

Anyways, this is not enough. Jim suggested to set up a SSH server on PyPI using Paramiko, so we could use a standard ssh connection and benefit from ssh-agent. But this is unfortunately not universal.

So let me get back to the idea I sent some time ago : http://mail.python.org/pipermail/python-ideas/2009-January/002465.html

What about having an option in getpass to store and reuse passwords in
system keyrings ?

    getpass(prompt[, stream])

would become:

    getpass(prompt[, stream, keyring])

where keyring would be a callable that can be use to retrieve the
password from a keyring system
and store it the first time.

The getpass module could provide some keyring support for:

- ssh-agent under Linux
- keychain under Mac OS X
- ...
ss

And let the developers use their own keyring system by providing a callable.

As Greg Smith said in the thread, the first task is to create a library that supports all standard keyring systems out there, including things like KWallet, Internet Explorer, Fireforx and so on…

I’ll mentor this project if any student would like to do it.

Tomorrow is the Language Summit, yeepee.

The package and distribution part of the Summit is going to be great since we have key people coming up.

zc.buildout and pip leaders (Jim Fulton and Ian Bicking) will be present, and many others. I’ll be representing Distutils, since I am its current maintainer. Unfortunately Philip Eby (setuptools) can’t make it, but he should be reachable via IRC (I am trying to set something up for tomorrow).

Anyway, one of the goal of the Summit is to validate the new features and enhancements we want to introduce in Distutils and PyPI. It’s important to make sure they play well with third-party tools like zc.buildout, setuptools and pip. We also need to make sure these tools will evolve in the same direction in the future.

We have reached a point in Python where we need to concentrate all the packaging effort to build a common standard in the standard library, because it is badly needed.

Here’s a draft of the slides I will present tomorrow, during the first 5/10 minutes, as a session leader:

• Packaging Survey results overview
• Topics to discuss
  • setting up an organized network of mirrors (see PEP 381)
  • discuss about other PyPI enhancements
  • improve the package installation / uninstallation  (see PEP 376)
  • discuss the package dependencies problem and see if we can come up with a PEP
  • discuss the multiple version problem and see if we can come up with a PEP
  • discuss the isolated environment vs the OS vendor approach and see what can be done to improve their coexistence.

Summit schedule:

• 13:20 -> 13:30 : presentation
• 13:30 -> 14:30 : discussions/work in small groups
• 14:30 -> 14:50 : “tour de table”
• 14:50 -> 15:10 : break
• 15:10 -> onwards: sprint !

I am not sure about the ‘work in small group’ part yet, because I don’t know how many people will show up, and what people will want to focus on.

Around 570 people answered the survey, which is a great number I didn’t expect. Thanks again to Massimo for his help on this.

I have a lot of work to read all the open question answers, and all the comments that goes with the “other” answer, but I wanted to publish the results of the closed questions before the summit.

I don’t want to comment the results yet. I will after I have studied all answers, so it’ll be a little while

Who are you ?

Where are you located ?

If you are a web programmer, what is the framework you use the most ?

How do you organize your application code most of the time ?

For libraries you don’t distribute publicly, do you you create a setup.py script ?

What is the main tool or combination of tools you are using to package and distribute your Python application ?

How do you install a package that does not provide a standalone installer (but provides a standard setup.py script) most of the time ?

How do you remove a package ?

How do you manage using more than one version of a library on a system ?

Do you work with setuptools’ namespace packages ?

Has PyPI become mandatory in your everyday work (if you use zc.buildout for example) ?

If you previously answered Yes, did you set up an alternative solution (mirror, cache..) in case PyPI is down ?

Do you register your packages on PyPI ?

Do you upload your package on PyPI ?

If you previously answered No, how do you distribute your packages ?

I have the opportunity to lead a session at the Python Language Summit at Pycon.

In order to prepare this event, I am currently building a survey because we need to know how people use Distutils, what is wrong with it, and so on…

The draft is here : http://wiki.python.org/moin/Packaging%20Survey

When it’s ready (hopefully soon), I’ll post it in a SurveyMonkey-like system online for people to take it, and synthetize the results, so what I am saying at the summit reflects hopefully the reality.

If you are a Python developer, an OS packager, come to the Distutils-SIG mailing list and help us build the best survey possible !

(while the draft is in a wiki, it’s better to discuss the changes in the mailing list before it is applied)

Happy New year all !

I am going to make a few posts on the things I would like to achieve in 2009. Each entry will focus on a topic. This one is about Distutils.

I was granted a commit privilege in Python, specifically to work on Distutils maintenance. This is a huge privilege, and I try will do my best in this job. I have worked on a few tickets already and closed some. I learnt the Python development process, which requires to backport and to forward-port changesets in various Python versions. While this can be taken care of automatically by someone else if you don’t do it, it’s better that every commiter takes the time to merge his own work.

So what’s next ?

• There are 132 tickets that are open in the Python tracker, that match the word distutils, and some of them are 5 years old !
• There’s a Python language summit to be held in Chicago right before Pycon, and I volunteered to champion the task about Distutils, PyPI and packaging matters.

I am planning to :

• review and classify all the tickets in the tracker;
• fix the maximum amount of them before the summit;
• make Distutils a first class citizen in test coverage;
• make Distutils code more modern.

Besides, I will try to build a roadmap for Distutils I will present in Chicago.

To build this roadmap, I will ask for input in the distutils-SIG mailing list in the coming days, and see what people will come up with. There’s no crowd in this list these days but sometimes some threads are hot when it comes to the future of packaging in Python.

The roadmap I am planning to build will not address all the issues people have when it comes to distribute a Python application, since there is no consensus yet on the best practices. It will rather try to see if the current version of Distutils can be enhanced to adress some problems, and at least be the bridge to something new in the future. Maybe by including some best practices from third-party tools (the pre-condition for all of this imho is to make the Distutils code base healthier).

Anyway, I hope that the lead developers of: zc.buildout, pip, setuptools, paver (and those projects I forget about right now) will  participate in this discussion, and that we will be able to find pragmatic enhancements.

I have 2 accepted talks at Pycon, that is great. I would like to say that the Pycon review system is awesome because you can see what the reviewers have said, and understand why your talk was accepted or declined.

I was a bit frustrated that my Atomisator talk was declined, but I think it makes sense : this is a new tool, and beside my user group and a few people, it is not really used yet.

One reviewer said that it had to be picked, and another one answered :

I agree that PyCon should not restrict itself to well-known projects, but it should definitely restrict itself to projects that are (a) in production use, (b) under active development, and (c) likely to still be so in a year. There are so many projects meeting these criteria that for me, the bar is very high indeed to spend a talk slot on one that does not.

Ok, fair enough : I will present this talk at Pycon 2010 and they won’t have any argument to decline it

The talks that made it:

• How AlterWay releases web applications using zc.buildout
• On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes

I will get into greater details later on.

The proposal acceptance date is in a few days.

Here are the four proposals I have made:

• The state of packaging in Python. This discussion resumes the current options when it comes to distribute your packages. It also explains the pitfalls and the gap between the Python developers and the OS vendors and packagers. I think this talk will not be picked because the topic is wide and vague. So I proposed to transform it into a panel where lead developers from various framework could explain their usage of distutils and what is missing to make them happy. No feedback yet on this.
• Atomisator, the agile data processing framework. This tool is starting to be useful, and I think it can be useful to others. Check http://atomisator.ziade.org for a quick overview.
• How AlterWay releases web applications using zc.buildout. That is the same talk I gave at the Plone conf but I present it in a way people understand zc.buildout is not tied to Zope and Plone and can be used with any other application. As a matter of fact, it has become a standard here, and we use it for Pylons, etc..
• On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes. That’s a long title. It presents my work on PyPI.

Last, I will go to the Python Language Summit the day before Pycon. I volunteered to be a “champion” on distutils matters.

Yesterday, I bumped into PostRank. This system is collecting data from various social systems like Twitter and provides a service where you can type in an url of a blog post or a entire blog. You get a PostRank depending on the popularity of the URL.

I wrote a plugin for Atomisator and ran it on my own blog. Here’s the result:  http://ziade.org/afpy/

And the Atomisator configuration for this is :

[atomisator]
sources =
    rss http://tarekziade.wordpress.com/feed/atom/

database = sqlite:///carpet.db

outputs =
    rss  public/rss.xml "http://tarekziade.wordpress.com/feed/atom/" "Carpet Python with PR" "Powered by Atomisator"

enhancers =
    postrank

How PostRank works

PostRank works with urls you provide, on their web interface or through their web services.

As long as these url are present in their big cloud-computing based system, they provide a rank that is calculated with the number of comments related to the blog, the number of tweet messages that refers to it, and so on. The complete algorithm they used is secret but this is not the point. I have secret algorithms too .

The point is that they are trying to categorize blog entries using social networks as indicators, and that they have a huge database.

Social indicators in Atomisator

This is one of the approach I have with Atomisator, when it is used to build a planet. For instance I have a Digg plugin that will inject in each entry the comments found on Digg if the entry was digged. It also present the number of Digg. Of course this is done live because I don’t have a cloud-computing based system where I store data. I use Digg webservice on the fly. (On the fly here doesn’t mean Atomisator make the calls to Digg from the Planet application of course. It means Atomisator calls them when it creates the merged feed on the system)

The benefit of this approach is that I can provide a social indicator on a post immediatly. Systems like PostRank will not work on entries that are too recent because their spiders have a lag of one week or so.

The pitfall of my approach is that I am unable to calculate trends because I don’t store the indicators as they vary.

But if someone wanted to build a BtoC application using Atomisator, they could implement a set of plugins based on Amazon tools to make them store data in a more scalable way and in time.

Next steps

So I have this new PostRank plugin, and this is awesome because I have added a treshold parameter in it. Basically if a post has a high PostRank value, it will appear in the Planet. If it’s low, it can be automatically removed. The fact that PostRanks are lagging for new entries is not a problem: interesting posts will eventually pop after a few days in the Planet.

This is perfect to reduce the number of entries in an aggregator.

But I do want to write my own PostRank that works live, with no storage at all. Because the whole point of Atomisator is to provide a framework where anyone can try out various filtering combinations.

So to be able to provide this power, it needs to work just by collecting data directly from the social services, like the PostRank plugin does with this PostRank “meta-service”. The next step is therefore to see if I can query services like Twitter to list the twits related to an url, without having to store the twitter feed myself.

In any case, if my talk on Atomisator at Pycon 2009 is selected, the PostRank plugin will be shown besides the Digg plugin.

We made the shirts at home for PyCon FR and that was a LOT of fun, even if we had to work hard to make sure all shirt were ready by the time the event started. We used a silk-screen technique and the output looks really professional. The ink used makes the shirt design last longer than shirts that uses transfer. Although the number of colors is limited.

Have a look at the pictures : http://www.flickr.com/photos/82007723@N00/sets/72157605181699082/

Guess what: some people ordered us some t-shirts for some other events, so some more fun ahead !

Pycon FR will take place in Paris, may 17/18, at the Cité des sciences et de la Villette.

A very rich program will be presented:

• Learn Python first ! (Eric LEBIGOT)
• Graphviz made easier with GvGen (Sebastien Tricaud)
• Genes research with Python (Andre Espaze)
• CouchDB (Benoit Chesneau)
• PyPy explained (Victor Stinner)
• Quality Assurance (Julien Jehannet)
• Zope 3 overview (Christophe Combelles)
• Scapy in action (Renaud Lifchitz)
• E-sonoclaste, multimedia annotations (Vincent Rioux)
• Why Django ? (David Larlet)
• PLUIE project (Michel Claveau)
• WSGI in practice using Paste (Gael Pasgrimaud)
• Distributed Version Control with Mercurial (Michael SCHERER)
• Create and deploy Python apps with zc.buildout (Tarek Ziadé)
• Django everyday: quality and performance (David Larlet)
• Python 3000 (Victor Stinner)

There will be lightning talks as well, and the French speaking user group (Afpy) will have its annual meeting on sunday morning. A second room will be available for sprints and BoF sessions as well. All the talks will be in French this year

If you are around Paris, please join use ! It free !

Otherwise, watch the talks in the live stream: http://fr.pycon.org/stream-live

Thanks to our partners, for making this happen:

• The Python Software Foundation
• Logilab
• Nerim
• Ingeniweb
• Resolver Systems
• Pilot Systems
• WingWare
• Makina Corpus
• Toonux
• ActiveState
• Gymglish
• Eyrolles
• Cité des Sciences et de la Vilette

edit: I have changed the blog title, which was not correct in English, thanks Maik

I wanted to react on Martinj’s blog to make things clear about Pycon FR

Background

First of all here’s a little background about Pycon FR.

Afpy (Association Francophone Python) is a french Python user group created four years ago. The group counts around 100 members and is the only one of this size in french-speaking countries as far as we know. It has members from France, Belgium, Canada and North Africa countries.

Many meetings are organized throughout the year, mainly in Paris, and sometimes in Charleroi in Belgium.

An important event took place at the “Cité des sciences” in Paris last summer, called “Journées Python” We had around 80 attendees and a rich program was given during two days. Topics where presented by french speaking people from France and Belgium, and we had talks about AI, Django, Zope, TuxDroid, etc

Some videos are available on the web, you can probably reach them from here: http://video.google.fr/videosearch?q=journees.afpy.org

The conference was 100% free for attendees. We also decided to chose carefully the dates so the conference would not compete with other events like Europython.

This event was a *true* success.

This event is now called Pycon FR, to be in the same line of what happens in some other countries. This makes a lot of sense indeed, to have national Python events, with standardized names.

Attendees

Pycon FR attendees are mainly people that speaks english but for some of them not enough to easily follow English talks. Most of those people would not make it to Europython because of this language barrier, and a national conference held in Paris is something that was quite missing at this time, for them.

So PyCon FR is just a sign of the growing interest about Python, and should be seen as a 100% positive and constructive thing, believe me.

Together with Europython and Pycon US, the “local” PyCons are in my opinion, a very good thing to spread the good word. We will, like last year, promote Europython there. Maybe we could even do a survey this year to know exactly who goes to Europython, etc.

Pycon FR expansion plans

When I said in my last blog entry that we were working on making it less french-centric, and having international attendees, I should have explained it better to make sure no one thinks Pycon FR might be a bad thing for Europython.

• If we do invite international speakers, it will be done only if we are able to provide a live translation system.
• The event date will always be picked carefully, as usual, so it does not compete with Europython.

Europython and local PyCons expansion

Here’s my opinion about what Europython strategy could be: since it is an international english-speaking event that gathers a lot of people. It could be the seed to set up a local Pycon event in places where it does not exist yet, using the momentum of the event.

This could be done by working with the local user group this way, by organizing maybe two years in a row Europython there. So, maybe we could see “Pycon Lithunia” next year ? Then move EuroPython to Amsterdam and see “Pycon The Netherlands” in two years ?

Europython exists to promote the language at an international level, so that could be a good way to spread imho.

But please, pretty please, don’t see local Pycons as Europython competitors, this is not true, this happens just because Python is growing.

Everyone should be happy about that I think. I disagree with the fact that each conference might fight to have the “best international speakers” for a very simple reason: the number of speakers is growing too, and I don’t think either Pycon either EuroPython will ever miss of interesting talks.

The next smart move I think is to organize and synchronize all these events, and that is where the Python Software Foundation has a role to play in my opinion. They have started anyway, and I am really confident about what will happen next.

PyCon FR is coming up ! We published yesterday the final program, and I must say that this second annual meeting of Pythoneers in Paris, looks very good.

Extracts:

• Why you should learn Python
• GraphViz with GvGen
• Gene search
• CouchDB
• PyPy
• Quality Assurance
• Scapy
• Zope 3
• zc.buildout
• Django
• WSGI
• etc…

This is going to be two intense days in Paris, so if you are around, please join us, it’s free.

Paris - Cité des Sciences et de la Vilette - 16/17 May .

The talks will be in french, but the organization team works hard to have english talks as well next year. Anyway, you don’t need to speak french to join us and have a good time