Happy New year all !

I am going to make a few posts on the things I would like to achieve in 2009. Each entry will focus on a topic. This one is about Distutils.

I was granted a commit privilege in Python, specifically to work on Distutils maintenance. This is a huge privilege, and I try will do my best in this job. I have worked on a few tickets already and closed some. I learnt the Python development process, which requires to backport and to forward-port changesets in various Python versions. While this can be taken care of automatically by someone else if you don’t do it, it’s better that every commiter takes the time to merge his own work.

So what’s next ?

• There are 132 tickets that are open in the Python tracker, that match the word distutils, and some of them are 5 years old !
• There’s a Python language summit to be held in Chicago right before Pycon, and I volunteered to champion the task about Distutils, PyPI and packaging matters.

I am planning to :

• review and classify all the tickets in the tracker;
• fix the maximum amount of them before the summit;
• make Distutils a first class citizen in test coverage;
• make Distutils code more modern.

Besides, I will try to build a roadmap for Distutils I will present in Chicago.

To build this roadmap, I will ask for input in the distutils-SIG mailing list in the coming days, and see what people will come up with. There’s no crowd in this list these days but sometimes some threads are hot when it comes to the future of packaging in Python.

The roadmap I am planning to build will not address all the issues people have when it comes to distribute a Python application, since there is no consensus yet on the best practices. It will rather try to see if the current version of Distutils can be enhanced to adress some problems, and at least be the bridge to something new in the future. Maybe by including some best practices from third-party tools (the pre-condition for all of this imho is to make the Distutils code base healthier).

Anyway, I hope that the lead developers of: zc.buildout, pip, setuptools, paver (and those projects I forget about right now) will  participate in this discussion, and that we will be able to find pragmatic enhancements.

I have 2 accepted talks at Pycon, that is great. I would like to say that the Pycon review system is awesome because you can see what the reviewers have said, and understand why your talk was accepted or declined.

I was a bit frustrated that my Atomisator talk was declined, but I think it makes sense : this is a new tool, and beside my user group and a few people, it is not really used yet.

One reviewer said that it had to be picked, and another one answered :

I agree that PyCon should not restrict itself to well-known projects, but it should definitely restrict itself to projects that are (a) in production use, (b) under active development, and (c) likely to still be so in a year. There are so many projects meeting these criteria that for me, the bar is very high indeed to spend a talk slot on one that does not.

Ok, fair enough : I will present this talk at Pycon 2010 and they won’t have any argument to decline it

The talks that made it:

• How AlterWay releases web applications using zc.buildout
• On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes

I will get into greater details later on.

The proposal acceptance date is in a few days.

Here are the four proposals I have made:

• The state of packaging in Python. This discussion resumes the current options when it comes to distribute your packages. It also explains the pitfalls and the gap between the Python developers and the OS vendors and packagers. I think this talk will not be picked because the topic is wide and vague. So I proposed to transform it into a panel where lead developers from various framework could explain their usage of distutils and what is missing to make them happy. No feedback yet on this.
• Atomisator, the agile data processing framework. This tool is starting to be useful, and I think it can be useful to others. Check http://atomisator.ziade.org for a quick overview.
• How AlterWay releases web applications using zc.buildout. That is the same talk I gave at the Plone conf but I present it in a way people understand zc.buildout is not tied to Zope and Plone and can be used with any other application. As a matter of fact, it has become a standard here, and we use it for Pylons, etc..
• On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes. That’s a long title. It presents my work on PyPI.

Last, I will go to the Python Language Summit the day before Pycon. I volunteered to be a “champion” on distutils matters.

Yesterday, I bumped into PostRank. This system is collecting data from various social systems like Twitter and provides a service where you can type in an url of a blog post or a entire blog. You get a PostRank depending on the popularity of the URL.

I wrote a plugin for Atomisator and ran it on my own blog. Here’s the result:  http://ziade.org/afpy/

And the Atomisator configuration for this is :

[atomisator]
sources =
    rss http://tarekziade.wordpress.com/feed/atom/

database = sqlite:///carpet.db

outputs =
    rss  public/rss.xml "http://tarekziade.wordpress.com/feed/atom/" "Carpet Python with PR" "Powered by Atomisator"

enhancers =
    postrank

How PostRank works

PostRank works with urls you provide, on their web interface or through their web services.

As long as these url are present in their big cloud-computing based system, they provide a rank that is calculated with the number of comments related to the blog, the number of tweet messages that refers to it, and so on. The complete algorithm they used is secret but this is not the point. I have secret algorithms too ;).

The point is that they are trying to categorize blog entries using social networks as indicators, and that they have a huge database.

Social indicators in Atomisator

This is one of the approach I have with Atomisator, when it is used to build a planet. For instance I have a Digg plugin that will inject in each entry the comments found on Digg if the entry was digged. It also present the number of Digg. Of course this is done live because I don’t have a cloud-computing based system where I store data. I use Digg webservice on the fly. (On the fly here doesn’t mean Atomisator make the calls to Digg from the Planet application of course. It means Atomisator calls them when it creates the merged feed on the system)

The benefit of this approach is that I can provide a social indicator on a post immediatly. Systems like PostRank will not work on entries that are too recent because their spiders have a lag of one week or so.

The pitfall of my approach is that I am unable to calculate trends because I don’t store the indicators as they vary.

But if someone wanted to build a BtoC application using Atomisator, they could implement a set of plugins based on Amazon tools to make them store data in a more scalable way and in time.

Next steps

So I have this new PostRank plugin, and this is awesome because I have added a treshold parameter in it. Basically if a post has a high PostRank value, it will appear in the Planet. If it’s low, it can be automatically removed. The fact that PostRanks are lagging for new entries is not a problem: interesting posts will eventually pop after a few days in the Planet.

This is perfect to reduce the number of entries in an aggregator.

But I do want to write my own PostRank that works live, with no storage at all. Because the whole point of Atomisator is to provide a framework where anyone can try out various filtering combinations.

So to be able to provide this power, it needs to work just by collecting data directly from the social services, like the PostRank plugin does with this PostRank “meta-service”. The next step is therefore to see if I can query services like Twitter to list the twits related to an url, without having to store the twitter feed myself.

In any case, if my talk on Atomisator at Pycon 2009 is selected, the PostRank plugin will be shown besides the Digg plugin.

We made the shirts at home for PyCon FR and that was a LOT of fun, even if we had to work hard to make sure all shirt were ready by the time the event started. We used a silk-screen technique and the output looks really professional. The ink used makes the shirt design last longer than shirts that uses transfer. Although the number of colors is limited.

Have a look at the pictures : http://www.flickr.com/photos/82007723@N00/sets/72157605181699082/

Guess what: some people ordered us some t-shirts for some other events, so some more fun ahead !

Pycon FR will take place in Paris, may 17/18, at the Cité des sciences et de la Villette.

A very rich program will be presented:

• Learn Python first ! (Eric LEBIGOT)
• Graphviz made easier with GvGen (Sebastien Tricaud)
• Genes research with Python (Andre Espaze)
• CouchDB (Benoit Chesneau)
• PyPy explained (Victor Stinner)
• Quality Assurance (Julien Jehannet)
• Zope 3 overview (Christophe Combelles)
• Scapy in action (Renaud Lifchitz)
• E-sonoclaste, multimedia annotations (Vincent Rioux)
• Why Django ? (David Larlet)
• PLUIE project (Michel Claveau)
• WSGI in practice using Paste (Gael Pasgrimaud)
• Distributed Version Control with Mercurial (Michael SCHERER)
• Create and deploy Python apps with zc.buildout (Tarek Ziadé)
• Django everyday: quality and performance (David Larlet)
• Python 3000 (Victor Stinner)

There will be lightning talks as well, and the French speaking user group (Afpy) will have its annual meeting on sunday morning. A second room will be available for sprints and BoF sessions as well. All the talks will be in French this year

If you are around Paris, please join use ! It free !

Otherwise, watch the talks in the live stream: http://fr.pycon.org/stream-live

Thanks to our partners, for making this happen:

• The Python Software Foundation
• Logilab
• Nerim
• Ingeniweb
• Resolver Systems
• Pilot Systems
• WingWare
• Makina Corpus
• Toonux
• ActiveState
• Gymglish
• Eyrolles
• Cité des Sciences et de la Vilette