We have a simple need for a customer project that runs a Zeo server and a few Zeo clients : being able to check the status of every Zeo client, and monitor what they are doing.

DeadlockDebugger almost provides this feature since it is able to produce a dump of the execution stack for every thread a Zope instance is running.

Based on this tool, I have developed ZopeHealthWatcher, that provides a console script to query a Zope instance, and get back a status for every running thread. It tells you if the thread is idling or if it’s running some code. The script also returns an exit code depending on the number of busy threads, so it can be used in tools like Nagios.

When there are 4 or more busy threads, the script will return the execution stacks for every busy thread and some extra info like the system load and memory info. The returned info will be extendable through plug-ins in the next version, but right now the provided info are enough for our needs.

I have also created an HTML version, so when the dump is requested from another tool than the console script (e.g. a browser), it displays a nice human-readable interface (check the PyPI page for more info and a screenshot).

Notice that DeadLockDebugger is hackish since it patches the Zope publisher at startup. But we won’t change this part: we need this tool to run from the oldest to the newest Zope 2 version. And the patch just works fine, so…

The provided version should run out of the box in a buildout-based Plone 3 application, but requires manual installation steps on older Plone or CPS versions.

I didn’t mention these manual steps in the documentation. I think I am the only person in the world interested in running this tool on the dead-but-still-in-production-in-many-places Nuxeo CPS.

By the way: kudos goes to Marc-Aurèle Darche, who is maintaining CPS for years now, making it one of the most bug-free and stable CMS solution out there.  Ok it’s probably easier to reach this level of quality since the platform is very stable and only evolves very slowly thanks to Georges Racinet.

I have 2 accepted talks at Pycon, that is great. I would like to say that the Pycon review system is awesome because you can see what the reviewers have said, and understand why your talk was accepted or declined.

I was a bit frustrated that my Atomisator talk was declined, but I think it makes sense : this is a new tool, and beside my user group and a few people, it is not really used yet.

One reviewer said that it had to be picked, and another one answered :

I agree that PyCon should not restrict itself to well-known projects, but it should definitely restrict itself to projects that are (a) in production use, (b) under active development, and (c) likely to still be so in a year. There are so many projects meeting these criteria that for me, the bar is very high indeed to spend a talk slot on one that does not.

Ok, fair enough : I will present this talk at Pycon 2010 and they won’t have any argument to decline it

The talks that made it:

• How AlterWay releases web applications using zc.buildout
• On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes

I will get into greater details later on.

Here’s a proposal I will send to the python-dev. What do you think ?

(Disclaimer : this proposal is highly inspired from the work done by people in various tools, it does not reinvent anything)

The problem

Python developers distribute and deploy their packages using myriads of dependencies. Some of them are not yet available as official OS python packages.  Even sometimes one package conflicts with the official version of a package installed in a given OS.

In any case, the cycle of development of most Python applications is shorter than the release cycle of Linux distributions, so it is impossible for application Foo to wait that Bar 5.6 is officialy available in Debian 4.x.

Therefore, there’s a need to provide or describe a specific list of dependencies for their application to work.

And this list of dependency might conflict with the existing list of packages installed in Python. In other words, even if this is not a wanted behavior from an os packager point of view, an application might need to provide its own execution context.

Right now, when Python is loaded, it uses the site module to browse the site-packages directory to populate the path with packages it find there.  .pth files are also parsed to provide extra paths.

Python 2.6 has introduced per-user site-packages directory, where you can define an extra directory, which is added in the path like the central one.

But both will append new paths to the environment without any rule of exclusion or version checking.

The workarounds

A few workarounds exist to be able to express what packages (and version) an application needs to run, or to set up an isolated environment for it:

• setuptools provides the install_requires mechanism where you can define dependencies directly inside the package, as a new metadata. It also provides a way to install two different versions of one package and let you pick by code or when the program starts, which one you want to activate.
• virtualenv will let you create an isolated Python environment, where you can define your own site-packages. This allows you to make sure you are not conflicting with a incompatible version of a given package.
• zc.buildout relies on setuptools and provides an isolated environment a bit similar in some aspects to virtualenv.
• pip provides a way to describe requirements in a file, which can be used to define bundles, which are very similar to what zc.buildout provides.

But they all aim at the same goal : define a specific execution context for a specific application, and declare dependencies with no respect to other applications or to the OS environment.

This proposal describes a solution that can be added to Python to provide that feature.

The solution

A isolated environment file that describes dependencies is added. This file can be tweaked by the application packager, or later by the OS packager if something goes wrong.

The isolated environment file

A new file called a  Python Isolated Environment file (PIE file) can be provided by any  application to define the list of dependencies and their versions.

It is a simple text file with a first line that provides :

• a list of paths, separated by ‘:’, on line 1
• then one package per line, starting at line 2. each package can be prefixed by a `!`

For example:

/var/myapp/myenv
lxml
sqlite
sqlalchemy
!sqlobject

This list of packages might or might not be installed in Python.

Versions can be provided as well in this file :

/var/myapp/myenv:/var/myapp/myenv2
lxml >= 0.9
sqlite > 1.8
sqlalchemy == 0.7
!sqlobject == 0.6

The file is saved with the pie extension,

Loading an isolated environment file

A new function called load_isolated_environment is added in site.py, that let you load a PIE file.

Loading a PIE file means:

• for each package defined, starting at line 2, load_isolated_environment will look into the environment if the package with the particular version exists. The version is given by the package.__version__ value or the PKG-INFO one when available. If the package exists but the version is not available, the version 0.0 is used.
• for packages without the ! prefix:
  • if the  package is not found, it will scan each path provided on line 1 of the file, using the site-packages method, looking for that package.
  • if the package is found, it is added in the path.
  • if the package is not found, a PackageMissing error is raised.
• for packages starting with the ! prefix:
  • if the  package is found, it is removed from the path

This function can be called by code like this:

>>> from site import load_isolated_environment
>>> load_isolated_environment('/path/to/context.pie')

From there, sys.path meets the requirements and the code that is executed after this call will benefit from this context.
Another context can be loaded in the same process :

>>> load_isolated_environment('/path/to/another_context.pie')

Limitations:

• if the new context brakes other programs in the process. It’s up to the application packager to fix the context file.
• it’s not the job of load_isolated_environment to resolve dependencies issues : if the foo package needs the bar package, it won’t complain.
• it is not the job of load_isolated_environment to get missing dependencies.

Using an isolated environment file

Typically, an isolated environment file can be used into high-level Python scripts. For example, any script an application provides to be launched :

# this script runs zope
from site import load_isolated_environment
load_isolated_environment('zope-3.4.pie')

import zope

if __name__ == '__main__':
    zope.run()

The proposal acceptance date is in a few days.

Here are the four proposals I have made:

• The state of packaging in Python. This discussion resumes the current options when it comes to distribute your packages. It also explains the pitfalls and the gap between the Python developers and the OS vendors and packagers. I think this talk will not be picked because the topic is wide and vague. So I proposed to transform it into a panel where lead developers from various framework could explain their usage of distutils and what is missing to make them happy. No feedback yet on this.
• Atomisator, the agile data processing framework. This tool is starting to be useful, and I think it can be useful to others. Check http://atomisator.ziade.org for a quick overview.
• How AlterWay releases web applications using zc.buildout. That is the same talk I gave at the Plone conf but I present it in a way people understand zc.buildout is not tied to Zope and Plone and can be used with any other application. As a matter of fact, it has become a standard here, and we use it for Pylons, etc..
• On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes. That’s a long title. It presents my work on PyPI.

Last, I will go to the Python Language Summit the day before Pycon. I volunteered to be a “champion” on distutils matters.

I love Packt. As soon as I have told them that some people liked the book but complained about the typos, they proposed to go ahead and launch a new print cycle.

Basically it means that the next buyers will have a typo-free book. At least for all the typos that were reported on my Trac here, or at Packt’s.

I am currently processing all the typos reported at Packt so I have a full list on my wiki, and will provide them the final list tomorrow.

Then, they will re-print it.

So if you already own the book, and you see a typo that is not listed, please let me know.

I found a bit of time to work on distribution matters. Here’s a status of what I am doing there.

There are two topics I am focusing on right now.

• clean up and enhance Python’s distutils package
• implement the mirroring infrastructure at PyPI

distutils work

Nathan Van Gheem proposed a cool patch in collective.dist, (this package is a port of the new features I have added in distutils so they are available in 2.4 and 2.5).

Nathan proposed a patch to be able to avoid the storage of the password in the .pypirc file. The prompt is used in that case. This is something that was in my pile for a long time.

I have added a few things to Nathan’s patch, and a test, and proposed it to Python. I am now waiting for its integration in 2.7 trunk: http://bugs.python.org/issue4394. If it’s accepted, I will backport it to collective.dist.

There are some other tickets I am waiting to be accepted:

• http://bugs.python.org/issue4400
• http://bugs.python.org/issue2461
• http://bugs.python.org/issue3992
• http://bugs.python.org/issue3985
• http://bugs.python.org/issue3986

I am not sure when those will be integrated. The average time for the integration of tickets in distutils in Python is between 6 months and 8 months. hihihi.

PyPI mirroring

The job I am doing in PyPI will be in three phase :

• Phase 1: implement the mirroring infrastructure in PyPI
• Phase 2: promote it, and propose patches for the mirroring tools out there so they use the protocol
• Phase 3: promote and propose patches for pip so it can use the mirrors efficiently (fail-over and nearest mirror infrastructure).

Phase 1: so far, so good.

With some insights from Richard Jones and Martin von Löwis, I am currently implementing the mirroring infrastructure for PyPI we have defined during the D.C. sprint (I still owe a blog entry about this sprint). The code lives in a branch on the python svn folder dedicated to PyPI.

The idea of the mirroring infrastructure is to be able to get a list of official mirrors for PyPI, that can be used as alternatives sources . (It is described here: http://wiki.python.org/moin/PEP_374). A great behavior could be that the client application interacts with the nearest mirror location automatically, and switch to another if it goes down.

So, a list of mirrors will be made available at /mirrors, and the client applications will be able from there to use an alternative location for every package. The hardest part concerns the stats : we want to display in PyPI the download counts for each package by summing downloads from every mirror.

So every mirror will have to provide its “local stats” that can be visited by PyPI. That’s the biggest part of the work I am doing. It will build the stats for PyPI by parsing its Apache log file. And hopefully, this code should be reusable by the mirrors themselve so they can build their stats the same way.

Of course this infrastructure could be used for any PyPI-compatible server even if is not a mirror of PyPI (like a private PyPI server)

Phase 2 will consist in promoting the infrastructure to the mirroring softwares out there. Maybe Pycon will be a good place for that.

Phase 3 is the most interesting one : make sure the client applications use the mirrors ! I think Ian Bicking’s pip project could be the right place for these innovations.

Next topics in the pile:

• index-merging: describe in a PEP-like document the index-merging feature that would allow clients to merge several indexes with a content that differe. For example: PyPI + a private PyPI server. I have written a first draft of such a patch in setuptools in the past (http://bugs.python.org/setuptools/issue32) but I have lost all my hopes to see this project moving forward lately.
• Brainstorming: try to understand the Python Packaging Paradox. That is = how come the community, which is composed of many briliant people, is unable to move forward in packaging matters.
• Distribute the return

I feel really bad about this comment on my book : How To Be Dissappointed in Something You Recommend.

Just a quick word about the try, return finally code pattern, since I had some feedback about it. I would like to mention that this code pattern is perfectly right:

def function():
    try:
      return something
    finally:
      do something

I should have explained it better, because this pattern is not used a lot by people, so you can think that “do something” is called after the return of the function, which is not the case.

For the typos now:

The first thing I did wrong: when I started the book, I wanted, as I did in my previous book, to run unit tests on the book itself to avoid those mistakes. That said, the previous one was in Latex, which is quite simple to interact with, and this one is in OpenOffice, because that is how the editor works. I had to write a script to extract the Python code from the Ooo file, to unit test it. I didn’t. I simply ran out of time, as usual when you have deadlines on books.

The second thing I did wrong: I should have told the editor to wait a bit, I didn’t.

But Packt does Print On Demand, so I know that the Errata page I am maintaining here : http://atomisator.ziade.org/wiki/Errata, is being processed by the editor, and that the typos will be removed from the book at some point, without having to wait for a second edition.

I’ll update this blog entry as soon as I know the status on this.

I am really sorry Calvin, and all the people that are suffering from these typos.

I like Google Alert, the idea of receiving a mail every day that summarizes all articles related to a given topic is really helpfull when you need to focus on a specific subject for a while.

But this is not enough. I want to receive a mail that points me to any mailing list or planet feed or blogs out there as well, that talks about the topic.

You can’t do it with Google Alerts as far as I know.

Let’s take an example:

I want to receive a daily mail that points me to any mail thread or blog entry, that is related to the word “buildout” or to the word “pycon”.

Basically, to do it manually, I need to read Planet Python, Planet Zope, then take a look at the Python, Zope and Plone mailing lists. It takes at least 10 minutes, and more if you want to read all entries to make sure you won’t miss anything.

Since online systems like Nabble provides RSS feed for mailing lists (don’t find yours ? just add it there !), it is easy to read them as they where regular feeds.

From there, a script that reads all the selected feeds and sends a mail pointing to the entries that match the selected words is simple to write as well, and fill the need.

But don’t code it : Atomisator will let you do this with a few lines of configuration.

Here’s a step-by-step tutorial.

Step 1 - install easy_install

• download ez_setup : http://peak.telecommunity.com/dist/ez_setup.py
• run it with your Python interpreter

Step 2 - install Atomisator and SQLite

• run the command : easy_install Atomisator
• make sure SQLite is installed on your system. If not, install it: http://www.sqlite.org/download.html
  

Step 3 - create an “atomisator.cfg” file

The content of the file has to be:

[atomisator]
store-entries = false

sources =
  rss http://www.nabble.com/Python---python-list-f2962.xml
  rss http://n2.nabble.com/Plone-f293351.xml
  rss http://www.nabble.com/Zope---General-f6715.xml
  rss http://planet.python.org/rss10.xml
  rss http://www.zope.org/Planet/planet_rss10.xml

filters =
  buzzwords words.txt

outputs =
  email email.cfg

This file will look into Planet Python, Planet Zope and various mailing lists (Python, Plone, Zope). Of course you can add or remove feeds in the sources option.

Step 4 - Create the words.txt file

This file contains regular expressions, one per line, that will be used to match the entries. The file has to be saved besides atomisator.cfg.

For our example:

buildout
pycon

You can put any expression you want in this file, as long as you have one matching expression per line.

Step 5 - add an email.cfg configuration file.

This is where you define the target emails that will receive the alerts (tos option). You can also specify the from email, or the smtp server location. The file has to be saved besides atomisator.cfg.

In our case it can be:

[email]
tos = tarek@ziade.org
from = tarek@ziade.org
smtp_server = smtp.neuf.fr

Step 6 - Run it !

The command to be called is atomisator (installed by easy_install) followed by the configuration file:

$ atomisator atomisator.cfg
Reading data.
Launching worker for rss - ('http://www.nabble.com/Python---python-list-f2962.xml',)
Launching worker for rss - ('http://n2.nabble.com/Plone-f293351.xml',)
Launching worker for rss - ('http://www.nabble.com/Zope---General-f6715.xml',)
Launching worker for rss - ('http://planet.python.org/rss10.xml',)
Launching worker for rss - ('http://www.zope.org/Planet/planet_rss10.xml',)
Retrieving from rss - ('http://www.nabble.com/Python---python-list-f2962.xml',)
Retrieving from rss - ('http://www.nabble.com/Zope---General-f6715.xml',)
Retrieving from rss - ('http://n2.nabble.com/Plone-f293351.xml',)
Retrieving from rss - ('http://planet.python.org/rss10.xml',)
Retrieving from rss - ('http://www.zope.org/Planet/planet_rss10.xml',)
.................................................................................................................................................
Writing outputs.
Data ready.

Check your mails. This call can be put in a daily cron.

Tested under Mac OS X and Linux.

I am back from the Plone Conference in D.C., and the jetlag is gone. The jetlag is gone for weeks now but it’s hard to find the time to blog these days :/

On the talks I have seen and topics I have chatted about

There were a lot of great talks in D.C., and it was hard to decide which one to look at. In any case it was easy to meet the speaker if I had missed the talk, because the Plone Conference, unlike big conferences like OSCON, is a place where everyone hangs around the same spot after a talk is over.

Here’s a list of some topics I have seen or I have talked about with some people.

Deliverance - Ian Bicking

If you look at what Ian has produced in the past 5 years, he is one of the most prolific contributor of tools that become standards in the Python web development web community. Think about Python Paste or virtualenv, and many others. Deliverance might be the next big one.

Take a bunch of micro web applications you want to join to build a full web system, for historical reasons or just because you believe a particular feature just won’t fit in Plone but will do great in Pylons.

Now ask a designer to glue everything together under the same look. He (or the guy that integrates his design) will probably hates you: he will have to learn how to integrate in heterogeneous environments. This is easy under some systems that let you stick a layout and a css in a simple way. This is not easy under Plone, unless you learn how to do it (but this will be improved in the future).

Deliverance is a proxy that let you skin any application that spits html content, by running some XPATH rules on the content and applying some changes to produce a new output. Basically, you have a simple html page that just provides the layout you want to have, without any content, and a xml file that explains how to extract some content from the page produced by the third-party application and where to inject it in your empty html page. The great thing is that you can call different third-party servers given the path you are in, and even call several servers to build one single page. This opens a lot of perspectives.

The first caveat of this approach is that you have to provide a Single-Sign On feature to avoid people having to connect several times. This can be a problem sometimes with some applications if they are not open enough to let you do it. But most of the time, it is not a problem : if the users are all located in a LDAP it is easiy.

Furthermore, if you use only Python-based applications, you can use a WSGI envrionment and a middleware like repoze.who to glue together let’s say, a Plone app and a Pylons app. Products.oopas is the PAS plugin that can be used for that on Plone side to grab the authentication context and use it.

The second problem I can see is about response headers. One example: if a page is composed of elements that comes from several pages, and if the page has a Last-Modifier header, I don’t think Deliverance handles this correctly yet, to make sure to present the newest Last-Modified header from all third-party servers that where called to build that page. But this more likely to be a detail compared to the single authentication problem.

In any case this is a very promising tool !

Content Mirror - Kapil Thangavelu

I didn’t see that talk, but I have talked about this tool with a few people. The idea is to serialize the content of a Plone instance into a relational database (eg Postgresql), as it happens, using events.

I need to give a try and check it deeper, to see how the overhead is dealt, and how the aggregator I have read about is doing (it collects mirorring operations to perform in a transaction, and optimize the calls at the end of the transaction to avoid redudant calls if I understood correctly). I don’t know yet for example if there’s a pool of jobs for the mirroring tasks to avoid a point of failure. But I am pretty sure this is taking care of. The other point I need to see if there’s a round trip. e.g. if there’s a way to apply a relational database change back into Plone.

But in any case I can already see various use cases for my customers. For instance, having a plone instance as a back office, with complex workflows for editors and contributors, and a lightweight Pylons application as the front application, that concentrates into displaying the relational database as fast as possible, makes a lot of sense in big environments. It just scales better.

So this is a interesting tool as well.

repoze.bfg - Chris McDonough

Chris gave a talk about repoze.bfg, which is a new web framework that takes back the good bits from Zope and push them into a WSGI world, using the Pylons approach I would say. That is : “here’s the template engine you can use in repoze, but really, use the one you like”.

Frankly, I am really seeing this new effort as one of the most promising one in the Zope community. Already, repoze.auth is a major middleware in WSGI : Zope’s Pluggable Authentication Service outside Zope, usable with any WSGI application. This is a blast !

And people are starting to contribute a lot of interesting middlewares under the repoze namespace.

Now I didn’t really try repoze.bfg itself yet, but given the people that are behind it, I am pretty sure this framework will meet success in the future. Having a MVC framework ala Pylons that let you use Zope packages with a “this zope package is repoze/wsgi compliant” label on each one of them is very cool.

collective.indexing - Andreas Zeidler and al

At the snow sprint, we worked with the Enfold crew that did a great work in integrating the Solr/Lucene system so it can be used from Plone. We replaced a few fields like the searchable text and indexed it on Solr side, just to give it a try. The snow work was really focusing on providing a buildout, a few recipes and a bench to say : “Hey, Plone community, this is a blast ! let’s do more of it”

Later Andreas Zeidler and a few other guys continued the work on indexing matter and they delivered collective.indexing, which provides two things:

• a queue that collects all indexing to be done, and optimize the call to the catalog
• a bridge to use collective.solr

I didn’t follow the latest development and I didn’t know how far the guys went, but I had the chance to hang around with Andreas and Tom Lazar in D.C., so now I know that this package is production ready

So in other words : I’ll probably use it as a mandatory package for all the big plones out there.

The queuing part imho, should go into the catalog itself because there’s no other way to make sure a third-party product is not calling the catalog during the transaction wile another product does the same.

Server-Side Include (SSI)

Tom Lazar worked during the Snow Sprint on lovely.remoteinclude to make Plone portlets accessible via unique URLs. From there, it is possible to push a page that contains a list of urls rather than the calculated page, to a front server that knows how to read SSI directive, and builds the page.

This is great for performances, and is a lot like ESI (Edge Side Include) we use to have in CPSSkins.

I am wondering if both could be implemented in the same tool in fact.

Tom told me that he will try to continue this work at the performance sprint in Bristol in december, so let’s keep an eye on this !

I have seen many other talks and topics, but these few ones where the ones I really needed to talk about.

On the conference organization

I am helping in the organization of Pycon FR in Paris since 2 years now. I know what is means to organize such events : it is a LOT OF WORK.

You know when an event is well organized when you don’t feel it is organized.

That was the case in D.C. Bravo Alex, Amy and all the others !

The only problem (wifi) was not the organizers fault, and I have never been to any event where it is not cahotic at some point (besides OSCON) so…

On the community

I love you all guys. It is an amazing community.

I was totally drowned into some customer projects since I came back from the Plone Conference. But things are looking better now, so I can take a bit of time to start blogging about the conference. I’ll probably do three blog posts: this one about my tutorial, the next one about the conference itself and last, an entry about the sprints.

So I gave a tutorial about zc.buildout. The length was a bit challenging, since I had 90 minutes. Enough time to explain more things than in a regular talk, but not enough time to get into great details, as a tutorial should be.

The other thing was about the topic: two talks were covering zc.buildout, Clayton Parker’s one and mine. So my goal was to make sure they were not overlapping too much.

I had the chance to meet Clayton before we both gave our talk, since the Six Feet Up crew gave me shelter in the house they rented (nicest guys in the block). Even if we didn’t exchange a lot on the slides themselves, I could figure what Clayton was going to present. So I…. started my slides from scratch two days before my tutorial and worked carefully on their scope

Alex Clark came and backed me up during the talk, since we are working together one plone.org for months now and since the talk presented the new plone.org that is coming up.

I think I did quite well during the talk, because we had a pause half way, and when we started back, the room stayed full

This is the second time I record all the console work in small screencasts, to avoid live problems, and I think this is the best way to go if you need to do some demos, so I’ll keep on doing it. Plus, it’s nice to provide them to people after the talk.

Anyway, the talk was videotaped so you can to judge by yourself:

• The video
• The slides.

And everytime you see a “get the screencast at http://ziade.org/ploneconf” in there, well get them

In detail:

1. Distutils demo
2. setuptools demo
3. collective.eggproxy demo
4. PloneSoftwareCenter installation
5. collective.dist demo
6. new.plone.org demo
7. collective.releaser demo
8. plone 3 buildout demo
9. Multiple target releasing demo

What’s next ?

• We need to finish the work with Alex on plone.org. It’s not hard, it just takes time, and we both are quit busy in our jobs
• I need to polish collective.releaser and collective.eggproxy. They are brut de fonderie, and the code suck a bit. If you are using them and want to help, or have some feedback/issues, please, pretty please, let me know.

Just a quick note:

We have started the sprint in D.C. on packaging matters. Our tasks for the moment:

• re-read all the threads in distutils-ML since mid-september.
• write a short list of actions that can be done NOW in distutils, setuptools and PyPI
• write a short list of writing that can be done to define a “new” tool

Please, join at any time in #distutils in freenode, and ping us, if you want to get involved.

JJ shirt

So I promised I would create a shirt for the top #5 contributors of my fund raising.

I think I found an idea. My friend JJ is wearing a shirt I made on the picture you see. That’s the phrase we wanted me to put on the shirt:

There’s no place like 127.0.0.1

(except maybe ::1)

That is one of the geekiest shirt I know of

Anyway, I want to borrow the idea for the plone shirt, and write, beside the Plone logo:

There’s no place like 127.0.0.1

(except maybe #plone on freenode)

That is, to express the fact that the Plone community is a really friendly one, and it’s great to be part of it.

EDIT: I have better proposals from the two Alex:

$ whois plone

127.0.0.1

and:

There is no place like Plone

and:

Plone, I wish I knew how to quit you

and:

Did you mean Plone ?

I am hesitating now….

(This post is a work in progress, as things still evolve)

In a way, I am glad I have made a fork, and I am glad this is going to be the shortest fork ever. A lot of people reacted on my proposal and I could get a very clear picture of what is wrong in the distutils/setuptools world, and how I could help on this.

This is how I interpret it:

Guido explained that the mistake made was not to integrate setuptools in the core from the very begining. The current distutils code did not evolve in the meantime and Phillip Eby worked on setuptools to make it evolve. setuptools now reaches a point where it needs other contributors to fit all the needs and solve more problems. It had one contributor in the past, (Jim Fulton) but it needs more. It also became a mandatory package for many folks in the Python world because Phillip did a great work on it : it solves most problems.

From what I could see, all the people that gets frustrated at some point with setuptools, either don’t use it, either try to see how it could be changed. But when they try that, they take the same paths others have taken before because there is a lack of info and documentation on setuptools current status and future. And these paths are quite long for the brain before you are able to provide a well-thaught idea or patch.

I have tried to help myself on setuptools, then I have started to blame Phillip saying that he did not have enough time to make setuptools evolve. But that was the wrong approach, because the only problem really, is a lack of visibility in setuptools development. And this is something that can be fixed quite easily I believe. The bug tracker added some months ago helped a lot. A roadmap and one or two page around it and that should be it.

Guido also stated that it was merely impossible to work on the next generation of distutils outside Python core. But in the meantime this means that we need some people from the core to help us in there. And they are really busy (I can understand that) on other matters in Python code. We can write patches for some fixes for sure, and there are hundreds of them to do in distutils.

But to boost it we would need a core developer that gives some love to that package. I mean, for example I still have a patch waiting for reviewing, wich only adds some unit tests that are lacking. There is no code there, just unit tests, and it is pending since 6 months… Guido said that I could try to become a core developer, that this was not impossible, if I started to contribute patches often to other parts of Python as well. But that’s quite a challenge.

What frustrated me is that some people like Jim Fulton said they were willing to work on a new distutils from scratch, and in the meantime I understand what Guido says. he pointed us to Joel entry on rewriting from scratch.

My Plans

I don’t have the brain to drive a fork of setuptools at this time. I knew it from the beginning, I just wanted to make people react. I think that setuptools can be the laboratory where we can experiment things, and distutils the place where we can apply them at some point, with Phillip help because he has a pretty big overview of all that. It is going to take years, but well, we are young.

To help on this I will:

• try to gather people at the PloneConf to talk about it, maybe even sprint
• try to document setuptools and distutils at my level, and write a roadmap for people to know what is going on. Either through a Sphinx site either on Python.org wiki.
• work on patches for distutils, and try to point things in setuptools that might be brought into distutils
• try to see if I have the brain to understand and work on other parts of Python.

They are many commenting systems out there for Plone, Zope 3, etc.

But let’s take this simple statement : why a content provider like any Zope-based application should deal with comments ? A comment is a simple piece of text, that refers to a page and maybe to another comment. There are no workflow needed in most cases. Maybe an authorization required to add the comment but that’s it. And how do you provide a comment on a page that is not generated by your web server, but that is displayed somewhere to your users nevertheless ? (like a static page)

So, looking at how Deliverance works, and because I want to have comments on Sphinx pages, I thaught of a simple way to deal with comments: A WSGI Middleware/Proxy.

The comment application stores comments in a SQL DB and use the pages id (the full URL) as references. I can even use services like Akismet to avoid spams. Then when the page is displayed, I can inject the comments associated to it on-the-fly, and provide a JS-enabled form for people to add comments (which can be caught when submited by the middleware too I guess, or sent through Ajax to a small JSON-enabled service that listens)

This is perfect for an hybrid environment where you have pages generated by various web applications.

Of course there are some caveats, like if the page is gone, or has moved. But dealing with orphan comments should be doable.

We might try that out soon for Afpy.org

EDIT: Read this new entry

That’s enough !

Setuptools is now a big part of our Python development and release infrastructure.

But we have been struggling with the setuptools development process for months around here. The project is run by one single man, who is really busy on other things as well. I am not blaming him, he’s doing a great job. I am just saying that setuptools needs to be more open.

Yesterday, I had, like the week before, some developers complaining about some incompatibility between setuptools and Subversion 1.5.

It is not like it is a hard to fix, and as a matter of fact it is fixed now in the trunk of the project. But not released since quite a long time now. So I have to ask my developers to checkout the trunk on their buildout, and use the ==dev tag elsewhere. I proposed some help to do the release, but my mails were just ignored.

There are also a lot of improvments to do in this tool, so Python gets the distribution tool it deserves.

But having one single person with the commiter rights is not possible on such an important package for the community. It has to be community-driven.

Enough talking, I am launching a setuptools fork, which will be community-driven, and wich will remain 100% compatible with setuptools at this point.

You can react, of follow the reaction on the distutils-SIG I started here : http://mail.python.org/pipermail/distutils-sig/2008-September/010031.html

Maybe my attempt to make such a tool belong to the Python community will fail, I don’t know. Maybe people will not like seeing someone forking like that and will continue to sleep and to unsubsribe themselves from the distutils-SIG.

But at least I am trying something because Python is suffering from a lack of a good, robust distribution tool at this point and there are a lot of people that could contribute if the development process is more open and the maintainer(s) more available and reactive.

I have contributed a bit in distutils in the past year, but the Python core development team has a lack of interest in this part of Python at this time. Thanks to some of the core team members, I could have some patch applied in the trunk, like the multiple servers pypirc. But we won’t be able to create what I would call ‘distutils 2′ in Python itself at this point. And setuptools seems to be the  best candidate for a distutils replacement in the future.

As long as the community can work on it of course ! We are the community, we are distributing large application to our customers.

So that’s what “Distribute” is about.

It seems that my book is now officialy available, so I should give more details about its content.

Who should get this book ?

I think Shannon described the targetship of my book quite well in the foreword:

If you’re looking to progress from knowing Python to mastering Python, this is the
book for you. In fact, this is exactly the type of book I wish I had had five years ago.
What took me years to discover by steadfastly attending talks at PyCon and my local
Python users’ group is now available in a succinct book form.

What is means is that this book does not only focus on Python syntax, but also covers how to use Python in a professional environment. Beyond writing a program that works, a good Python developer uses continous integration principles and tries to think about the maintainability of his code. Taking care of choosing good names for instance, will naturally make the code better. Test-Driven Development make the code better too.

The book tries to synthesize these good practices and explain why they are good. So if you are using Python and know how to write a program, and want to push it further, this is a book for you.

On the code

All the code of the book lives here: http://atomisator.ziade.org. So it can evolve.

On my English

I am French. My english is far from being perfect. The Packt team did a great work on improving it, but you will probably feel my french touch in the book. I hope you won’t mind.

On the structure

Depending on your needs, you might feel that the ordering of the chapters is not what you were expecting. The current ordering is the one I would use when writing an application, but this is my own vision. If you are a manager you probably have your own way. I worked this out with one concern in mind : every chapter is independant. So feel free to jump to any chapter when you have finished one.

Sample chapter

I am giving away Chapter 10, as an appetizer: chapter 10 of “Expert Python Programming”

This chapter is about documentation. It gives principles and good practices to document your Python projects. It is an invitation to use Sphinx and reStructuredText, together with a set of good practices.

More details on each chapter

Chapter 1, Getting Started. I took me a long time to decide whether I should drop or not this chapter. It describes how to install an environment to work with Python. Since the book is for people that already knows how to use Python a bit, it seemed out of topic.

A few things convinced me to let it in: I have a friend that works with Python under Windows for years, but don’t know anything about distutils or setuptools and how to set the proper MinGW environment to be able to install packages without having any trouble. Secondly: I am making some assumptions there for the rest of the book examples to work and I am a setuptools fan. Last but not least: this is a small chapter compared to the size of the book (less than 10%)

Chapter 2, Syntax Best Practices—Below the Class Level, I had a lot of fun there. I am talking about topics like coroutines and contextlib. Pure Python joy !

Chapter 3, Syntax Best Practices—Above the Class Level, This chapter explains amongst other topics why super() is dangerous, how the MRO works, meta-programming, etc. Lots of fun too. I think people will enjoy it a lot.

Chapter 4, Choosing good names, Writing a program that works is a good thing. Writing a program that can evolve and that is comprehensible by other developers is harder. This chapter will give you some clues on choosing good names, beyond the PEP8 guide, and how to organize and build a modular application by working out the API.

Chapter 5, Writing a package. The main ideas here are: write and distribute all your packages the same way, so use templates and distutils.

Chapter 6, Writing an application. Same as chapter 5, but at the application level. This field is very different depending on what frameworks you use and what community you are part of. I tried to come up with something that seems to be the way all developers are tending to take. Maybe I am a bad visonnary, maybe I am wrong. I don’t think I am. If you disagree, you will still find interesting stuff in there.I am presenting a micro case study to make things clearerild a modular application by working out the API.

Chapter 7, Working with zc.buildout. Buildout is widely use in Plone and Zope. There are other tools out there of course. But buildout rocks imho. This chapter show how it work and how to use it to work and distributeyour application.

Chapter 8, Managing code. Its starts with a state-of-the-art of version control systems and continuous integration principles. I am explaining how you can work using mercurial and buildbot.

Chapter 9, Managing lifecyle. Same as chapter 8, but focusing on software lifecycles, and why the iterative approach rocks. Also show how trac can be used.

Chapter 10, Documenting your projects, Just read it

Chapter 11, Test-Driven Development, This is my TDD manifesto

Chapter 12 and 13, Optimisation, Ever wondered how to calculate the complexity of the code ? how to benchmark and optimize ?

Chapter 14, Useful design patterns, Design patterns revisited, because the GoF did not know enough about Python

Feedback !

It is really frustrating in some way to write a printed book on a topic like Python. By the time I have finished the book and gave it back to the editor, I was already thinking on some changes and some improvments I could make.

I created a website for the readers : http://atomisator.ziade.org

You can add a ticket there for the v2 !

I just wanted to announce the release of collective.eggproxy (http://pypi.python.org/pypi/collective.eggproxy/0.2.0)

collective.eggproxy is a smart mirror for PyPI, thaught and coded by my colleague Bertrand Mathieu.

It will collect packages on PyPI only when a program like easy_install or zc.buildout asks for it. In other words, unlike some mirrors that act like rsync and get the whole PyPI base (more than 5 gigas) collective.eggproxy will only get what you need.

At first run collective.eggproxy downloads pypi index and builds a page of links. When a software asks for a specific package, version, etc. collective.eggproxy downloads it if needed and stores it locally.

Want to give a try ? try it in two lines with easy_install:

easy_install collective.eggproxy
eggproxy_run

That’s it !

This is probably obvious for the people that uses Repoze or Pylons, or early adopters in the Plone world, but from a Plone or a Zope developer perspective, you could live without it until now.

Now WSGI is everywhere.

I remember when Martijn Faassen brought the idea in 2006, of hooking Grok and Zope 3 into the WSGI. Maybe someone else talked about it before but that was the first time I could picture what WSGI could bring.

Now with the work done by people like:

• The repoze team, that made it easier to run a Plone-based application in WSGI
• The Paste Script / Paste Deploy team, that provided a simple way to describe a WSGI chain

And major WSGI middlewares like :

• repoze.who which allows you to deal with authentication separately
• Deliverance, which let you theme any application and let this application focus on delivering a content
• Things like Beaker, which let you use memcached for instance, to store session data and cache arbitrary things

From a CTO point of view, a WSGI environment brings me the ability to think about a web application and build it without having to stick into one framework and try to bend all technologies inside it.

For instance:

• I can write a Plone application and use Beaker to deal with sessions, without having to wrap Memcached into a custom plone package.
• I can ask a graphic designer to work on a CSS and a layout without having to do it into Plone. It’s not that Plone design tools are bad, but the learning curve of writing a rule file in Deliverance and apply it to any piece of application makes the designer more productive than becoming a specialist of one skinning tool.

• If my customer use moinmoin as a Wiki, I can put it into my Plone site transparently by defining a composite section in my Paste configuraton file.
• …

You could do all the mentioned thing without WSGI, just by importing the packages and/or dealing with proxies at Apache level. But that is not the point.

The point is that WSGI brought the idea of making all web frameworks and libraries interact together to build one web application.

It is not the silver bullet of course, but my gut feeling is that this will create some kind of reunification in Python Web development communities: people are starting to look at a wider range of package, beyond the framework they use everyday.

I don’t have time to talk in detail about it at this time, but I wanted to say that the Bobun Paris Sprint was a great event.

Together with the sprint we have done a few month before here at AlterWay, this a great momentum for the French Plone community.

Those events will probably be recurrent from now on (the one we organized at AlterWay will be organize every year for sure, and I think the Bobun team wants to do it again as well).

Maybe we will be able at some point in a few years to organize the PloneConf in Paris ?

That would be a blast

Sphinx is now used by quite a few communities : Python, Django, Pylons, Grok (not sure about the current status), …

No wonder, it’s a blast.

We are now starting to use it to produce customer documentation for our buildout-based applications. Basically, a Sphinx structure is created in the buildout using sphinx-quickstart, and a few tweaks are made so the HTML and PDF outputs have a custom look.

Managing the documentation like the code makes life easier. This is one of the basic rule of agile documentation : separate the content from the layout, so you can provide documentation in any shape (html, pdf) with a single source.

To make things easier, I have released collective.recipe.sphinxbuilder.

This recipe:

• creates for you a Sphinx-based documentation in your buildout
• creates a single script in the bin folder to build the documentation with one command
• provides an extensive set of options to drive Sphinx from buildout

Adding it in your buildout is as simple as :

[buildout]
parts =
  sphinx

[sphinx]
recipe = collective.recipe.sphinxbuilder

I have also customized the look and feel of the output so it uses the Plone logo and a custom css. This is configurable from the buildout configuration file of course. By the way, if someone from the Plone community wants to improve the CSS, please dot it ! (I am not good at this )

Notice that if you use LaTex or PDF rendering, you will need to install pdflatex. Furthermore, the recipe script will not work under Windows unless you install a Linux-like environment, since it uses the Makefile provided by Sphinx. I guess MSYS+MinGW should make it work, but I didn’t try.