I like Google Alert, the idea of receiving a mail every day that summarizes all articles related to a given topic is really helpfull when you need to focus on a specific subject for a while.

But this is not enough. I want to receive a mail that points me to any mailing list or planet feed or blogs out there as well, that talks about the topic.

You can’t do it with Google Alerts as far as I know.

Let’s take an example:

I want to receive a daily mail that points me to any mail thread or blog entry, that is related to the word “buildout” or to the word “pycon”.

Basically, to do it manually, I need to read Planet Python, Planet Zope, then take a look at the Python, Zope and Plone mailing lists. It takes at least 10 minutes, and more if you want to read all entries to make sure you won’t miss anything.

Since online systems like Nabble provides RSS feed for mailing lists (don’t find yours ? just add it there !), it is easy to read them as they where regular feeds.

From there, a script that reads all the selected feeds and sends a mail pointing to the entries that match the selected words is simple to write as well, and fill the need.

But don’t code it : Atomisator will let you do this with a few lines of configuration.

Here’s a step-by-step tutorial.

Step 1 - install easy_install

• download ez_setup : http://peak.telecommunity.com/dist/ez_setup.py
• run it with your Python interpreter

Step 2 - install Atomisator and SQLite

• run the command : easy_install Atomisator
• make sure SQLite is installed on your system. If not, install it: http://www.sqlite.org/download.html
  

Step 3 - create an “atomisator.cfg” file

The content of the file has to be:

[atomisator]
store-entries = false

sources =
  rss http://www.nabble.com/Python---python-list-f2962.xml
  rss http://n2.nabble.com/Plone-f293351.xml
  rss http://www.nabble.com/Zope---General-f6715.xml
  rss http://planet.python.org/rss10.xml
  rss http://www.zope.org/Planet/planet_rss10.xml

filters =
  buzzwords words.txt

outputs =
  email email.cfg

This file will look into Planet Python, Planet Zope and various mailing lists (Python, Plone, Zope). Of course you can add or remove feeds in the sources option.

Step 4 - Create the words.txt file

This file contains regular expressions, one per line, that will be used to match the entries. The file has to be saved besides atomisator.cfg.

For our example:

buildout
pycon

You can put any expression you want in this file, as long as you have one matching expression per line.

Step 5 - add an email.cfg configuration file.

This is where you define the target emails that will receive the alerts (tos option). You can also specify the from email, or the smtp server location. The file has to be saved besides atomisator.cfg.

In our case it can be:

[email]
tos = tarek@ziade.org
from = tarek@ziade.org
smtp_server = smtp.neuf.fr

Step 6 - Run it !

The command to be called is atomisator (installed by easy_install) followed by the configuration file:

$ atomisator atomisator.cfg
Reading data.
Launching worker for rss - ('http://www.nabble.com/Python---python-list-f2962.xml',)
Launching worker for rss - ('http://n2.nabble.com/Plone-f293351.xml',)
Launching worker for rss - ('http://www.nabble.com/Zope---General-f6715.xml',)
Launching worker for rss - ('http://planet.python.org/rss10.xml',)
Launching worker for rss - ('http://www.zope.org/Planet/planet_rss10.xml',)
Retrieving from rss - ('http://www.nabble.com/Python---python-list-f2962.xml',)
Retrieving from rss - ('http://www.nabble.com/Zope---General-f6715.xml',)
Retrieving from rss - ('http://n2.nabble.com/Plone-f293351.xml',)
Retrieving from rss - ('http://planet.python.org/rss10.xml',)
Retrieving from rss - ('http://www.zope.org/Planet/planet_rss10.xml',)
.................................................................................................................................................
Writing outputs.
Data ready.

Check your mails. This call can be put in a daily cron.

Tested under Mac OS X and Linux.

I am back from the Plone Conference in D.C., and the jetlag is gone. The jetlag is gone for weeks now but it’s hard to find the time to blog these days :/

On the talks I have seen and topics I have chatted about

There were a lot of great talks in D.C., and it was hard to decide which one to look at. In any case it was easy to meet the speaker if I had missed the talk, because the Plone Conference, unlike big conferences like OSCON, is a place where everyone hangs around the same spot after a talk is over.

Here’s a list of some topics I have seen or I have talked about with some people.

Deliverance - Ian Bicking

If you look at what Ian has produced in the past 5 years, he is one of the most prolific contributor of tools that become standards in the Python web development web community. Think about Python Paste or virtualenv, and many others. Deliverance might be the next big one.

Take a bunch of micro web applications you want to join to build a full web system, for historical reasons or just because you believe a particular feature just won’t fit in Plone but will do great in Pylons.

Now ask a designer to glue everything together under the same look. He (or the guy that integrates his design) will probably hates you: he will have to learn how to integrate in heterogeneous environments. This is easy under some systems that let you stick a layout and a css in a simple way. This is not easy under Plone, unless you learn how to do it (but this will be improved in the future).

Deliverance is a proxy that let you skin any application that spits html content, by running some XPATH rules on the content and applying some changes to produce a new output. Basically, you have a simple html page that just provides the layout you want to have, without any content, and a xml file that explains how to extract some content from the page produced by the third-party application and where to inject it in your empty html page. The great thing is that you can call different third-party servers given the path you are in, and even call several servers to build one single page. This opens a lot of perspectives.

The first caveat of this approach is that you have to provide a Single-Sign On feature to avoid people having to connect several times. This can be a problem sometimes with some applications if they are not open enough to let you do it. But most of the time, it is not a problem : if the users are all located in a LDAP it is easiy.

Furthermore, if you use only Python-based applications, you can use a WSGI envrionment and a middleware like repoze.who to glue together let’s say, a Plone app and a Pylons app. Products.oopas is the PAS plugin that can be used for that on Plone side to grab the authentication context and use it.

The second problem I can see is about response headers. One example: if a page is composed of elements that comes from several pages, and if the page has a Last-Modifier header, I don’t think Deliverance handles this correctly yet, to make sure to present the newest Last-Modified header from all third-party servers that where called to build that page. But this more likely to be a detail compared to the single authentication problem.

In any case this is a very promising tool !

Content Mirror - Kapil Thangavelu

I didn’t see that talk, but I have talked about this tool with a few people. The idea is to serialize the content of a Plone instance into a relational database (eg Postgresql), as it happens, using events.

I need to give a try and check it deeper, to see how the overhead is dealt, and how the aggregator I have read about is doing (it collects mirorring operations to perform in a transaction, and optimize the calls at the end of the transaction to avoid redudant calls if I understood correctly). I don’t know yet for example if there’s a pool of jobs for the mirroring tasks to avoid a point of failure. But I am pretty sure this is taking care of. The other point I need to see if there’s a round trip. e.g. if there’s a way to apply a relational database change back into Plone.

But in any case I can already see various use cases for my customers. For instance, having a plone instance as a back office, with complex workflows for editors and contributors, and a lightweight Pylons application as the front application, that concentrates into displaying the relational database as fast as possible, makes a lot of sense in big environments. It just scales better.

So this is a interesting tool as well.

repoze.bfg - Chris McDonough

Chris gave a talk about repoze.bfg, which is a new web framework that takes back the good bits from Zope and push them into a WSGI world, using the Pylons approach I would say. That is : “here’s the template engine you can use in repoze, but really, use the one you like”.

Frankly, I am really seeing this new effort as one of the most promising one in the Zope community. Already, repoze.auth is a major middleware in WSGI : Zope’s Pluggable Authentication Service outside Zope, usable with any WSGI application. This is a blast !

And people are starting to contribute a lot of interesting middlewares under the repoze namespace.

Now I didn’t really try repoze.bfg itself yet, but given the people that are behind it, I am pretty sure this framework will meet success in the future. Having a MVC framework ala Pylons that let you use Zope packages with a “this zope package is repoze/wsgi compliant” label on each one of them is very cool.

collective.indexing - Andreas Zeidler and al

At the snow sprint, we worked with the Enfold crew that did a great work in integrating the Solr/Lucene system so it can be used from Plone. We replaced a few fields like the searchable text and indexed it on Solr side, just to give it a try. The snow work was really focusing on providing a buildout, a few recipes and a bench to say : “Hey, Plone community, this is a blast ! let’s do more of it”

Later Andreas Zeidler and a few other guys continued the work on indexing matter and they delivered collective.indexing, which provides two things:

• a queue that collects all indexing to be done, and optimize the call to the catalog
• a bridge to use collective.solr

I didn’t follow the latest development and I didn’t know how far the guys went, but I had the chance to hang around with Andreas and Tom Lazar in D.C., so now I know that this package is production ready

So in other words : I’ll probably use it as a mandatory package for all the big plones out there.

The queuing part imho, should go into the catalog itself because there’s no other way to make sure a third-party product is not calling the catalog during the transaction wile another product does the same.

Server-Side Include (SSI)

Tom Lazar worked during the Snow Sprint on lovely.remoteinclude to make Plone portlets accessible via unique URLs. From there, it is possible to push a page that contains a list of urls rather than the calculated page, to a front server that knows how to read SSI directive, and builds the page.

This is great for performances, and is a lot like ESI (Edge Side Include) we use to have in CPSSkins.

I am wondering if both could be implemented in the same tool in fact.

Tom told me that he will try to continue this work at the performance sprint in Bristol in december, so let’s keep an eye on this !

I have seen many other talks and topics, but these few ones where the ones I really needed to talk about.

On the conference organization

I am helping in the organization of Pycon FR in Paris since 2 years now. I know what is means to organize such events : it is a LOT OF WORK.

You know when an event is well organized when you don’t feel it is organized.

That was the case in D.C. Bravo Alex, Amy and all the others !

The only problem (wifi) was not the organizers fault, and I have never been to any event where it is not cahotic at some point (besides OSCON) so…

On the community

I love you all guys. It is an amazing community.

I was totally drowned into some customer projects since I came back from the Plone Conference. But things are looking better now, so I can take a bit of time to start blogging about the conference. I’ll probably do three blog posts: this one about my tutorial, the next one about the conference itself and last, an entry about the sprints.

So I gave a tutorial about zc.buildout. The length was a bit challenging, since I had 90 minutes. Enough time to explain more things than in a regular talk, but not enough time to get into great details, as a tutorial should be.

The other thing was about the topic: two talks were covering zc.buildout, Clayton Parker’s one and mine. So my goal was to make sure they were not overlapping too much.

I had the chance to meet Clayton before we both gave our talk, since the Six Feet Up crew gave me shelter in the house they rented (nicest guys in the block). Even if we didn’t exchange a lot on the slides themselves, I could figure what Clayton was going to present. So I…. started my slides from scratch two days before my tutorial and worked carefully on their scope

Alex Clark came and backed me up during the talk, since we are working together one plone.org for months now and since the talk presented the new plone.org that is coming up.

I think I did quite well during the talk, because we had a pause half way, and when we started back, the room stayed full

This is the second time I record all the console work in small screencasts, to avoid live problems, and I think this is the best way to go if you need to do some demos, so I’ll keep on doing it. Plus, it’s nice to provide them to people after the talk.

Anyway, the talk was videotaped so you can to judge by yourself:

• The video
• The slides.

And everytime you see a “get the screencast at http://ziade.org/ploneconf” in there, well get them

In detail:

1. Distutils demo
2. setuptools demo
3. collective.eggproxy demo
4. PloneSoftwareCenter installation
5. collective.dist demo
6. new.plone.org demo
7. collective.releaser demo
8. plone 3 buildout demo
9. Multiple target releasing demo

What’s next ?

• We need to finish the work with Alex on plone.org. It’s not hard, it just takes time, and we both are quit busy in our jobs
• I need to polish collective.releaser and collective.eggproxy. They are brut de fonderie, and the code suck a bit. If you are using them and want to help, or have some feedback/issues, please, pretty please, let me know.

Just a quick note:

We have started the sprint in D.C. on packaging matters. Our tasks for the moment:

• re-read all the threads in distutils-ML since mid-september.
• write a short list of actions that can be done NOW in distutils, setuptools and PyPI
• write a short list of writing that can be done to define a “new” tool

Please, join at any time in #distutils in freenode, and ping us, if you want to get involved.

JJ shirt

So I promised I would create a shirt for the top #5 contributors of my fund raising.

I think I found an idea. My friend JJ is wearing a shirt I made on the picture you see. That’s the phrase we wanted me to put on the shirt:

There’s no place like 127.0.0.1

(except maybe ::1)

That is one of the geekiest shirt I know of

Anyway, I want to borrow the idea for the plone shirt, and write, beside the Plone logo:

There’s no place like 127.0.0.1

(except maybe #plone on freenode)

That is, to express the fact that the Plone community is a really friendly one, and it’s great to be part of it.

EDIT: I have better proposals from the two Alex:

$ whois plone

127.0.0.1

and:

There is no place like Plone

and:

Plone, I wish I knew how to quit you

and:

Did you mean Plone ?

I am hesitating now….

(This post is a work in progress, as things still evolve)

In a way, I am glad I have made a fork, and I am glad this is going to be the shortest fork ever. A lot of people reacted on my proposal and I could get a very clear picture of what is wrong in the distutils/setuptools world, and how I could help on this.

This is how I interpret it:

Guido explained that the mistake made was not to integrate setuptools in the core from the very begining. The current distutils code did not evolve in the meantime and Phillip Eby worked on setuptools to make it evolve. setuptools now reaches a point where it needs other contributors to fit all the needs and solve more problems. It had one contributor in the past, (Jim Fulton) but it needs more. It also became a mandatory package for many folks in the Python world because Phillip did a great work on it : it solves most problems.

From what I could see, all the people that gets frustrated at some point with setuptools, either don’t use it, either try to see how it could be changed. But when they try that, they take the same paths others have taken before because there is a lack of info and documentation on setuptools current status and future. And these paths are quite long for the brain before you are able to provide a well-thaught idea or patch.

I have tried to help myself on setuptools, then I have started to blame Phillip saying that he did not have enough time to make setuptools evolve. But that was the wrong approach, because the only problem really, is a lack of visibility in setuptools development. And this is something that can be fixed quite easily I believe. The bug tracker added some months ago helped a lot. A roadmap and one or two page around it and that should be it.

Guido also stated that it was merely impossible to work on the next generation of distutils outside Python core. But in the meantime this means that we need some people from the core to help us in there. And they are really busy (I can understand that) on other matters in Python code. We can write patches for some fixes for sure, and there are hundreds of them to do in distutils.

But to boost it we would need a core developer that gives some love to that package. I mean, for example I still have a patch waiting for reviewing, wich only adds some unit tests that are lacking. There is no code there, just unit tests, and it is pending since 6 months… Guido said that I could try to become a core developer, that this was not impossible, if I started to contribute patches often to other parts of Python as well. But that’s quite a challenge.

What frustrated me is that some people like Jim Fulton said they were willing to work on a new distutils from scratch, and in the meantime I understand what Guido says. he pointed us to Joel entry on rewriting from scratch.

My Plans

I don’t have the brain to drive a fork of setuptools at this time. I knew it from the beginning, I just wanted to make people react. I think that setuptools can be the laboratory where we can experiment things, and distutils the place where we can apply them at some point, with Phillip help because he has a pretty big overview of all that. It is going to take years, but well, we are young.

To help on this I will:

• try to gather people at the PloneConf to talk about it, maybe even sprint
• try to document setuptools and distutils at my level, and write a roadmap for people to know what is going on. Either through a Sphinx site either on Python.org wiki.
• work on patches for distutils, and try to point things in setuptools that might be brought into distutils
• try to see if I have the brain to understand and work on other parts of Python.