By Ingeniweb. A Django site.
Août 12, 2008
» iw.eggproxy, a smart PyPI mirror


Mirroring PyPI becomes a recurrent need for Zope development, because zc.buildout makes a lot of package downloads to build one application.

This is useful when you are working in an intranet with limited web access or when you want to speed up download times. It also makes things safer: if PyPI is down and if developers computers don’t have caches, having a mirror will save your day.

While PyPI has proven its robustness (it is 100% up for months now as far as I can see), having mirrors makes a lot of sense.

We have created a small mirror application here at Ingeniweb, that we use for our buildouts needs. This work was thaught and created by my colleague Bertrand Mathieu.

It is a smart proxy that will download packages at PyPI everytime they have been asked by a buildout or an easy_install client. When the package is downloaded, it is kept in the proxy side for any new requests. This means that after a while, the proxy has its own collection of packages that corresponds to the real needs and will not query PyPI anymore.

This approach avoids having to download and synchronize PyPI with crons, which is a heavy process since PyPI weight several gigas. The caveat of course, is that it won’t be able to get a new package if PyPI is down.

Take a look ! http://pypi.python.org/pypi/iw.eggproxy

By the way there is an interesting sprint coming up on all these topics, in Germany :

http://www.openplans.org/projects/black-forest-sprint/project-home

Août 8, 2008
» A new Python book : “Expert Python Programming”


The Packt team added a page about my new book, which will be out sometimes in September, so I guess it is the right time for me to announce it here !

So here comes “Expert Python Programming”, where I explain how we work with Python every day to create software.

This is my first book in English, and writing in another language was quite challenging ;)

Anyway, this book is intended for developers that already have a background in Python and covers only advanced topics (see the editor details). But as I said, it explains how we develop our applications in Python so topics like continuous integration, documentation, testing, releasing, refactoring, etc. are covered.
Managers will also have a good overview of how a Python project can be run and managed, using modern tools like Distributed Version Control Systems (Mercurial for instance) or Buildbot.

Even if I am working on Zope and Plone these days, I have focused on writing a book that is only about Python, to make it useful to any developer. The fact that my friend Shannon did the technical reviewing helped on this : he doesn’t use Zope too much so he was close to the target readership.

I will take more time when it comes out to announce it in the various mailing lists, and to get into greater details about the content that will be available online besides the book (some Python packages, etc)

I can’t wait to see it out and hold it :D

Juillet 29, 2008
» OSCON report #1: the city of Portland


Covering OSCON when you are a speaker is quite hard. Until my talk was done, I was more in a mood of reviewing my slides and not really thinking about other talks or blogging.

Convention center from the Bridge (the two twin towers)

I will try to write a few reports now that everything is over. This first report is about Portland, the city and is not technical at all.

This is the first non-technical entry in my blog, but let’s this it for once. So if you are looking for technical content, skip this entry :)

Portland is an amazing place. The weather was nice throughout the whole week: sunny and not too hot. Some evenings were a bit chilly but that felt nice to me. The convention center where OSCON took place was located on the other side of the Willamette river, in front of the city center. It is a bit far from the center, but finding places were you can put so many people must be hard I guess.

The top six things I have noticed about Portland:

Local beers are great. You can find many places where they brew their own beers, that are comparable to Belgian beers in quality and taste, like the Lucky Lab (Where Jon, a member of the local Plone user group took us, thanks!).  Some place have samples so you can taste several kinds of beers. I also went to the Oregon Brewers Festival right after OSCON was over, where you could taste beers from many places.

Beer Festival

Oregon Brewers Festival

Parking a car is a pain. I think the best way to travel in Portland is a mix of Tram, Bus and Bike. You can put your bike on the nose of the buses or inside the trams. Smart thing. This would be useful in Paris. We ended up parking our car far from the center and used the tram (called Max, and free in the center). Portland should set up a bike rental system like in Paris (Velib’), I am sure this would rock.

Organics: people here seem to be really concerned about sustainability and organics. I’ve been told there are a lot of organics farm around Portland. Supermarkets have a nice amount of organics stuff as well. This is nice.

Starbucks Coffee owns the streets: this is scary. there’s a Starbucks almost on each block here. Anyway, local coffee shops have better coffees and they provide nice places to chill out, read a book or talk with people. I think this is where you can feel the real Portland way of living.

The food is good : I have to admit, when we, french people, travel, we feel a bit superior to some countries on food matters. We tend to show off about it :). But Portland has great places to eat. If you like donuts, the place you must go to is called Voodoo Doughnut. They are creative !

Voodoo Donuts

Voodoo Doughnut

Big open spaces: I went to Mount Hood where people are skying in… July. I wanted to visit this place because it is where Stanley Kubrick did the outside shots of the famous motel in The Shining. I also had a walk at the Hood river sandbar, where people do kite surfing. One tip: don’t walk there with shorts on, the wind is so strong that you get slapped by the sand. So if you like big open spaces and sports, Portland is the place to be: all those place are one or two hours drive from the city center.

Mount Hood

Mt Hood

End of the aparté ! The next entry will focus on OSCON

Juillet 26, 2008
» Updating your form code to latest version of plone.z3cform


I am glad to see more work is happening for using z3c.form in CMF and Plone, and Daniel Nouri updated us this week with the latest changes. I just updated my buildout to use the new packages and got plone.z3cform 0.4 and plone.app.z3cform 0.3.2.

With the small API changes that happened, you can see below that there is less to write to expose the form in the Plone site (our ContactForm example from my
previous post):

# my.example/my/example/browser.py import datetime from zope import schema import zope.component import z3c.form import plone.app.z3cform from plone.app.z3cform import layout from plone.i18n.normalizer.interfaces import IIDNormalizer from my.example import interfaces from my.example.contact import MyContact class ContactForm(z3c.form.form.Form): """ Contact Form """ fields = z3c.form.field.Fields(interfaces.IContactData) message_field = z3c.form.field.Fields(schema.TextLine(__name__ = 'message', title=u"Message", required=False) ) fields += message_field ignoreContext = True @z3c.form.button.buttonAndHandler(u'Send', name='send') def handle_send(self, action): data, errors = self.extractData() if errors: self.status = z3c.form.form.EditForm.formErrorsMessage return # Add the contact data, if not in yet id = data['firstname'] + data['lastname'] id = zope.component.queryUtility(IIDNormalizer).normalize(id) if id not in self.context.objectIds(): self._name = id contact = MyContact(self._name, **data) self.context[self._name] = contact # Complete the code so it sends the message to the site admin... message = data['message'] print "Message from %s %s (%s):\n%s" % (data['firstname'], data['lastname'], data['email'], data['message']) self.request.RESPONSE.redirect(self.context.absolute_url()) # New way to provide the wrapping View ContactFormView = layout.wrap_form(ContactForm, label="Contact Form")

The related ZCML code does not change.
By the way, on my TODO list, is moving from ZCML to grokcore.component registration ;)

Juillet 25, 2008
» My OSCON slides online (zc.buildout)


After a few attempts to make my screencast look nice under Google Video, I decided I would just upload the original ones (.mov and .m4v files) and a PDF export of my presentation.

So everything is available here: http://ziade.org/oscon

Slides: http://ziade.org/oscon/oscon.pdf

The .mov files are streamed automatically in your browser I believe, not the m4v ones.

Thanks to Jim Fulton for the quick feedback.

» Faire du virtualhosting avec zope façon wsgi

J'en avais ras le bol que Zope nécessite des url complètement tordues pour faire du virtual hosting. Ça m'empêchais entre autre d'utiliser Paste#urlmap pour dispatcher certaines url sur d'autres applis que Zope.

Du coup, j'ai tenté un truc tout con: plutôt que d'utiliser les RewriteRule d'Apache, récrire le PATH_INFO en englobant l'application Zope dans une autre. Et ça marche. Fiesta !

Voilà donc à quoi ça ressemble. J'utilise zopeproject. J'ai donc modifier le machin qui créer l'application Zope. A savoir le fichier startup.py comme ceci:

def application_factory(global_conf, conf='zope.conf', vhost='www.gawel.org'):
    vhost = '/++vh++http:%s:80/++' % vhost
    zopeapp = zope.app.wsgi.getWSGIApplication(zope_conf)
    def zopewrapper(environ, start_response):
        environ['PATH_INFO'] = vhost + environ['PATH_INFO']
        return zopeapp(environ, start_response)
    return zopewrapper

Et hop, ça roule. L'avantage, en plus d'avoir une url propre en entrée, c'est que vu que je développe aussi derrière Apache, j'ai juste eu à changer mon fichier debug.ini pour prendre en compte mon virtual host de développement.

En fait j'ai fais un peu mieux que tout ça, car comme dit au début, le but était d'utiliser Paste#urlmap. La source de la bidouille en question est ici.

Aller, pendant que j'y suis, j'en chiais aussi pas mal pour déterminer vers quel backend rediriger les requêtes dans varnish. Tester des ++ dans l'url, ça lui plaisait pas du tout. Vu que j'utilise Apache devant (surtout pour subversion), j'ai trouvé le truc. Il suffit d'activer le module headers:

# a2enmod headers

Puis rajouter un truc du genre dans votre virtualhost Apache:

RequestHeader set VARNISH_BACKEND gawel_org

Vous l'aurez compris, ceci ajoute un header à la requête. Ensuite, dans varnish, on test ce header:

if (req.http.VARNISH_BACKEND ~ "gawel_org") {
    set req.backend = gawel_org;
}

Et le tour est joué. Il faut bien sur que toutes les requêtes entrantes aient ce header. Pour moi ce n'est pas un problème vu que tout passe par apache.

Juillet 19, 2008
» A variation on my previous z3c.form example


Another example of form logic, adding to the one discussed in my previous post.

Here is the idea: If you need a more flexible solution for your use case, you can inherit directly from z3c.form.form.Form.
Let’s say you want the form to be used for both adding contact information (making sure later that this only works when the user is authenticated) and sending a mail message to the site administrator. A site feedback form that will at the same time be used to populate your database of contacts.

Your form class could be defined as follows (I’m skipping the imports part):

# my.example/my/example/browser.py class ContactForm(z3c.form.form.Form): """ Contact Form """ fields = z3c.form.field.Fields(interfaces.IContactData) message_field = z3c.form.field.Fields(schema.TextLine(__name__ = 'message', title=u"Message", required=False) ) fields += message_field ignoreContext = True @z3c.form.button.buttonAndHandler(u'Send', name='send') def handle_send(self, action): data, errors = self.extractData() if errors: self.status = z3c.form.form.EditForm.formErrorsMessage return # Add the contact data, if not in yet id = data['firstname'] + data['lastname'] id = zope.component.queryUtility(IIDNormalizer).normalize(id) if id not in self.context.objectIds(): self._name = id contact = MyContact(self._name, **data) self.context[self._name] = contact # Do something else, e.g. send the message to the site admin. # Complete the code as needed... message = data['message'] print "Message from %s %s (%s):\n%s" % (data['firstname'], data['lastname'], data['email'], data['message']) # Redirect self.request.RESPONSE.redirect(self.context.absolute_url())

The main points of the used pattern:

  • You define the fields for the form ; new fields can be added to the already defined list (based on the IContactData schema in interfaces.py) using the “+” operator.
  • You define your specific form button(s) and handler method(s). In the case of a standard AddForm, there is already a handle_add() defined for the “add” button for you.
  • Also, you need to set the form attribute ‘ignoreContext’ to True, so that the form has the same behaviour as an AddForm, i.e. it does not have to get/set data on attributes on the context. Note that by default, an AddForm has this attribute set to True, and an EditForm has it set to False.

Now, the final touch with the wrapping view…

class ContactFormView(base.FormWrapper): form = ContactForm label= "Contact Form"

… and its configuration:

<configure xmlns="http://namespaces.zope.org/zope" xmlns:browser="http://namespaces.zope.org/browser" xmlns:five="http://namespaces.zope.org/five" i18n_domain="my.example" > <browser:page for="Products.CMFPlone.Portal.PloneSite" name="contact_form" class=".browser.ContactFormView" permission="zope2.View" /> </configure>

Juillet 18, 2008
» Going to OSCON


I am leaving tomorrow, heading to Portland, OR, to OSCON. My talk on zc.buildout and Plone will be thurdsay the 24th, and I’ll be there the whole week.

If you are going there and want to meet leave me a note, I am looking forward to meet other geeks there. :D

Juillet 17, 2008
» nose doctest plugin sucks

En ce moment je bosse sur une application en Pylons. J'adore ce petit framework, mais y a un truc que je pouvais pas encadrer, c'est de faire des tests avec des TestCase. Je préfère de loin les doctests.

Me voilà donc partit à la recherche de docs pour pouvoir écrire mes tests comme j'aime les écrire. Pylons utilise nose comme framework de test. Je découvre alors avec joie que nose fournit un plugin pour parcourir les doctests. Chouet !

Le problème, c'est que ce plugin est carrément rudimentaire. En gros, il choppe vos doctest et les initialise ultra basiquement. Comprendre: impossible de passer des options telles que optionflag, setUp ou tearDown. En bref, ça pu. Comment je fais pour initialiser mon framework Pylons pour mes tests moi ? Hein ?

J'ai finalement trouvé une solution en surclassant la classe doctest.DocFileCase afin de faire ce que je veux. Voici le code en question. Il suffit de le placer dans le fichier tests/functional/test_docs.py de votre application Pylons:

# -*- coding: utf-8 -*-
import os
import doctest
import mypylonsapp
from mypylonsapp.tests import *

optionflags = (doctest.ELLIPSIS |
               doctest.NORMALIZE_WHITESPACE |
               doctest.REPORT_ONLY_FIRST_FAILURE)

dirname = os.path.join(os.path.dirname(mypylonsapp.__file__), 'docs')


def build_testcase(filename):
    name = os.path.splitext(filename)[0]
    path = os.path.join(dirname, filename)

    class Dummy(doctest.DocFileCase, TestController):
        def __init__(self, *args, **kwargs):
            # init pylons stuff
            TestController.__init__(self, *args, **kwargs)

            # get tests from file
            parser = doctest.DocTestParser()
            doc = open(self.path).read()
            test = parser.get_doctest(doc, globals(), name, self.path, 0)

            # init doc test case
            doctest.DocFileCase.__init__(self, test, optionflags=optionflags)

        def setUp(self):
            """init pylons stuff and make app available in doctest
            """
            TestController.setUp(self)
            test = self._dt_test
            test.globs['app'] = self.app

        def tearDown(self):
            """cleaning
            """
            TestController.tearDown(self)
            test = self._dt_test
            test.globs.clear()

    # generate a new class for the file
    return ("Test%s" % name.title(),
            type('Test%sClass' % name.title(), (Dummy,), dict(path=path)))

for filename in os.listdir(dirname):
    if filename == '.svn':
        continue
    name, klass = build_testcase(filename)
    exec "%s =  klass" % name

# clean namespace to avoid test duplication
del build_testcase, filename, name, klass

Vous admirerez la ruse qui est de générer une nouvelle classe pour chaque fichier trouvé dans le répertoire contenant les doctests.

On peut ensuite créer un fichier texte dans docs/ et y écrire des tests du genre:

>>> response = app.get(url_for(controller='main', action="index"))
>>> print response
Response: 200
...

Ce qui est tout de même vachement plus convi qu'un test classique.

Juillet 16, 2008
» Using z3c.form for our forms in Plone


I’ve been developing complex forms for a Plone project these days, and I get the job done with z3c.form, the form framework that was first used in the Plone world by the Singing & Dancing project.
Here is the first episode of a small serie to show interesting tips & tricks and patterns that I’ve been learning in the process.

To get started, install your buildout using fake eggs and the required dependencies (plone.z3cform, z3c.form…) You might want to follow the howto contributed by Daniel Nouri on plone.org.

(Note: The code snippets are simplified for easier reading.)

First things first !

z3c.form’s first concept, as you guess, is the “form”, which basically has the list of fields defined for it using an attribute called ‘fields’. From the form, it is also possible to access the list of field widgets using the ‘widgets’ attribute.
The framework is smartly engineered using Zope Component Architecture, so you have “separation of concerns” in every bit, and works with zope.schema for the fields definition and validation.

Note that, at least with the core, you can define a basic Form or a specific Add/Edit/DisplayForm, and other cases such as Group (a group of fields part of a Form) and Subform.

To define the list of fields for a form class, we must provide a schema (for example, IContactData), which I like to think of as the “data model” specification.

# my.example/my/example/interfaces.py from zope.interface import Interface, Invalid, invariant from zope import schema class IContactData(Interface): """Contact data interface """ firstname = schema.TextLine(title=u"Firstname", required=True) lastname =schema.TextLine(title=u"Lastname", required=True) email =schema.TextLine(title=u"Email", required=False) @invariant def email_format(obj): if obj.email.find('@') == -1: raise Invalid(u"Not a valid email")

Defining a storage… if you need to

Since, we generally need to store something, let’s choose the data storage. Though you could choose to do that later.
There are many options (including RDB-based), but the immediate one for us is using the ZODB.
The most simple way to do that, in most real-world apps, is by defining an object class that brings persistency, traversal, security and all the goodies we get in Zope for “free”, i.e. by inheriting from OFS.SimpleItem.SimpleItem (or OFS.Folder.Folder if we want containment) and defining the attributes it needs.

# my.example/my/example/contact.py from zope import interface from zope.schema.fieldproperty import FieldProperty import OFS from my.example.interfaces import IContactData class MyContact(OFS.SimpleItem.SimpleItem): """Contact model class, with ZODB-based storage. """ interface.implements(IContactData) firstname = FieldProperty(IContactData['firstname']) lastname = FieldProperty(IContactData['lastname']) email = FieldProperty(IContactData['email']) def __init__(self, id, **kw): self.id = id for key, value in kw.items(): setattr(self, key, value) super(MyContact, self).__init__(id) @property def title(self): return "%s %s" % (self.firstname, self.lastname)

What I like in this pattern: It’s simple, it’s pythonic ! And it does the job for most cases where we don’t need a full-featured Plone content type.
Of course, if we need to manage a full-featured content type, we can inherit from plone.app.content base classes and bring the required Plone mechanisms on the table.

For those who are new to zope.schema, the Field Property mechanism, is very handy too. It does the job of providing data validation based on the information found in the schema.

Defining an “add” form for our Contact objects

Now the really new stuff starts !

An AddForm (and EditForm) could be used by a Content Manager to maintain a list of contacts in a folder within the site. You do that by inheriting from z3c.form.form.AddForm and providing your create(), add() and nextURL() methods.

# my.example/my/example/browser.py import datetime from zope import schema import zope.component import z3c.form from plone.z3cform import base from plone.i18n.normalizer.interfaces import IIDNormalizer from my.example import interfaces from my.example.contact import MyContact class ContactAddForm(z3c.form.form.AddForm): """ Contact Add Form """ fields = z3c.form.field.Fields(interfaces.IContactData) def create(self, data): id = data['firstname'] + data['lastname'] id = zope.component.queryUtility(IIDNormalizer).normalize(id) self._name = id contact = MyContact(self._name, **data) return contact def add(self, obj): # Add the object within context, and persist it ! context = self.context context[self._name] = obj def nextURL(self): return "%s/%s" % (self.context.absolute_url(), self._name)

There is one last thing to do, to make sure that our form can be rendered via Plone’s presentation machinery like any other page ; we define a special View by inheriting from the FormWrapper class provided by the plone.z3cform package, as follows :

# my.example/my/example/browser.py class ContactAddFormView(base.FormWrapper): form = ContactAddForm label= "Contact Add Form"

And don’t forget to add the configuration for this in the right ZCML file, something along the lines of:

<configure xmlns="http://namespaces.zope.org/zope" xmlns:browser="http://namespaces.zope.org/browser" xmlns:five="http://namespaces.zope.org/five" i18n_domain="my.example" > <browser:page for="Products.CMFPlone.Portal.PloneSite" name="contact_add_form" class=".browser.ContactAddFormView" permission="cmf.AddPortalContent" /> </configure>

Update: Of course, we render our form after restarting Zope, via the URL http://plonesite/@@contact_add_form.

Of course, in a real case you would use another context for the container, by affecting the right interface to the ‘for’ attribute in the configure.zcml, instead of the ‘Products.CMFPlone.Portal.PloneSite’ value.

Juillet 9, 2008