By Ingeniweb. A Django site.
Juillet 19, 2008
» A variation on my previous z3c.form example


Another example of form logic, adding to the one discussed in my previous post.

Here is the idea: If you need a more flexible solution for your use case, you can inherit directly from z3c.form.form.Form.
Let’s say you want the form to be used for both adding contact information (making sure later that this only works when the user is authenticated) and sending a mail message to the site administrator. A site feedback form that will at the same time be used to populate your database of contacts.

Your form class could be defined as follows (I’m skipping the imports part):

# my.example/my/example/browser.py class ContactForm(z3c.form.form.Form): """ Contact Form """ fields = z3c.form.field.Fields(interfaces.IContactData) message_field = z3c.form.field.Fields(schema.TextLine(__name__ = 'message', title=u"Message", required=False) ) fields += message_field ignoreContext = True @z3c.form.button.buttonAndHandler(u'Send', name='send') def handle_send(self, action): data, errors = self.extractData() if errors: self.status = z3c.form.form.EditForm.formErrorsMessage return # Add the contact data, if not in yet id = data['firstname'] + data['lastname'] id = zope.component.queryUtility(IIDNormalizer).normalize(id) if id not in self.context.objectIds(): self._name = id contact = MyContact(self._name, **data) self.context[self._name] = contact # Do something else, e.g. send the message to the site admin. # Complete the code as needed... message = data['message'] print "Message from %s %s (%s):\n%s" % (data['firstname'], data['lastname'], data['email'], data['message']) # Redirect self.request.RESPONSE.redirect(self.context.absolute_url())

The main points of the used pattern:

  • You define the fields for the form ; new fields can be added to the already defined list (based on the IContactData schema in interfaces.py) using the “+” operator.
  • You define your specific form button(s) and handler method(s). In the case of a standard AddForm, there is already a handle_add() defined for the “add” button for you.
  • Also, you need to set the form attribute ‘ignoreContext’ to True, so that the form has the same behaviour as an AddForm, i.e. it does not have to get/set data on attributes on the context. Note that by default, an AddForm has this attribute set to True, and an EditForm has it set to False.

Now, the final touch with the wrapping view…

class ContactFormView(base.FormWrapper): form = ContactForm label= "Contact Form"

… and its configuration:

<configure xmlns="http://namespaces.zope.org/zope" xmlns:browser="http://namespaces.zope.org/browser" xmlns:five="http://namespaces.zope.org/five" i18n_domain="my.example" > <browser:page for="Products.CMFPlone.Portal.PloneSite" name="contact_form" class=".browser.ContactFormView" permission="zope2.View" /> </configure>

Juillet 18, 2008
» Going to OSCON


I am leaving tomorrow, heading to Portland, OR, to OSCON. My talk on zc.buildout and Plone will be thurdsay the 24th, and I’ll be there the whole week.

If you are going there and want to meet leave me a note, I am looking forward to meet other geeks there. :D

Juillet 17, 2008
» nose doctest plugin sucks

En ce moment je bosse sur une application en Pylons. J'adore ce petit framework, mais y a un truc que je pouvais pas encadrer, c'est de faire des tests avec des TestCase. Je préfère de loin les doctests.

Me voilà donc partit à la recherche de docs pour pouvoir écrire mes tests comme j'aime les écrire. Pylons utilise nose comme framework de test. Je découvre alors avec joie que nose fournit un plugin pour parcourir les doctests. Chouet !

Le problème, c'est que ce plugin est carrément rudimentaire. En gros, il choppe vos doctest et les initialise ultra basiquement. Comprendre: impossible de passer des options telles que optionflag, setUp ou tearDown. En bref, ça pu. Comment je fais pour initialiser mon framework Pylons pour mes tests moi ? Hein ?

J'ai finalement trouvé une solution en surclassant la classe doctest.DocFileCase afin de faire ce que je veux. Voici le code en question. Il suffit de le placer dans le fichier tests/functional/test_docs.py de votre application Pylons:

# -*- coding: utf-8 -*-
import os
import doctest
import mypylonsapp
from mypylonsapp.tests import *

optionflags = (doctest.ELLIPSIS |
               doctest.NORMALIZE_WHITESPACE |
               doctest.REPORT_ONLY_FIRST_FAILURE)

dirname = os.path.join(os.path.dirname(mypylonsapp.__file__), 'docs')


def build_testcase(filename):
    name = os.path.splitext(filename)[0]
    path = os.path.join(dirname, filename)

    class Dummy(doctest.DocFileCase, TestController):
        def __init__(self, *args, **kwargs):
            # init pylons stuff
            TestController.__init__(self, *args, **kwargs)

            # get tests from file
            parser = doctest.DocTestParser()
            doc = open(self.path).read()
            test = parser.get_doctest(doc, globals(), name, self.path, 0)

            # init doc test case
            doctest.DocFileCase.__init__(self, test, optionflags=optionflags)

        def setUp(self):
            """init pylons stuff and make app available in doctest
            """
            TestController.setUp(self)
            test = self._dt_test
            test.globs['app'] = self.app

        def tearDown(self):
            """cleaning
            """
            TestController.tearDown(self)
            test = self._dt_test
            test.globs.clear()

    # generate a new class for the file
    return ("Test%s" % name.title(),
            type('Test%sClass' % name.title(), (Dummy,), dict(path=path)))

for filename in os.listdir(dirname):
    if filename == '.svn':
        continue
    name, klass = build_testcase(filename)
    exec "%s =  klass" % name

# clean namespace to avoid test duplication
del build_testcase, filename, name, klass

Vous admirerez la ruse qui est de générer une nouvelle classe pour chaque fichier trouvé dans le répertoire contenant les doctests.

On peut ensuite créer un fichier texte dans docs/ et y écrire des tests du genre:

>>> response = app.get(url_for(controller='main', action="index"))
>>> print response
Response: 200
...

Ce qui est tout de même vachement plus convi qu'un test classique.

Juillet 16, 2008
» Using z3c.form for our forms in Plone


I’ve been developing complex forms for a Plone project these days, and I get the job done with z3c.form, the form framework that was first used in the Plone world by the Singing & Dancing project.
Here is the first episode of a small serie to show interesting tips & tricks and patterns that I’ve been learning in the process.

To get started, install your buildout using fake eggs and the required dependencies (plone.z3cform, z3c.form…) You might want to follow the howto contributed by Daniel Nouri on plone.org.

(Note: The code snippets are simplified for easier reading.)

First things first !

z3c.form’s first concept, as you guess, is the “form”, which basically has the list of fields defined for it using an attribute called ‘fields’. From the form, it is also possible to access the list of field widgets using the ‘widgets’ attribute.
The framework is smartly engineered using Zope Component Architecture, so you have “separation of concerns” in every bit, and works with zope.schema for the fields definition and validation.

Note that, at least with the core, you can define a basic Form or a specific Add/Edit/DisplayForm, and other cases such as Group (a group of fields part of a Form) and Subform.

To define the list of fields for a form class, we must provide a schema (for example, IContactData), which I like to think of as the “data model” specification.

# my.example/my/example/interfaces.py from zope.interface import Interface, Invalid, invariant from zope import schema class IContactData(Interface): """Contact data interface """ firstname = schema.TextLine(title=u"Firstname", required=True) lastname =schema.TextLine(title=u"Lastname", required=True) email =schema.TextLine(title=u"Email", required=False) @invariant def email_format(obj): if obj.email.find('@') == -1: raise Invalid(u"Not a valid email")

Defining a storage… if you need to

Since, we generally need to store something, let’s choose the data storage. Though you could choose to do that later.
There are many options (including RDB-based), but the immediate one for us is using the ZODB.
The most simple way to do that, in most real-world apps, is by defining an object class that brings persistency, traversal, security and all the goodies we get in Zope for “free”, i.e. by inheriting from OFS.SimpleItem.SimpleItem (or OFS.Folder.Folder if we want containment) and defining the attributes it needs.

# my.example/my/example/contact.py from zope import interface from zope.schema.fieldproperty import FieldProperty import OFS from my.example.interfaces import IContactData class MyContact(OFS.SimpleItem.SimpleItem): """Contact model class, with ZODB-based storage. """ interface.implements(IContactData) firstname = FieldProperty(IContactData['firstname']) lastname = FieldProperty(IContactData['lastname']) email = FieldProperty(IContactData['email']) def __init__(self, id, **kw): self.id = id for key, value in kw.items(): setattr(self, key, value) super(MyContact, self).__init__(id) @property def title(self): return "%s %s" % (self.firstname, self.lastname)

What I like in this pattern: It’s simple, it’s pythonic ! And it does the job for most cases where we don’t need a full-featured Plone content type.
Of course, if we need to manage a full-featured content type, we can inherit from plone.app.content base classes and bring the required Plone mechanisms on the table.

For those who are new to zope.schema, the Field Property mechanism, is very handy too. It does the job of providing data validation based on the information found in the schema.

Defining an “add” form for our Contact objects

Now the really new stuff starts !

An AddForm (and EditForm) could be used by a Content Manager to maintain a list of contacts in a folder within the site. You do that by inheriting from z3c.form.form.AddForm and providing your create(), add() and nextURL() methods.

# my.example/my/example/browser.py import datetime from zope import schema import zope.component import z3c.form from plone.z3cform import base from plone.i18n.normalizer.interfaces import IIDNormalizer from my.example import interfaces from my.example.contact import MyContact class ContactAddForm(z3c.form.form.AddForm): """ Contact Add Form """ fields = z3c.form.field.Fields(interfaces.IContactData) def create(self, data): id = data['firstname'] + data['lastname'] id = zope.component.queryUtility(IIDNormalizer).normalize(id) self._name = id contact = MyContact(self._name, **data) return contact def add(self, obj): # Add the object within context, and persist it ! context = self.context context[self._name] = obj def nextURL(self): return "%s/%s" % (self.context.absolute_url(), self._name)

There is one last thing to do, to make sure that our form can be rendered via Plone’s presentation machinery like any other page ; we define a special View by inheriting from the FormWrapper class provided by the plone.z3cform package, as follows :

# my.example/my/example/browser.py class ContactAddFormView(base.FormWrapper): form = ContactAddForm label= "Contact Add Form"

And don’t forget to add the configuration for this in the right ZCML file, something along the lines of:

<configure xmlns="http://namespaces.zope.org/zope" xmlns:browser="http://namespaces.zope.org/browser" xmlns:five="http://namespaces.zope.org/five" i18n_domain="my.example" > <browser:page for="Products.CMFPlone.Portal.PloneSite" name="contact_add_form" class=".browser.ContactAddFormView" permission="cmf.AddPortalContent" /> </configure>

Update: Of course, we render our form after restarting Zope, via the URL http://plonesite/@@contact_add_form.

Of course, in a real case you would use another context for the container, by affecting the right interface to the ‘for’ attribute in the configure.zcml, instead of the ‘Products.CMFPlone.Portal.PloneSite’ value.

Juillet 9, 2008
» shutil.copytree small improvement


When you have to work with directories and files, there’s a very common pattern: copying a directory into another, but filtering out a few files and directories.

For instance, if you want to copy a directory that contains source code, you will probably remove .pyc files and .svn directories if you work with Subversion. In that case, shutil.copytree cannot be used, so os.walk is the usual way to go (rough example):

import os
from os.path import join, splitext, split, exists
from shutil import copyfile

def copy_directory(source, target):
    if not os.path.exists(target):
        os.mkdir(target)
    for root, dirs, files in os.walk(source):
        if '.svn' in dirs:
            dirs.remove('.svn')  # don't visit .svn directories           
        for file in files:
            if splitext(file)[-1] in ('.pyc', '.pyo', '.fs'):
                continue
            from_ = join(root, file)           
            to_ = from_.replace(source, target, 1)
            to_directory = split(to_)[0]
            if not exists(to_directory):
                os.makedirs(to_directory)
            copyfile(from_, to_)

This is a lot of boiler-plate code, so I usually create a small function that accepts more arguments to filter out files and directory. But Python should provide this pattern in the standard library.

I have proposed a patch for shutil.copytree, to integrate filtering capability. It has been reviewed and commited in the trunk this week, so we will have it in Python 2.6.  Now copytree comes with an ignore argument that has to be a callable. If given it will be called on each visited directory to decide what is copied and what is not.

There’s a default callable in shutil called ignore_patterns, that can be used to filter out files with glob-style patterns. I have added this example to Python doc:

from shutil import copytree, ignore_patterns
copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*'))

Pretty straight forward ! :D
More info and examples here : http://docs.python.org/dev/library/shutil.html#shutil.copytree

Juillet 8, 2008
» Packager ses scripts Python

Bon nombre de gens utilise python pour faire de petits scripts. Le problème c'est que pour les distribuer, ensuite, c'est pas le top.

Heureusement il y a distutils !!

distutils est un paquet inclus dans les distributions python permettant de créer des paquet python.

Le principe est simple. On englobe un module python dans un paquet contenant un fichier setup.py

Le plus simple est d'utiliser paste pour créer son paquet. Renseignez bien les information demandées. Elles seront visible si vous décidez de distribuer votre paquet par la suite. Donc:

$ easy_install -U Paste
$ paster create monscript
$ cd monscript
$ ls
monscript/ monscript.egg-info/ setup.cfg setup.py

Ceci nous créer un répertoire monscript contenant un setup.py et un sous répertoire destiné à recevoir le code python.

Nous devons maintenant créer un point d'entrée pour notre script. Pour cela, nous allons modifier monscript/__init__.py pour qu'il ressemble à ça:

def main():
    print 'Yeah !'

Ensuite, en modifiant le fichier setup.py, nous pouvons associer ce point d'entrée à un véritable script qui sera installé à l'installation du paquet. Modifiez la section entry_points du setup.py pour qu'il ressemble à quelque chose du du genre:

entry_points="""
# -*- Entry points: -*-
[console_scripts]
mon_super_script = monscript:main
""",

Voilà, le tour est joué. Alors, pourquoi tout cela pour un simple script ? C'est simple. Vous pouvez maintenant aisément le distribuer.

Voici les principales commandes qui vous serons utiles:

  • créer un tarball:

    $ python setup.py sdist
    
  • créer un egg:

    $ python setup.py bdist_egg
    
  • rendre le paquet disponible sur pypi:

    $ python setup.py sdist bdist_egg register upload
    

Un utilisateur lambda pourra ensuite l'installer simplement:

  • via le tarball:

    $ wget http://exemple.com/monscript-0.1.tar.gz
    $ tar monscript-0.1.tar.gz
    $ cd monscript
    $ python setup.py install
    
  • via pypi:

    $ easy_install -U monscript
    

Moralité, distutils rends la vie plus facile.

» iw.recipe.pound is dead, viva plone.recipe.pound


Please notice that iw.recipe.pound was renamed to plone.recipe.pound. So if you have some buildouts with iw.recipe.pound please their configuration so they use plone.recipe.pound. iw.recipe.pound is not longer maintained.

New option in 0.5 release : socket path (thanks Mathieu) , fixed some doctests (Cheettah requirement), added a run script (as runzope) in bin directory (thanks to Rocky)

Regards Youenn.

Juillet 4, 2008
» Write your first Zope 2 product


‘The topic is to make understandable what is a Zope product in the python world.

What is a python module?

A Python module is a directory on your filesystem that contain at least a file called __init__.py. This file can be empty, but the name is fixed.

How Python can find such a directory?
The standard path where python search for modules is in the directory site-packages of your Python installation. By default you find it in C:Python2.4Libpython under windows or in /usr/lib/python2.4 un Unix-like systems (Linux, Mac OS, BSD, etc). You can extend this search by adding the PYTHONPATH variable in your environment.

@set PYTHONPATH=C:Zope2.9libpython

or

export PYTHONPATH=/usr/lib/python2.4:/opt/Zope2.9/lib/python

In a Python module every sub-directory that contains a file named __init__.py is considered as a sub-module.
On these points Zope doesn’t differ from Python

What is a Zope Products?

You can divide your Zope installation in three parts: the Zope framework, the Zope Server and the ZODB. A Zope product is an application for the Zope Framework. You can use it alone, without the Zope Server or the ZODB. Actually it’s the failure of Zope to not have any public software based on the Zope Framework without the Zope Server.
Zope 2 is using a specific code to load products to register them as specific applications. You can read it in your Zope installation in the file OFS/Application.py.

During the warm up the Zope Server is looking in the zope.conf file for the ‘products‘ keywork. Each time it is defined this keywork is following by a filesystem path where Zope Products are uncompressed.
This list of filesystem paths is initialized with the two paths:

  1. the lib/python/Products directory of your Zope installation
  2. the Products directory of the Zope instance

The first one contains base application you need to work like ZCatalog, PythonScripts (for ZMI). You should never add a product in there and read the documentation available in these products will help you to understand Zope Framework standards.
The second one is empty and is designed to store cusomer Zope Products.

Implement your first Zope 2 product

To create your product you create a Python module in the Products directory. You should Have something like this:

Products

> MyProduct

> __init__.py

To be recognized as a Zope Product the __init__.py file must contain a function called initialize. This function is mainly declarative. You can read a full sample on the document ” Write a toolbox for a Plone 2.5 product “.
The main goal of the initialize function is to register your new classes into the good drawer in the Zope Application. In the sample we register a new tool object but you will do the same for content types, i18n domains, etc.

» Write a toolbox for a Plone 2.5 product


For years we have had the need to embed a utility in Plone products that allows us to execute private methods in restricted Python mode. You need such a utility when using a ZMI Python script in a Page Template or in an Expression. The last one is a CMFCore class that is used in workflow transition condition or in Archetypes field condition for example. To be able to do this utility you need to create a ‘portal tool’ for your site with some public methods. Here I will show you how to build a very basic utility that I can use in the next blog entry.

What is a tool

In pure Zope 2/CMF, a tool is a unique item with a forced id that inherits from the SimpleItem class. An instance of a tool is always created on the root of your Plone site to be acquirable from everywhere in the site.

The CMFCore product provides a function that allows us to call a tool in an unique way: ‘getToolByName’.
In python modules, external methods, or python scripts, you can use following code:

from Products.CMFCore.utils import getToolByName
mytool = getToolByName(obj or context, 'tool id', default value is not found - None if not specified)

You can use it in Page Template too, using the ‘module’ call:

python:modules['Products.CMFCore.utils'].getToolByName(here, 'tool id')

Implementation

What should a tool module contain

Now back to our implementation.

First you need to have a product to write in. To create a new product please read this howto , or download someone else’s to play with.
In your product create a file called ‘tool.py,’ if there will be only one tool in your product, or ‘myfunctiontool.py’ if there will be more than one. As a best practice, I recommend to always choose the second way.

A simple tool code could be:

from OFS.SimpleItem import SimpleItemfrom Products.CMFCore.utils import UniqueObject

from AccessControl import ClassSecurityInfo

from Globals import InitializeClass

class ProductNameTool(SimpleItem, UniqueObject):

    """ Description of what your tool handle
    """

    id = 'portal_productname'

    meta_type = 'Product Name Tool'

    manage_options = SimpleItem.manage_options
    security = ClassSecurityInfo()

InitializeClass(ProductNameTool)

Description of imported items:

  • SimpleItem is a class coming from Zope that ensures us that we have persistence and base views available in ZMI.
  • UniqueObject is a utility class coming from CMF that ensures us that nobody can rename an instance of it. It’s important for us because our tool must never be renamed by error.
  • ClassSecurityInfo is the class to use in every Zope persistent class in order to define the security of each method.
  • InitializeClass is the helper function that registers our class in the Zope context.

Description of class attributes:

  • The id must be unique on your site root and should start with ‘portal_’ or end with ‘_tool’. If your tool unherit from ZCatolog the id must be suffixed by ‘_catalog’.
  • All classes must have a meta type. You can just extend your class name with spaces.
  • manage_options contains the tabs displayed in the ZMI.
  • security defines the security in your class. All classes must contain such a declaration.

Initialize our tool at Zope startup

Now that Zope knows that you have a tool, what about Plone? Plone requires an extra step in order to register our tool (full details in: “Write your first Zope 2 product“.
We have to edit the __init__.py file of our product and add following lines:
 

from Products.CMFCore.utils import ToolInit

from myfunctiontool import ProductNameTool

tools = (ProductNameTool,)

def initialize( context ):

    ToolInit( 'My Product Tools'
                , tools=tools
                , product_name='ProductName'
                , icon='tool.gif'
                ).initialize( context )

ToolInit is a class that handles tool registration for us. We can register all tools defined in our product in only one pass.
ProductNameTool is our tool class. Here we need to import all tool classes we want to register.

At last, we call the ToolInit constructor and run its ‘initialize‘ method to build our tool factories:

  • the first parameter is the name viewed in the Add dropdown menu in the ZMI
  • tools is a tuple of classes
  • product_name should contain the name of your product
  • icon is the name of the icon displayed in the ZMI (16×16) that must be on the root of your product

We can restart Zope then add the tool in the ZMI. After that, we can add methods to this tool. Most often these methods are public or protected by ‘View’ or ‘Manage portal’ permissions.

Future of Zope 2 tools

In Zope 3 tools are becoming utilities. In Plone 3 we are in limbo, with all old Zope 2 tools registered both as tools and as utilities. In the near future, however, only the utilities will be used. Use Tool for Plone 2.5 development only.

You can read more about utilities in Zope 3 and Plone 3 documentation.

Juillet 2, 2008
» plone.org migration


Plone.org migration to Plone 3 is taking a bit longer than expected, but it should turn into reality soon.

There will be many improvements on the set of packages the website uses (I am thinking in particular about Maurits’s work on POI that will speed up the trackers, but it is just an example), and blobs should be used for the products section (more than 700 projects are registered there).

I worked last week-end on the products section, by finishing collective.psc.mirroring, which will copy all packages that are uploaded at plone.org into a file system directory. This directory will be published directly by Apache so the website will become a new package location for zc.buildout (find-links section) and easy_install calls without invoking the Plone instance.

Now I am focusing on PloneSoftwareCenter (PSC) migration. It is a pretty interesting topic: for every project located in the products folder with releases, I am going to extract its “distutils ids”. These ids are the name set in the setup.py file for each release.

I will then look at PyPI through XML-RPC if the package is also released, using the id. In that case, and if the author email is the same on both side, I will validate that the project on plone.org “owns” the given distutils id. From there PSC will act like PyPI and will reject uploads of packages if the user is not the owner of the project that owns the package id.

Of course there will be errors and some people might feel like their package has been hijacked if they cannot upload their packages. But this should be minor and should be OK after a while. A mail will soon be send to the community to ask people to check that they are synced between PyPI and plone.org.

I am really excited about this work because plone.org will then be compatible with distutils register and upload commands, which means that people will be able to update plone.org products section like they do with PyPI : through a single commande line.

Hey Sidnei, what you thought of several years ago is about to turn true. ;)

Edit: My apologies goes to Maurits, who did the work on POI, not Reinout, his brother ;)

Juin 24, 2008
Bertrand Mathieu
bertrand
Paste here is about »
» Five 1.5.6 testbrowser and recent version of mechanize

Doing development with Plone 3.1, I encountered a KeyError when I tried to use Five.testbrowser.Browser:

Browser()
KeyError: '_seek'
I eventually found that my system (currently Ubuntu 8.04) had the package "python-mechanize" 0.1.7b, which masks the one provided by Zope 2.10 (0.1.2b). Unfortunatly there has been API changes between 0.1.2b and 0.1.7b... (see classes UserAgent/UserAgentBase)

The solution is to tell buildout to get mechanize egg 0.1.2b: this way it will mask the system library.
[buildout]
versions = versions
eggs=
...
mechanize

[versions]
mechanize = 0.1.2b
Et voila!

Juin 21, 2008