By Ingeniweb. A Django site.
Août 23, 2010
» Current job: Enter phase 2

I’ve now spent 10 months in my current job, and I owe to myself to stop today for a quick restrospective before moving to another phase.

The first period (phase 1) helped me understand the context and what can be done to bring value. Fortunately, a lot can be done, so the challenge is interesting. Also, I was lucky to quickly understand and accept that, in the current context, change could only happen progressively.

Saying that nothing interesting has happened so far with the projects I am in charge of, would not be true. But the whole process is too slow (or sometimes chaotic), for unnecessary reasons.

My phase 2 is going to involve more initiatives to pick up the pace on the projects and get things moving. Wish me good luck or my next related post will be about asking if someone has a job opportunity for me ;)

As for other news, I restarted developing for personal projects with my favourite CMS (which is preparing a new release). I wish all success to Plone 4 and the community. If you need a CMS for a critical/complex site project, Plone is undoubtly in the top 2 or 3. Also, we have a great community which opens to various technologies and development practices besides Plone such as Zope Toolkit, ZODB, Deliverance, deco.gs, and BFG, to only name a few.


Avril 29, 2010
» AllowedContentType in Plone2.5

Hello,

I notice that the method allowedContentTypes cost time when you have a lot of type in plone2.5.I don’t know if plone3 or plone4 are impacted of that. I have 120 types and the time of execution of context.allowedContentTypes is about 0.32sec.

The path of allowedContentTypes is :

allowedContentTypes -> portal_types.listTypeInfo -> for each content type: portal_types.isConstructionAllowed -> portal_types._queryFactoryMethod -> Products.Five.pythonproducts.patch_ProductDispatcher__bobo_traverse__ -> Products.Five.pythonproducts.product_packages

and Products.Five.pythonproducts.product_packages time call is 0,003s

When you have 100 content type only product_packages is responsible of 0,3sec , this method is a performance bottleneck.

So add this patch fix the problem :

from Products.Five import pythonproducts
old_product_packages = pythonproducts.product_packages
pythonproducts.product_packages =  forever.memoize(old_product_packages)

Regards Youenn


Avril 7, 2010
» New Release of iw.fss

I’ve just released a new version of iw.fss (2.8rc2 for plone 3) and FileSystemStorage (2.6.3 the same but for plone2.5).

Those releases change the behaviour of getData method which retrieve data from filesystem.
This change is important in case of big file because now all data are handled by filestream_iterator (in all case). So no memory is consumed when we access directly to the data (if you respect this new API)

For the developper point of view , getData works like OFS.Pdata. So to get all data (don’t do that please) just call str(field.get(instance)), and to get the first block of data call field.get(instance).data and you have a pointer to the next block in calling field.get(instance).next.

So if you are using iw.fss for yours projects please update it, our plone will thank you after that !!

Regards Youenn


Mars 4, 2010
» Resizing and cropping thumbnails with jquery

Resizing and cropping images with jquery

Sometimes, we need images with fixed dimensions (carousels, photo albums …), but all images don’t have the good dimensions or good ratio height /width.

This little jquery script will help you for this usecase.

In this example we will fix the dimension and position of thumbnails placed in blocks using the class « imageContainer ». « imageContainer » have a fixed width and height (in this example 250px/150px). The size of thumbs will be increased or reduced, and the thumb will be moved inside its container to get a centered cropping. Of course images and its containers must have a relative position.

Your html template code looks like :

  <a class="imageContainer" href="the_link">
    <img src="the_image_src" alt="" width="450" height="300" />
  </a>
  <a class="imageContainer" href="another_link">
    <img src="another_image_src" alt="" width="200" height="100" />
  </a>

In your css, you must fix at least :

.imageContainer,
.imageContainer img {
  position: relative;
}

.imageContainer {
  width: 250px;
  height: 150px;
  overflow: hidden;
  border: 1px solid grey;
}

The javascript code is simple :

var canImproveResolution = false;

thumbResize = function(thumb, min_width, min_height, orientation) {
    twidth = jQuery(thumb).width();
    theight = jQuery(thumb).height();
    new_height = theight;
    new_width = twidth;
    // strange 1px bug on MSIE
    if (jQuery.browser.msie) {
        min_width = min_width+1;
        min_height = min_height+1;
    }
    //calculate the good size
    if (orientation=='landscape') {
        ratio = min_width/min_height;
        tratio = twidth/theight;
        if (tratio>ratio) {
            new_height = min_height;
            new_width = parseInt(new_height*tratio);
        }
        else {
            new_width = min_width;
            new_height = parseInt(new_width/tratio);
        }
    }
    else {
        ratio = min_height/min_width;
        tratio = theight/twidth;
        if (tratio>ratio) {
            new_width = min_width;
            new_height = parseInt(new_width*tratio);
        }
        else {
            new_height = min_height;
            new_width = parseInt(new_height/tratio);
        }
    }

    // resize thumb
    if (theight!=new_height || twidth!=new_width) {
        jQuery(thumb).height(new_height);
        jQuery(thumb).width(new_width);
    }
    // adjust position (vertical and horizontal centering for css cropping)
    if (new_width > min_width) {
        moveleft = parseInt((new_width-min_width)/2);
        jQuery(thumb).css('left', '-'+moveleft+'px');
    }
    if (new_height > min_height) {
        movetop = parseInt((new_height-min_height)/2);
        jQuery(thumb).css('top', '-'+movetop+'px');
    }
    // improve resolution
    if (canImproveResolution && (new_width > twidth || new_height > theight)) improveResolution(thumb);
}

resizeThumbs = function(thumbs, min_width, min_height) {
    orientation = (min_height > min_width) ? 'portrait' : 'landscape';
    // hide images before resizing to avoid bad visual effect
    thumbs.each( function() {
            jQuery(this).css('visibility', 'hidden');
        })
    // use window.load because on document.ready images are not always loaded
    jQuery(window).load(function() {
        thumbs.each( function() {
            thumbResize(this, min_width, min_height, orientation );
            jQuery(this).css('visibility', 'visible');
        })
    });
}

// change image src for a better resolution
// this is just an example for Plone, adapt it with
// your own sizes rules

improveResolution = function(img) {
    img_src = img.src;
    img_src = img_src.replace(/\/image_preview/gi, '/image');
    img_src = img_src.replace(/\/image_mini/gi, '/image_preview');
    img_src = img_src.replace(/\/image_thumb/gi, '/image_mini');
    img.src = img_src;
}

For Plone users, there is a small improvement, if you fix the variable :

canImproveResolution = true;

you will get a better resolution when increasing image size  (works only with standard plone image thumb sizes …).

At last launch the script for the wanted thumbnails :

jQuery(document).ready(function(){
  resizeThumbs(jQuery('.imageContainer img'), 250, 150)});

Février 6, 2010
» Plone and multilingual sites

Usually we build multilingual Plone sites with LinguaPlone.

This solution has a big advantage, it’s generic and very easy to implement in a plone site.

But there are many inconvenients, due to the design of this product (translations are independent by design) :

  • Each translation is a new Archetype object, and it could be a big problem on sites with many contents, the portal objects number is increased by the number of available languages.
  • Translations uses plone references catalog to be linked to the original (called canonical) object, but when moving objects translations are not moved, when copying pasting objects, translations are not pasted, when deleting objects translations are not deleted, when reordering contents, translations are not reordered, when publishing objects translations are not published, … For web masters maintaining a site  with LinguaPlone inside could be a challenge.
  • When translating folders with LP, all translated contents are moved from the canonical folder to the translated folder with same language, translating low level folders on big depth tree sites could take a long long time, don’t be surprise if you get errors.
  • If a content is neutral (with no language attribute), inside a translated folder, it could not be seen when browsing a translation of the parent folder.
  • At last a lower problem :  the translation edit forms are not pretty to use, they show a table with two columns, the first column with the « canonical » content inside in « view » mode, the second column with the translation edit form, in fixed width sites the translated form width is sometimes ridiculously small.

But since many years we use LinguaPlone because it was the only easy way to make multilingual Plone sites.

By the past we were using a LP patch called LinguaFace to reduce the number of problems with LP (synchronisation on reorder, copy-paste, delete, or move – see neutral contents inside all translated folders – more usable translation edit forms …), but LF add a new layer of complexity and maintaining it with all LP versions becomes complicated. See some examples on how it works :

  • when a content is copied, all translations are copied
  • when a content is pasted or moved all translations are pasted or moved at the good place (not so easy)
  • when a content is published or retracted translations follow the same workflow transition
  • when a folder is translated, we don’t see only objects inside but also objects with same canonical Path (a new catalog index) to see also neutral contents.
  • Navtree is patched, breadcrumbs are patched, to use canonical path
  • and so on …

A big nightmare.

To day a new solution exists that store translations inside each Archetypes field, raptus.multilingualfields, and a Plone integration of raptus.multilanguagefields called raptus.multilingualplone that extends the schema of all Plone Content Types making them translatable. raptus.multilingualfields also provides multilingual catalog indexes that return the good translated data when searching for contents or displaying trees, and multilingual criterions for topics.

A LP feature not provided by raptus.multilingualfields is the internationalization of urls, if you really need this feature, i think it’s not a big challenge to add some traversal rules, for me it’s not essential.

AnotherLP  feature that can’t be provided when storing translations in fields is to get different workflows or security settings for each translation. If you need this feature use LinguaPlone, LinguaPlone is done for that, it makes all translated contents independent, but i’m curious to know the number of users who really want this feature, all clients i had never ask for that but finally, after some LP experience, always wanted the exact opposite use-case.

To make your Plone site translatable with raptus.multilanguagefields, you have two choices :

  1. Add raptus.multilanguageplone in your buildout and install it in your Plone Site using extensions products control panel, to make all your Plone content types and derived translatable (all fields for which translation make a sense are translatable).
  2. I prefer integrate by myself raptus.multilanguagefields inside a product, since we could want just some fields or content types translatable, as example i don’t need to translate the images or the files contents, just their titles and descriptions.

How to implement the second solution ?

Just take a look at raptus.multilanguageplone code, it’s easy :

  1. Make your archetype extenders to make your wanted fields translatable, example can be found here
  2. register your extenders in setuphandlers (Generic Setup import step), example here
  3. replace the standard catalog  indexes with multilingual indexes in Generic Setup profile, example here

That’s all, you will get superb edit forms with translatable fields inside, you also will get a google help to translate contents (a pleasant gadget). I say « Bravo » to raptus developpers.

Important :

  • these products are young, and there’s still many work todo to make it work without problems (tests are needed …). Last 0.6 releases have bugs under plone3.3, use the svn versions below instead.
  • raptus.multilanguageplone 0.6  has a bug in extenders with primary fields, tested in Plone 3.3  (fixed in branch aws_evols, not tested with images at this time)
  • raptus.multilanguagefields 0.6 has a bug, doctests are broken in Plone3.3 (fixed in trunk )
  • At this time these products don’t have unit tests or functional tests, it’s the only reproach i can make.  I started the work  here, and here

Décembre 23, 2009
» Migrer vos vieux sites vers Plone 3 ou Plone 4


Vous êtes nombreux à avoir ce souci, et c’est pareil pour moi, de migrer vos anciens sites CPS ou Plone2 vers une technologie plus mature et plus pérenne, à savoir Plone3 ou Plone4.

Ce billet n’a pas la prétention de résoudre ces questions avec un framework sophistiqué et néanmoins intéressant appelé Products.contentmigration mais plus humblement de vous donner quelques astuces bien pratiques.

Par exemple vous ne savez pas comment supprimer ces maudites Access Rules, qui, c’est pas faute de vous avoir prévenu étaient dangereuses, alors c’est facile, vous avez besoin d’un petit import :

from ZPublisher.BeforeTraverse import unregisterBeforeTraverse

et sur chaque objet migré :

rules = unregisterBeforeTraverse(obj, 'AccessRule')
if rules:
     try: del getattr(obj, rules[0].name).icon
     except: pass

Passons à une chose stupide mais lourde de conséquences, lorsque vous migrez des contenus d’un meta_type vers un autre, la plupart du temps il faut copier-coller l’objet et lui ré-attribuer tous ses anciens attributs, mais supposons que l’objet est structurellement le même ou presque, pourquoi s’embêter (imaginez dans le cas d’un dossier à la racine d’un site comme ça peut faire mal), il suffit de :

obj.__class__ = new_class

vous êtes assez malins pour trouver new_class et new_portal_type…

et dans le cas d’un Archetypes object à l’arrivée

obj.updateSchema()

C’est un exemple à ne pas suivre, mais comme tous les exemples de ce type il est bien pratique (ça se compte en jours/semaines de prise de tête en moins)

Sans doute qu’il y aura une suite à ce ticket.

Août 31, 2009
» How to configure an custom vary tag for squid

You want to make an authenticated cache with apache/squid-varnish/plone and you don’t
know how do that : it’s possible with the vary tag.

Vary header tell to proxy cache what’s headers is variant for an object for a cache.
For example if you tell to the cache that the variant is Cookie , then for a same url with different cookie value the result of the cache is different.
The Server send to the proxy (in the response) which header is considered for vary by sending Vary: list of request header name

In Cachefu, you can configure that by rule with varyExpression.

In global configuration of cache fu you can also configure an global vary header. By default this configuration is send with rule.portal_cache_settings.getVaryHeader()

You can activate or desactivate vary with the header_set configuration ( vary field).

Vary headers must be present in the request (not response) of the browser in order to be considered to be variant for the proxy cache. So we are limited with the standard header of the protocol http.

But with cookie and apache (apache is in front of squid) we can elaborate strategy to construct a vary tag more efficient.

The second aspect of the cache work is purge content when the content change.

PURGE of Vary objects is still very poorly supported in squid, and you can only purge one variant at a time and need to get the URL cached again before being able to purge another variant. So how to deal with that also ?

First , how build our vary tag ?

The trick is to construct an custom vary tag with apache.
We can to do this with RewriteRule::

RewriteCond %{HTTP_COOKIE} mycookie="([^"]+) [NC]
RewriteRule ^(.*)$ - [E=mycookie:%1]

So in this example mycookie contains the value of cookie_key

You can add a cookie for the language , a cookie for group , a cookie for a permission and so on and then construct your custom vary tag with values of this specifics cookies with mod_headers

RequestHeader append MyVary %{mycookie}e

And then the value of mycookie is considered to be variant..

If you want have a specific vary tag for anonymous you can test the presence of
__ac cookie and send a custom MyVary in this case

RewriteCond %{HTTP_COOKIE} __ac="([^"]+) [NC]
RewriteRule ^(.*)$ - [E=authenticated:1]

RequestHeader append MyVary %{mycookie}e env=authenticated
RequestHeader append MyVary anonymous env=!authenticated

So now with that you can vary cache as you want. Now how to treat the big deal of purge.

The trick is  have an image (or a ajax request or ..) in content that is never in cache. This image is serve by a browser view (in case of zope application) that set a cookie. This cookie value is added to Vary tag. So the Vary tag change if the value of this cookie change and then the content is updated (for all request).

For example we can construct a cookie with the value of the catalog change

catalog_count = pcs.getCatalogCount()
context.REQUEST.RESPONSE.appendHeader('Pragma','no-cache')
context.REQUEST.RESPONSE.appendHeader('Cache-control', 'no-cache')
cookie = context.REQUEST.cookies.get('X_CACHE_CATALOG', 0)

if cookie != str(catalog_count) :
context.REQUEST.RESPONSE.setCookie('X_CACHE_CATALOG',
catalog_count ,
path="/")
return catalog_count

And in apache we add
RewriteCond %{HTTP_COOKIE} X_CACHE_CATALOG=([^"]+) [NC]
RewriteRule ^(.*)$ - [E=X_CACHE_CATALOG:%1]
RequestHeader append MyVary %{mycookie}e:%{X_CACHE_CATALOG}e env=authenticated

And when catalog change, the vary also (in the second request) and the cache is updated. You can elaborate other strategies for purging vary object with this technique.

The last point is to combined Etag and Vary Header in response. IE with a Vary header don’t treat correctly Etag header and If-None-Match is never sending. So in apache remove the tag Vary and then Etag work well for all browser

Header unset Vary


Août 1, 2009
» The power of Deliverance and PyQuery

Some times ago i made a new skin for the French-speaking PYthon Association (AFPY) web site, using Deliverance and PyQuery. I made this exercice on my free time with many pleasure.

You can see the Deliverance power here :

- The home page without Deliverance, the same using Deliverance proxy : all dynamic contents are taken from original Plone site, try to find it, you will notice the Deliverance + PyQuery magic in action :-)

- The association pages without Deliverance (a sphinx site), the same contents injected in the site with Deliverance

Thanks to Gaël Pasgrimaud, a friend of mine and coworker, for his big work which really help me :

Here, we use a specific Gael’s buildout and a specific package done for our Deliverance projects used at Ingeniweb, our company. With these tools (if some people are interested in, it could be released on pypi, tell us …), we get the best flexibility for Deliverance :

  • PyQuery integration
  • The good base rules for Deliverance to avoid theming bugs with Ajax and HTML editor (*)
  • Dynamic css : css are mako templates served by a skin server, colors fonts and other properties are set in a « .ini » file
  • A base mako css for Plone which permits to start the work of externalize the entire css :
    in our projects, all portal_css content links from Plone are stripped when the Plone site is called through Deliverance (just add a condition for Plone stylesheets in portal_css : python:request.HTTP_HOST in request.SERVER_URL), the final css is lighter, the Deliverance css is really clean without many ‘!important’ bad overloads)
  • Supervisor deamon to control skin servers
  • We can use eggs for our Deliverance skins.

(*) to get real wysiwyg rendering in Plone behind Deliverance ‘s  HTML editor areas, use also FCKeditor.Plone 2.6.4, in FCKeditor control panel you will find the way to inject Deliverance css in wysiwyg area, it’s an important feature for end users.

Now i can take my summer holidays in peace with myself :-)


Mai 23, 2009
» Some Deliverance tips

I really like to use Deliverance for skinning Plone or any web application. In this post, i just want to help Deliverance beginners to configure their server, and to talk about PyQuery, which could be a good Deliverance companion.

Apache 2.2 + Deliverance : configuration

I spent some time to configure Apache in front of Deliverance, you will find here a possible configuration.

Deliverance Server configuration :

In your rules.xml, the first declaration relates to server config, use ’0.0.0.0′  as server address to be able to serve any ip address :

<server-settings>
 <server>0.0.0.0:8000</server>
 <execute-pyref>true</execute-pyref>
 <display-local-files>false</display-local-files>
 <dev-allow>
 localhost
 </dev-allow>
 <dev-user username="dev" password="dev" />
 <dev-expiration>60</dev-expiration>
</server-settings>

Always in rules.xml you need to write the proxy rule for Zope Virtual Host or another server.

If you want to serve a plone site located at http://localhost:8080/myplonesite :

<proxy path="/" rewrite-links="1">
     <dest href="http://localhost:8080/VirtualHostBase/http/{SERVER_NAME}:8000/myplonesite/VirtualHostRoot" />
</proxy>

If you need a more complex configuration in front of Zope, you can always use Apache + Varnish or SQUID + Pound, or IIS + EEP, and ZEO in front of Zope using another port (90 in this example) and use this Deliverance proxy rule :

<proxy path="/ rewrite-links="1">
    <dest href="http://{SERVER_NAME}:90" />
</proxy>

And if you want to serve anything else for another path :

<proxy path="/anotherpath" rewrite-links="1">
    <dest href="http://anythingelse" />
</proxy>

Apache VirtualHost configuration

You can use Deliverance on port 80 (just replace 8000 by 80 in rules.xml) to directly serve your site, but you would prefer Apache for that in many cases.

Suppose you want to rewrite all requests to « www.mysite.org » with a Deliverance response (on www.mysite.org:8000), you can use the mod_rewrite like this :

<VirtualHost *:80>
    ServerName www.mysite.org
    ServerAlias mysite.org
    ErrorLog "logs/error.mysite.org.log"
    CustomLog "logs/access.mysite.org.log" combined

    # do not forget this declaration
    ProxyPreserveHost On
    RewriteEngine On
    RewriteLog "logs/rewrite.mysite.org.log"
    RewriteCond %{HTTP:Authorization} ^(.*)
    RewriteRule ^/(.*) http://%{SERVER_NAME}:8000/$1 [P]

    # etc ...

</VirtualHost>

That’s all.

If you want to serve urls starting with a particular path (/mypath),  just change your rewrite rule like this :

    RewriteRule ^/mypath(.*) http://%{SERVER_NAME}:8000/_vh_mypath$1 [P]

Just take care in this last case to put the good urls in your Deliverance static contents, if you need it .

In fact you can do anything you want with rewrite rules :-)

Deliverance and PyQuery

PyQuery is a jQuery-like library for Python which help you to play easily with html elements. Combined with Deliverance you will be able, with a python rule, to modify content or theme on the fly, at any point of your Deliverance transformation.

To learn how to add PyQuery rules to Deliverance, read this post  (many thanks to Gaël who help me to learn Deliverance) :
http://www.gawel.org/weblog/en/2008/12/skinning-with-pyquery-and-deliverance

Read also  :
http://www.openplans.org/projects/deliverance/lists/deliverance-discussion/archive/2008/12/1229591136002/forum_view

And now just try these example :

You have a floating right toolbar with floating right tabs inside in your html template theme :

<ul id="right-bar">
    <li><a href="/abcd">ABCD</a></li>
    <li><a href="/efgh">EFGH</a></li>
</ul>

You have a floating left toolbar with floating left tabs inside in your html source content :

<ul id="left-bar">
    <li><a href="/abcd">ABCD</a></li>
    <li><a href="/efgh">EFGH</a></li>
</ul>

With Deliverance xml rules you want to replace right tabs by left tabs like this :

    <replace content="children:#left-bar" theme="children:#righ-bar" />

But of course you need to reverse tabs order before or after xml transformation.
To modify the theme  just add a pyquery rule after the replace declaration :

    <pyquery pyref="myskinproduct.rules:pqrules" />

In your rules.py,  create your pqrules function like this :

def pqrules(content, theme, resource_fetcher, log) :

    # if you prefer to work on content
    # before Deliverance transform
    # replace theme('#right-bar') by content('#left-bar')
    nav = theme('#right-bar')
    navitems = nav('li')
    navitems.reverse()
    # i'm not sure that it's needed :
    nav.empty()
    nav.append(navitems)

Other examples :

    # replace a search button text with 'OK' in content
    content('#portal-searchbox .searchButton').val('OK')

    # remove icons from a navigation portlet
    # replace icons by "> "
    nav = content('#navigation')
    navitems = nav('li')
    navitems('img').remove()
    navitems.append('<span>&gt;&nbsp;</span>')

    # add a "last-item" class to last tab in a toolbar
    from pyquery import PyQuery as pq
    navitems = content('#navbar li')
    if navitems :
        pq(navitems[len(navitems-1)]).addClass('last-item')

    # if you have inserted external contents in content area with Deliverance xml rules
    # you want to change a navitem class in result
    if theme("#photos') :
        selectNavigationItem('#photos')

    #etc ...

To do this kind of template manipulations, changing content on source application could be a best practice, it depends of  your particular situation.
But i’m sure that it’s a fast way  ;-)

The problem is how to test the result ? twill could be a solution …

Do not transform some contents with deliverance

If you dont like bad surprises, you need to abort ajax html content transformation. And in some other cases you need also to abort Deliverance transformation (see example)

In your rules.xml, just add :

    <match pyref="myproduct.rules:match_notheme" abort="1" />

and in your rules.py

def match_notheme(req, resp, headers, *args):

    match = False
    if req.headers.get('X-Requested-With', '').startswith('XML'):
        match = True
    # the html template of FCKEditor iframe must be unchanged
    if req.path_info.startswith('/editor'):    
        match = True
    if 'notheme' in req.GET:
        match = True
    return match

Mai 7, 2009
» Install a maintenance page without restarting apache


These days we are working with fabric to remove some hazard in maintenance.
Recently someone gzip a Data.fs file in production and our customer loose 2 days of work. We choose to use fabric (http://www.nongnu.org/fab/) to limit command line working to the strict minimum.

Now, almost all procedures were added in a fabric factory, from the installation of a development environment to the upgrade of the production server. In this last part, we search a solution to put a maintenance page without have to gain root privileges to restart the apache used in frontend. For static or CGI sites you can use a .htaccess file to override the global rules. For a stopped Zope server it doesn’t help much.

The goal is to use RewriteCond and RewriteRule in a such way that a static HTML file would be displayed when it exists. We can call it ‘maintenance.html‘ and rename it ‘maintenance.html-disabled‘ to let the site be displayed normally.

We start with a virtual host generated by iw.recipe.squid:

<VirtualHost *:80>
   ServerName www.gosseyn.fr

   RewriteEngine On
   RewriteLog /my/path/to/squid/parts/squid/log/rewrite_www.gosseyn.fr.log
   RewriteLogLevel 0

   CustomLog /my/path/to/squid//parts/squid/log/access_www.gosseyn.fr.log common
   ErrorLog /my/path/to/squid//parts/squid/log/error_www.gosseyn.fr.log

   ## common rules for squid rewrite rules
   RewriteRule ^(.*)$ - [E=BACKEND_LOCATION:127.0.0.1]
   RewriteRule ^(.*)$ - [E=BACKEND_PORT:8081]
   RewriteRule ^(.*)$ - [E=BACKEND_PATH:site]

   RewriteRule  ^/(.*)/$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]
   RewriteRule  ^/(.*)$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]

   ## specific rules base on cookie

</VirtualHost>

First we need to declare a document root and to bypass all security rules. We create a new path in the root of our buildout folder called ‘maintenances_pages’.

   DocumentRoot /my/path/to/buildout/prod/maintenance_pages

   <Directory "/my/path/to/buildout/prod/maintenance_pages">
       Options None
       AllowOverride None
       Order allow,deny
       allow from all

       <LimitExcept GET>
           Order allow,deny
           allow from all
       </LimitExcept>
   </Directory>

These declaration should satisfy all paranoid BOFH.
Now create a file called maintenance.html within the folder. All code (javascript, CSS and images) must be embeded, this is a limitation of this approach.
Well, how to detect that a file exist on the filesystem from apache. You can do this with the flag ‘-f’ from the rewrite module:

    RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html -f

Then redirect all pages to our maintenance page:

   RewriteCond %{REQUEST_URI} !/maintenance.html
   RewriteRule $ /maintenance.html [R=302,L]

We also need to disable the standard rewrite rules. The easier way to do it is to use ‘!-f‘.

   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)/$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]
   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]

These rules will not be applied anymore when our maintenance file will be name ‘maintenance.html’. Our goal is reached.

The whole virtualhost file look like this:

<VirtualHost *:80>
   ServerName www.gosseyn.fr

   RewriteEngine On
   RewriteLog /my/path/to/squid/parts/squid/log/rewrite_www.gosseyn.fr.log
   RewriteLogLevel 0

   CustomLog /my/path/to/squid//parts/squid/log/access_www.gosseyn.fr.log common
   ErrorLog /my/path/to/squid//parts/squid/log/error_www.gosseyn.fr.log

   DocumentRoot /my/path/to/buildout/prod/maintenance_pages

   <Directory "/my/path/to/buildout/prod/maintenance_pages">
       Options None
       AllowOverride None
       Order allow,deny
       allow from all

       <LimitExcept GET>
           Order allow,deny
           allow from all
       </LimitExcept>
   </Directory>

   ## our maintenance page
   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html -f
   RewriteCond %{REQUEST_URI} !/maintenance.html
   RewriteRule $ /maintenance.html [R=302,L]

   ## common rules for squid rewrite rules
   RewriteRule ^(.*)$ - [E=BACKEND_LOCATION:127.0.0.1]
   RewriteRule ^(.*)$ - [E=BACKEND_PORT:8081]
   RewriteRule ^(.*)$ - [E=BACKEND_PATH:site]

   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)/$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]
   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]

   ## specific rules base on cookie

</VirtualHost>

It will be difficult to include this in a recipe as we modify an existing virtualhost in a very specific way.

As usual, all feedbacks are appreciated.

Mars 21, 2009
» Scratching my own documentation itch

A week ago, I finished updating the Archetypes Developer Manual for Plone 3. In case you have not seen it, the new version is interesting for any developer who has to develop or update an Archetypes-based product ; it tries to present our current coding practices (GenericSetup, ZCA techniques…) The manual is accompanied by a new example package. It is also useful, directly, for the “what does what/how to” when using the ZopeSkel “archetype” template with paster to bootstrap your product package.

What’s next ?

I started another manual to give an overview of the Zope Component Architecture techniques and tools we use for Plone development. Part of the initial work is refactoring/merging existing howtos and tutorials written by the other documentation contributors, and updating them. Hopefully, a first cut of this manual will be reviewed and published in a few weeks.

Sprinting at Pycon

Next week, I am going to Pycon in Chicago and will have 2 days to participate to the post-conference sprints. My current plan is to just dedicate my time to documentation. Topics that I may start contributing on are “Dexterity content types” and “WSGI setup for Plone (3/4) + using middlewares to augment your deployment”. There may be other topics to give priority to, and I will discuss with others to find out. And of course, I will have some time to make progress on the ZCA manual.

Looking forward to meet other “Python Web” community members there!


Mars 15, 2009
» How to upload your package to Plone.org


We have been working hard in the past few months with my Plone friends to make it happen.

It’s now available : Plone.org is acting like PyPI in its Products section, which means that you can use Distutils to register and upload your packages in both places now.

I have written a small tutorial on plone.org to explain how to use it: http://plone.org/documentation/tutorial/how-to-upload-your-package-to-plone.org

I will also explain during my Pycon presentation (http://us.pycon.org/2009/conference/schedule/event/44) how this new feature at Plone.org will help people that are working with Plone and Python packages.

Mars 5, 2009
» ‘Practical Plone 3′ review


I finished to read the long waited (read updates here :) ) ‘Practical Plone 3′ book. It reprensents a big amount of very good work.

If you want to begin in Plone or if you want a webmaster guide for your brand new Plone site the two first parts are designed for you. Each step of your needs are describe in a very good teaching way.

Parts are growing in difficulties. The first will show you how to install and what in installed in a default Plone site. The second part will learn you how to become the owner of your site by creating content and configuring it. The third part is for those who wanted to learn how to use addon products for Plone or want to customize Plone site: zc.buildout, PloneFormGen and ArchGenXML are presented here. The last part will show you some general needs around Plone: put Plone in production behind Apache or IIS with speed optimizations, and LDAP/Active Directory integration.

Even if the target of this book is beginners it was a very interresting reading for an old developper like me. If you don’t use it for yourself, you will use it to give answers to your end-users. I can only recommand everyone to buy it. Nice work guys.

Février 24, 2009
» Don’t forget to write tests for your patches


As all new major releases Plone 3.2 is heavy tested by early integrators that want the new version. They report bugs and sometimes patches. Hopefully most of them are using ’svn diff’ or ‘diff -u’ to prepare those patchesinstead of saying: “Replace the line XX by this and add that line here”…

But the one thing they never do it is to add a test to demonstrate their bugs. Then one of the maintainer have to take time to write a test around what (s)he think to be the bug. Often, the patch is just applied, Plone is growing with another untested code and the life continue.

Please help maintainers: if you are able to write a patch you should be able to write the associated test.

Décembre 16, 2008
» Pycon 2009 talks


I have 2 accepted talks at Pycon, that is great. I would like to say that the Pycon review system is awesome because you can see what the reviewers have said, and understand why your talk was accepted or declined.

I was a bit frustrated that my Atomisator talk was declined, but I think it makes sense : this is a new tool, and beside my user group and a few people, it is not really used yet.

One reviewer said that it had to be picked, and another one answered :

I agree that PyCon should not restrict itself to well-known projects, but it should definitely restrict itself to projects that are (a) in production use, (b) under active development, and (c) likely to still be so in a year. There are so many projects meeting these criteria that for me, the bar is very high indeed to spend a talk slot on one that does not.

Ok, fair enough : I will present this talk at Pycon 2010 and they won’t have any argument to decline it ;)

The talks that made it:

  • How AlterWay releases web applications using zc.buildout
  • On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes

I will get into greater details later on.

Décembre 15, 2008
» Python Isolated Environment (PIE)


Here’s a proposal I will send to the python-dev. What do you think ?

(Disclaimer : this proposal is highly inspired from the work done by people in various tools, it does not reinvent anything)

The problem

Python developers distribute and deploy their packages using myriads of dependencies. Some of them are not yet available as official OS python packages.  Even sometimes one package conflicts with the official version of a package installed in a given OS.

In any case, the cycle of development of most Python applications is shorter than the release cycle of Linux distributions, so it is impossible for application Foo to wait that Bar 5.6 is officialy available in Debian 4.x.

Therefore, there’s a need to provide or describe a specific list of dependencies for their application to work.

And this list of dependency might conflict with the existing list of packages installed in Python. In other words, even if this is not a wanted behavior from an os packager point of view, an application might need to provide its own execution context.

Right now, when Python is loaded, it uses the site module to browse the site-packages directory to populate the path with packages it find there.  .pth files are also parsed to provide extra paths.

Python 2.6 has introduced per-user site-packages directory, where you can define an extra directory, which is added in the path like the central one.

But both will append new paths to the environment without any rule of exclusion or version checking.

The workarounds

A few workarounds exist to be able to express what packages (and version) an application needs to run, or to set up an isolated environment for it:

  • setuptools provides the install_requires mechanism where you can define dependencies directly inside the package, as a new metadata. It also provides a way to install two different versions of one package and let you pick by code or when the program starts, which one you want to activate.
  • virtualenv will let you create an isolated Python environment, where you can define your own site-packages. This allows you to make sure you are not conflicting with a incompatible version of a given package.
  • zc.buildout relies on setuptools and provides an isolated environment a bit similar in some aspects to virtualenv.
  • pip provides a way to describe requirements in a file, which can be used to define bundles, which are very similar to what zc.buildout provides.

But they all aim at the same goal : define a specific execution context for a specific application, and declare dependencies with no respect to other applications or to the OS environment.

This proposal describes a solution that can be added to Python to provide that feature.

The solution

A isolated environment file that describes dependencies is added. This file can be tweaked by the application packager, or later by the OS packager if something goes wrong.

The isolated environment file

A new file called a  Python Isolated Environment file (PIE file) can be provided by any  application to define the list of dependencies and their versions.

It is a simple text file with a first line that provides :

  • a list of paths, separated by ‘:’, on line 1
  • then one package per line, starting at line 2. each package can be prefixed by a `!`

For example:

/var/myapp/myenv
lxml
sqlite
sqlalchemy
!sqlobject

This list of packages might or might not be installed in Python.

Versions can be provided as well in this file :

/var/myapp/myenv:/var/myapp/myenv2
lxml >= 0.9
sqlite > 1.8
sqlalchemy == 0.7
!sqlobject == 0.6

The file is saved with the pie extension,

Loading an isolated environment file

A new function called load_isolated_environment is added in site.py, that let you load a PIE file.

Loading a PIE file means:

  • for each package defined, starting at line 2, load_isolated_environment will look into the environment if the package with the particular version exists. The version is given by the package.__version__ value or the PKG-INFO one when available. If the package exists but the version is not available, the version 0.0 is used.
  • for packages without the ! prefix:
    • if the  package is not found, it will scan each path provided on line 1 of the file, using the site-packages method, looking for that package.
    • if the package is found, it is added in the path.
    • if the package is not found, a PackageMissing error is raised.
  • for packages starting with the ! prefix:
    • if the  package is found, it is removed from the path

This function can be called by code like this:

>>> from site import load_isolated_environment
>>> load_isolated_environment('/path/to/context.pie')

From there, sys.path meets the requirements and the code that is executed after this call will benefit from this context.
Another context can be loaded in the same process :

>>> load_isolated_environment('/path/to/another_context.pie')

Limitations:

  • if the new context brakes other programs in the process. It’s up to the application packager to fix the context file.
  • it’s not the job of load_isolated_environment to resolve dependencies issues : if the foo package needs the bar package, it won’t complain.
  • it is not the job of load_isolated_environment to get missing dependencies.

Using an isolated environment file

Typically, an isolated environment file can be used into high-level Python scripts. For example, any script an application provides to be launched :

# this script runs zope
from site import load_isolated_environment
load_isolated_environment('zope-3.4.pie')

import zope

if __name__ == '__main__':
    zope.run()

Décembre 14, 2008
» Looking for beta testers for Atomisator


I am looking for beta testers, interested in customized rss feeds or email alerts experimentations.

Here’s a list of services Atomisator can provide :

  • You run a project and you would like to receive a daily summary in your mailbox on what is being said about it in blogs, tweets, etc
  • You have a list of feeds you want to aggregate, with specific filters and you can’t manage to do with Yahoo pipes or any tools out ther, because it is too specific.
  • You want to annotate entries in a feed with extra information
  • etc..

What you get as a beta-tester:

  • a custom Atomisator configuration that fills your needs
  • I am hosting the service, and you get
    • either an url on my server to an xml file you can read in your aggregator
    • either one mail per day

What you are not getting as a beta tester:

  • you don’t get any guarantee on the output or the reliability, these are just experimentations.
  • if it’s down I can’t promise when it will be up again

Let me know by mail if you are interested

      

Décembre 12, 2008
» Pycon 2009 proposals


The proposal acceptance date is in a few days.

Here are the four proposals I have made:

  • The state of packaging in Python. This discussion resumes the current options when it comes to distribute your packages. It also explains the pitfalls and the gap between the Python developers and the OS vendors and packagers. I think this talk will not be picked because the topic is wide and vague. So I proposed to transform it into a panel where lead developers from various framework could explain their usage of distutils and what is missing to make them happy. No feedback yet on this.
  • Atomisator, the agile data processing framework. This tool is starting to be useful, and I think it can be useful to others. Check http://atomisator.ziade.org for a quick overview.
  • How AlterWay releases web applications using zc.buildout. That is the same talk I gave at the Plone conf but I present it in a way people understand zc.buildout is not tied to Zope and Plone and can be used with any other application. As a matter of fact, it has become a standard here, and we use it for Pylons, etc..
  • On the importance of PyPI in delivering and building Python softwares - mirroring, fail-over and third-party package indexes. That’s a long title. It presents my work on PyPI.

Last, I will go to the Python Language Summit the day before Pycon. I volunteered to be a “champion” on distutils matters.

      

Décembre 9, 2008
» How to make binary distribution of buildouts


The Problem

I need to distribute pre-compiled buildouts because some projects don’t allow us to have gcc installed on the production system for security reasons.

Fair enough, we need to provide a pre-compiled buildout.

If you want to distribute your buildout-based Plone application in a binary form, so it can be installed without requiring any compiler on the platform, you need to compile all .c modules before you provide a tarball of your buildout folder.

This is easy : just run your buildout and all .so files will be created in the zope 2 installation. (.pyd under windows)

But this will work only if you compile in a directory that is located within the same path on the target machine, because zc.buildout uses absolute paths when it builds scripts.

Furthermore, if the python interpreter is not located in the same place, your buildout script itself is screwed.

Last but not least, plone.recipe.zope2install is not clever enough. It will remove your zope2 installation when it detects that the path has changed. This is pretty annoying even if you have gcc : what is the point of compiling the c extension again since they
are statically compiled in-place ?

The solution

I have changed plone.recipe.zope2install and added a new option called `smart-recompile` (in trunk right now, not released).

If you use it, the recipe will check for .so or .pyd files before trying to ditch your zope 2 installation and recompile it. Even if you don’t use it to build binary distributions, it will make your buildout build faster if you already have zope compiled in there.

Next, I have created a special bootstrap.py, who is clever enough to rebuild the buildout script with the right path to the used interpreter, and with offline-mode capabilities. To make it short : boostrap.py works no matter if you have an internet connection or not. Grab it here : http://ziade.org/bootstrap.py

So now, basically you can compile your buildout and deploy it on any system, on any path, without any internet connection, like this:

$ python bootstrap.py    # will rebuild the buildout script   
$ bin/buildout

Of course this doesn’t work if you have dynamically compiled extensions like python-ldap. For theses, the best pick is to rely on the system ones.

      

Décembre 7, 2008
» A PostRank plugin for Atomisator


Yesterday, I bumped into PostRank. This system is collecting data from various social systems like Twitter and provides a service where you can type in an url of a blog post or a entire blog. You get a PostRank depending on the popularity of the URL.

I wrote a plugin for Atomisator and ran it on my own blog. Here’s the result:  http://ziade.org/afpy/

And the Atomisator configuration for this is :

[atomisator]
sources =
    rss http://tarekziade.wordpress.com/feed/atom/

database = sqlite:///carpet.db

outputs =
    rss  public/rss.xml "http://tarekziade.wordpress.com/feed/atom/" "Carpet Python with PR" "Powered by Atomisator"

enhancers =
    postrank

How PostRank works

PostRank works with urls you provide, on their web interface or through their web services.

As long as these url are present in their big cloud-computing based system, they provide a rank that is calculated with the number of comments related to the blog, the number of tweet messages that refers to it, and so on. The complete algorithm they used is secret but this is not the point. I have secret algorithms too ;) .

The point is that they are trying to categorize blog entries using social networks as indicators, and that they have a huge database.

Social indicators in Atomisator

This is one of the approach I have with Atomisator, when it is used to build a planet. For instance I have a Digg plugin that will inject in each entry the comments found on Digg if the entry was digged. It also present the number of Digg. Of course this is done live because I don’t have a cloud-computing based system where I store data. I use Digg webservice on the fly. (On the fly here doesn’t mean Atomisator make the calls to Digg from the Planet application of course. It means Atomisator calls them when it creates the merged feed on the system)

The benefit of this approach is that I can provide a social indicator on a post immediatly. Systems like PostRank will not work on entries that are too recent because their spiders have a lag of one week or so.

The pitfall of my approach is that I am unable to calculate trends because I don’t store the indicators as they vary.

But if someone wanted to build a BtoC application using Atomisator, they could implement a set of plugins based on Amazon tools to make them store data in a more scalable way and in time.

Next steps

So I have this new PostRank plugin, and this is awesome because I have added a treshold parameter in it. Basically if a post has a high PostRank value, it will appear in the Planet. If it’s low, it can be automatically removed. The fact that PostRanks are lagging for new entries is not a problem: interesting posts will eventually pop after a few days in the Planet.

This is perfect to reduce the number of entries in an aggregator.

But I do want to write my own PostRank that works live, with no storage at all. Because the whole point of Atomisator is to provide a framework where anyone can try out various filtering combinations.

So to be able to provide this power, it needs to work just by collecting data directly from the social services, like the PostRank plugin does with this PostRank “meta-service”. The next step is therefore to see if I can query services like Twitter to list the twits related to an url, without having to store the twitter feed myself.

In any case, if my talk on Atomisator at Pycon 2009 is selected, the PostRank plugin will be shown besides the Digg plugin.