While the Python African Tour (a.k.a. PAT) events are now underway, here is a wrap-up of some feedback and current thoughts about contributing through the organization of Python learning activities in Africa.

The New vs. the Old, again

When you talk to most people, you find that the Python language is underrated or new to them. C, Java, and PHP… it’s okay. But Python ?

Python could get a decent base of users in the medium to long term, if some influential guys in the university and entrepreneur spheres are aware of its advantages and decide to introduce it in their toolset or work environment.

Now, there is hope ; we’ve met guys from this new generation in Morocco, Senegal… and we are meeting others in Nigeria in a few weeks. In fact, these are the people who have been initiators and resource people in the process of organizing PAT events.

PAT Senegal team - (c) DakarLUG

User Groups

The Senegal PAT event was hosted with the help of DakarLUG, a small but very active group of people spreading the Linux and Open Source message there (install parties, demos, and all that).

Seems we were lucky… From my perception, it is hard to find active local geek or learning communities, at least in the francophone world which I am more connected to.  Though this might be less true today, with the local barcamps, LUGs and Google Technology User Groups that have been launched these last two years.

Community is key to spreading Open Source ideas, practices and solutions, and if we can’t have local communities I am afraid this is going to be even more difficult.

We are helping to improve that situation ourselves, since in addition to the Python workshop, we have been organizing a camp day for a larger public, with presentations of interesting technologies and development practices.

Tools and applications, a.k.a. solutions

Ok, people want solutions… They have problems to solve.

While we want to not simply consume solutions, but be able to extend and hack around them, it helps a lot to quickly introduce interesting tools and applications (and platforms) to the people.

The current focus of PAT for this includes two domains where Python is really strong:

• Web Frameworks (Django but not limited to). In the future, we should extend this with the “client” side of things (mobile, JavaScript, HTML5…)

• Scientific tools (SciPy, NumPy)… for people coming from research departments.

PAT Senegal / Emmanuelle Gouillart presenting Scientific tools - (c) DakarLUG

Sustaining and going forward

Last but not least, we need that students having participated to the workshop become contributors. This means they need to practice after the PAT event, so they need ongoing mentoring and small projects to work on.

One trick is to convince the head of the CS/Research department we meet in the universtity or school, to introduce Python in the curriculum, or allow students to use Python for their school projects. This gave encouraging results in Dakar.

An idea I have been nurturing is running our own kind of SoC program. This is the next phase, and more details coming soon.

Contact me if you want to share more thoughts, experiences, or contribute to any aspect of this.

Hello,

Today I have some problems to install multiprocessing-2.6.2.1 on my mac (in dependance of collective.releaser) for python 2.4

So if you have the same problems , here goes some tips about that

first multiprocessing failed like that:

Modules/mmapmodule.c: In function 'new_mmap_object':
Modules/mmapmodule.c:947: warning: implicit declaration of function 'open'
Modules/mmapmodule.c:947: error: 'O_RDWR' undeclared (first use in this function)
Modules/mmapmodule.c:947: error: (Each undeclared identifier is reported only once
Modules/mmapmodule.c:947: error: for each function it appears in.)
error: Setup script exited with error: command '/usr/bin/gcc-4.2' failed with exit status 1

So I download multiprocessing by hand

$ wget http://pypi.python.org/packages/source/m/multiprocessing/multiprocessing-2.6.2.1.tar.gz

I edit Modules/mmapmodule.c and add

81 #include <fcntl.h>

$ python setup.py install

succeed after that.

This bug is similar of this issue : http://bugs.python.org/issue3266

Regards Youenn.

Usually we build multilingual Plone sites with LinguaPlone.

This solution has a big advantage, it’s generic and very easy to implement in a plone site.

But there are many inconvenients, due to the design of this product (translations are independent by design) :

• Each translation is a new Archetype object, and it could be a big problem on sites with many contents, the portal objects number is increased by the number of available languages.
• Translations uses plone references catalog to be linked to the original (called canonical) object, but when moving objects translations are not moved, when copying pasting objects, translations are not pasted, when deleting objects translations are not deleted, when reordering contents, translations are not reordered, when publishing objects translations are not published, … For web masters maintaining a site  with LinguaPlone inside could be a challenge.
• When translating folders with LP, all translated contents are moved from the canonical folder to the translated folder with same language, translating low level folders on big depth tree sites could take a long long time, don’t be surprise if you get errors.
• If a content is neutral (with no language attribute), inside a translated folder, it could not be seen when browsing a translation of the parent folder.
• At last a lower problem :  the translation edit forms are not pretty to use, they show a table with two columns, the first column with the « canonical » content inside in « view » mode, the second column with the translation edit form, in fixed width sites the translated form width is sometimes ridiculously small.

But since many years we use LinguaPlone because it was the only easy way to make multilingual Plone sites.

By the past we were using a LP patch called LinguaFace to reduce the number of problems with LP (synchronisation on reorder, copy-paste, delete, or move – see neutral contents inside all translated folders – more usable translation edit forms …), but LF add a new layer of complexity and maintaining it with all LP versions becomes complicated. See some examples on how it works :

• when a content is copied, all translations are copied
• when a content is pasted or moved all translations are pasted or moved at the good place (not so easy)
• when a content is published or retracted translations follow the same workflow transition
• when a folder is translated, we don’t see only objects inside but also objects with same canonical Path (a new catalog index) to see also neutral contents.
• Navtree is patched, breadcrumbs are patched, to use canonical path
• and so on …

A big nightmare.

To day a new solution exists that store translations inside each Archetypes field, raptus.multilingualfields, and a Plone integration of raptus.multilanguagefields called raptus.multilingualplone that extends the schema of all Plone Content Types making them translatable. raptus.multilingualfields also provides multilingual catalog indexes that return the good translated data when searching for contents or displaying trees, and multilingual criterions for topics.

A LP feature not provided by raptus.multilingualfields is the internationalization of urls, if you really need this feature, i think it’s not a big challenge to add some traversal rules, for me it’s not essential.

AnotherLP  feature that can’t be provided when storing translations in fields is to get different workflows or security settings for each translation. If you need this feature use LinguaPlone, LinguaPlone is done for that, it makes all translated contents independent, but i’m curious to know the number of users who really want this feature, all clients i had never ask for that but finally, after some LP experience, always wanted the exact opposite use-case.

To make your Plone site translatable with raptus.multilanguagefields, you have two choices :

1. Add raptus.multilanguageplone in your buildout and install it in your Plone Site using extensions products control panel, to make all your Plone content types and derived translatable (all fields for which translation make a sense are translatable).
2. I prefer integrate by myself raptus.multilanguagefields inside a product, since we could want just some fields or content types translatable, as example i don’t need to translate the images or the files contents, just their titles and descriptions.

How to implement the second solution ?

Just take a look at raptus.multilanguageplone code, it’s easy :

1. Make your archetype extenders to make your wanted fields translatable, example can be found here
2. register your extenders in setuphandlers (Generic Setup import step), example here
3. replace the standard catalog  indexes with multilingual indexes in Generic Setup profile, example here

That’s all, you will get superb edit forms with translatable fields inside, you also will get a google help to translate contents (a pleasant gadget). I say « Bravo » to raptus developpers.

Important :

• these products are young, and there’s still many work todo to make it work without problems (tests are needed …). Last 0.6 releases have bugs under plone3.3, use the svn versions below instead.
• raptus.multilanguageplone 0.6  has a bug in extenders with primary fields, tested in Plone 3.3  (fixed in branch aws_evols, not tested with images at this time)
• raptus.multilanguagefields 0.6 has a bug, doctests are broken in Plone3.3 (fixed in trunk )
• At this time these products don’t have unit tests or functional tests, it’s the only reproach i can make.  I started the work  here, and here

I notice that in a zope application that I have to maintain is slow because of DateTime class.
The profile in this application test give the top time to this class.
So I want to test an other implementation which is name mx.DateTime. The difference is that mx.DateTime is writen in C.
So in a terminal , I install the two eggs via easy install :

./bin/easy_install DateTime
./bin/easy_install egenix-mx-base

And do a little script for testing the two api:

import sys
from mx import DateTime as mxDateTime
from DateTime import DateTime
from datetime import datetime
from time import time

def create_mxdatetime():
    return mxDateTime.now()

def create_zopedatetime():
    return DateTime()

def create_datetime():
    return datetime.now()

def bf(f, i):
    t1 = time()
    for i in xrange(i):
        f()
    t2 = time()
    print "bench for %s is %s" % (str(f), t2 - t1)

bf(create_mxdatetime, int(sys.argv[1]))
bf(create_zopedatetime, int(sys.argv[1]))
bf(create_datetime, int(sys.argv[1]))

This script just create Date in the three implementation : zope, mx and the standard library

And the results:

bash-3.2$ bin/python bench.py 1000
bench for function create_mxdatetime at 0x7db70 is 0.00120091438293
bench for function create_zopedatetime at 0x4215f0 is 0.84446310997
bench for function create_datetime at 0x4214b0 is 0.00220394134521
bash-3.2$ bin/python bench.py 10000
bench for function create_mxdatetime at 0x7db70 is 0.0117778778076
bench for function create_zopedatetime at 0x4215f0 is 8.81699991226
bench for function create_datetime at 0x4214b0 is 0.041069984436
bash-3.2$ bin/python bench.py 100000
bench for function create_mxdatetime at 0x7db70 is 0.11746096611
bench for function create_zopedatetime at 0x4215f0 is 87.8845770359
bench for function create_datetime at 0x4214b0 is 0.222129106522

No comment.. and in memory ?

So

bash-3.2$ bin/easy_install pympler

and now::

>>> from pympler import asizeof
>>> from datetime import datetime
>>> from mx import DateTime as mxDateTime
>>> from DateTime import DateTime
>>> asizeof.asizesof(DateTime() , mxDateTime.now(), datetime.now())
(1760, 56, 32)

Hoaaa !! a zope DateTime is 798 time slower than mxDateTime and it consume 31 more space than mxDateTime

I think ReplacingDateTime could be a good performance issue for zope, no ?

With DateTimeNG (zope DateTime with mx.DateTime) performance is 10 time better than DateTime. Memory consume is the same

From mx.DateTime documentation
Comparing the types to time-module based routines is not really possible,
since the used strategies differ. You can compare them to tuple-based
date/time classes though: DateTime[Delta] are much faster on creation, use
less storage and are faster to convert to the supported other formats than
any equivalent tuple-based implementation written in Python.
Creation of time-module values using time.mktime() is much slower than
doing the same thing with DateTime(). The same holds for the reverse
conversion (using time.localtime()).
The storage size of ticks (floats, which the time module uses) is about 1/3
of the size a DateTime instance uses. This is mainly due to the fact that
DateTime instances cache the broken down values for fast access.
To summarize: DateTime[Delta] are faster, but also use more memory than
traditional time-module based techniques.

I notice that in charge , counter in Zope2 can generate some conflict error.

Why ?

Because two thread want to change the value of a variable. Conflict error are exposed here :
http://wiki.zope.org/zope2/ConflictErrors

But in certain case it’s useful to have an global counter that increment in certain operation. Cache Fu have those counter for caching purpose. Or in charge those counter generate some Conflict Error. There is an solution : resolve the conflict by hand.

In zope source I notice that there is an class that implement this use case. I try to add an counter with this implementation. And I test two implementation of the two counter : one wich was simply an int and one wich was Products.Transience.Transience.Increaser.

The init code of the first counter looks like that :

tool.counter = Products.Transience.Transience.Increaser(0)

The second counter looks like that:

tool.counter2 = 0

I test under siege the incrementation of the two implementation:

First test (Increaser)

Transactions:		         200 hits
Availability:		      100.00 %
Elapsed time:		        6.68 secs
Data transferred:	        0.00 MB
Response time:		        0.10 secs
Transaction rate:	       29.94 trans/sec
Throughput:		        0.00 MB/sec
Concurrency:		        2.97
Successful transactions:         200
Failed transactions:	           0
Longest transaction:	        0.44
Shortest transaction:	        0.01

O conflicts errors

Second test (int)

## 20 user

Transactions:		         200 hits
Availability:		      100.00 %
Elapsed time:		       11.35 secs
Data transferred:	        0.06 MB
Response time:		        0.26 secs
Transaction rate:	       17.62 trans/sec
Throughput:		        0.00 MB/sec
Concurrency:		        4.56
Successful transactions:         197
Failed transactions:	           0
Longest transaction:	        6.09
Shortest transaction:	        0.01

49 conflicts (3 unresolved)

I have recently quit Ingeniweb for a job in an international organization involved in the promotion of cultural diversity and sustainable development, to only mention two of its activity fields.

I am in charge of website projects (in the context of the Communication department). There is a lot to do, the first thing being understanding the context, the way things work here, and the people’s expectations.

As for the tools and technologies side of things, there is not currently anything Python-based here. PHP (SPIP) rules ! Well, I guess it’s perceived as easy and more importantly, skills exist everywhere (me looking in the direction of “web agencies”).
So, one of the interesting things will be to introduce Python, where it makes sense, in the not-too-far (hopefully) future. Anyway, I am confident that will happen one day. After all, it’s not always bad to be ahead of the time. You just have to wait, monitor, and talk to people everytime there is an opportunity. And you might be lucky and convince those who take the time to listen

I just release a little tool to detect Memory Leak in zope2 call Products.MemoryProfiler .

It use heapy (http://guppy-pe.sourceforge.net/#Heapy) in internal. It’s just an interface to this tool.

It provide an http interface in zope control panel to see the current memory .

When you start profiling, you take an snapshot of the memory at instant t.
When you click to updateSnapshot, memory profiling  tell you what objects are added between the start and the updateSnashot click. It will be usefull to detect Memory Leak.
Each snapshot is store (as string) in MemoryProfiler to be consult later (link to the date).

The button clear db cache clear all zeo cache of all mounting point so you can see the impact of the memory of those cache.

For windows users, you must compile guppy. There is egg for python 2.6 but
no for python 2.4. I have fatal error with Mingw to compile guppy. I hope that we have soon a binary egg to for python 2.4.

I hope that this tool give to us usefull  information to the memory consume by zope.

Some times ago i made a new skin for the French-speaking PYthon Association (AFPY) web site, using Deliverance and PyQuery. I made this exercice on my free time with many pleasure.

You can see the Deliverance power here :

- The home page without Deliverance, the same using Deliverance proxy : all dynamic contents are taken from original Plone site, try to find it, you will notice the Deliverance + PyQuery magic in action

- The association pages without Deliverance (a sphinx site), the same contents injected in the site with Deliverance

Thanks to Gaël Pasgrimaud, a friend of mine and coworker, for his big work which really help me :

Here, we use a specific Gael’s buildout and a specific package done for our Deliverance projects used at Ingeniweb, our company. With these tools (if some people are interested in, it could be released on pypi, tell us …), we get the best flexibility for Deliverance :

• PyQuery integration
• The good base rules for Deliverance to avoid theming bugs with Ajax and HTML editor (*)
• Dynamic css : css are mako templates served by a skin server, colors fonts and other properties are set in a « .ini » file
• A base mako css for Plone which permits to start the work of externalize the entire css :
  in our projects, all portal_css content links from Plone are stripped when the Plone site is called through Deliverance (just add a condition for Plone stylesheets in portal_css : python:request.HTTP_HOST in request.SERVER_URL), the final css is lighter, the Deliverance css is really clean without many ‘!important’ bad overloads)
• Supervisor deamon to control skin servers
• We can use eggs for our Deliverance skins.

(*) to get real wysiwyg rendering in Plone behind Deliverance ‘s  HTML editor areas, use also FCKeditor.Plone 2.6.4, in FCKeditor control panel you will find the way to inject Deliverance css in wysiwyg area, it’s an important feature for end users.

Now i can take my summer holidays in peace with myself

Sometimes the hard part of a python application is to integrate sso because there is an unknown : what rules is defined to get the user !

In windows, apache mod_sspi or enfold proxy give to us an http header ( name X_REMOTE_USER) to deal with the active directory. This header is like that:

Domain\user

If you have one domain it’s pretty simple. Your userid is unique

But in big company there is multiple domain controler. And user is not unique ! So how retrieve an unique user id for active directory and use it in my windows python application !

The response is get UserPrincipalName with COM and NameTranslate interface.

import win32com.client
d = win32com.client.Dispatch('NameTranslate')
d.Init(3,'')
d.Set(3,'domain\\user')
userPrincipalName = d.Get(9)

Now if you use COM with zope as me , COM is not thread safe. So init the client at zope starting and lock yours calls to the API

import win32com.client
import threading
D = win32com.client.Dispatch('NameTranslate')
D.Init(3,'')
COMLOCK = threading.Lock()

And in a function use the global D

def getUserPrincipalName(sso_header):

     try:
            COMLOCK.acquire()
            D.Set(3,sso_header)
            userPrincipalName = D.Get(9)
     finally:
            COMLOCK.release()
     return userPrincipalName

Youpi , thanks win32com !!

I really like to use Deliverance for skinning Plone or any web application. In this post, i just want to help Deliverance beginners to configure their server, and to talk about PyQuery, which could be a good Deliverance companion.

Apache 2.2 + Deliverance : configuration

I spent some time to configure Apache in front of Deliverance, you will find here a possible configuration.

Deliverance Server configuration :

In your rules.xml, the first declaration relates to server config, use ’0.0.0.0′  as server address to be able to serve any ip address :

<server-settings>
 <server>0.0.0.0:8000</server>
 <execute-pyref>true</execute-pyref>
 <display-local-files>false</display-local-files>
 <dev-allow>
 localhost
 </dev-allow>
 <dev-user username="dev" password="dev" />
 <dev-expiration>60</dev-expiration>
</server-settings>

Always in rules.xml you need to write the proxy rule for Zope Virtual Host or another server.

If you want to serve a plone site located at http://localhost:8080/myplonesite :

<proxy path="/" rewrite-links="1">
     <dest href="http://localhost:8080/VirtualHostBase/http/{SERVER_NAME}:8000/myplonesite/VirtualHostRoot" />
</proxy>

If you need a more complex configuration in front of Zope, you can always use Apache + Varnish or SQUID + Pound, or IIS + EEP, and ZEO in front of Zope using another port (90 in this example) and use this Deliverance proxy rule :

<proxy path="/ rewrite-links="1">
    <dest href="http://{SERVER_NAME}:90" />
</proxy>

And if you want to serve anything else for another path :

<proxy path="/anotherpath" rewrite-links="1">
    <dest href="http://anythingelse" />
</proxy>

Apache VirtualHost configuration

You can use Deliverance on port 80 (just replace 8000 by 80 in rules.xml) to directly serve your site, but you would prefer Apache for that in many cases.

Suppose you want to rewrite all requests to « www.mysite.org » with a Deliverance response (on www.mysite.org:8000), you can use the mod_rewrite like this :

<VirtualHost *:80>
    ServerName www.mysite.org
    ServerAlias mysite.org
    ErrorLog "logs/error.mysite.org.log"
    CustomLog "logs/access.mysite.org.log" combined

    # do not forget this declaration
    ProxyPreserveHost On
    RewriteEngine On
    RewriteLog "logs/rewrite.mysite.org.log"
    RewriteCond %{HTTP:Authorization} ^(.*)
    RewriteRule ^/(.*) http://%{SERVER_NAME}:8000/$1 [P]

    # etc ...

</VirtualHost>

That’s all.

If you want to serve urls starting with a particular path (/mypath),  just change your rewrite rule like this :

    RewriteRule ^/mypath(.*) http://%{SERVER_NAME}:8000/_vh_mypath$1 [P]

Just take care in this last case to put the good urls in your Deliverance static contents, if you need it .

In fact you can do anything you want with rewrite rules

Deliverance and PyQuery

PyQuery is a jQuery-like library for Python which help you to play easily with html elements. Combined with Deliverance you will be able, with a python rule, to modify content or theme on the fly, at any point of your Deliverance transformation.

To learn how to add PyQuery rules to Deliverance, read this post  (many thanks to Gaël who help me to learn Deliverance) :
http://www.gawel.org/weblog/en/2008/12/skinning-with-pyquery-and-deliverance

Read also  :
http://www.openplans.org/projects/deliverance/lists/deliverance-discussion/archive/2008/12/1229591136002/forum_view

And now just try these example :

You have a floating right toolbar with floating right tabs inside in your html template theme :

<ul id="right-bar">
    <li><a href="/abcd">ABCD</a></li>
    <li><a href="/efgh">EFGH</a></li>
</ul>

You have a floating left toolbar with floating left tabs inside in your html source content :

<ul id="left-bar">
    <li><a href="/abcd">ABCD</a></li>
    <li><a href="/efgh">EFGH</a></li>
</ul>

With Deliverance xml rules you want to replace right tabs by left tabs like this :

    <replace content="children:#left-bar" theme="children:#righ-bar" />

But of course you need to reverse tabs order before or after xml transformation.
To modify the theme  just add a pyquery rule after the replace declaration :

    <pyquery pyref="myskinproduct.rules:pqrules" />

In your rules.py,  create your pqrules function like this :

def pqrules(content, theme, resource_fetcher, log) :

    # if you prefer to work on content
    # before Deliverance transform
    # replace theme('#right-bar') by content('#left-bar')
    nav = theme('#right-bar')
    navitems = nav('li')
    navitems.reverse()
    # i'm not sure that it's needed :
    nav.empty()
    nav.append(navitems)

Other examples :

    # replace a search button text with 'OK' in content
    content('#portal-searchbox .searchButton').val('OK')

    # remove icons from a navigation portlet
    # replace icons by "> "
    nav = content('#navigation')
    navitems = nav('li')
    navitems('img').remove()
    navitems.append('<span>&gt;&nbsp;</span>')

    # add a "last-item" class to last tab in a toolbar
    from pyquery import PyQuery as pq
    navitems = content('#navbar li')
    if navitems :
        pq(navitems[len(navitems-1)]).addClass('last-item')

    # if you have inserted external contents in content area with Deliverance xml rules
    # you want to change a navitem class in result
    if theme("#photos') :
        selectNavigationItem('#photos')

    #etc ...

To do this kind of template manipulations, changing content on source application could be a best practice, it depends of  your particular situation.
But i’m sure that it’s a fast way 

The problem is how to test the result ? twill could be a solution …

Do not transform some contents with deliverance

If you dont like bad surprises, you need to abort ajax html content transformation. And in some other cases you need also to abort Deliverance transformation (see example)

In your rules.xml, just add :

    <match pyref="myproduct.rules:match_notheme" abort="1" />

and in your rules.py

def match_notheme(req, resp, headers, *args):

    match = False
    if req.headers.get('X-Requested-With', '').startswith('XML'):
        match = True
    # the html template of FCKEditor iframe must be unchanged
    if req.path_info.startswith('/editor'):    
        match = True
    if 'notheme' in req.GET:
        match = True
    return match