By Ingeniweb. A Django site.
Janvier 12, 2010
» DateTime against mx.DateTime

I notice that in a zope application that I have to maintain is slow because of DateTime class.
The profile in this application test give the top time to this class.
So I want to test an other implementation which is name mx.DateTime. The difference is that mx.DateTime is writen in C.
So in a terminal , I install the two eggs via easy install :

./bin/easy_install DateTime
./bin/easy_install egenix-mx-base

And do a little script for testing the two api:

import sys
from mx import DateTime as mxDateTime
from DateTime import DateTime
from datetime import datetime
from time import time

def create_mxdatetime():
    return mxDateTime.now()

def create_zopedatetime():
    return DateTime()

def create_datetime():
    return datetime.now()

def bf(f, i):
    t1 = time()
    for i in xrange(i):
        f()
    t2 = time()
    print "bench for %s is %s" % (str(f), t2 - t1)

bf(create_mxdatetime, int(sys.argv[1]))
bf(create_zopedatetime, int(sys.argv[1]))
bf(create_datetime, int(sys.argv[1]))

This script just create Date in the three implementation : zope, mx and the standard library

And the results:

bash-3.2$ bin/python bench.py 1000
bench for function create_mxdatetime at 0x7db70 is 0.00120091438293
bench for function create_zopedatetime at 0x4215f0 is 0.84446310997
bench for function create_datetime at 0x4214b0 is 0.00220394134521
bash-3.2$ bin/python bench.py 10000
bench for function create_mxdatetime at 0x7db70 is 0.0117778778076
bench for function create_zopedatetime at 0x4215f0 is 8.81699991226
bench for function create_datetime at 0x4214b0 is 0.041069984436
bash-3.2$ bin/python bench.py 100000
bench for function create_mxdatetime at 0x7db70 is 0.11746096611
bench for function create_zopedatetime at 0x4215f0 is 87.8845770359
bench for function create_datetime at 0x4214b0 is 0.222129106522

No comment.. and in memory ?

So

bash-3.2$ bin/easy_install pympler

and now::

>>> from pympler import asizeof
>>> from datetime import datetime
>>> from mx import DateTime as mxDateTime
>>> from DateTime import DateTime
>>> asizeof.asizesof(DateTime() , mxDateTime.now(), datetime.now())
(1760, 56, 32)

Hoaaa !! a zope DateTime is 798 time slower than mxDateTime and it consume 31 more space than mxDateTime

I think ReplacingDateTime could be a good performance issue for zope, no ?

With DateTimeNG (zope DateTime with mx.DateTime) performance is 10 time better than DateTime. Memory consume is the same

From mx.DateTime documentation
Comparing the types to time-module based routines is not really possible,
since the used strategies differ. You can compare them to tuple-based
date/time classes though: DateTime[Delta] are much faster on creation, use
less storage and are faster to convert to the supported other formats than
any equivalent tuple-based implementation written in Python.
Creation of time-module values using time.mktime() is much slower than
doing the same thing with DateTime(). The same holds for the reverse
conversion (using time.localtime()).
The storage size of ticks (floats, which the time module uses) is about 1/3
of the size a DateTime instance uses. This is mainly due to the fact that
DateTime instances cache the broken down values for fast access.
To summarize: DateTime[Delta] are faster, but also use more memory than
traditional time-module based techniques
.


Décembre 9, 2009
» How to add a counter without conflict error in zope ?

I notice that in charge , counter in Zope2 can generate some conflict error.

Why ?

Because two thread want to change the value of a variable. Conflict error are exposed here :
http://wiki.zope.org/zope2/ConflictErrors

But in certain case it’s useful to have an global counter that increment in certain operation. Cache Fu have those counter for caching purpose. Or in charge those counter generate some Conflict Error. There is an solution : resolve the conflict by hand.

In zope source I notice that there is an class that implement this use case. I try to add an counter with this implementation. And I test two implementation of the two counter : one wich was simply an int and one wich was Products.Transience.Transience.Increaser.

The init code of the first counter looks like that :

tool.counter = Products.Transience.Transience.Increaser(0)

The second counter looks like that:

tool.counter2 = 0

I test under siege the incrementation of the two implementation:

First test (Increaser)

Transactions:		         200 hits
Availability:		      100.00 %
Elapsed time:		        6.68 secs
Data transferred:	        0.00 MB
Response time:		        0.10 secs
Transaction rate:	       29.94 trans/sec
Throughput:		        0.00 MB/sec
Concurrency:		        2.97
Successful transactions:         200
Failed transactions:	           0
Longest transaction:	        0.44
Shortest transaction:	        0.01

O conflicts errors

Second test (int)

## 20 user

Transactions:		         200 hits
Availability:		      100.00 %
Elapsed time:		       11.35 secs
Data transferred:	        0.06 MB
Response time:		        0.26 secs
Transaction rate:	       17.62 trans/sec
Throughput:		        0.00 MB/sec
Concurrency:		        4.56
Successful transactions:         197
Failed transactions:	           0
Longest transaction:	        6.09
Shortest transaction:	        0.01

49 conflicts (3 unresolved)


Novembre 13, 2009
» Memory Profiler for zope

I just release a little tool to detect Memory Leak in zope2 call Products.MemoryProfiler .

It use heapy (http://guppy-pe.sourceforge.net/#Heapy) in internal. It’s just an interface to this tool.

It provide an http interface in zope control panel to see the current memory .

When you start profiling, you take an snapshot of the memory at instant t.
When you click to updateSnapshot, memory profiling  tell you what objects are added between the start and the updateSnashot click. It will be usefull to detect Memory Leak.
Each snapshot is store (as string) in MemoryProfiler to be consult later (link to the date).

The button clear db cache clear all zeo cache of all mounting point so you can see the impact of the memory of those cache.

For windows users, you must compile guppy. There is egg for python 2.6 but
no for python 2.4. I have fatal error with Mingw to compile guppy. I hope that we have soon a binary egg to for python 2.4.

I hope that this tool give to us usefull  information to the memory consume by zope.

memory detail

Août 31, 2009
» How to configure an custom vary tag for squid

You want to make an authenticated cache with apache/squid-varnish/plone and you don’t
know how do that : it’s possible with the vary tag.

Vary header tell to proxy cache what’s headers is variant for an object for a cache.
For example if you tell to the cache that the variant is Cookie , then for a same url with different cookie value the result of the cache is different.
The Server send to the proxy (in the response) which header is considered for vary by sending Vary: list of request header name

In Cachefu, you can configure that by rule with varyExpression.

In global configuration of cache fu you can also configure an global vary header. By default this configuration is send with rule.portal_cache_settings.getVaryHeader()

You can activate or desactivate vary with the header_set configuration ( vary field).

Vary headers must be present in the request (not response) of the browser in order to be considered to be variant for the proxy cache. So we are limited with the standard header of the protocol http.

But with cookie and apache (apache is in front of squid) we can elaborate strategy to construct a vary tag more efficient.

The second aspect of the cache work is purge content when the content change.

PURGE of Vary objects is still very poorly supported in squid, and you can only purge one variant at a time and need to get the URL cached again before being able to purge another variant. So how to deal with that also ?

First , how build our vary tag ?

The trick is to construct an custom vary tag with apache.
We can to do this with RewriteRule::

RewriteCond %{HTTP_COOKIE} mycookie="([^"]+) [NC]
RewriteRule ^(.*)$ - [E=mycookie:%1]

So in this example mycookie contains the value of cookie_key

You can add a cookie for the language , a cookie for group , a cookie for a permission and so on and then construct your custom vary tag with values of this specifics cookies with mod_headers

RequestHeader append MyVary %{mycookie}e

And then the value of mycookie is considered to be variant..

If you want have a specific vary tag for anonymous you can test the presence of
__ac cookie and send a custom MyVary in this case

RewriteCond %{HTTP_COOKIE} __ac="([^"]+) [NC]
RewriteRule ^(.*)$ - [E=authenticated:1]

RequestHeader append MyVary %{mycookie}e env=authenticated
RequestHeader append MyVary anonymous env=!authenticated

So now with that you can vary cache as you want. Now how to treat the big deal of purge.

The trick is  have an image (or a ajax request or ..) in content that is never in cache. This image is serve by a browser view (in case of zope application) that set a cookie. This cookie value is added to Vary tag. So the Vary tag change if the value of this cookie change and then the content is updated (for all request).

For example we can construct a cookie with the value of the catalog change

catalog_count = pcs.getCatalogCount()
context.REQUEST.RESPONSE.appendHeader('Pragma','no-cache')
context.REQUEST.RESPONSE.appendHeader('Cache-control', 'no-cache')
cookie = context.REQUEST.cookies.get('X_CACHE_CATALOG', 0)

if cookie != str(catalog_count) :
context.REQUEST.RESPONSE.setCookie('X_CACHE_CATALOG',
catalog_count ,
path="/")
return catalog_count

And in apache we add
RewriteCond %{HTTP_COOKIE} X_CACHE_CATALOG=([^"]+) [NC]
RewriteRule ^(.*)$ - [E=X_CACHE_CATALOG:%1]
RequestHeader append MyVary %{mycookie}e:%{X_CACHE_CATALOG}e env=authenticated

And when catalog change, the vary also (in the second request) and the cache is updated. You can elaborate other strategies for purging vary object with this technique.

The last point is to combined Etag and Vary Header in response. IE with a Vary header don’t treat correctly Etag header and If-None-Match is never sending. So in apache remove the tag Vary and then Etag work well for all browser

Header unset Vary


Juin 9, 2009
» Retrieving UserPrincipalName with NetBiosDomain\NetBiosLogin

Sometimes the hard part of a python application is to integrate sso because there is an unknown : what rules is defined to get the user !

In windows, apache mod_sspi or enfold proxy give to us an http header ( name X_REMOTE_USER) to deal with the active directory. This header is like that:

Domain\user

If you have one domain it’s pretty simple. Your userid is unique

But in big company there is multiple domain controler. And user is not unique ! So how retrieve an unique user id for active directory and use it in my windows python application !

The response is get UserPrincipalName with COM and NameTranslate interface.

import win32com.client
d = win32com.client.Dispatch('NameTranslate')
d.Init(3,'')
d.Set(3,'domain\\user')
userPrincipalName = d.Get(9)

Now if you use COM with zope as me , COM is not thread safe. So init the client at zope starting and lock yours calls to the API

import win32com.client
import threading
D = win32com.client.Dispatch('NameTranslate')
D.Init(3,'')
COMLOCK = threading.Lock()

And in a function use the global D

def getUserPrincipalName(sso_header):

     try:
            COMLOCK.acquire()
            D.Set(3,sso_header)
            userPrincipalName = D.Get(9)
     finally:
            COMLOCK.release()
     return userPrincipalName

Youpi , thanks win32com !!


Septembre 17, 2008
» How to install zope.testrecorder with buildout


Hi,

zope.testrecorder is a tool to record navigation browser for doctest. In this post I explain how install it with buildout.

In your buildout.cfg :

[buildout]

parts =
  zope2
  fakezope2eggs
  ...

eggs =
  ...
  zope.testrecorder

[fakezope2eggs]
recipe = z3c.recipe.fakezope2eggs
additional-fake-eggs = ZODB3

[instance]
...

eggs =
  ...
  zope.testrecorder

zcml =
  ...
  zope.testrecorder

Relaunch buildout and restart instance and now go to
http://<zope host>:<zope port>/++resource++recorder/index.html
You are ready to record you’re doctest.

Juin 13, 2008
» Mock For ldap


Hi !

I work at the present time to an project with a lot data provided by an ldap server. The schema of this ldap is really elaborate . To reproduce it in a real ldap server is very difficult. How can I test my terrific code witch consist on an specific vocabularies and a set of PAS plugins ?

The response : a mock ldap object that give to my eggs a real API for ldap.

You can found it here : http://products.ingeniweb.com/catalog/iw.mock.ldap

The goal of this egg : testing component witch use ldap server without ldap. But also provide some good data in order to have some good test. So iw.mock.ldap read an real ldif file and you can search via ldap api (search_s).

It’s not perfect but for me I found very easy to test ldap component.

So how use it ?

In your buildout

[buildout]
eggs =

iw.mock.ldap

[zinstance]

eggs =

iw.mock.ldap

zcml =

iw.mock.ldap

After in you’re project where you want have some test you create a structure like that::

path-to-your-egg/mock
path-to-your-egg/mock/__init__.py
path-to-your-egg/mock/data.ldif

In __init__.py you put this boiler code::

import sys

from zope.interface import implements
from zope.component import adapts
from zope.component import provideAdapter

from iw.mock.ldap.interfaces import ILdifFile
from iw.mock.ldap.interfaces import ILdifReader
import iw.mock

class LdifFile(object):

implements(ILdifFile)
adapts(ILdifReader)

def __init__(self, context):
self.context = context

def open(self):
return file(os.path.join(os.path.dirname(__file__),’data.ldif’))

provideAdapter(LdifFile)

sys.path.insert(0, os.path.dirname(iw.mock.__file__))
from iw.mock import ldap
reload(ldap)


You can also register your adapter in an zcml (example test.zcml)

<configure xmlns=”http://namespaces.zope.org/zope”>
<adapter factory=”.mock.LdifFile”/>
</configure>

And now in you’re test you import path-to-your-egg/mock and all connections and request ldap pass
by iw.mock.ldap and use your data in data.ldif

» iw.recipe.pound

Hi !

I’m pleased to announce a new version of iw.recipe.pound.
The recipe is divided now in two part:

  1. A build part (iw.recipe.pound:build)
  2. A config part (iw.recipe.pound:config)

The build part accept new options as localisation of libssl.
If you have again need of compilation option (to disable some module perhaps) the extra-options keyword is for you .

The config part accept now all general configuration of pound server (see man pound). You can define TimeOut globally ( a big mistake I know ! ). The recipe give choice also to define Priority and TimeOut by backend server. The bind ip adresss of http server is required now ( before it’s always 127.0.0.1 !! so if apache or squid is located in another server the recipe serves to nothing ).

Now the backends configuration looks like that :

one  127.0.0.1:80 127.0.0.1:8080 127.0.0.1:8081,1

where :
one is the name of cluster
127.0.0.1 is the bind ip adress
80 the port of pound http server

After there is a list of backend of this pound server. Each backend have this form :

<ip backend>:<port backend>[,<priority backend>,[TimeOut backend]]

Notice that if you want define timeout you must define also priority

I hope that this recipe give you satisfaction. Don’t hesitate to submit an issue at http://trac.ingeniweb.com/ if you have problem.

Regards Youenn


Mai 13, 2008
» plone.recipe.apache

In this post we describe how to install and configure an apache server in front of zope with buildout.
It’s very simple. Ok let’s go

1 – Create an zope instance with buildout

If you have a paster in your system we create an buildout skeleton with it:

$ paster create -t plone3_buildout

This create an buildout.cfg witch create an zope server (listen in 8080)

$ python2.4 bootstrap.py
$ bin/buildout

Create an instance named plone

$ bin/instance start

Ok now with your favourite browser you can see in http://localhost:8080/plone
the plone instance..
We can declare a localdns (for test purpose) that says that www.testit.com go to localhost..
So http://www.testit.com:8080/plone works also.

2 – Apache serve your plone instance

It’s possible now with buildout
We add an apache server to serve our zope in http port
Edit buildout.cfg and put this config in buildout::

[buildout]
parts =
...
apachebuild
apacheconf
...
[apachebuild]
recipe = plone.recipe.apache:build
url = http://mir2.ovh.net/ftp.apache.org/dist/httpd/httpd-2.2.8.tar.gz

[apacheconf]
recipe = plone.recipe.apache:config
bind = 80
backends =
 www.testit.com:127.0.0.1:8080
zope2_vhm_map =
 www.testit.com:/plone

You have already an httpd installed in your system . No problem
delete apachebuild conf and add in apacheconf a new directive mainconfig as this

[apacheconf]
mainconfig = /etc/apache2/apache2.conf
recipe = plone.recipe.apache:config
bind = 80
backends =
 www.testit.com:127.0.0.1:10080
zope2_vhm_map =
 www.testit.com:/plone

BE CAREFUL : the user that launch buildout must write in mainconfig so give to him the good right !! Also in order to work the current apache server must have rewrite and proxy module allowed.

So now relaunch buildout::

# bin/buildout
Getting distribution for 'plone.recipe.apache'.
Got plone.recipe.apache 0.1.0.
...
apachebuild: Downloading apache tarball.
apachebuild: Compiling Apache
...
apachebuild: Unpacking and configuring
checking for chosen layout... Apache
...
Installing apacheconf.

And after, start (or restart) apache server

# sudo bin/apachectl start

and now::
http://www.testit.com/ go to your plone instance.. Viva buildout !