By Ingeniweb. A Django site.
Juillet 3, 2009
» Dropping PEP 386 (versions comparison)


As you might know, we are working hard on Distutils side for Python 2.7 and 3.2 upcoming releases. The biggest work is on PEP 376, that will introduce among other things a uninstaller function and functions to query installed distributions.

The other “big” work is on PEP 345. We want to introduce a new metadata field called “install_requires” to be able to express requirements. That’s from the setuptools project and is quite used by the community. Notice that there were several attempts to define requirements in the past in Distutils, but none of them really made it through.

For instance, if you want to define docutils as a dependency for your distribution but a version less or equal to 0.4, you can say :

docutils <= 0.4

But as long as you want to work with such dependencies and provide a way to express them with operators, you have to be able to compare versions. For instance if you want to compare an installed docutils distribution to see of it is compatible with 0.4.

That’s another big topic we have been working on for the last few months, with people from various communities (Fedora, Ubuntu, etc). And I have started to write down PEP 386.

But comparing version appears to be a topic that cannot be generic. It seems that Distutils, therefore Python, shouldn’t enforce any rule on this.

Furthermore, since we have said that Distutils should be a lighter package, it will not implement a complete package managment system, like setuptools or zc.buildout does.

So I’ve decided to propose to drop PEP 386, and stick on a very simple rule in PEP 345, saying that requirements can be defined, with :

distribution_name OPERATOR version

where OPERATOR is in >, <, ==, !=, >= or <=.

Last, so the work done at Pycon and in Distutils-SIG is not lost, I will publish the library we wrote. This could be a very good basis for packaging managment systems out there.

» Distutils nighlty builds


If you want to work (or try to find some bugs in it to help out) with the latest bleeding-edge Distutils trunk version that will be shipped with Python 2.7 and Python 3.2, you can do it now !

I am creating a nighlty build every day here now : http://nightly.ziade.org/

This distribution can be installed in your existing Python 2.6 installation, and will probably work with 2.5.

I’m not planning to publish standalone versions yet, except nighlty build, at least until 2.7 and 3.2 are starting to be released as alphas.

Juin 30, 2009
» mod_ntlm in FreeBSD


We had to install mod_ntlm under a FreeBSD 6.4 server with a colleague, but the port for this package seems broken at least for FreeBSD 6.2 and 6.4. It just doesn’t work when it initiates an NTLM session through SMB, and it seems to be compiled without the log support so the last log you get doesn’t give any useful hint on what’s wrong.

So we had to recompile it to activate the log and try understand what the problem was. But Apache doesn’t not support threaded mode under FreeBSD 6.x (or I couldn’t manage to make it work howsoever), so mod_ntlm failed to compile since it uses Apache’s thread mutexes to perform some locks on some operations.

We’ve deactivated them and recompiled the module, and now it works now like a charm. It’s weird because it means that the binary package for this module is completely broken. I couldn’t find any place mentioning this.

Anyways, I know this is an old software combination, but since Google doesn’t give any hint on this problem, I wanted to blog about it and join the diff I made out of mod_ntlm2 at https://modntlm.svn.sourceforge.net/svnroot/modntlm/trunk here: diff file

So if you bump into this problem, you will hopefully reach this page and save a few hours.

Juin 29, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» Facebook connect with python

A few weeks ago I've discover Facebook Connect. So I decide to try to use it in a new project. It look like an easy way for new user to register in my application.

The first step is to follow the Quick start guide. Then whe need to take care of Facebook in our application. There is already a cool python library to play with Facebook called pyfacebook. So you need to install it.

In my project I aleady use repoze.what for authentification. It just rocks. So i decide to write an IIDentifier plugin for Facebook:

import sha
import facebook
from webob import Request
from zope.interface import implements
from repoze.who.interfaces import IIdentifier

DEFAULT_FIELDS = ['uid', 'name', 'first_name', 'birthday', 'relationship_status',
                  'proxied_email', 'sex', 'hometown_location',
                  'pic', 'pic_big', 'pic_small', 'pic_square']

class Params(dict):
    def __getattr__(self, attr):
        return self.get(attr, '')
    def __html__(self):
        return repr(self)

class Facebook(object):
    implements(IIdentifier)

    def __init__(self, api_key, secret_key):
        self.api_key = api_key
        self.secret_key = secret_key

    def identify(self, environ):
        req = Request(environ)

        # initialize the api object
        fbapi = facebook.Facebook(self.api_key, self.secret_key) # init pyfacebook instance
        if fbapi.check_session(req):
            environ['repoze.who.fb'] = fbapi
            user = fbapi.users.getInfo([fbapi.uid], DEFAULT_FIELDS)[0]

            # we used the proxied_email and a generated password to retriave
            # the user from our DB
            user.update(
                login=user['proxied_email'],
                email=user['proxied_email'],
                password=sha.new('%s%s' % (fbapi.uid, self.secret_key)).hexdigest(),
                )

            # we store the facebook data in environ
            environ['repoze.who.fbuser'] = Params(user.items())
            return user

    def remember(self, environ, identity): pass
    def forget(self, environ, identity): pass

At the first time there is no user in our DB. So the user isn't really connected to our interface. The trick is to use a javascript callback to show a registration form to the user with fields pre-filled with Facebook's data.

<html xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://www.facebook.com/2008/fbml">
  <head></head>
  <body>
   <script src="http://static.ak.connect.facebook.com/js/api_lib/v0.4/FeatureLoader.js.php" type="text/javascript">
   </script>
   <fb:login-button onlogin="facebook_onlogin();"></fb:login-button>
   <script type="text/javascript">
     FB.init("YOUR_API_KEY_HERE", "xd_receiver.htm");
     var facebook_onlogin() function() {
         // here is the trick
         window.location.href = '/register/facebook';
     }
   </script>
</body> </html>

When the form is submitted, you need to create the real user in your application. That's all !

» collective.releaser missing releases and upload bug


Since few weeks we have problems to use collective.releaser when we use release-packages: packages are not uploaded to our private pypi or are published in pypi when they should be private and public packages are not published at all.
collective.releaser seems be a victim of this bug as the 0.6.2 release was never published on pypi even if the tag was done 2 months ago.

Today ready this:
collective/releaser/commands.py:L107

Line 110, when the expression is matching something the loop continues, then in the line 117 an empty list is in found sometimes.

If you add these two lines after the line 112 you can again upload your releases in cheeseshops:

if founded != []:
    break

Juin 9, 2009
» Retrieving UserPrincipalName with NetBiosDomain\NetBiosLogin


Sometimes the hard part of a python application is to integrate sso because there is an unknown : what rules is defined to get the user !

In windows, apache mod_sspi or enfold proxy give to us an http header ( name X_REMOTE_USER) to deal with the active directory. This header is like that:

Domain\user

If you have one domain it’s pretty simple. Your userid is unique

But in big company there is multiple domain controler. And user is not unique ! So how retrieve an unique user id for active directory and use it in my windows python application !

The response is get UserPrincipalName with COM and NameTranslate interface.

import win32com.client
d = win32com.client.Dispatch('NameTranslate')
d.Init(3,'')
d.Set(3,'domain\\user')
userPrincipalName = d.Get(9)

Now if you use COM with zope as me , COM is not thread safe. So init the client at zope starting and lock yours calls to the API

import win32com.client
import threading
D = win32com.client.Dispatch('NameTranslate')
D.Init(3,'')
COMLOCK = threading.Lock()

And in a function use the global D

def getUserPrincipalName(sso_header):

     try:
            COMLOCK.acquire()
            D.Set(3,sso_header)
            userPrincipalName = D.Get(9)
     finally:
            COMLOCK.release()
     return userPrincipalName

Youpi , thanks win32com !!

Juin 3, 2009
» Heads up: new plone.recipe.pound release


Since few weeks plone.recipe.pound 0.5.4 was broken with an output like this:

An internal error occured due to a bug in either zc.buildout or in a recipe being used:
Traceback (most recent call last):
File "/tmp/tmpVcq-b1/zc.buildout-1.2.1-py2.4.egg/zc/buildout/buildout.py", line 1509, in main
File "/tmp/tmpVcq-b1/zc.buildout-1.2.1-py2.4.egg/zc/buildout/buildout.py", line 473, in install
File "/tmp/tmpVcq-b1/zc.buildout-1.2.1-py2.4.egg/zc/buildout/buildout.py", line 1091, in _call
File "/home/encolpe/.virtualenvs/internal/preprod/plone.recipe.pound/plone/recipe/pound/build.py", line 74, in install
installed = CMMIRecipe.install(self)
File "/home/encolpe/.virtualenvs/internal/preprod/downloads/dist/zc.recipe.cmmi-1.2.0/src/zc/recipe/cmmi/__init__.py", line 155, in install
system("make install")
File "/home/encolpe/.virtualenvs/internal/preprod/downloads/dist/zc.recipe.cmmi-1.2.0/src/zc/recipe/cmmi/__init__.py", line 27, in system
raise SystemError("Failed", c)
SystemError: ('Failed', 'make install')

This recipe directly unherit from zc.recipe.cmmi and act as a wrapper around the CMMIRecipe.install method. Most arguments used are transmitted to the recipe are self attributes or keys in self.options. The version 1.2.0 of this recipe publish the May, 18 moved the way that extra arguments are transmitted:

  • zc.recipe.cmmi 1.1.x:  self.options['extra_options']
  • zc.recipe.cmmi 1.2.0: self.extra_options (an empty string by default)

There’s no regression tests, neither an entry in the release history. Youenn did a new release of plone.recipe.pound to fix this new behavior.

Morality:

  • if you are using zc.recipe.cmmi 1.1.x you can still use plone.recipe.pound 0.5.4
  • if you are using zc.recipe.cmmi 1.2.0 or above use plone.recipe.pound 0.5.5
  • other recipes using zc.recipe.cmmi may be broken

Mai 30, 2009
» Smoke testing in Distutils


I am not really sure smoke testing is the best name to describe what I am trying to set up.

Wikipedia says:

In computer programming and software testing, smoke testing is a preliminary to further testing, which should reveal simple failures severe enough to reject a prospective software release. In this case, the smoke is metaphorical.

But I think it describes well the need I have : being able to test the output of several releases of Distutils, without relying on unit tests for that. Simply because they did not exist on early versions of Distutils.

Why do I need “smoke tests”

Lately, I was working on Distutils, removing some old bugs and adding some tests. Working on an under-tested package is like opening a can of worms : when you change something you can introduce some regressions. It’s hard to avoid for sure, because even if you add some tests subsequently for some features, you are not really doing proper TDD, where the tests and the code grow together. So there might be some uncovered cases left even if the coverage is improving slowly.

Regression also happens when you correct a behavior that was broken for years : if third party tools used a broken behavior without knowing it, fixing the behavior at some point becomes a regression from them.

But these pitfalls are unavoidable. Hopefully they don’t last for too long in a big project like Python. They are quickly reported, either by buildbots, either by someone from the community.

The last behavior problem I had was on the build_ext command. There’s a string option called “compiler”, were you can force the compiler type. You can put for instance ‘linux‘ and build_ext will use the UnixCompiler compiler class.

Although this option name is misleading. It should have been called “compiler_type” in the first place.  And the build_ext command adds another problem : it turns the compiler attribute of the class where the option value is stored, into a compiler instance. No need to say a class attribute value should not have several types.

The first application to suffer from this is Python itself. It uses build_ext.compiler, as a compiler instance to build its modules.

This means that “compiler” is now doomed to be both an option and a compiler attribute. It’s not easy to fix and will require a deprecation step.

So the idea of the smoke test is to make sure Distutils still knows how to produce releases of existing projects.

The Smoke testing buildbot

I have built a buildbot instance using collective.buildbot. The script the slaves run is quite simple: it gets some projects source and run their setup.py script, using various versions of the Python interpreter, then the trunk itself. The commands run so far are sdist and bdist.

Then it compares every file contained in the archive created to make sure produced archives are similar.

I will encouter some exceptions, when the package I am testing includes different files in its distribution depending on the Python version, but those exception will be managed.

Last, I cannot run this test on every package out there, for security reasons. Until I have the proper virtual environment to do that, I’ll work on a white list of packages I “trust”. So far, there’s just a package of mine, and NumPy. But this list will grow.

If you are curious to know if your package builds under Python trunk, get in touch with me, I’ll probably add it if I know I can trust it.

The last problem I have with this system yet is the fact that setuptools is not working anymore with the Distutils trunk, so I am unable to test setuptools-based packages.

The buildbot is located here : http://buildbot.ziade.org/waterfall

Mai 23, 2009
» Some Deliverance tips


I really like to use Deliverance for skinning Plone or any web application. In this post, i just want to help Deliverance beginners to configure their server, and to talk about PyQuery, which could be a good Deliverance companion.

Apache 2.2 + Deliverance : configuration

I spent some time to configure Apache in front of Deliverance, you will find here a possible configuration.

Deliverance Server configuration :

In your rules.xml, the first declaration relates to server config, use ‘0.0.0.0′  as server address to be able to serve any ip address :

<server-settings>
 <server>0.0.0.0:8000</server>
 <execute-pyref>true</execute-pyref>
 <display-local-files>false</display-local-files>
 <dev-allow>
 localhost
 </dev-allow>
 <dev-user username="dev" password="dev" />
 <dev-expiration>60</dev-expiration>
</server-settings>

Always in rules.xml you need to write the proxy rule for Zope Virtual Host or another server.

If you want to serve a plone site located at http://localhost:8080/myplonesite :

<proxy path="/" rewrite-links="1">
     <dest href="http://localhost:8080/VirtualHostBase/http/{SERVER_NAME}:8000/myplonesite/VirtualHostRoot" />
</proxy>

If you need a more complex configuration in front of Zope, you can always use Apache + Varnish or SQUID + Pound, or IIS + EEP, and ZEO in front of Zope using another port (90 in this example) and use this Deliverance proxy rule :

<proxy path="/ rewrite-links="1">
    <dest href="http://{SERVER_NAME}:90" />
</proxy>

And if you want to serve anything else for another path :

<proxy path="/anotherpath" rewrite-links="1">
    <dest href="http://anythingelse" />
</proxy>

Apache VirtualHost configuration

You can use Deliverance on port 80 (just replace 8000 by 80 in rules.xml) to directly serve your site, but you would prefer Apache for that in many cases.

Suppose you want to rewrite all requests to “www.mysite.org” with a Deliverance response (on www.mysite.org:8000), you can use the mod_rewrite like this :

<VirtualHost *:80>
    ServerName www.mysite.org
    ServerAlias mysite.org
    ErrorLog "logs/error.mysite.org.log"
    CustomLog "logs/access.mysite.org.log" combined

    # do not forget this declaration
    ProxyPreserveHost On
    RewriteEngine On
    RewriteLog "logs/rewrite.mysite.org.log"
    RewriteCond %{HTTP:Authorization} ^(.*)
    RewriteRule ^/(.*) http://%{SERVER_NAME}:8000/$1 [P]

    # etc ...

</VirtualHost>

That’s all.

If you want to serve urls starting with a particular path (/mypath),  just change your rewrite rule like this :

    RewriteRule ^/mypath(.*) http://%{SERVER_NAME}:8000/_vh_mypath$1 [P]

Just take care in this last case to put the good urls in your Deliverance static contents, if you need it .

In fact you can do anything you want with rewrite rules :-)

Deliverance and PyQuery

PyQuery is a jQuery-like library for Python which help you to play easily with html elements. Combined with Deliverance you will be able, with a python rule, to modify content or theme on the fly, at any point of your Deliverance transformation.

To learn how to add PyQuery rules to Deliverance, read this post  (many thanks to Gaël who help me to learn Deliverance) :
http://www.gawel.org/weblog/en/2008/12/skinning-with-pyquery-and-deliverance

Read also  :
http://www.openplans.org/projects/deliverance/lists/deliverance-discussion/archive/2008/12/1229591136002/forum_view

And now just try these example :

You have a floating right toolbar with floating right tabs inside in your html template theme :

<ul id="right-bar">
    <li><a href="/abcd">ABCD</a></li>
    <li><a href="/efgh">EFGH</a></li>
</ul>

You have a floating left toolbar with floating left tabs inside in your html source content :

<ul id="left-bar">
    <li><a href="/abcd">ABCD</a></li>
    <li><a href="/efgh">EFGH</a></li>
</ul>

With Deliverance xml rules you want to replace right tabs by left tabs like this :

    <replace content="children:#left-bar" theme="children:#righ-bar" />

But of course you need to reverse tabs order before or after xml transformation.
To modify the theme  just add a pyquery rule after the replace declaration :

    <pyquery pyref="myskinproduct.rules:pqrules" />

In your rules.py,  create your pqrules function like this :

def pqrules(content, theme, resource_fetcher, log) :

    # if you prefer to work on content
    # before Deliverance transform
    # replace theme('#right-bar') by content('#left-bar')
    nav = theme('#right-bar')
    navitems = nav('li')
    navitems.reverse()
    # i'm not sure that it's needed :
    nav.empty()
    nav.append(navitems)

Other examples :

    # replace a search button text with 'OK' in content
    content('#portal-searchbox .searchButton').val('OK')

    # remove icons from a navigation portlet
    # replace icons by "> "
    nav = content('#navigation')
    navitems = nav('li')
    navitems('img').remove()
    navitems.append('<span>&gt;&nbsp;</span>')

    # add a "last-item" class to last tab in a toolbar
    from pyquery import PyQuery as pq
    navitems = content('#navbar li')
    if navitems :
        pq(navitems[len(navitems-1)]).addClass('last-item')

    # if you have inserted external contents in content area with Deliverance xml rules
    # you want to change a navitem class in result
    if theme("#photos') :
        selectNavigationItem('#photos')

    #etc ...

To do this kind of template manipulations, changing content on source application could be a best practice, it depends of  your particular situation.
But i’m sure that it’s a fast way  ;-)

The problem is how to test the result ? twill could be a solution …

Do not transform some contents with deliverance

If you dont like bad surprises, you need to abort ajax html content transformation. And in some other cases you need also to abort Deliverance transformation (see example)

In your rules.xml, just add :

    <match pyref="myproduct.rules:match_notheme" abort="1" />

and in your rules.py

def match_notheme(req, resp, headers, *args):

    match = False
    if req.headers.get('X-Requested-With', '').startswith('XML'):
        match = True
    # the html template of FCKEditor iframe must be unchanged
    if req.path_info.startswith('/editor'):    
        match = True
    if 'notheme' in req.GET:
        match = True
    return match

Mai 19, 2009
» Monitoring a Zope 2 application


We have a simple need for a customer project that runs a Zeo server and a few Zeo clients : being able to check the status of every Zeo client, and monitor what they are doing.

DeadlockDebugger almost provides this feature since it is able to produce a dump of the execution stack for every thread a Zope instance is running.

Based on this tool, I have developed ZopeHealthWatcher, that provides a console script to query a Zope instance, and get back a status for every running thread. It tells you if the thread is idling or if it’s running some code. The script also returns an exit code depending on the number of busy threads, so it can be used in tools like Nagios.

When there are 4 or more busy threads, the script will return the execution stacks for every busy thread and some extra info like the system load and memory info. The returned info will be extendable through plug-ins in the next version, but right now the provided info are enough for our needs.

I have also created an HTML version, so when the dump is requested from another tool than the console script (e.g. a browser), it displays a nice human-readable interface (check the PyPI page for more info and a screenshot).

Notice that DeadLockDebugger is hackish since it patches the Zope publisher at startup. But we won’t change this part: we need this tool to run from the oldest to the newest Zope 2 version. And the patch just works fine, so…

The provided version should run out of the box in a buildout-based Plone 3 application, but requires manual installation steps on older Plone or CPS versions.

I didn’t mention these manual steps in the documentation. I think I am the only person in the world interested in running this tool on the dead-but-still-in-production-in-many-places Nuxeo CPS.

By the way: kudos goes to Marc-Aurèle Darche, who is maintaining CPS for years now, making it one of the most bug-free and stable CMS solution out there.  Ok it’s probably easier to reach this level of quality since the platform is very stable and only evolves very slowly thanks to Georges Racinet.

Mai 10, 2009
» Distutils state


Since Pycon, a lot of discussion and work has been going on.

During the summit, we made a few decisions (see http://mail.python.org/pipermail/python-dev/2009-March/087834.html) but this topic is so wide and so complex that a lot of discussion still needs to be done to have a clear and complete picture of where we want to go and how we are going to do it

But we have made significant progress and reached consensus on some key points. This entry is a short list of these key points.

Being able to compare project versions

If you take a look in Distutils code, there’s a version and a versionpredicate module that provides a way to compare versions. This feature is barely used by Distutils itself, and a very few number of projects out there are using it (If you do so, please let me know).

This is probably because Distutils provides two different ways to compare versions: a “strict” one and a “loose” one. In other words Distutils clearly states that it is unable to provide a unique version comparison algorithm that can be used by anyone out there. Anyone means here : Python developers, OS packagers, etc.

Setuptools did a better job by providing an algorithm that covers most cases, but suffers from this universality: it’s too heuristic.

So one of the main topic during Pycon was to try to find a version comparison algorithm that would just work for everyone and in the meantime would be strict enough to be useful. That’s a pretty tough task, but I think we have reached something that is “good enough” for everybody. We had the chance to have people from Ubuntu, RedHat during Pycon to work on this task, and Trent Mick took the lead during the sprints to come up with a prototype.

It’s described here : http://wiki.python.org/moin/Distutils/VersionComparison and I have put the code here http://bitbucket.org/tarek/distutilsversion.

Two more things to take care of:

  • Philip Eby came up with an edge-case : being able to do a development version of a post release.
  • Jean-Paul Calderone proposed to add a constructor that would take explicit arguments to describe the version, rather than a string representation

There’s a branch (tarek-postdev) on bitbucket to work on these two cases. But basically, it seems that we have a consensus on a unique way to compare versions in Python ! This is a great step forward.

Another point that Toshio Kuratomi raised during a hall discussion was the fact that some Python developers are not having good practices when versioning their releases. So we agreed that a good “How to properly use version numbers for your projects releases” document will have to be delivered in Distutils documentation, besides the new algorithm. I am pretty confident that people will eventually follow it.

APIs to access installed project metadata, and an egg-info standard

Once a project is installed in Python, you don’t have a way to get its metadata and to answer to basic questions like:

  • what are the installed packages ?
  • what is the version ?
  • how is the author ?

Of course you can dig in your installation and get back most of these, but depending how you installed your package (manual, easy_install, pip) it might be located in different places.

So we agreed on a standard for this, described in PEP 376 : http://www.python.org/dev/peps/pep-0376

An API will be introduced to be able to get the info for a installed package, as long as it was installed using PEP 376 standard.

An uninstall feature !

A feature that is claimed for a long time now should be introduced in Distutils : uninstallation !

Distutils will provide a basic, reference uninstall feature that will remove all files that where previously installed for a project. This will be doable as long as this list of files is recorded at installation time, and not used by another project.

See the details in PEP 376.

Standardize PKG-INFO

New fields will be added in the metadata of a project. The most important one is install_requires from setuptools, which let you list the projects your project depend on and their versions.

This is for informational purpose, and Distutils will not provide an install feature that will fetch and install dependencies. This can be done by third-party tools like easy_install  as long as they use the installation standard described earlier.

We started this work with Jim Fulton at Pycon, and Tres Seaver took the lead on this task since then. The plan is to update PEP 345 . The work is done in a branch here : http://svn.python.org/view/peps/branches/jim-update-345/pep-0345.txt?view=markup

But the remaining work is focusing on practical details. Just remember that PKG-INFO will evolve and install_requires will be integrated amongst other changes.

Distutils code cleanup, test coverage

The test coverage is starting to look good and most modules are covered around 80%. I guess this is the average in most Test-Driven Development projects out there. So Distutils is becoming a good citizen of the standard library  ;)

The remaining work for a good test coverage is mostly on compilers side and os specific commands.

My black list of untested (or nearly untested) modules :

  • bcppcompiler
  • cygwinccompiler
  • emxccompiler
  • msvc9compiler
  • msvccompiler
  • command.bdist_msi
  • command.bdist_rpm

I need to figure out how to properly test them. Last, I need to set up a buildbot that will try out Distutils trunk on a list of projects out there, like numpy for example.

Other topics

Check this ! http://wiki.python.org/moin/Distutils

Mai 7, 2009
» Install a maintenance page without restarting apache


These days we are working with fabric to remove some hazard in maintenance.
Recently someone gzip a Data.fs file in production and our customer loose 2 days of work. We choose to use fabric (http://www.nongnu.org/fab/) to limit command line working to the strict minimum.

Now, almost all procedures were added in a fabric factory, from the installation of a development environment to the upgrade of the production server. In this last part, we search a solution to put a maintenance page without have to gain root privileges to restart the apache used in frontend. For static or CGI sites you can use a .htaccess file to override the global rules. For a stopped Zope server it doesn’t help much.

The goal is to use RewriteCond and RewriteRule in a such way that a static HTML file would be displayed when it exists. We can call it ‘maintenance.html‘ and rename it ‘maintenance.html-disabled‘ to let the site be displayed normally.

We start with a virtual host generated by iw.recipe.squid:

<VirtualHost *:80>
   ServerName www.gosseyn.fr

   RewriteEngine On
   RewriteLog /my/path/to/squid/parts/squid/log/rewrite_www.gosseyn.fr.log
   RewriteLogLevel 0

   CustomLog /my/path/to/squid//parts/squid/log/access_www.gosseyn.fr.log common
   ErrorLog /my/path/to/squid//parts/squid/log/error_www.gosseyn.fr.log

   ## common rules for squid rewrite rules
   RewriteRule ^(.*)$ - [E=BACKEND_LOCATION:127.0.0.1]
   RewriteRule ^(.*)$ - [E=BACKEND_PORT:8081]
   RewriteRule ^(.*)$ - [E=BACKEND_PATH:site]

   RewriteRule  ^/(.*)/$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]
   RewriteRule  ^/(.*)$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]

   ## specific rules base on cookie

</VirtualHost>

First we need to declare a document root and to bypass all security rules. We create a new path in the root of our buildout folder called ‘maintenances_pages’.

   DocumentRoot /my/path/to/buildout/prod/maintenance_pages

   <Directory "/my/path/to/buildout/prod/maintenance_pages">
       Options None
       AllowOverride None
       Order allow,deny
       allow from all

       <LimitExcept GET>
           Order allow,deny
           allow from all
       </LimitExcept>
   </Directory>

These declaration should satisfy all paranoid BOFH.
Now create a file called maintenance.html within the folder. All code (javascript, CSS and images) must be embeded, this is a limitation of this approach.
Well, how to detect that a file exist on the filesystem from apache. You can do this with the flag ‘-f’ from the rewrite module:

    RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html -f

Then redirect all pages to our maintenance page:

   RewriteCond %{REQUEST_URI} !/maintenance.html
   RewriteRule $ /maintenance.html [R=302,L]

We also need to disable the standard rewrite rules. The easier way to do it is to use ‘!-f‘.

   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)/$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]
   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]

These rules will not be applied anymore when our maintenance file will be name ‘maintenance.html’. Our goal is reached.

The whole virtualhost file look like this:

<VirtualHost *:80>
   ServerName www.gosseyn.fr

   RewriteEngine On
   RewriteLog /my/path/to/squid/parts/squid/log/rewrite_www.gosseyn.fr.log
   RewriteLogLevel 0

   CustomLog /my/path/to/squid//parts/squid/log/access_www.gosseyn.fr.log common
   ErrorLog /my/path/to/squid//parts/squid/log/error_www.gosseyn.fr.log

   DocumentRoot /my/path/to/buildout/prod/maintenance_pages

   <Directory "/my/path/to/buildout/prod/maintenance_pages">
       Options None
       AllowOverride None
       Order allow,deny
       allow from all

       <LimitExcept GET>
           Order allow,deny
           allow from all
       </LimitExcept>
   </Directory>

   ## our maintenance page
   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html -f
   RewriteCond %{REQUEST_URI} !/maintenance.html
   RewriteRule $ /maintenance.html [R=302,L]

   ## common rules for squid rewrite rules
   RewriteRule ^(.*)$ - [E=BACKEND_LOCATION:127.0.0.1]
   RewriteRule ^(.*)$ - [E=BACKEND_PORT:8081]
   RewriteRule ^(.*)$ - [E=BACKEND_PATH:site]

   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)/$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]
   RewriteCond /my/path/to/buildout/prod/maintenance_pages/maintenance.html !-f
   RewriteRule  ^/(.*)$ http://127.0.0.1:3128/%{ENV:BACKEND_LOCATION}/%{ENV:BACKEND_PORT}/http/%{SERVER_NAME}/80/%{ENV:BACKEND_PATH}/__original_url__/$1 [L,P]

   ## specific rules base on cookie

</VirtualHost>

It will be difficult to include this in a recipe as we modify an existing virtualhost in a very specific way.

As usual, all feedbacks are appreciated.

Mai 3, 2009
» Gsoc : Keyring library work started !


I am very proud to be a Gsoc mentor this year on a very interesting topic : an universal keyring library for Python.

About the topic

In Distutils, if you want to interact with PyPI, you have to register in the website and you get a login/password so you can register and/or upload your packages.

Before Python 2.6, the only way you could interact with PyPI was by storing these info into your .pypirc file in clear text. This was not the best solution. For example we have in my company some staging servers we share, and from whom we upload packages to various PyPI-like servers. So we have to store PyPI login/password info in them. This means that if Bob wants to push his package from that server, he has to put his password into a clear text file which is most of the time readable by everyone. It’s not such a big deal in our company since we can trust each other, but it’s a very bad practice.

So I ended up changing this in Python 2.6 so people could type their password in a prompt when working with packages, using getpass.getpass. So they wouldn’t have to store them anymore.

But this is not enough : we need to provide something better. We need getpass.getpass to be able to interact with keyring libraries like KeyChain, Gnome Keyring, etc. So the login/password info are safely stored and can be reused.

This service will be useful for Distutils, but also for any Python application.

The idea of the GSOC task is to provide an unified keyring library for Python, and it’s harder that it sounds. For instance, we need to find a way to provide something that works under Windows. So the whole work is quite challenging and interesting, and the goal is to end up with a keyring library we can use into Distutils and propose for inclusion in getpass.

About Kang

Kang Zhang is the student that was selected for this work. Congrats ! He has started to work on it. You can follow his work in his blog. I have a strong feeling that he will succeed in this work and come up with something good.

Take a look at the Python Soc planet too, where all students involved in Python GSOC are blogging about their ongoing work.

Mai 2, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» collective.releaser rocks !

Tarek have made some great works on collective.releaser and collective.dist the last months. There is currently a refactor branch.

The goal of collective.releaser is to normalize the release process of a buildout based project.

I've worked on it the last week to implement the latest needed steps to allow an easy way to release a buildout project.

So here is how it works. collective.releaser came with a paster template to generate a buildout skel. You got a dev.cfg for development with a develop option with all your develop eggs in it and a prod.cfg for production without develop and all versions fixed in the versions section.

I've modified the project_release so you just need to run:

$ bin/project_release --version=1.0

This will release eggs founded in the dev.cfg's develop option on a pypi server and fix the versions in buildout.cfg. If no version is founded for an egg in the versions section of the prod.cfg then the egg is released at the version passed to the command line. If a version is founded then the egg is released at this version. Here is an example:

The dev.cfg:

[buildout]
parts = eggs

develop =
  ../collective.releaser
  ../my.project

[eggs]
recipe = zc.recipe.egg
eggs =
  collective.releaser
  my.project

The prod.cfg:

[buildout]
parts = eggs
versions = versions

[versions]
collective.releaser=0.6.1

[eggs]
recipe = zc.recipe.egg
eggs =
  collective.releaser
  my.project

collective.releaser package is already at 0.6.1 so no release is needed. my.project as no versions specified so a new 1.0 release is generated and uploaded on your pypi server.

The generated buildout.cfg will look like this:

[buildout]
extends = prod.cfg
versions = versions

[versions]
collective.releaser=0.6.1
my.project=1.0

After all eggs are released, project_release will generate a tarball with the tagged version of the project and all eggs founded in the develop option.

Then to upgrade your project you just need to untar the archive on your production server. Launch bin/buildout. And that's it.

This will probably make life easier for developers ! Hope we can make a new self generated release soon :)

» Basic plugin system using ABCs and the “extensions” package


I need a very simple plugin system for one of my projects. The project is a small WSGI application called mysysadmin that allows you to launch some commands on your system to manage some applications. It also allows you to view log files in your web browser.

It’s similar in some ways to WebMin,

So in my application, every tab is a plugin that manages one application. I have a plugin for Apache, another one for MySQL, and so one.

Back to my plugin system. So every plugin that is registered becomes a tab in the WSGI application, as long as it provides all the methods my web application needs to interact with it. So I want to check that each plugin strictly provides the API needed by the main program.

The first tools that came in mind were :

But I find both projects a bit complex to implement such a simple plugin system.I could use the standalone Plugins package Phillip provides instead of setuptools, but it still does too much things imho. That’s someting I am currently learning by working on packaging matters : one library should not provide too many features.

Extensions : a simple plugin system

So I have started to implement a light-weight plugin system called extensions, which reuses setuptools entry points principles but is more simple to use. The goal of this project is to provide very simple APIs to handle plugins, and to make it work without introducing a new argument into the setup.py setup method, like setuptools does.

For instance, if you want to define an apache function in your modules module, in your myapp package, you just call the register function :

from extensions import register

register('mysysadmin.modules', 'apache', 'myapp.modules:apache')

That’s it !

And to use it, the mysysadmin application can use a simple API called get, that iterates over all plugins defined for “mysysadmin.modules” :

from extensions import get

for plugin in get(group='mysysadmin.modules'):
    # do something with the plugin

The magic is done by writing in the .egg-info directory installed for the package that contains each plugin, a PLUGIN file that contains the list of registered elements. It’s an idea borrowed from setuptoools entry points. So get iterates over all .egg-info directories in your path and load the PLUGIN files it finds.  Nothing new here. That’s how setuptools does, and that’s perfect.

If you have any feedback on extensions, let me know !

Strict plugins

The other need is to strictly check that every plugin provides the API needed, e.g. fulfill the requirements. This is what we could call Design by contract .

You can provide a base class for this, and ask the plugins to inherit from it. Or you can ask the Plugin to provide a marker to specify it implements a given behavior. zope.interface can do a nice job for the latter,and let you check that a given object implements an interface.

But I wanted to give a shot to the brand new Python ABCs and make sure anyone can write a plugin in plain Python, without having to rely on any kind of marker system. ABCs will let you check that a class implements some methods without requiring it to inherit from a specific class, to implement a specific interface or provide a custom marker. Pure duck typing !

So let’s define for our application a Plugin class, that gives the signature every plugin will need to provide. It uses ABCMeta as its meta class, and the abstractmethod for every method that should be implemented by every plugin.

Here’s an extract :

from abc import ABCMeta, abstractmethod

class Plugin(object):
    __metaclass__ = ABCMeta
    @abstractmethod
    def get_command_list(self):
        return []

    @abstractmethod
    def run_command(self, name):
        pass

    @classmethod
    def __subclasshook__(cls, klass):
        if cls is Plugin:
            for method in cls.__abstractmethods__:
                if any(method in base.__dict__ for base in klass.__mro__):
                    continue
                return NotImplemented
            return True
        return NotImplemented

The __subclasshook__ method is a class method that will be called everytime a class is tested using issubclass(klass, Plugin). In that case, it will check that every method marked with the abstractmethod decorator is provided by the class.

So basically, the application can discover and use the plugins, with:

from extensions import get

for plugin in get(group='mysysadmin.modules'):
    klass = plugin.load()
    if not issubclass(klass, Plugin):
        logging.info('sorry, not a suitable plugin')
        continue
    # do something with the plugin
    xxx

Abstract Base Classes are one honking great idea — let’s do more of those!

Avril 19, 2009
» URLs in books


I received some complaints about the fact that some links in my books were dead by the time they were printed.

For the next book I am working on, I have proposed to my editor to set up a website to keep track of all references mentioned.

By using unique short ascii references throughout the book, it’s easy to provide a simple redirect service to the target URL, and to fix it when it changes (just by setting up a mail alert if your redirect reaches a 404).

For example, if I am referring to mod_wsgi in my book, I can write this reference: #mod_wsgi, and provide a redirection to http://code.google.com/p/modwsgi into my website, through a unique, mnemotechnic permanent URL : http://ziade.org/urls/mod_wsgi.

This small service, à la Tiny URL is not a burden for the reader imho : he is using his computer anyway when he visits an URL mentioned in a book.

It’s a simple idea I am sure a lot of people have thaught about before, but I fail to see it applied in the books I am buying these days. Is there any good reason I fail to see ?

Avril 15, 2009
» new blog location, new design, update your bookmarks


I was thinking about doing this change for a while, and I took the time to do it last week-end:

My personal website at http://ziade.org is now powered by Pylons and Atomisator. It’s both in French and English (the urls are translated too).

It contains:

  • a home page with my latest twitter entries
  • a blog (this one)
  • some other pages like a résumé, a list of the books I wrote, a projects page, etc.

I am still using WordPress to write my blog entries but they are now processed with Atomisator and displayed at http://ziade.org/blog. The feed is now located at http://ziade.org/blog/xml and I will make sure all Planets use this URL for now on.

Next I’ll probably :

  • move to another blogging tool, and my readers will not suffer from the change as long as they use the ziade.org feed url.
  • add some portlets into the new site, like the ones I have on WordPress.

How do you like this new look ? Any comment / advice ? :)

Avril 10, 2009
» Distutils: introducing the check command (reStructuredText control)


I am introducing the check command in Distutils. This command will check your package metadata, like the sdist and the register command already do (they display warnings).

But the new thing is that it will also allow you to check if long_description is reStructuredText compliant.

Its usage will be:

$ python setup.py check --restructuredtext
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)

And there’s also a strict mode, that raises an error in case something is wrong

$ python setup.py check --restructuredtext --strict
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)
error: Please correct your package.

Last, this command will be used by register, and sdist, so you can stop the process in case the metadata are not correct. This is useful to make sure your PyPI home page is not broken for example, since it parses long_description to build it. So a good practice will be to use strict when registering a package:

$ python setup.py register --strict
running register
running check
warning: check: Title underline too short. (line 32)
warning: check: Title underline too short. (line 32)
warning: check: Could finish the parsing. (line None)
error: Please correct your package.

This feature will land in Python 2.7 (I am working on it and it should be commited this week end). Of course, it will not introduce a hard dependency on docutils in Python, neither it will change the current default behavior.

Until then, you can use collective.dist 0.2.3, that provides this feature for Python 2.4 to 2.6

Mars 30, 2009
» Pycon hallway session #2: thoughts for multiple versions in Python


We had an excellent brainstorming session today in the hall, with Toshio, Georg, Martin, Thomas, etc.. (sorry we were so many I don’t have the full list) with some insights from Guido and Brett. We tried to think about a way to handle multiple versions of a same package.

Here’s the two most important concepts :

  • Unicity: There should be one and only one instance of a Python package at a given version on a system
  • Combination: One Python application combines several packages to run

About unicity

A Python package is a component that can be installed on a system. If you use the standard Distutils approach, it will end up in the Python site-packages directory and be importable by the interpreter. This package comes with a version number and is unique.

This unicity is important for security and maintainability. For instance, if there’s a security hole in a package, the fix is applied in one place and the system maintainer knows it can’t be present elsewhere on the system.

This is the system administrator point of view

About Combination

What defines a Python application is the fact that is selects a list of packages it needs to run. And this varies for every application. So two Python applications might need a different version of a given package and that is normal.

What’s important is to have the right list of packages when an application loads. Tools like zc.buildout or virtualenv are perfect for this need : they create an isolated environment for your application to run with the right set of packages.

So the simplest way to release an application is to ship it with everything required, regardless the unicity.

This is the application developer point of view

The idea

zc.buildout and virtualenv are a blast for developers, and another thing system packagers might dislike. This is because they break the unicity by allowing developers to ship their applications as black boxes. Of course one may say that this is perfectly fine since what’s inside an application is not the problem of the system packager. But since this application is made of packages that may be shared on the system by other applications, that is redondant.

Forget site-packages for a moment. And let’s think about a new loading system for packages. This approach is similar in some ways to setuptools’ multiple version system.

Storing multiple versions

First of all, let’s store the packages in a directory, and for each version of the package, under a sub-directory which name is the version.

For example:

  • SQLAlchemy
    • 0.4
      • package code is here
      • package egg-info here
    • 0.5
      • package code is here
      • package egg-info here
  • jsonlib
    • 1.2.6
      • package code is here
      • package egg-info here
    • 1.3.10
      • package code is here
      • package egg-info here

Given this structure, some mechanism can provide to the interpreter the latest available version of a package, as long as Distutils knows how to handle version comparisons correctly (which will be the case in a near future)

Choosing specific versions

Back to our application. Let’s call it MyApp, version 1.0.

It needs specific versions for some packages. Forget zc.buildout and pip for a moment. And let’s think about a different way to express the packages (and versions) its needs.

Let’s make a Python package for this application and let’s make a few assumptions on some features in Distutils:

  • setuptools’ install_requires has made it into Distutils, as part of metadata.
  • metadata are defined statically in a package, apart from setup.py.

So basically, the application is mainly a static list of dependencies defined into install_requires. For example:

  • SQLAlchemy > 0.4
  • jsonlib == 1.2.6

When MyApp will get installed by Distutils, it will be added in the packages tree.

When it is used, it will need to load the versions of SQLAlchemy and jsonlib it needs. The ones that are inside its metadata.

To make it possible, the script that launches the application calls a built-in function called read_deps, that takes the metadata and reads them to know which versions fit:

# the script
read_deps('PKG-INFO')

import SQLAlchemy
import jsonlib
...

This call will load the right versions in the packages tree:

  • SQLAlchemy
    • 0.4
    • 0.5
  • jsonlib
    • 1.2.6
    • 1.3.10
  • MyApp
    • 1.0

What’s next

I just dropped here the rough idea, and a lot of details are missing. But I think it’s a good thing to share this idea here in its early stage.

We have decided we would try to write a prototype using importlib and sys.meta_path. Maybe Georg will start it during the Pycon sprint, and start to digg into the details. He was working on this when I left the open session at Pycon.

Stay tuned

Mars 27, 2009
» Pycon hallway session #1: a keyring library for Python


Before I sit down and clean up my summit notes to send them to python-dev, I wanted to post an entry about a small project which I think could be a great task for a student at the Summer of Code (I doubt it can fill 4 months of work but it could be done amongst other tasks).

Yesterday, we did a late session with Martin von Loewis, Jim Fulton and Georg Brandl about PyPI and the fact that it needed a better way to handle passwords on client side. That is, the fact that you have to store your password in the .pypirc file if you want to upload your package to PyPI.

This is unsafe and unwanted. A few months ago, I have made a small change in Distutils so it would prompt for the password using the getpass module if it doesn’t find it in the .pypirc file. (This was a contrbution of Nathan Van Gheem).

Anyways, this is not enough. Jim suggested to set up a SSH server on PyPI using Paramiko, so we could use a standard ssh connection and benefit from ssh-agent. But this is unfortunately not universal.

So let me get back to the idea I sent some time ago : http://mail.python.org/pipermail/python-ideas/2009-January/002465.html

What about having an option in getpass to store and reuse passwords in
system keyrings ?

    getpass(prompt[, stream])

would become:

    getpass(prompt[, stream, keyring])

where keyring would be a callable that can be use to retrieve the
password from a keyring system
and store it the first time.

The getpass module could provide some keyring support for:

- ssh-agent under Linux
- keychain under Mac OS X
- ...
ss

And let the developers use their own keyring system by providing a callable.

As Greg Smith said in the thread, the first task is to create a library that supports all standard keyring systems out there, including things like KWallet, Internet Explorer, Fireforx and so on…

I’ll mentor this project if any student would like to do it.