By Ingeniweb. A Django site.
Mars 6, 2010
Gael Pasgrimaud
gawel
Gawel's blurb
» Using restkit proxy in your WSGI app

Here is my use case. A few days ago I've wrote an application to mirror my Flickr accounts. The pics are downloaded on file system and the metadata are stored in CouchDB. Now I use the Flickr interface to upload and tags pics but I got my own data on my own server. That's always cool.

Well done but now it can be useful to have a small web app to see the pics. Right ? That's what I've do. Since I love jQuery and CouchDB is full json compliant I don't want to write a complex app with tones of python code. So the idea is to have a small wsgi app to only serve static javascripts/html/css files and a proxy app to serve json by proxying CouchDB.

For now I'm using WSGIProxy. The code is very simple and look like this:

from wsgiproxy.exactproxy import proxy_exact_request
from webob import Request

class Proxy(object):
    def __init__(self, db=None, **kwargs):
        self.db = db
    def __call__(self, environ, start_response):
        req = Request(environ)
        if req.method == 'GET':
            req.server_name = '127.0.0.1'
            req.server_port = 5984
            req.script_name = ''
            req.path_info = '/%s%s' % (self.db, req.path_info)
            resp = req.get_response(proxy_exact_request)
            resp.content_type = 'text/jasacsript'
        else:
            resp = exc.HTTPForbidden()
        return resp(environ, start_response)


def make_app(global_conf, **local_conf):
    conf = global_conf.copy()
    conf.update(local_conf)
    return Proxy(**conf)

That's cool and simple and ok for small files. But if you want to handle large request WSGIProxy will raise a MemoryError. This is no longer fun. I've already sent a patch but don't want to bother the Paste team with that.

Then I've try to use restkit and thought that it can be the best library to wrote a proxy app. restkit manage a pool of http connection and handle large file request in a clean way.

The result is a small contribution. A set of WSGI applications included in the wsgi_proxy extention.

Here is a simple Paste config file to show own to use the proxies:

[server:main]
use = egg:Paste#http
port = 4969

[app:main]
use = egg:Paste#urlmap
/couchdb = couchdb
/ = proxy

[app:couchdb]
use = egg:restkit#host_proxy
uri = http://localhost:5984/mydb

[app:proxy]
use = egg:restkit#host_proxy
uri = http://benoitc.github.com/restkit/
max_connections=50
allowed_methods = get head post

No code needed. Cheers.

You can also use the Proxy class to proxify clients request and transform the response on the fly:

from webob import Request
from restkit.ext.wsgi_proxy import Proxy

proxy = Proxy()

def application(environ, start_response):
    req = Request(environ)
    req.environ['SERVER_NAME'] = 'example.com'
    req.environ['SERVER_PORT'] = '80'
    # do stuff
    ...
    resp = req.get_response(proxy)

    # do stuff ...
    ...

    return resp(environ, start_response)

I've also tried to replace WSGIProxy in deliverance with an ugly monkey patch:

from restkit.ext.wsgi_proxy import Proxy
from wsgiproxy import exactproxy
print 'Patching exactproxy with restkit proxy'
exactproxy.proxy_exact_request = Proxy(max_connections=4, allowed_methods=['GET', 'HEAD', 'POST']).__call__

It work perfect.

Février 4, 2010
Gael Pasgrimaud
gawel
Gawel's blurb
» Avoid CSRF in the Ajax world

Most (good) web frameworks have a way to avoid Cross Site Request Forgery but in the Ajax world it's not so easy.

Here is my solution. First generate a secret key (sha1 of the current date or whatever) store it in user's session and show an hidden field in the requested webpage:

<input type="hidden" id="_req" name="_req" value="your secret key" />

Here is a snippet for Pylons:

def secure_field(with_id=False):
    value = request.environ.get('_req', None)
    if value is None:
        # generate a key if not already done for the request
        value = sha.new('%s-%s' % (datetime.now(), random.random())).hexdigest()
        session['_req'] = value
        request.environ['_req'] = value
        log.info('setting secure key to %r', value)
    if with_id:
        return hidden('_req', id='_req', value=value)
    return hidden('_req', value=value)

Notice that the field can be render multiple time with the same key (but only one with an id because of XHTML). The key is generated per request. This allow to have multiple _req fields in the same page so you can also add it to non-Ajax forms (and secure them too).

Then you need to POST this key on each Ajax request. Here is a wrapper for jQuery's post method:

post: function(url, data, callback, dataType) {
    // wrap $.post to add _req field
    if (!dataType) dataType = 'html';
    if (typeof(data) == typeof('')) {
        // $(form).serialize() return a string
        data += '&_req='+$('#_req').val();
    } else {
        data['_req'] = $('#_req').val();
    }
    $.post(url, data, callback, dataType);
}

The _req key is added to each POST request.

Last thing. You need to check that the key stored in user's session is also in the POST data. Here two Pylons decorators to avoid illegal requests:

@decorator
def secure_post(func, *args, **kwargs):
    """return html"""
    if request.method == 'POST':
        _req = session.get('_req', None)
        if _req is not None and _req == request.POST.get('_req'):
            del request.POST['_req']
            data = func(*args, **kwargs)
            return data
    if request.environ.get('paste.testing') is True:
        return func(*args, **kwargs)
    return _('Forbidden')

@decorator
def secure_json(func, *args, **kwargs):
    """return json"""
    if request.method == 'POST':
        _req = session.get('_req', None)
        if _req is not None and _req == request.POST.get('_req'):
            del request.POST['_req']
            data = func(*args, **kwargs)
    if request.environ.get('paste.testing') is True:
        data = func(*args, **kwargs)
    else:
        data = dict(error=_('Forbidden'))
    response.content_type = 'application/json'
    return json.dumps(data)

That's it. Now you are sure that all Ajax requests came from the user's web page. Cheers.

Of course the key is valid for more than one Ajax request. You may regenerate it for each main html page. May be this can be improved to change the key for each request including Ajax's but... It's already secure. Right ?

Octobre 11, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» FormAlchemy 1.3 status

FormAlchemy 1.3 is released. From the website:

FormAlchemy eliminates boilerplate by autogenerating HTML input fields from a
given model. FormAlchemy will try to figure out what kind of HTML code should
be returned by introspecting the model's properties and generate ready-to-use
HTML code that will fit the developer's application.

Why I choose FormAlchemy instead of another form library ? Everybody knows that explicit is better than implicit. So most of form libraries use a schema to define how widgets are render. FormAlchemy avoid that by using the schema of the data model (SQLAlchemy mappers at the origin). So it generate forms implicitly using your explicit data model.

Another reason is that FormAlchemy is independent. This mean that you can use it in Pylons, Django, repoze.bfg, bobo and (put your favorite framework here).

Thats cool. But we can do more. And that's what we tried to do. SQLAlchemy is not the only library to define data model. So let use the others !

Using couchdbkit

couchdbkit allow to define a schema to store data in CouchDB. CouchDB is a project of the Apache foundation emerging as one of the good modern non-sql database solutions. Let's define a Pet document using couchdbkit:

>>> from formalchemy.ext import couchdb
>>> from couchdbkit import schema
>>> class Pet(couchdb.Document):
...     name = schema.StringProperty()

What about the form ? Here it is:

>>> fs = FieldSet(Pet)
>>> fs.bind(Pet())
>>> print fs.render()

So easy.

Why couchdbkit and not couchdb-python ? Don't know. I guess it's doable with couchdb-python too. The only reason is the same as Jean Sarkosy's potential election at the Epad's presidency. I know Nicolas, HAHA. No. I know Benoît Chesneau (aka benoitc). Benoît release some good stuff related to CouchDB both in python and erlang. Have a look to his bitbucket account.

Using zope.schema

I came from the zope world so I know zope.schema. Most python coders are afraid by the zope word. But they are wrong. At this time we can say that zope is no longer a framework but a set of well tested and well documented libraries. So let's define a small schema:

>>> from zope import interface
>>> from zope import schema

>>> class IPet(interface.Interface):
...     name = schema.TextLine(u'name')

Now we need an object to store values:

>>> class Pet(object):
...     interface.implements(IPet)

Let's use FormAlchemy to render a form for this pet:

>>> from formalchemy.ext.zope import FieldSet
>>> fs = FieldSet(IPet)
>>> fs = fs.bind(Pet())
>>> print fs.render()

That's it. We (at Alterway) use it in a customer project based on repoze.bfg and zope's ZODB as backend. It just work.

Using RDFAlchemy

RDFAlchemy define schemas to describe a RDF node. I know really nothing about RDF but implementing a FormAlchemy extension to support RDFAlchemy was easy so it's now in FormAlchemy. It's tagged as experimental but it work AFAIK.

One of the first implementation I like to have is a FOAF profile editor using FormAlchemy's RESTController (see bellow). I don't know how hard it can be but this can be awesome.

Pylons CRUD interface

I love Pylons. Just because it's simple and have full WSGI support. FormAlchemy have a pylons extensions for a while to generate an admin UI ala Django. This was cool but not perfect. I've added a new module in FA's pylons extension to allow to generate RESTFul CRUD interface based on a data model. Data model mean all models supported by FormAlchemy.

At this time I assume that this work with SQLAlchemy and couchdbkit's models. So I guess it's also usable with RDF stuff. I will try soon.

There is two controllers: RESTController and ModelsController. RESTController render a CRUD interface for a single model. ModelsController render an admin UI for all models found.

This is highly customisable. You just need to change one template.

Have a look at the documentation to read more about that.

fa.jquery

fa.jquery is a standalone package that provide a set of widgets based on jquery.ui. Have a look at the demo page. It also have a plugin registry to allow you to write your own widgets with a few lines of javascript.

You can change the default jquery.ui theme (redmond) and use your own. jquery.ui provide a theme editor

Shabti

Shabti is a set of pylons templates initiated by Graham Higgins. There is now a FormAlchemy template in Shabti to quick initialize a pylons project with FormAlchemy and fa.jquery. The template also initialize a CRUD interface in the admin controller for you.

Here is some screenshots of the admin UI using fa.jquery:

User listing:

http://www.gawel.org/thumbs/blog/shabti_users.png

User edit form:

http://www.gawel.org/thumbs/blog/shabti_user_edit.png

Notice that at this time Shabti require pylons-dev.

An now...

So what future for FormAlchemy ? This is not discussed yet but I like to reduce the amount of code. This mean using external dependencies.

FormAlchemy have a helpers.py module which mostly came from WebHelpers so I guess we can remove it and use WebHelpers instead.

FormAlchemy's validation stuff is simple but FormEncode is popular and powerful so it can be a good thing to use it for validation stuff.

I also like to add some pylons related stuff in fa.jquery to improve the CRUD interface. For example adding some ajax stuff to allow to add new record from a relation widget and maybe inline editing.

Last thing. At this time I'm the only contributor. Alex and Jonathan don't have time to contribute or are involved in other projects. If you like to contribute you can fork the FormAlchemy repository on bitbucket and use the pull request feature to submit patches. (Google code still to be the official repository) But please, FormAlchemy is a well tested library so run the tests before patch submission and be sure that your changes will don't break anything. Well tested patches are always welcome. Thanks !

That's it for now. Hope you're enjoyed it.

Août 25, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» buildout vs pip. Why I choose buildout

pip and buildout are two way to deal with python eggs. Both have the same functionality eg. Install python eggs in a isolated environment.

I've learn buildout first because I came from the zope world and it's the standard way to install zope for a while now. But when pip appears I give it a try.

At the first look pip seems interesting because it allow to install a bundle of packages with fixed versions. But the main problem is that pip use virtualenv to isolate packages so each time you need a new environment you need a new virtualenv and fetching all packages again. This can take a lot of time if you are using lxm or python library with C code. And more and more if you have a lot of projects.

Instead, you can share eggs between buildout's directories. You just need to tell where they are. Add this to your ~/.buildout/default.cfg:

[buildout]
eggs-directory=/home/gawel/eggs

That's all. When buildout need a egg he'll try to find it in this directory before fetching. If no version is found the egg is fetched and installed in this directory. Of course, you can have more than one version per package. You can tell buildout which version to use (see bellow).

I'm a Pylons fan so I already have all packages needed in my ~/eggs directory to install a new pylons environment. Let's create a new project in an isolated environment:

gawel:~/tmp% date
Mar 25 aoû 2009 21:49:11 CEST
gawel:~/tmp% mkdir pylons
gawel:~/tmp% cd pylons
gawel:~/tmp/pylons% vi buildout.cfg
gawel:~/tmp/pylons% buildout
Creating directory '/Users/gawel/tmp/pylons/bin'.
Creating directory '/Users/gawel/tmp/pylons/parts'.
Creating directory '/Users/gawel/tmp/pylons/develop-eggs'.
Installing eggs.
Generated script '/Users/gawel/tmp/pylons/bin/paster'.
Generated script '/Users/gawel/tmp/pylons/bin/sphinx-build'.
Generated script '/Users/gawel/tmp/pylons/bin/sphinx-quickstart'.
Generated script '/Users/gawel/tmp/pylons/bin/sphinx-autogen'.
gawel:~/tmp/pylons% ./bin/paster create -t pylons myproject
Selected and implied templates:
  Pylons#pylons  Pylons application template

Variables:
  egg:      myproject
  package:  myproject
  project:  myproject
Enter template_engine (mako/genshi/jinja2/etc: Template language) ['mako']:
(...)
  Copying templates/default_project/test.ini_tmpl to ./myproject/test.ini
Running /Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python setup.py egg_info
gawel:~/tmp/pylons% date
Mar 25 aoû 2009 21:50:24 CEST

This take 1mn13s. Now try with pip. I'll not ;)

That's the main reason why I choose buildout.

buildout also make unit testing easyer. Just put this config file in your package root:

[buildout]
newest = false
parts = eggs
develop = .

[eggs]
recipe = zc.recipe.egg
eggs =
  YourPackageName
  nose

Run buildout. And you'll be able to run ./bin/nosetests in an isolated environment with your package installed in develop mode (develop = .). I have one in all my projects if you need some examples.

Another reason is that buildout can be extended easily. One feature that exist in pip but not in buildout is the ability to fetch eggs from VCS's urls. This is not a builtin feature in buildout. But I've created a buildout extension for that (gp.vcsdevelop). And you know what ? This extension use pip !! ;) By the way, there is no plugin system in pip AFAIK.

Now the last reason. I wonder how pip's users upgrade an existing project. Do they need to install another environment ? With buildout i'ts easy. There is an extension to list all packages versions used by a buildout project. You just need to use the generated file as a buildout's version.cfg and tell buildout to use it.

[buildout]
versions = versions.cfg
...

Then update this file on your production server. Run bin/buildout again. That's it. Your project is up to date and use the correct versions just because buildout create is own sys.path with required eggs.

gawel:~/tmp/pylons% cat bin/paster
#!/Library/Frameworks/Python.framework/Versions/2.6/Resources/Python.app/Contents/MacOS/Python

import sys
sys.path[0:0] = [
  '/Users/gawel/eggs/Pylons-0.9.7-py2.6.egg',
  '/Users/gawel/eggs/PasteScript-1.7.3-py2.6.egg',
  '/Users/gawel/eggs/setuptools-0.6c9-py2.6.egg',
  (...)
  '/Users/gawel/eggs/WebHelpers-0.6.4-py2.6.egg',
  '/Users/gawel/eggs/Routes-1.10.3-py2.6.egg',
  ]

import paste.script.command

if __name__ == '__main__':
    paste.script.command.run()

For all those reason, I will not use pip for now.

I know that buildout is more complicated than pip. But it's also more powerful. So, are you planning to learn how buildout works ? If so, I've wrote a How To for Pylons. You can also find it on pylonshq. I think this can help you to learn buildout even if you don't plan to use Pylons (but you should too ;).

Août 24, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» How to use jQuery.getJson in a Firefox extension

jQuery work fine in a Firefox extension and it's easy to manipulate the DOM of the current document with it.

If you need to call some web services then it will fail. I've solved the problem and tried to submit a path on the jquery-devel mailing list.

The problem is not solved since it require some too specific Firefox code. So if you need this, check the patch I've submitted and apply the changes in the ajax function by yourself.

Août 4, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» Serving both mercurial repository and sphinx docs at the same time

A few month ago I've started a small project named MercurialApp.

The main goal is to serve my Mercurial repositories as a wsgi application with Paste. It work fine and I use it on https://hg.gawel.org.

What I'm thinking now is that it was cool to serve the project documentations with the same application. I always have a docs/ folder in all my project with the Sphinx documentation in it. So let's use it.

Now you can add a sphinx_docs option in the configuration. Then MercurialApp add a changegroup hook (applied when a push occurs) in all repositories. This hook look for a docs/conf.py in the repository and if it exist try to rebuild the documentation in {sphinx_docs}/html/{reponame}/docs/.

Then if you look at the code in MercurialApp you will see something like this:

self.app = Cascate([StaticURLParser(os.path.join(c.sphinx_docs, 'html')), hgwebdir])

As you can see the newly generated application will try to serve a static file in sphinx_docs and if it does not exist serve the hgwebdir application. This way if you try to fetch /projectname/docs/ you'll see the sphinx documentation. /projectname will serve the hgwebdir application.

Here is a sample config file for MercurialApp:

[server:main]
use = egg:Paste#http
port = 5000

[app:main]
use = egg:MercurialApp

[hg:main]
# this is a public repo served at /
# everybody can read. Only gawel can push
hgwebdir = %(here)s/public_repositories
allow_read = *
allow_push = gawel

[hg:private]
# this is a private repo served at /private
# Only gawel can read and push
hgwebdir = %(here)s/public_repositories
allow_read = gawel
allow_push = gawel

The work still in progress but I like this project because I can browse my code as usual and the docs is always up to date without any wiki and/or WYSIWYG editor.

By the way Sphinx and Mercurial are two great python projects. A bunch of non python projects use them to write documentation or store source code. But what I love is that with python you can combine two great application in one. Python give me the power !

Juillet 23, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» Add an a.pwal.fr command to vimperator

I've writen a cool application to tinyfy urls and post them to social website. The result is http://a.pwal.fr.

there is a Firefox plugin for newbies but I'm not. I'm using the great Firefox plugin Vimperator as a 100% pur geek.

So I've tried to add a command ti Vimperator to quickly post links on Twitter.

It's a bit easy since I'm now familiar with XUL. Here the result. Just add it to your ~/.vimperatorrc:

" add :twit <alias> command
js << EOF
apwalfr = {
    addScript: function(path) {
        var doc = window.content.document;
        var s = doc.createElement('script');
        s.setAttribute('src', 'http://a.pwal.fr'+path);
        s.setAttribute('type', 'text/javascript');
        doc.body.appendChild(s);
    },
    quickTwit: function(alias) {
        var doc = window.content.document;
        var url =  "http://twitter.com/home/?status=$t+$u";
        var path = '/json/?callback=apwal.quickPost&alias='+alias+'&arg='+encodeURIComponent(url)+'&url='+encodeURIComponent(doc.location.href)
        this.addScript('/_static/api.js')
        this.addScript(path)
    }
}
EOF
command! twit -nargs=? :js apwalfr.quickTwit('<args>')

This post is a simple note for myself and can help people to extend Vimperator since the docs is hard to find for now.

Juin 29, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» Facebook connect with python

A few weeks ago I've discover Facebook Connect. So I decide to try to use it in a new project. It look like an easy way for new user to register in my application.

The first step is to follow the Quick start guide. Then whe need to take care of Facebook in our application. There is already a cool python library to play with Facebook called pyfacebook. So you need to install it.

In my project I aleady use repoze.what for authentification. It just rocks. So i decide to write an IIDentifier plugin for Facebook:

import sha
import facebook
from webob import Request
from zope.interface import implements
from repoze.who.interfaces import IIdentifier

DEFAULT_FIELDS = ['uid', 'name', 'first_name', 'birthday', 'relationship_status',
                  'proxied_email', 'sex', 'hometown_location',
                  'pic', 'pic_big', 'pic_small', 'pic_square']

class Params(dict):
    def __getattr__(self, attr):
        return self.get(attr, '')
    def __html__(self):
        return repr(self)

class Facebook(object):
    implements(IIdentifier)

    def __init__(self, api_key, secret_key):
        self.api_key = api_key
        self.secret_key = secret_key

    def identify(self, environ):
        req = Request(environ)

        # initialize the api object
        fbapi = facebook.Facebook(self.api_key, self.secret_key) # init pyfacebook instance
        if fbapi.check_session(req):
            environ['repoze.who.fb'] = fbapi
            user = fbapi.users.getInfo([fbapi.uid], DEFAULT_FIELDS)[0]

            # we used the proxied_email and a generated password to retriave
            # the user from our DB
            user.update(
                login=user['proxied_email'],
                email=user['proxied_email'],
                password=sha.new('%s%s' % (fbapi.uid, self.secret_key)).hexdigest(),
                )

            # we store the facebook data in environ
            environ['repoze.who.fbuser'] = Params(user.items())
            return user

    def remember(self, environ, identity): pass
    def forget(self, environ, identity): pass

At the first time there is no user in our DB. So the user isn't really connected to our interface. The trick is to use a javascript callback to show a registration form to the user with fields pre-filled with Facebook's data.

<html xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://www.facebook.com/2008/fbml">
  <head></head>
  <body>
   <script src="http://static.ak.connect.facebook.com/js/api_lib/v0.4/FeatureLoader.js.php" type="text/javascript">
   </script>
   <fb:login-button onlogin="facebook_onlogin();"></fb:login-button>
   <script type="text/javascript">
     FB.init("YOUR_API_KEY_HERE", "xd_receiver.htm");
     var facebook_onlogin() function() {
         // here is the trick
         window.location.href = '/register/facebook';
     }
   </script>
</body> </html>

When the form is submitted, you need to create the real user in your application. That's all !

Mai 2, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» collective.releaser rocks !

Tarek have made some great works on collective.releaser and collective.dist the last months. There is currently a refactor branch.

The goal of collective.releaser is to normalize the release process of a buildout based project.

I've worked on it the last week to implement the latest needed steps to allow an easy way to release a buildout project.

So here is how it works. collective.releaser came with a paster template to generate a buildout skel. You got a dev.cfg for development with a develop option with all your develop eggs in it and a prod.cfg for production without develop and all versions fixed in the versions section.

I've modified the project_release so you just need to run:

$ bin/project_release --version=1.0

This will release eggs founded in the dev.cfg's develop option on a pypi server and fix the versions in buildout.cfg. If no version is founded for an egg in the versions section of the prod.cfg then the egg is released at the version passed to the command line. If a version is founded then the egg is released at this version. Here is an example:

The dev.cfg:

[buildout]
parts = eggs

develop =
  ../collective.releaser
  ../my.project

[eggs]
recipe = zc.recipe.egg
eggs =
  collective.releaser
  my.project

The prod.cfg:

[buildout]
parts = eggs
versions = versions

[versions]
collective.releaser=0.6.1

[eggs]
recipe = zc.recipe.egg
eggs =
  collective.releaser
  my.project

collective.releaser package is already at 0.6.1 so no release is needed. my.project as no versions specified so a new 1.0 release is generated and uploaded on your pypi server.

The generated buildout.cfg will look like this:

[buildout]
extends = prod.cfg
versions = versions

[versions]
collective.releaser=0.6.1
my.project=1.0

After all eggs are released, project_release will generate a tarball with the tagged version of the project and all eggs founded in the develop option.

Then to upgrade your project you just need to untar the archive on your production server. Launch bin/buildout. And that's it.

This will probably make life easier for developers ! Hope we can make a new self generated release soon :)

Janvier 28, 2009
Gael Pasgrimaud
gawel
Gawel's blurb
» Fetching packages from various vcs in your buildout

I've recently released a buildout extension to checkout packages in a buildout from various vcs.

Here is a buildout config used to install this blog's skin package. This checkout deliverance from svn and pyquery from his bitbucket repository:

[buildout]
newest=false
parts = eggs
index=http://pypi.zopyx.com/
develop=
    .
extensions=gp.vcsdevelop
vcs-extend-develop=
    svn+http://codespeak.net/svn/z3/deliverance/trunk#egg=deliverance
    hg+ssh://bb/olauzanne/pyquery/@tip#egg=pyquery

[eggs]
recipe=zc.recipe.egg
eggs=
    GawelOrgSkin

This extension is less than 100 lines of code. All checkout stuff is handle by pip which support many vcs.

Décembre 16, 2008
Gael Pasgrimaud
gawel
Gawel's blurb
» Skinning with pyquery and deliverance

I'm using deliverance since a few month now to skin this blog and afpy.org. I also contribute to pyquery since i like the idea of manipulating xml in python like with jQuery. So the next step is to use a kind of pyquery rule in deliverance.

After looking at the deliverance code it seems that adding new rules is easy. You just need to register it in the deliverance rules. So here is the result:

from deliverance import rules
from pyquery import PyQuery as pq

class PyQuery(rules.AbstractAction):
    """PyQuery rule for deliverance"""
    name = 'pyquery'
    def __init__(self, source_location, callback=None):
        self.source_location = source_location
        self.callback = callback

    def apply(self, content_doc, theme_doc, resource_fetcher, log):
        """apply the rule"""
        self.callback(pq([content_doc]), pq([theme_doc]), resource_fetcher, log)

    @classmethod
    def from_xml(cls, tag, source_location):
        """Parses and instantiates the class from an element"""
        use = tag.attrib['use']
        modname, funcname = use.split(':')
        mod = __import__(modname, globals(), locals(), [''])
        callback = getattr(mod, funcname)
        return cls(source_location, callback)

# register the new rule
rules._actions['pyquery'] = PyQuery

Now i'm able to use my rule. I just need to import the above module before any deliverance one:

from myproject import pyquery_rule
from deliverance.middleware import DeliveranceMiddleware
# initialize your middleware

Then here is the rules:

<ruleset>
  <theme href="/theme.html" />
  <rule class="default">
      <pyquery use="myproject.rules:default" />
  </rule>
</ruleset>

Where myproject/rules.py look like this:

def default(content, theme, resource_fetcher, log):
    """rule used for testing"""
    content('title').text('My site - '+content('title').text())
    content('#footer').remove()
    theme('body').append(content('body'))

content and theme are pyquery objects. Feel free to do whatever you want with them. Enjoy !!

Notice that this work only with deliverance's trunk (or >=0.3).

Décembre 7, 2008
Gael Pasgrimaud
gawel
Gawel's blurb
» Combine zc.buildout and pip benefits

I've just released a new zc.buildout recipe which allow to install packages with pip.

I am getting really excited about it because it has many advantages taken from both components.

  1. The recipe adds a virtualenv in the parts/ directory of your buildout then use this binary to generate executable python scripts. So you have a clean sandbox.
  2. The recipe is based on zc.recipe.egg#scripts so you can share your eggs between buildouts as usual.
  3. Of course, you can install some .pybundle files.
  4. You can build package from svn with the editables option.
  1. Each line found in the install option is the last part of a pip command. This allow you to build eggs with dependencies. For example to install lxml in a pure sandbox without libxml2 and libxslt installed you need Cython installed and this command line python setup.py install --static-deps to install lxml. This is easy with the recipe. Here is a sample configuration file for this case:
[buildout]
# the cache dir is used by buildout & pip
download-cache = download
parts = eggs

[eggs]
recipe = gp.recipe.pip

# eggs installed by pip (also add the Deliverance bundle)
install =
    Cython
    --install-option=--static-deps lxml==2.2alpha1
    http://deliverance.openplans.org/dist/Deliverance-snapshot-latest.pybundle

# eggs installed by zc.recipe.egg
eggs =
    Paste
    pyquery

That's all !! This works perfectly on my Mac OSX and should works on most system.

In fact zc.recipe.egg will work in most cases but when you need a clean sandbox and some extra options, gp.recipe.pip is a good alternative.

Novembre 18, 2008
Gael Pasgrimaud
gawel
Gawel's blurb
» Get thumbnails the WSGI way

A few month ago I've created a middleware to generate and cache images thumbnails. I need this to evaluate the image size from the browser's window on the client side and then get the thumbnail at an url based on this size.

The result is iw.thumbs. The package provide a highly configurable middleware to serve images thumbnails.

The principle is to map an uri to a file system directory. Then if an uri match the specified regexp, a thumbnail is served by the middleware.

For example, with the default configuration, the url http://localhost/thumbs/100x100/image.png will render a thumbnail of /var/mapped_dir/image.png.

There is also a "size" uri parser to use named size and prevent potential DoS attacks.

That what i'm using here, for my blog. I have two size: blog (490x490) and large (750x750). Here the Paste configuration:

[app:thumbs]
use = egg:iw.thumbs
url_regexp = ^/(?P<size>%s)(?P<path>/.+)
url_parser = iw.thumbs.url:size_parser
image_dir = %(here)s/var/rst/images
cache_dir = %(here)s/data/thumbs
sizes =
    blog = 490x490
    large = 750x750

As you can see, the package also provide an application factory in case of you just want to math a specific prefix.

Then I just have to map it in my url mapper section:

[app:main]
use = egg:Paste#urlmap
/thumbs = thumbs
/ = pylons # my pylons application

Another middleware exist to generate thumbnail with a different approach. repoze.bitblt allow you to generate thumbnail from the size specified in your image tag. It also add a secure part to the image url to prevent DoS attacks. I think both package have their place.

What I like with iw.thumbs is that I only need to change a configuration parameter instead of all image tags if the width of my blog column change in the future. The other good point is that you can generate image tags on the client side wich seems not really possible with repoze.bitblt.

But well, both middleware are another good reason to use WSGI applications.

And as an example, here is Alain, the AFPy mascot render with the blog size. If you click on it, you'll get the large size in a popup.

http://www.gawel.org/thumbs/blog/alain_at_rennes.jpg

Novembre 3, 2008
Gael Pasgrimaud
gawel
Gawel's blurb
» Wonderful world of WSGI

This time it's really winter. When it's cold the only thing i like is geeking at home. So I decide to rewrite my website with a set of WSGI applications.

The first thing done was to initialize a Pylons project for my blog. I got something working after a few hours. Pylons really speed developments.

Then I need a skin. I decide to give a try to Deliverance. Skinning with Deliverance is quite easy. After reading the docs, I've create a layout based on my old html code and a set of rules. Add this to my configuration file:

[filter:deliverance]
use = egg:Deliverance
theme_uri = file:///%(here)s/themes/layout.html
rule_uri = file:///%(here)s/themes/rules.xml

That's it. My Pylons app looks good without any style sheet as you can see ;)

Another part of my website is my projects documentations which are generated by Sphinx and served by a static app. Those pages are now skinned with Deliverance and you can't really know that you browse another application.

I also use Trac to browse my svn repository so I've try to put it in my WSGI stack. Trac now support WSGI except that it do something strange with the output and/or stderr so you'd better not using an error middleware with it else you get a beautiful blank page. You can create a WSGI application with a few lines of code:

# -*- coding: utf-8 -*-
import os
from trac.web.main import dispatch_request
def make_trac(conf, trac_env):
    def application(environ, start_request):
        environ['trac.env_path'] = trac_env
        return dispatch_request(environ, start_request)
    return application

and skin it with Deliverance. Notice that in my case, Trac seems inserting different encoding in the same page so I need to hack it a bit.

The last thing done is to use a ProxyApp to serve and skin my buildbot. Of course, this part is also skinned with Deliverance.

I now have only one process to serve all my python apps so I can use repoze.who to manage authentication for all apps. I have a ldap server, so I've write a small plugin for it. This work perfect, and I'm now able to log in Pylons and Trac.

I'm using Deliverance as a WSGI middleware and it seems a bit different than when you use it as a proxy. I encounter a few bugs. I have to only use XPath to find nodes in the DOM. Other expressions just don't work. You also can't remove nodes. But, well, this seems promising.