I have a number of plone sites on a server, generally one site is associated with one zope instance. I am using buildout and iw.recipe.squid to add easily a new virtualhost with squid proxying. Im my case the server is running Ubuntu server 8.04, but this should apply without change to a Debian server. This may look overkill for juste one or two sites, but in my case I may have up to 30 different sites (of course all zope instances are not on localhost!).

Let's suppose I have these sites:

• www.site1.net, zope instance on localhost (127.0.01) port 8080, plone path is located at /site1/portal
  
• www.someothersite.com, zope instance on 10.2.0.5 port 9080, plone path /somepath/portal

buildout.cfg:

[buildout]
parts = squid
versions = versions

[versions]
iw.recipe.squid = 0.9

[squid]
recipe = iw.recipe.squid
squid_owner = proxy
squid_visible_hostname = myservername
squid_cache_dir = /var/cache/squid
squid_log_dir = /var/log/squid

squid_accelerated_hosts =
 www.site1.net: 127.0.0.1:8080/site1/portal
 www.someothersite.com: 10.2.0.5:9080/somepath/portal

After running buildout for the first time:

• make a symbolic link "/etc/squid.conf" pointing to parts/squid/etc/squid.conf
• run "bin/squidctl createswap" if required
• check that squid starts normally and that helper processes are running, too (iRedirector.py, squidAcl.py, squidRewriteRules.py)
  

After having added one or more site:

• in /etc/apache2/sites-available create a symbolic link for all config files located in parts/squid/apache. In our case they should be named "vhost_www.site1.net_80.conf", etc
• run a2ensite to active sites ("a2ensite vhost_www.site1.net_80.conf")
• reload apache
  

Adding a new site will be just a matter of adding a new line in squid_accelerated_hosts, running buildout, making the symbolic links and reloading apache.

As of iw.recipe.squid 0.9, squid and apache log reside in the same directory, but the next release should allow to have different directory; it should also allow you to set "combined" rather than "common" log format for apache.

Of course do not forget to configure properly CacheSetup on every plone site.

We have released iw.eggproxy 0.2.0. Bugs fixed:

• package index/download files: skip modules installed in local system (resulted
  in copying a directory instead of downloading a file)
• update script crashed with invalid/obsolete package name
• get eggs distributions for all versions/platforms, instead of system ones
  
• malformed tag in generated indexes

We are going to rename it to collective.eggproxy. The next release will provide a standalone server and (hopefully) a WSGI application. This should make it easier to start with it.

A sprint occured this summer in the topic, a few months after iw.eggproxy was released, and some people chose to create a full mirorring tool called z3c.pypimirror.
Here are some differences with z3c.pypimirror:

• eggproxy relies on setuptools. It sees what "easy_install" can see: no more, no less. Also it does not include its own machinery to read pypi indexes and follow links.
  
• eggproxy provides on-demand any egg provided at pypi. You don't need to know in advance what packages you will need, you don't need to download them before: just ask for them as if you were directly on pypi server
• OTOH if the server has network problems to reach pypi eggproxy may not be able to serve an egg, where z3.pypimirror would have already downloaded it. This case happens only if eggproxy has not already served once the egg.
• z3c.pypimirror creates a static directory layout suitable for any HTTP server; because of its nature eggproxy needs apache and mod_python (next release should change this strict requirement by providing a standalone service and/or WSGI).

Doing development with Plone 3.1, I encountered a KeyError when I tried to use Five.testbrowser.Browser:

Browser()
KeyError: '_seek'

I eventually found that my system (currently Ubuntu 8.04) had the package "python-mechanize" 0.1.7b, which masks the one provided by Zope 2.10 (0.1.2b). Unfortunatly there has been API changes between 0.1.2b and 0.1.7b... (see classes UserAgent/UserAgentBase)

The solution is to tell buildout to get mechanize egg 0.1.2b: this way it will mask the system library.

[buildout]
versions = versions
eggs=
   ...
   mechanize

[versions]
mechanize = 0.1.2b

Et voila!

In my previous post I described how to get a buildout part to get PloneLDAP with an up-to-date LDAPUserFolder. Things are even simpler now since all involved products have been released as eggs. So just list theses in your buildout along with other eggs:

eggs =
...
Products.LDAPUserFolder
Products.LDAPMultiPlugins
Products.PloneLDAP

Second important thing: LDAPUserFolder 2.9 has been released. If you used 2.9-beta you must upgrade, since an important bug related to the negative cache has been fixed. PloneLDAP documents usage with LDAPUserFolder 2.8, but 2.9 is definitely worth it: The negative cache feature avoids doing too many ldap request, especially useless/unsuccessful ones! Many thanks to Jens Vagelpohl (http://www.dataflake.org/software/ldapuserfolder).

Today we have released iw.eggproxy. It's a module for apache mod_python. Its purpose is to serve as a pypi proxy.

The first motivation to make it was because we have had to work in a private network with a very very slow internet access: for example updating a plone buildout could take more than one hour (just checking eggs freshness) when it should be no more than a few minutes. This condition also prevent to run some kind of rsync against pypi. So the obvious solution was to proxy the eggs we need, on-demand.

The module is installed as a handler on a Location. When accessing this location, eggproxy will serve an index similar to pypi simple view.

This is done like this:

• we already have the information in index.html: just let apache serve the file
• or, we use setuptools to fetch index information, build index.html, and let apache serve the file
  

This allows eggproxy to serve all available eggs from pypi, without actually having to download the whole pypi + content.

Then easy_install can see a package is available on our server, and tries to fetch information on available eggs:

• the subdirectory and its index.html already exists: just let apache serve the file
• or, we use setuptools again to get package information, make the subdirectory (package name) and build index.html
  

Finally, when trying to fetch an egg we do the same:

• the file is already present and is served by apache
• or we use setuptools to get the file on the server
  

iw.eggproxy also provides an update script: "eggproxy_update". This script refreshes the main index and all proxied eggs, if their index.html is older than the interval specified in the configuration file (24h by default).

We have installed it here: http://release.ingeniweb.com/pypi.python.org-mirror

Known bugs: some packages on pypi don't have eggs, then eggproxy does not respond. This is the case with "reportlab" for example.

Enhancements:

• indexes aggregation. At ingeniweb we plan to install it on local server and agregate pypi and some private eggs indexes.
• Standalone/pluggable server. Currently we are bound to apache + mod_python, which may not suit to anyone.
  

UPDATE: all products are available as eggs. See this post.

PloneLDAP is a great product, but its bundle ships a not so recent version of LDAPUserFolder. LDAPUserFolder and LDAPMultiPlugins download urls are both ending with "/download". That seems to confuse plone.recipe.distros, because it installs just one of them. Furthermore, at plone.org PloneLDAP single product (as opposed to PloneLDAP bundle) download url ends with... a space! (%20).
So we have uploaded theses tarballs on release.ingeniweb.com. Here is a working part for buildout:

[ldap_products]
recipe = plone.recipe.distros
urls =
http://release.ingeniweb.com/third-party-dist/LDAPUserFolder-2.9-beta.tgz
http://release.ingeniweb.com/third-party-dist/LDAPMultiPlugins-1.5.tgz
http://release.ingeniweb.com/third-party-dist/PloneLDAP-1.0.tar.gz

zc.buildout est un utilitaire créer par Jim Fulton, le papa de Zope. L'étendue des capacités de zc.buildout est énorme et mériterais un livre entier.

Cependant, voici un petit exemple pour se monter un environnement de paquets/binaire python avec un simple petit fichier de configuration.

Tout d'abord, nous allons nous créer un virtualenv afin de ne vraiment pas polluer notre distribution:

gawel@Stacy:~$ virtualenv --no-site-packages py
New python executable in py/bin/python
Installing setuptools.............done.

Si vous n'avez pas virtualenv, installez setuptools et virtualenv sur votre système. D'après moi, ce sont les deux seuls packages qui mérite d'être installé globalement, pour tout le système.

On va ensuite installer zc.buildout dans notre environnement:

gawel@Stacy:~$ cd py
gawel@Stacy:~/py$ ./bin/easy_install zc.buildout
Searching for zc.buildout
...
Finished processing dependencies for zc.buildout

Parfait, maintenant, initialisons l'environnement de zc.buildout:

gawel@Stacy:~/py$ ./bin/buildout init
Creating '/Users/gawel/py/buildout.cfg'.
Creating directory '/Users/gawel/py/parts'.
Creating directory '/Users/gawel/py/develop-eggs'.
Generated script '/Users/gawel/py/bin/buildout'.

Puis on édite le fichier buildout.cfg pour qu'il ressemble à ça:

gawel@Stacy:~/py$ cat buildout.cfg
[buildout]
parts = eggs

[eggs]
recipe = zc.recipe.egg
eggs=
    ipython
    i18ndude
    ZopeSkel
    IngeniSkel
    iw.releaser

Vous pouvez mettre absolument n'importe quel egg dans l'option eggs. La seule contrainte est que ceux-ci se trouvent sur pypi.

On lance la construction du buildout:

gawel@Stacy:~/py$ ./bin/buildout
Installing eggs.
...
Generated script '/Users/gawel/py/bin/ipython'.
Generated script '/Users/gawel/py/bin/pycolor'.
Generated script '/Users/gawel/py/bin/i18ndude'.
Generated script '/Users/gawel/py/bin/project_deploy'.
Generated script '/Users/gawel/py/bin/project_release'.
Generated script '/Users/gawel/py/bin/project_diff'.

J'ai maintenant tout mes binaires dans le dossier bin/ de l'environnement:

gawel@Stacy:~/py$ ls bin/
activate          easy_install-2.4* project_deploy*   pycolor*
buildout*         i18ndude*         project_diff*     python*
easy_install*     ipython*          project_release*  python2.4@

Le must, c'est que je peux importer chacun de ses paquets dans mon ipython:

gawel@Stacy:~/py$ ./bin/ipython
In [1]: import i18ndude

In [2]: dir i18ndude
------> dir(i18ndude)
Out[2]: ['__builtins__', '__doc__', '__file__', '__name__', '__path__']

Vous pouvez bien sur ajouter des eggs dans votre buildout.cfg puis relancer ./bin/buildout pour qu'il soit prit en compte.

Ceci est une simple introduction à zc.buildout. On peut faire bien mieux, je sais. Mais ça deviendrais compliquer de tout expliquer dans un simple post :)

Simplement, Je trouve ça tellement pratique pour tester rapidement un package que je voulais en faire bénéficier tous ceux qui ne connaissent même pas l'existence de zc.buildout.