Mirroring PyPI becomes a recurrent need for Zope development, because zc.buildout makes a lot of package downloads to build one application.

This is useful when you are working in an intranet with limited web access or when you want to speed up download times. It also makes things safer: if PyPI is down and if developers computers don’t have caches, having a mirror will save your day.

While PyPI has proven its robustness (it is 100% up for months now as far as I can see), having mirrors makes a lot of sense.

We have created a small mirror application here at Ingeniweb, that we use for our buildouts needs. This work was thaught and created by my colleague Bertrand Mathieu.

It is a smart proxy that will download packages at PyPI everytime they have been asked by a buildout or an easy_install client. When the package is downloaded, it is kept in the proxy side for any new requests. This means that after a while, the proxy has its own collection of packages that corresponds to the real needs and will not query PyPI anymore.

This approach avoids having to download and synchronize PyPI with crons, which is a heavy process since PyPI weight several gigas. The caveat of course, is that it won’t be able to get a new package if PyPI is down.

Take a look ! http://pypi.python.org/pypi/iw.eggproxy

By the way there is an interesting sprint coming up on all these topics, in Germany :

http://www.openplans.org/projects/black-forest-sprint/project-home

The Packt team added a page about my new book, which will be out sometimes in September, so I guess it is the right time for me to announce it here !

So here comes “Expert Python Programming”, where I explain how we work with Python every day to create software.

This is my first book in English, and writing in another language was quite challenging

Anyway, this book is intended for developers that already have a background in Python and covers only advanced topics (see the editor details). But as I said, it explains how we develop our applications in Python so topics like continuous integration, documentation, testing, releasing, refactoring, etc. are covered.
Managers will also have a good overview of how a Python project can be run and managed, using modern tools like Distributed Version Control Systems (Mercurial for instance) or Buildbot.

Even if I am working on Zope and Plone these days, I have focused on writing a book that is only about Python, to make it useful to any developer. The fact that my friend Shannon did the technical reviewing helped on this : he doesn’t use Zope too much so he was close to the target readership.

I will take more time when it comes out to announce it in the various mailing lists, and to get into greater details about the content that will be available online besides the book (some Python packages, etc)

I can’t wait to see it out and hold it

Covering OSCON when you are a speaker is quite hard. Until my talk was done, I was more in a mood of reviewing my slides and not really thinking about other talks or blogging.

Convention center from the Bridge (the two twin towers)

I will try to write a few reports now that everything is over. This first report is about Portland, the city and is not technical at all.

This is the first non-technical entry in my blog, but let’s this it for once. So if you are looking for technical content, skip this entry

Portland is an amazing place. The weather was nice throughout the whole week: sunny and not too hot. Some evenings were a bit chilly but that felt nice to me. The convention center where OSCON took place was located on the other side of the Willamette river, in front of the city center. It is a bit far from the center, but finding places were you can put so many people must be hard I guess.

The top six things I have noticed about Portland:

Local beers are great. You can find many places where they brew their own beers, that are comparable to Belgian beers in quality and taste, like the Lucky Lab (Where Jon, a member of the local Plone user group took us, thanks!).  Some place have samples so you can taste several kinds of beers. I also went to the Oregon Brewers Festival right after OSCON was over, where you could taste beers from many places.

Oregon Brewers Festival

Parking a car is a pain. I think the best way to travel in Portland is a mix of Tram, Bus and Bike. You can put your bike on the nose of the buses or inside the trams. Smart thing. This would be useful in Paris. We ended up parking our car far from the center and used the tram (called Max, and free in the center). Portland should set up a bike rental system like in Paris (Velib’), I am sure this would rock.

Organics: people here seem to be really concerned about sustainability and organics. I’ve been told there are a lot of organics farm around Portland. Supermarkets have a nice amount of organics stuff as well. This is nice.

Starbucks Coffee owns the streets: this is scary. there’s a Starbucks almost on each block here. Anyway, local coffee shops have better coffees and they provide nice places to chill out, read a book or talk with people. I think this is where you can feel the real Portland way of living.

The food is good : I have to admit, when we, french people, travel, we feel a bit superior to some countries on food matters. We tend to show off about it :). But Portland has great places to eat. If you like donuts, the place you must go to is called Voodoo Doughnut. They are creative !

Voodoo Doughnut

Big open spaces: I went to Mount Hood where people are skying in… July. I wanted to visit this place because it is where Stanley Kubrick did the outside shots of the famous motel in The Shining. I also had a walk at the Hood river sandbar, where people do kite surfing. One tip: don’t walk there with shorts on, the wind is so strong that you get slapped by the sand. So if you like big open spaces and sports, Portland is the place to be: all those place are one or two hours drive from the city center.

Mt Hood

End of the aparté ! The next entry will focus on OSCON

After a few attempts to make my screencast look nice under Google Video, I decided I would just upload the original ones (.mov and .m4v files) and a PDF export of my presentation.

So everything is available here: http://ziade.org/oscon

Slides: http://ziade.org/oscon/oscon.pdf

• Screencast #1, playing with distutils: http://ziade.org/oscon/demo1.m4v
• Screencast #2, playing with setuptools: http://ziade.org/oscon/demo2.m4v
• Screencast #3, playing with zc.buildout: http://ziade.org/oscon/demo3.mov
• Screencast #4, creating a recipe: http://ziade.org/oscon/demo4.mov
• Screencast #5, creating a plone buildout and adding a develop package in it: http://ziade.org/oscon/demo5.mov
• Screencast #6, collective.buildbot demo (no coding, just showing buildbot.ingeniweb.com): http://ziade.org/oscon/demo6.mov

The .mov files are streamed automatically in your browser I believe, not the m4v ones.

Thanks to Jim Fulton for the quick feedback.

I am leaving tomorrow, heading to Portland, OR, to OSCON. My talk on zc.buildout and Plone will be thurdsay the 24th, and I’ll be there the whole week.

If you are going there and want to meet leave me a note, I am looking forward to meet other geeks there.

When you have to work with directories and files, there’s a very common pattern: copying a directory into another, but filtering out a few files and directories.

For instance, if you want to copy a directory that contains source code, you will probably remove .pyc files and .svn directories if you work with Subversion. In that case, shutil.copytree cannot be used, so os.walk is the usual way to go (rough example):

import os
from os.path import join, splitext, split, exists
from shutil import copyfile

def copy_directory(source, target):
    if not os.path.exists(target):
        os.mkdir(target)
    for root, dirs, files in os.walk(source):
        if '.svn' in dirs:
            dirs.remove('.svn')  # don't visit .svn directories           
        for file in files:
            if splitext(file)[-1] in ('.pyc', '.pyo', '.fs'):
                continue
            from_ = join(root, file)           
            to_ = from_.replace(source, target, 1)
            to_directory = split(to_)[0]
            if not exists(to_directory):
                os.makedirs(to_directory)
            copyfile(from_, to_)

This is a lot of boiler-plate code, so I usually create a small function that accepts more arguments to filter out files and directory. But Python should provide this pattern in the standard library.

I have proposed a patch for shutil.copytree, to integrate filtering capability. It has been reviewed and commited in the trunk this week, so we will have it in Python 2.6.  Now copytree comes with an ignore argument that has to be a callable. If given it will be called on each visited directory to decide what is copied and what is not.

There’s a default callable in shutil called ignore_patterns, that can be used to filter out files with glob-style patterns. I have added this example to Python doc:

from shutil import copytree, ignore_patterns
copytree(source, destination, ignore=ignore_patterns('*.pyc', 'tmp*'))

Pretty straight forward !
More info and examples here : http://docs.python.org/dev/library/shutil.html#shutil.copytree

Plone.org migration to Plone 3 is taking a bit longer than expected, but it should turn into reality soon.

There will be many improvements on the set of packages the website uses (I am thinking in particular about Maurits’s work on POI that will speed up the trackers, but it is just an example), and blobs should be used for the products section (more than 700 projects are registered there).

I worked last week-end on the products section, by finishing collective.psc.mirroring, which will copy all packages that are uploaded at plone.org into a file system directory. This directory will be published directly by Apache so the website will become a new package location for zc.buildout (find-links section) and easy_install calls without invoking the Plone instance.

Now I am focusing on PloneSoftwareCenter (PSC) migration. It is a pretty interesting topic: for every project located in the products folder with releases, I am going to extract its “distutils ids”. These ids are the name set in the setup.py file for each release.

I will then look at PyPI through XML-RPC if the package is also released, using the id. In that case, and if the author email is the same on both side, I will validate that the project on plone.org “owns” the given distutils id. From there PSC will act like PyPI and will reject uploads of packages if the user is not the owner of the project that owns the package id.

Of course there will be errors and some people might feel like their package has been hijacked if they cannot upload their packages. But this should be minor and should be OK after a while. A mail will soon be send to the community to ask people to check that they are synced between PyPI and plone.org.

I am really excited about this work because plone.org will then be compatible with distutils register and upload commands, which means that people will be able to update plone.org products section like they do with PyPI : through a single commande line.

Hey Sidnei, what you thought of several years ago is about to turn true.

Edit: My apologies goes to Maurits, who did the work on POI, not Reinout, his brother