By Ingeniweb. A Django site.
Septembre 24, 2008
» Annoucing collective.eggproxy, the smart PyPI mirror


I just wanted to announce the release of collective.eggproxy (http://pypi.python.org/pypi/collective.eggproxy/0.2.0)

collective.eggproxy is a smart mirror for PyPI, thaught and coded by my colleague Bertrand Mathieu.

It will collect packages on PyPI only when a program like easy_install or zc.buildout asks for it. In other words, unlike some mirrors that act like rsync and get the whole PyPI base (more than 5 gigas) collective.eggproxy will only get what you need.

At first run collective.eggproxy downloads pypi index and builds a page of links. When a software asks for a specific package, version, etc. collective.eggproxy downloads it if needed and stores it locally.

Want to give a try ? try it in two lines with easy_install:

easy_install collective.eggproxy
eggproxy_run

That’s it !

      

Septembre 22, 2008
Bertrand Mathieu
bertrand
Paste here is about »
» iw.eggproxy 0.2.0

We have released iw.eggproxy 0.2.0. Bugs fixed:

  • package index/download files: skip modules installed in local system (resulted
    in copying a directory instead of downloading a file)
  • update script crashed with invalid/obsolete package name
  • get eggs distributions for all versions/platforms, instead of system ones
  • malformed tag in generated indexes
We are going to rename it to collective.eggproxy. The next release will provide a standalone server and (hopefully) a WSGI application. This should make it easier to start with it.

A sprint occured this summer in the topic, a few months after iw.eggproxy was released, and some people chose to create a full mirorring tool called z3c.pypimirror.
Here are some differences with z3c.pypimirror:
  • eggproxy relies on setuptools. It sees what "easy_install" can see: no more, no less. Also it does not include its own machinery to read pypi indexes and follow links.
  • eggproxy provides on-demand any egg provided at pypi. You don't need to know in advance what packages you will need, you don't need to download them before: just ask for them as if you were directly on pypi server
  • OTOH if the server has network problems to reach pypi eggproxy may not be able to serve an egg, where z3.pypimirror would have already downloaded it. This case happens only if eggproxy has not already served once the egg.
  • z3c.pypimirror creates a static directory layout suitable for any HTTP server; because of its nature eggproxy needs apache and mod_python (next release should change this strict requirement by providing a standalone service and/or WSGI).

Août 12, 2008
» iw.eggproxy, a smart PyPI mirror


Mirroring PyPI becomes a recurrent need for Zope development, because zc.buildout makes a lot of package downloads to build one application.

This is useful when you are working in an intranet with limited web access or when you want to speed up download times. It also makes things safer: if PyPI is down and if developers computers don’t have caches, having a mirror will save your day.

While PyPI has proven its robustness (it is 100% up for months now as far as I can see), having mirrors makes a lot of sense.

We have created a small mirror application here at Ingeniweb, that we use for our buildouts needs. This work was thaught and created by my colleague Bertrand Mathieu.

It is a smart proxy that will download packages at PyPI everytime they have been asked by a buildout or an easy_install client. When the package is downloaded, it is kept in the proxy side for any new requests. This means that after a while, the proxy has its own collection of packages that corresponds to the real needs and will not query PyPI anymore.

This approach avoids having to download and synchronize PyPI with crons, which is a heavy process since PyPI weight several gigas. The caveat of course, is that it won’t be able to get a new package if PyPI is down.

Take a look ! http://pypi.python.org/pypi/iw.eggproxy

By the way there is an interesting sprint coming up on all these topics, in Germany :

http://www.openplans.org/projects/black-forest-sprint/project-home

Mai 12, 2008
» The new .pypirc format in Python / distutils


Thanks to the sprinters in Washington D.C., (Andrew who worked on merging it, Alex, etc.) and people that helped in the discussion (Fred, Martin), my patch to Python for multiple servers handling in .pypirc file has been integrated in 2.6 trunk.

This patch will allow using distutils register and upload commands with several servers, using the -r option and storing for each one of them the appropriate username/password in the configuration file. See http://wiki.python.org/moin/EnhancedPyPI for more details.

The collective.dist package provides the same feature for Python 2.4 and 2.5, through new commands: mregister and mupload.

The new format is also pretty convenient to store several profiles per servers. For instance, if you have several accounts on PyPI, one for your company (acme) and one for your own usage, you can define them like this:

[distutils]
index-servers =
    pypi
    acme

[pypi]
username:user
password:password

[acme]
username:acme_user
password:password

Then use them depending on the package you work in:

$ python setup.py register -r acme
$ python setup.py register    # default, which is pypi

When plone.org will go Plone 3 (this is happening soon), the pypirc file can be extended with this new target:

[distutils]
index-servers =
    pypi
    acme
    plone

[pypi]
username:user
password:password
[acme]
username:acme_user
password:password
[plone]
repository:http://plone.org/products
username:plone_user
password:password

From there, setuptools aliases can be used in each package to simplify things:

$ python setup.py alias plone_org register -r plone sdist bdist_egg upload -r plone

Which will allow you to upload the package to the website in one simple command:

$ python setup.py plone_org

The patch aslo corrects a few minor bugs in distutils, such as:

  • .pypirc was not found on Windows
  • pydistutils.cfg was not found on Windows