Modularising your code deployment

13Apr09

Most software projects start with a nice, clean, compartmentalised architecture, whether real or imagined. As implementation progresses, the lines between components tend to blur as unforeseen dependencies emerge and edge cases are dealt with.

However, by the time it comes to deployment, you’ll probably still have a number of separate packages, with some (hopefully acyclic) dependency graph binding them together.

At WebMynd, the web tier runs on Turbogears, a Python web framework. Turbogears is by its nature very modular, with various options for “plugging in” alternative tools and extensions, which has led us to be quite modular with our own code.

Dependencies between these packages is managed via the install_requires setuptools parameters, e.g.:

    install_requires=[
        "TurboGears",
        "SQLAlchemy",
        "MiniMock >= 1.2.2",
        "Boto >= 1.5",
        "Sphinx",
        "WMQueueLib",
        "WMModel",
    ],

Here, the “WM…” packages are internal, and we don’t really want to share them on PyPI. So how best to get them installed onto the machines where they’re required?

One option is to grab the source code directly, build and install it into place. Even if your code is in a DVCS, this process can complex, and you’re going to have to store somewhere the URLs and/or levels that each package depends on from the others. But this information is already encoded in a much more concise and flexible way: the install_requires declarations!

We’ve found it convenient to take advantage of this version-controlled dependency graph by hosting our own little package index internally. It’s nice and easy: all that’s required is some easy_install configuration like this:

[easy_install]
find_links = http://internal_server.webmynd.com/packages/

Our internal_server is only accessible from a restricted set of IPs, but you could use other security measures – I’ve just tried basic HTTP authentication and it works: just prepend username:password@ to the domain.

There are a few places you can put this configuration, but we include it in setup.cfg in all our packages, so that install dependencies just take care of themselves, with no hassle and no changes required on the machines. Installing a package is as simple as:

easy_install WMWebTier

Rather than making sure that the right source is pulled down on the right machine at the right time, now you can safely push all your good builds up onto your internal package index and trust that the client selects the right one. You’ve already encoded dependencies in your package metadata – relax and let easy_install do the hard work for you!

Advertisements


4 Responses to “Modularising your code deployment”

  1. Small but important typo here: install_requires is not (yet) a Distutils option, but a setuptools one.

    Cheers
    Tarek

  2. Is it true that WebMynd now supports Bing search? I saw it mentioned on an on10.net post, but this site doesn’t indicate that.

    If WebMynd does, that’d be really neat.

    • 4 WebMynd Team

      The WebMynd extension for OneRiot does support Bing search, you can download it here.

      So with that installed, when you go to Bing to do a search you will get the WebMynd widgets on the right-hand side.

      This will be supported in our main extension when we launch the next version in a couple of weeks’ time.



%d bloggers like this: