Use buildout.dumppickedversions version 0.5 or later:

If you're using buildout.dumppickedversions, use version 0.5. This release includes a fix I contributed which yields a 3-4 fold decrease in runtime if buildout is run with just the -N option and no -v options. If your're using buildout-versions instead of buildout.dumppickedversions, unfortunately it suffers from the same problem. The author knows about this and may address it but until then you may want to switch to buildout.dumppickedversions.

Use zc.buildout version 1.5 or later:

This seems to yield a 2-3 fold decrease in buildout runtime when run with -N and addresses a similar hot loop as found in buildout.dumppickedversions. In fact, it's resolution of required distributions seems to be much more efficient in general. If you're stuck with 1.4, I've contributed the same fix as for buildout.dumppickedversions to the 1.4 branch so it should make it into a next release if someone makes a new 1.4 release.

Use the -N option:

With the above newest releases of zc.buildout and buildout.dumppickedversions, you can now exercise a lot of control of your run times depending on what command-line options you use. In particular, make sure newest is false and that verbosity is not increased. Both of these can be influenced by options in the buildout configuration, so keep an eye out for that, but if no such options are interfering then this means just invoking buildout like $ bin/buildout -N -c buildout.cfg. IOW, use one -N option and no -v options. Before the improvements in zc.buildout 1.5 and the fix to buildout.dumppickedversions, a lot of time was wasted doing work that would only actually be used if -v had been given.

Clean out your egg cache and use virtualenv --no-site-packages:

Some of the remaining inefficiencies in zc.buildout are proportional to the number of distributions available on sys.path. IOW if you reduce the number of distributions to be scanned, you can increase performance. If you're using a shared eggs-directory or have a buildout that has been around for a long time, there may be a lot of different versions of the same packages in eggs-directory. By cleaning those out you reduce the number of distributions that buildout needs to scan. I recommend just emptying your eggs-directory and then letting buildout re-download all the eggs it actually needs now. I takes a while once, but then it's done. Similarly, using virtualenv --no-site-packages can reduce the number of dists buildout needs to scan.

Help me get zc.buildout/branches/env-cache merged and released:

Having said the above about cleaning out the egg cache, buildout, distribute, and setuptools shouldn't be scanning these paths multiple time anyways. Along with package indexes, these paths are global in nature and so the dists found there should only be scanned for once and cached globally. I've started this work on the env-cache branch of zc.buildout. In my timing tests, this branch seems to yield another ~60% decrease in run time. The zc.buildout tests, however, are a PITA and Jim Fulton is a very busy man with little time to evaluate this branch. If anyone else can help me shepherd this branch through to merge and then release, that would be great. I've written up how to use this branch with your existing buildouts.

With all this in effect I took a production buildout and another development buildout from 2 minute run times down to 20 second run times. This really makes re-running buildout something I no longer feel an urge to avoid. So use zc.buildout 1.5.2, buildout.dumppickedversions 0.5, -N and do what you can to help get the env-cache branch merged.

We in Plone land have been long frustrated about how easily a poorly behaved add-on could break a Plone site. Simply installing the add-on and removing it can leave your site inaccessible, even the ZMI! Often times add-ons that provide only minimal additional object behavior can break the entire object when removed.

Now this isn't actually the ZODB's fault at all. The ZODB actually has quite robust handling of broken object and it works quite well. I'm afraid that fault here lies in zope.interface and zope.component. They're simply both too central to almost all ZODB applications to fail to provide graceful handling of broken objects. So I've taken advantage of the sprints following the Plone Conference here in San Francisco to work on this.

The experimental.broken package provides patches for zope.interface and zope.component to make the services they provide more tolerant of broken objects in the ZODB and allows them to function as if the interfaces and components aren't there when the add-on has been removed. Even better, if the add-on is restored, the full behavior of the interfaces and components is restored.

That said, this is all highly experimental so don't use it. If you do use it, never use it in production and only use it on a copy of your ZODB. And if you do use it, definitely report your experience. If you do have a broken site I'd love to hear how it works for you. Just add experimental.broken to your instance eggs.

If this approach tests well and the core Zope developers think it's the right approach I'll be merging this into zope.interface and zope.component.

[build_ext]
library_dirs = /usr/lib:/usr/lib/i386-linux-gnu:/usr/local/lib

Consulting/Contract Opportunities

I'm perfectly happy to keep working independently or as a sub-contractor with a larger consultancy on Plone projects. I know Plone very well and am very happy to continue working with it on projects such as:

• Intranet workflow/business-process applications
• Custom web publishing CMS projects
• Quickly solving critical problems as a Plone "hired gun"
• Building a project while also cross-training internal developers
• Social justice related projects
• Etc., pretty much any Plone project

Opportunities with a Team

I'm eager to join a team, either as an employee or as a contractor, that can offer some of the following:

• More collaboration as a part of a team that genuinely like each other
• More technological diversity outside of Plone/Zope
• Opportunity to work on social justice related projects

I ran across this video on a "Planet ..." feed of some sort recently describing what I've been looking for. This guy also has a lot of other good git-svn thoughts. I'm just not sure it's worth it. I guess it all depends on how quickly people migrate to github or some such natively. If there's any shared interest, leave a comment or email me. If there's interest I'll start a thread somewhere.

At any rate, thanks to a Planet Emacsen blog post I decided to give it a try. It's a clear win for me. Better group manaagement, key binding system, and plays nicely with the Gnome/Unity-2D panel when I need it. It's all around more modern and less of a fight with modern desktops like Gnome/Unity-2D. I don't think I'll be going back as long as the instability everyone mentions is manageable.

But wait, hidden in a corner of mr.developer, and totally undocumented is a mr.developer.gitsvn module that does exactly that! You can use a "gitsvn" checkout type and you're in business:

[sources]
...
collective.formcriteria = gitsvn https://svn.plone.org/svn/collective/collective.formcriteria/trunk
...

I was going to start modifying all my local.cfg buildout files to change all the "svn" checkouts to "gitsvn" checkouts when I noticed something about how mr.developer registers checkout types. At the very bottom of mr/developer/gitsvn.py I changed:

common.workingcopytypes['gitsvn'] = GitSVNWorkingCopy

to:

common.workingcopytypes['gitsvn'] = GitSVNWorkingCopy
common.workingcopytypes['svn'] = GitSVNWorkingCopy

Viola! Now mr.developer uses gitify for all svn checkouts. Now I'm getting close to being able to pretend svn doesn't exist.

I would love to see mr.developer use a global switch so I don't have to modify the egg like this, maybe something in ~/.buildout/default.cfg. If there's interest in that, I'd be happy to contribute such a thing. I should also mention that gitify is often pretty rough and leaves svn and git in an inconsistent state which is only really repairable by someone who is very familiar with both svn and git. The next time I run into that, I'll be trying to run it to ground and contribute fixes.

I've known about zope.testing/zope.testrunner's "--profile=cProfile" command line option for sometime. When you use it, it spits out the "top 50 stats, sorted by cumulative time and number of calls". Under a featureful application like Plone, however, the top 50 stats are usually consumed with the top publisher and template rendering calls and don't actually tell you anything useful.

Recently I needed to do some profiling of the baseline performance of an application. Since I use functional zope.testbrowser tests a lot (some would say too much), my functional tests are actually a pretty good representation of the real world application usage. So I was thrilled to find that the testrunner actually leaves the raw profiling data behind after a test run! It leaves the data in the working directory in which the test run was invoked. If you're doing this in buildout using a zc.recipe.testrunner part named "test", as is the convention, the file will be in parts/test/tests_profile.??????.prof.

In order to avoid the test fixture setup from polluting the profiling data, I use the "--repeat" command line option for the test runner to make sure that profiling data from the actual tests are much louder in the profiling data than the test setup. For a large functional test suite I tend to use "--repeat=3" since the suite is long enough as it is and any more repetitions just take too long:

$ bin/test --profile=cProfile --repeat=3

If I'm profiling one test or a small number of tests, however, I like to use "--repeat=10" to be very sure that my profiling data isn't skewed:

$ bin/test -t my/functional/test.txt --profile=cProfile --repeat=10

Once the tests are finished I can fire up python and use the pstats module to browse my stats. Here I discovered another surprising convenience. Since I use mr.developer everywhere right now, all the code I generally have an interest in optimizing is in "src". I can use that fact to get much more targeted stats reporting by limiting the stats to those for code in "src":

>>> import os
>>> import pstats
>>> stats = pstats.Stats('parts/test/tests_profile.??????.prof').sort_stats('cumulative')
>>> stats.print_stats(os.getcwd()+'/src', 50)
...    parts/test/tests_profile.??????.prof
         ... function calls (... primitive calls) in ... CPU seconds
   Ordered by: cumulative time
   List reduced from ... to ... due to restriction <'.../src'>
   List reduced from ... to 50 due to restriction <50>
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   ...

I'm actually wondering if this might be a better default for the testrunner output when "--profile" is used. If the ZTK people think I should do that, just let me know and I'd be happy to contribute that.

You should also, however, be careful when just looking at the code in "src". For example, if your code is rendering templates when it doesn't need to, then limiting the stats to "src" will hide that from you. Filtering by "src" is a good start, but be sure to also check out the top 50 overall and to look into single tests in detail so you don't miss anything. See the print_stats docs to see everything you can do.

Finally, one should never just assume that even functional tests represent real-world load profiles. I would love to see a tool for capturing and sanitizing request data from production along with the response times for those requests which can then turn that sanitized real-world data into functional load test suites. This would be useful for doing continuous integration that can alert developers when they introduce performance regressions but it would also be very useful for targeted profiling where you could be nearly certain that your optimizations would get you the biggest bang for your users' bucks. This is something I'd love to work on in a sprint sometime.

Over my years of using ratpoison, I've resorted to some truly horrific hacks to try and approximate this goal. I've tried stalonetray and trayer. I've written ugly wrapper scripts that waited a while before killing metacity and starting ratpoison. I couldn't figure why I couldn't just let gnome-panel and nautilus (in the case of Ubuntu Desktop) or netbook-launcher-efl (in the case of Ubuntu Netbook) live in peace alongside ratpoison. Finally, I decided to dig into how Gnome does session startup and to try to get at the root of it.

The first hurdle was getting Gnome to use ratpoison as it's window manager. Gnome consults the /desktop/gnome/session/required_components/windowmanager gconf key to decide what window manager to use. In the case of Ubuntu Netbook Edition 2D, you can't set this gconf key without first unlocking it since UNE locks down a bunch of configurability. So the first order of business under UNE is to unlock the key. As root, edit /var/lib/gconf/une-efl.mandatory/%gconf-tree.xml and remove the "<entry name="windowmanager"..." under the same path for the key in the XML <dir> elements, it's towards the top.

Next we have to tell Gnome to accept ratpoison as a provider of the "windowmanager" required component. To do this, as root, copy /usr/share/applications/metacity.desktop to /usr/share/applications/ratpoison.desktop and in the new file replace, case insensitive, all occurrences of "metacity" with "ratpoison".

At this point you must log out and log back in so gdm will restart and gconf will see the changes. Now you can, as your normal user, set the key to ratpoison using gconf-editor or the following command line:

$ gconftool-2 --type string --set /desktop/gnome/session/required_components/windowmanager ratpoison

Now that Gnome will use ratpoison as it's window manager, we need to tell ratpoison to leave gnome-panel and nautilus/netbook-launcher-efl (Gnome/UNE 2D) windows alone. Tell ratpoison not to manage those windows by adding the following to your normal user's ~/.ratpoisonrc:

unmanage Top Expanded Edge Panel
unmanage Bottom Expanded Edge Panel
unmanage Netbook Launcher EFL
unmanage x-nautilus-desktop

Now we get into the part where somewhat less ugly hacks than my previous solutions, but still ugly hacks are required. Firstly, the gnome-panel and nautilus/netbook-launcher-efl (Gnome/UNE 2D) windows need to be opened after ratpoison has already fired up. Otherwise, ratpoison mangles the windows. Secondly, we need to make sure gnome-panel starts up last so that it will appear on top of the unmanaged windows. Gnome has discussed setting up startup dependencies, which is the right solution to this, but it's not there yet at least as far as I could tell.

Without such dependencies, the hacks required are so ugly, I suggest using gnome-session-properties ("Startup Applicatons" in the Gnome Sytem Preferences menu) instead of ~/.ratpoisonrc to do your app startup to solve the first problem. If ratpoison isn't opening your apps, it starts up lightning fast when set as the gnome window manager and window mangling isn't an issue at least for me.

The second issue, getting gnome-panel to startup after nautilus/netbook-launcher-efl, still requires an ugly hack, at least as far as I could tell. Make sure ~/bin is on your normal user's PATH, the default in recent Ubuntus at least. Then add a wrapper around gnome-panel so that it waits for nautilus/netbook-launcher-efl to start before it does by adding an executable ~/bin/gnome-panel with the following contents:

#! /bin/bash
set -x
IFS='
'

command=${0##*/}

# Find the next path for command
for path in $( which -a $command )
do
    if [ $next ]
    then
        break
    elif [ $path == $0 ]
    then
        next=1
    fi
done

# Determine command dependencies
case $DESKTOP_SESSION in
    gnome)
        depends_name="nautilus"
        ;;
    une-efl)
        depends_name="netbook-launcher-efl"
        ;;
    *)
        exec $path "$@"
        ;;
esac

# Wait for the dependency to start
while [ 1 ]
do
    # check for the process, if present continue
    ps -C "$depends_name" -o pid,user | grep $USER && break || sleep 1
done

# pause for window initialization
sleep 1

# Start the real command
exec $path "$@"

Now you should be able to log out of Gnome and back in, ratpoison will startup and tell you so immediately. Sometime later, nautilus/netbook-launcher-efl and gnome-panel will startup as unmanaged windows and your gnome-session startup apps will open. The windows ratpoison doesn't manage can be seen by clearing all the ratpoison manged windows with "C-t -". IOW, when using ratpoison under gnome, "C-t -" is equivalent to switching to the desktop.

This all only works under Ubuntu Netbook Edition 2D, since under 3D it appears everything, gnome-panel, the launcher, everything is actually all pulled into the mutter window manager used by unity. I think this is fine since I don't see much point in using ratpoison under unity. Under the 2D version of UNE, however, it works beautifully.

The one hitch I've found is that since the Gnome desktop windows are not managed by ratpoison, many completed actions, such as completing a dialog, will return you back to the previous ratpoison managed window. Most of the time, this is actually best for keyboard junkies since we tend to want to have as little to do with the desktop interface as possible. It can be annoying, however, when you have multiple interactions to have with the desktop in a row since you have to keep doing "C-t -" after each operation to get back to the desktop.

Since I was figuring this out as I went, I didn't do this setup in this on a clean install, so there may be some hitches. Sign in via openid and comment if you run into problems and I'll try to cleanup the procedure for anyone else.

The sprint was focused on "achievable tasks that provide immediate wins for Plone-the-product or Plone-the-community", meaning low-hanging fruit mostly on the plone.org site itself. With Limi, I took on merging the plone.net service providers site into plone.org itself. For my part, this involved:

• updating Products.PloneServicesCenter to Plone 4 along other fixes, cleanup and improvements
• exporting the content from the plone.net then importing it on plone.org
• doing a few selective catalog updates
• re-creating some content
• setting up some content rules

Pretty easy stuff and it was mostly done by Friday. It was also the first time I got my hands on plone.org itself. Call me a fan-boy but I still get giddy when I get one step closer to the core of Plone, a project I was once totally on the outside of.

I also spent some time walking through the creation of a new custom AT content type with Bill and Spanky to see how hard it is to approach this task as a newcomer to Plone. I always enjoy helping get started and happily, Bill even posted his code to github.

Not to toot my own horn, but my mixology seems to have been a hit. My Manhattan was particularly popular and since Bill Deegan asked, here's the recipe:

Cioppino Manhattan

1.5 oz      Buffalo Trace Bourbon (good bang for buck, not quite top-shelf)
1.5 oz      Carpano Antica Formula (really good sweet vermouth)
1-2 dashes  Blood Orange Bitters

Stir with lots of ice for a good while to melt plenty and mellow
out the cocktail.  Strain and serve.

In summary, the Cioppino Sprint was quite a success, I had a great time, and I very much hope it becomes an annual tradition!

I've been getting he following log message ever since I migrated to ZODB BLOBs:

ZODB.blob WARNING (PID) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout.

It's not obvious what to do about this, even after initial googling, so I thought I'd document it. Firstly, we need to use the ZODB.scripts.migrateblobs script so create a console script for it that has all the necessary eggs available. Unfortunately, the ZODB3 egg doesn't define a console_scripts entry point for this script so I used zc.recipe.egg to work around this:

[buildout]
...
parts +=
    zeo
...
    migrateblobs
...
[migrateblobs]
recipe = zc.recipe.egg
eggs = ZODB3
entry-points = migrateblobs=ZODB.scripts.migrateblobs:main

After re-running buildout and shutting down ZEO, I used the generated script to migrate the blobstorage:

$ mv var/blobstorage var/blobstorage-lawn
$ bin/migrateblobs var/blobstorage-lawn/ var/blobstorage
(PID) Blob directory `var/blobstorage-lawn/` has layout marker set. Selected `lawn` layout.
(PID) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout.
(PID) Blob directory '.../var/blobstorage/' does not exist. Created new directory.
(PID) Blob temporary directory 'var/blobstorage/tmp' does not exist. Created new directory.
Migrating blob data from `var/blobstorage-lawn/` (lawn) to `var/blobstorage` (bushy)
        OID: 0x###### - 1 files
...

Now I'm able to fire up ZEO and I can access AT file and image objects fine without the log warning.

I use Ubuntu Linux for my development platform (and everything else). As such, I've had the luxury of a pretty good OS installed python development environment. I really don't know much about virtualenv. I do know its one of those tools that many people swear by with a bit of zeal. Some won't assist with a problem unless you first setup a virtualenv to eliminate variables. The thing is, it's never actually solved a problem of mine that I can think of by eliminating a variable. I have on a very few occasions found it useful to smash my way out of a horrible environment when I'm not under one of my trusty Ubuntu environments, but even then just to get up and running. So for me, it's a tool I don't know well that I will use only when I need to to meet someone else's requirements or to solve a problem quickly when finding the root cause isn't worth my time.

Plone 3, unfortunately (go Plone 4, GO!), still requires Python 2.4 and Ubuntu has, quite appropriately, recently dropped 2.4. While struggling to keep my Plone 3 buildouts running and with some RHEL deployments, I found myself reaching for virtualenv. The problems were a mess of path issues (PATH, PYTHONPATH, LD_LIBRARY_PATH, etc.). I don't know much about these issues and I don't want to. I prefer to delegate to my trusty OS and to buildout. In the end, though virtualenv helped somewhat, I still had to come to understand far more about that path issues than I feel I should.

It occurred to me, however, that if I'd been working on something other than Ubuntu all these years, I would have run into many more path and similar problems and I probably would have fallen in love with virtualenv. It also occurred to me, that in the process of learning to use virtualenv to help resolve these issues, I would probably have learned a lot more about all the path issues and would probably be fairly competent at comprehending such issues. I would probably also not be aware that I had gained such further familiarity and might attribute my new-found greater ease with my Python environments disproportionately to virtualenv.

This reminded me of my experience with zope.formlib and z3c.form. A while back I'd built a number of forms with zope.formlib/zope.app.form. I found it initially much more productive than Archetypes, mostly due to much more appropriate decoupling, but as time went on it seemed likely I would have to put at least as much time into understanding things I don't want to as I have had to do with Archetypes. Given I already have the requisite familiarity with Archetypes and can be quite productive in it, I limited my investment in zope.formlib. When z3c.form came around, I was very excited at the prospect that I might get much of the benefit of zope.fomrlib forms without having to know as many things I don't want to know. I'm now considerably more invested in z3c.form than I ever was in zope.formlib and I can safely say I need to know just about as much I don't feel I should have to as I did with zope.formlib.

I have noted that the advocates of z3c.form often seem to find the appropriate approach to a given problem to be very clear to them and take a lot of pride in being able to point to the right place in the docs. For my experience, however, I find that even after reading the right place in the docs and reviewing several other docs, I still have to do way too much UTSL to solve my problem. I also have the strong sense that after putting this much time into learning z3c.form, the obviousness of "The Right Way" will only come to me once I've had to learn nearly as much I shouldn't have to as I've had to do with Archetypes. I take from this that it may be another case of somewhat erroneously attributing the value of comprehension, understanding, and experience to the tool whose use lead to this familiarity largely incidentally.

(To be very clear, I don't want to discuss z3c.form here. I think investing in some successor to AT is very necessary at this point and we may not be able to afford waiting for something else. The documentation is also absolutely worth being proud of. I think most of the value of z3c.form will be in putting our collective weight behind something. I use z3c.form and will continue to do so.)

Now for the disturbing part, I know I've done this very thing many times but I can't yet list them. It's much easier to come to see a pattern like this when looking critically at or feeling frustrated with tools I'm just using but not heavily invested in than it is to do so with tools I'm building. I hope to temper my strategic decisions for adopting and suggesting tools with this critical awareness. I also hope to temper my own perspective of the value of tools I'm developing or becoming deeply invested in. I also hope our development community can learn to better ask whether the perceived value of a given tool is being unduly influenced by the incidental familiarity we've developed in building or using that tool.

The upgrade went pretty smoothly for my main site. I had some other, older, Plone portals where I've installed various add-ons in the past and for those I had to do some manual cleanup of some broken objects.

The unified folder implementation makes removing broken objects very difficult. The manage_delObjects(), _delObject, and _delOb methods are all broken before the folders have been migrated but the migration won't complete if there are broken objects in the folder. I had to do something like this where "idx" is the index in _objects of the broken object and 'foo' is the id:

>>> folder._objects = folder._objects[:idx]+folder._objects[idx+1:]
>>> delattr(folder, 'foo')
>>> import transaction
>>> transaction.commit()

On two of my portals I also had some regular TextIndex'es in the catalog, which I guess are no longer around. Those I was able to just delete in the "Indexes" tab of the ZMI for portal_catalog. Since these were core indexes like "Title" I used portal_setup to install just the catalog import step from the Plone base profile. Being a base profile this means running the import step cleared the indexes so I also had to update those catalogs.

After all that was done I just uninstalled the classic theme and Kupu and then installed and configured plone.app.caching. Everything seems to be working well.

Just to polish things off, I wanted to make sure I was getting the full benefit of Hanno's ExtensionClass/ZODB fix. The one thing I can't find is a clear migration procedure. As far as I understand the ZODB, the objects created before the fix would need to be written to the ZODB again to see the benefit from the fix. So I used the following at a "bin/instance debug" prompt:

>>> app.ZopeFindAndApply(app, search_sub=1, apply_func=lambda obj, path: setattr(obj, '_p_changed', True))
>>> import transaction
>>> transaction.commit()

That should do it, but if anyone knows how that's insufficient or knows a better way to accomplish the migration, please do leave a comment.