Now, as the sprint mixologist, I'm off to put together the shopping list for the bar. I'm sure Steve and others are thinking of the shopping lists for the actual cioppino. The rest of the sprint attendees are engaged in similar preparation. And the rest of you can just go stew in envy. :-)

I have a room checking in on the 18th and checking out on the 25th at the Nittany Lion Inn with two double beds. Does anyone want to share the room? Also, that leaves the night of the 17th when I fly in and the night of the 25th before the last day of sprints. Does anyone have a couch, bit of floor, or a spare bed they'd be willing to let me use either of those nights?

Hope to see a lot of you there!

Really my conference review should be taken worth a grain of salt, because I never go for the talks, it's all about the sprints. Having so thoroughly enjoyed sprinting since I came out of my shell and joined the Plone community in earnest, it's always the only thing I'm really focused on. What can I say? In this wonderful age where, through open-source communities and the recognition of the ambiguous stuff that helps them thrive, we all recognize that things like docs and diversity are as important as the quality of code or the power of features, I'm still a code monkey at heart.

Since there were no Plone sprints that I heard of or saw on the boards, I took the chance to sprint on Pyramid, part of the Pylons project and love child of repoze. I've been wanting to learn Pyramid for quite some time now since I noticed how many of my favorite Zope people have moved to Pyramid. So I went to the Pyramid room, sat down and asked what someone with plenty of zope experience could do.

The Pyramid sprinters were a great group, very welcoming and friendly. They remind me a lot of the things I most love about the Plone community. It also gave me the chance to work closely with Chris McDonough. Chris is very smart, builds great software, and is just very kind-hearted.

Here's some of the stuff I worked on:

• add IP range/network restriction to pyramid_debugtoolbar

  I used this simple feature task to cut my teeth on the Pyramid development environment and tool-chain. It was the first time in a long time I worked outside a buildout environment and it made for some interesting comparisons. They keep things really simple relying only on distribute's setuptools and virtualenv.

  I have to say, I still prefer working with buildout. Buildout's caching is much better than easy_install's and virtualenv so updating the environment after changes is smoother and faster. For testing against multiple Python versions and implementations, such as Jython and PyPy, Pyramid uses tox which is pretty slick and very useful. I'd love to see something like this for deploying the same buildout with different versions of Python.

• support Python 3 in repoze.sendmail

  This was my first time tackling Python 2 and 3 compatibility issues. I have to say it's just not as hard as I feared.

• improve and centralize email message serializing in repoze.sendmail

  The one place where there was a lot of Python 2 and 3 compatibility pain was in the encoding of email messages. In this case, as in others I suspect, the problem was that encoding was being done all over the place in a not-very-well-architected way and the real solution was to clean things up and put in more clearly defined boundaries and interfaces.

  That task would have been impossible without R. David Murray. I pestered him endlessly and he helped me learn all the deep, dark corners of the email protocols and the Python email package, both of which have considerable quirks which some would call bugs. At the end of it, I had a clear sense of all the quirks of encoding emails and all the things that the email package doesn't already do. I baked those into repoze.sendmail.encoding and added that support as a part sending mail through repoze.sendmail.

  Once that was in place, finishing Python 2 and 3 compatibility was a snap.

• add Python 3 support to pyramid_mailer

  Again, once repoze.sendmail.encoding was in-place, I just ripped all the email encoding support out of pyramid_mailer and Python 2 and 3 compatibility was a cinch.

• fix Cc/Bcc witout To handling in pyramid_mailer

  While I had my hands in pyramid_mailer, I fixed a bug reported by Michael Merickel.

• add support for multiple request parameters when registering Pyramid views

  I'm not even sure what this stuff does, but Pyramid has such great test coverage, I was able to get started. I'm still waiting for feeback to see what I'm not yet understanding.

• updating warnings handling in Pyramid tests

  Just a simple test fixture update to cleanup output.

gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include   -DPy_BUILD_CORE -DSVNVERSION=\"`LC_ALL=C svnversion .`\" -o Modules/getbuildinfo.o ./Modules/getbuildinfo.c
gcc: error: directory": No such file or directory
make: *** [Modules/getbuildinfo.o] Error 1

What's happening here is that Python is using svnversion in it's build process but the 1.7 version has different output than previous versions. I ran into this problem with Python 2.5 but not with Python 2.4. I'm not sure about Python 2.6+ because I just downgraded subversion temporarily before I built the other versions of Python.

Since running the upgrade procedure can take a long time, the iterations take a long time. This is compounded by the amount of time it can take to transfer the DB, no matter how you do it. Using ZEO over remote is very slow, I assume due to the latency, from looking at iftop output. Syncing the BLOBs, even with a well tuned rsync, script can take forever.

I finally found a set of approaches that brings my development of upgrade steps up to a tolerable speed. It's still slower than "normal" development, but it no longer feels maddening. Firstly, I developed an upgrade runner that commits upgrades per profile version increment so that I don't have to start the whole procedure over. It's called collective.upgrade and I plan to cut a release along with a more detailed blog post once I've deployed this current upgrade.

Secondly, I used a union mount (UnionFS, AUFS, etc.) to get copy-on-write behavior for my var/filestorage/Data.fs and var/blobstorage/. IOW, whenever the upgrade procedure reads a BLOB, it gets it from the production blobstorage directory via a network filesystem (SSHFS, CIFS/SMB, NFS, etc.). When writing to a BLOB, however, it writes it to a local directory and will use that version of the BLOB in the future. Since BLOBs are very often compressed images and files, I find no penalty in just letting the network FS do a dumb transfer of BLOBs as opposed to compressing with rsync -z. I also use the same setup for Data.fs, but since it's much smaller than the BLOBs and it's much more heavily used than the BLOBs, I've found it best to just rsync the Data.fs and Data.fs.index. With this setup I can test the upgrade at nearly local speeds and my upgrade step development is much faster for having all my favorite tools.

Here's some scripts I'm using to do this. It's very important that you mount the prod newtork FS as read-only with the -o ro mount option:

sudo mount -v -t <network fs type> <remote network fs URL> var/prod -o ro,<fs options>
sudo mount -v -t aufs none var/filestorage -o dirs=var/filestorage.prod:var/prod/var/filestorage=rr
sudo mount -v -t aufs none var/blobstorage -o dirs=var/blobstorage.prod:var/prod/var/blobstorage=rr

When it's time to refresh to the latest prod, shutdown Zope and ZEO:

rsync -Paz <remote rsync URL>/var/filestorage/Data.fs <remote rsync URL>/var/filestorage/Data.fs.index var/filestorage.prod/
rsync -Paz --existing <remote rsync URL>/var/blobstorage/ var/blobstorage.prod/

The second command there will only revert BLOBs that were already copied locally back to the prod version.

Enjoy, but be very careful that you don't accidentally apply changes to prod. Backup prod and make sure your prod network mount is read-only.

Use buildout.dumppickedversions version 0.5 or later:

If you're using buildout.dumppickedversions, use version 0.5. This release includes a fix I contributed which yields a 3-4 fold decrease in runtime if buildout is run with just the -N option and no -v options. If your're using buildout-versions instead of buildout.dumppickedversions, unfortunately it suffers from the same problem. The author knows about this and may address it but until then you may want to switch to buildout.dumppickedversions.

Use zc.buildout version 1.5 or later:

This seems to yield a 2-3 fold decrease in buildout runtime when run with -N and addresses a similar hot loop as found in buildout.dumppickedversions. In fact, it's resolution of required distributions seems to be much more efficient in general. If you're stuck with 1.4, I've contributed the same fix as for buildout.dumppickedversions to the 1.4 branch so it should make it into a next release if someone makes a new 1.4 release.

Use the -N option:

With the above newest releases of zc.buildout and buildout.dumppickedversions, you can now exercise a lot of control of your run times depending on what command-line options you use. In particular, make sure newest is false and that verbosity is not increased. Both of these can be influenced by options in the buildout configuration, so keep an eye out for that, but if no such options are interfering then this means just invoking buildout like $ bin/buildout -N -c buildout.cfg. IOW, use one -N option and no -v options. Before the improvements in zc.buildout 1.5 and the fix to buildout.dumppickedversions, a lot of time was wasted doing work that would only actually be used if -v had been given.

Clean out your egg cache and use virtualenv --no-site-packages:

Some of the remaining inefficiencies in zc.buildout are proportional to the number of distributions available on sys.path. IOW if you reduce the number of distributions to be scanned, you can increase performance. If you're using a shared eggs-directory or have a buildout that has been around for a long time, there may be a lot of different versions of the same packages in eggs-directory. By cleaning those out you reduce the number of distributions that buildout needs to scan. I recommend just emptying your eggs-directory and then letting buildout re-download all the eggs it actually needs now. I takes a while once, but then it's done. Similarly, using virtualenv --no-site-packages can reduce the number of dists buildout needs to scan.

Help me get zc.buildout/branches/env-cache merged and released:

Having said the above about cleaning out the egg cache, buildout, distribute, and setuptools shouldn't be scanning these paths multiple time anyways. Along with package indexes, these paths are global in nature and so the dists found there should only be scanned for once and cached globally. I've started this work on the env-cache branch of zc.buildout. In my timing tests, this branch seems to yield another ~60% decrease in run time. The zc.buildout tests, however, are a PITA and Jim Fulton is a very busy man with little time to evaluate this branch. If anyone else can help me shepherd this branch through to merge and then release, that would be great. I've written up how to use this branch with your existing buildouts.

With all this in effect I took a production buildout and another development buildout from 2 minute run times down to 20 second run times. This really makes re-running buildout something I no longer feel an urge to avoid. So use zc.buildout 1.5.2, buildout.dumppickedversions 0.5, -N and do what you can to help get the env-cache branch merged.

We in Plone land have been long frustrated about how easily a poorly behaved add-on could break a Plone site. Simply installing the add-on and removing it can leave your site inaccessible, even the ZMI! Often times add-ons that provide only minimal additional object behavior can break the entire object when removed.

Now this isn't actually the ZODB's fault at all. The ZODB actually has quite robust handling of broken object and it works quite well. I'm afraid that fault here lies in zope.interface and zope.component. They're simply both too central to almost all ZODB applications to fail to provide graceful handling of broken objects. So I've taken advantage of the sprints following the Plone Conference here in San Francisco to work on this.

The experimental.broken package provides patches for zope.interface and zope.component to make the services they provide more tolerant of broken objects in the ZODB and allows them to function as if the interfaces and components aren't there when the add-on has been removed. Even better, if the add-on is restored, the full behavior of the interfaces and components is restored.

That said, this is all highly experimental so don't use it. If you do use it, never use it in production and only use it on a copy of your ZODB. And if you do use it, definitely report your experience. If you do have a broken site I'd love to hear how it works for you. Just add experimental.broken to your instance eggs.

If this approach tests well and the core Zope developers think it's the right approach I'll be merging this into zope.interface and zope.component.

[build_ext]
library_dirs = /usr/lib:/usr/lib/i386-linux-gnu:/usr/local/lib

Consulting/Contract Opportunities

I'm perfectly happy to keep working independently or as a sub-contractor with a larger consultancy on Plone projects. I know Plone very well and am very happy to continue working with it on projects such as:

• Intranet workflow/business-process applications
• Custom web publishing CMS projects
• Quickly solving critical problems as a Plone "hired gun"
• Building a project while also cross-training internal developers
• Social justice related projects
• Etc., pretty much any Plone project

Opportunities with a Team

I'm eager to join a team, either as an employee or as a contractor, that can offer some of the following:

• More collaboration as a part of a team that genuinely like each other
• More technological diversity outside of Plone/Zope
• Opportunity to work on social justice related projects

I ran across this video on a "Planet ..." feed of some sort recently describing what I've been looking for. This guy also has a lot of other good git-svn thoughts. I'm just not sure it's worth it. I guess it all depends on how quickly people migrate to github or some such natively. If there's any shared interest, leave a comment or email me. If there's interest I'll start a thread somewhere.

At any rate, thanks to a Planet Emacsen blog post I decided to give it a try. It's a clear win for me. Better group manaagement, key binding system, and plays nicely with the Gnome/Unity-2D panel when I need it. It's all around more modern and less of a fight with modern desktops like Gnome/Unity-2D. I don't think I'll be going back as long as the instability everyone mentions is manageable.

But wait, hidden in a corner of mr.developer, and totally undocumented is a mr.developer.gitsvn module that does exactly that! You can use a "gitsvn" checkout type and you're in business:

[sources]
...
collective.formcriteria = gitsvn https://svn.plone.org/svn/collective/collective.formcriteria/trunk
...

I was going to start modifying all my local.cfg buildout files to change all the "svn" checkouts to "gitsvn" checkouts when I noticed something about how mr.developer registers checkout types. At the very bottom of mr/developer/gitsvn.py I changed:

common.workingcopytypes['gitsvn'] = GitSVNWorkingCopy

to:

common.workingcopytypes['gitsvn'] = GitSVNWorkingCopy
common.workingcopytypes['svn'] = GitSVNWorkingCopy

Viola! Now mr.developer uses gitify for all svn checkouts. Now I'm getting close to being able to pretend svn doesn't exist.

I would love to see mr.developer use a global switch so I don't have to modify the egg like this, maybe something in ~/.buildout/default.cfg. If there's interest in that, I'd be happy to contribute such a thing. I should also mention that gitify is often pretty rough and leaves svn and git in an inconsistent state which is only really repairable by someone who is very familiar with both svn and git. The next time I run into that, I'll be trying to run it to ground and contribute fixes.

I've known about zope.testing/zope.testrunner's "--profile=cProfile" command line option for sometime. When you use it, it spits out the "top 50 stats, sorted by cumulative time and number of calls". Under a featureful application like Plone, however, the top 50 stats are usually consumed with the top publisher and template rendering calls and don't actually tell you anything useful.

Recently I needed to do some profiling of the baseline performance of an application. Since I use functional zope.testbrowser tests a lot (some would say too much), my functional tests are actually a pretty good representation of the real world application usage. So I was thrilled to find that the testrunner actually leaves the raw profiling data behind after a test run! It leaves the data in the working directory in which the test run was invoked. If you're doing this in buildout using a zc.recipe.testrunner part named "test", as is the convention, the file will be in parts/test/tests_profile.??????.prof.

In order to avoid the test fixture setup from polluting the profiling data, I use the "--repeat" command line option for the test runner to make sure that profiling data from the actual tests are much louder in the profiling data than the test setup. For a large functional test suite I tend to use "--repeat=3" since the suite is long enough as it is and any more repetitions just take too long:

$ bin/test --profile=cProfile --repeat=3

If I'm profiling one test or a small number of tests, however, I like to use "--repeat=10" to be very sure that my profiling data isn't skewed:

$ bin/test -t my/functional/test.txt --profile=cProfile --repeat=10

Once the tests are finished I can fire up python and use the pstats module to browse my stats. Here I discovered another surprising convenience. Since I use mr.developer everywhere right now, all the code I generally have an interest in optimizing is in "src". I can use that fact to get much more targeted stats reporting by limiting the stats to those for code in "src":

>>> import os
>>> import pstats
>>> stats = pstats.Stats('parts/test/tests_profile.??????.prof').sort_stats('cumulative')
>>> stats.print_stats(os.getcwd()+'/src', 50)
...    parts/test/tests_profile.??????.prof
         ... function calls (... primitive calls) in ... CPU seconds
   Ordered by: cumulative time
   List reduced from ... to ... due to restriction <'.../src'>
   List reduced from ... to 50 due to restriction <50>
   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   ...

I'm actually wondering if this might be a better default for the testrunner output when "--profile" is used. If the ZTK people think I should do that, just let me know and I'd be happy to contribute that.

You should also, however, be careful when just looking at the code in "src". For example, if your code is rendering templates when it doesn't need to, then limiting the stats to "src" will hide that from you. Filtering by "src" is a good start, but be sure to also check out the top 50 overall and to look into single tests in detail so you don't miss anything. See the print_stats docs to see everything you can do.

Finally, one should never just assume that even functional tests represent real-world load profiles. I would love to see a tool for capturing and sanitizing request data from production along with the response times for those requests which can then turn that sanitized real-world data into functional load test suites. This would be useful for doing continuous integration that can alert developers when they introduce performance regressions but it would also be very useful for targeted profiling where you could be nearly certain that your optimizations would get you the biggest bang for your users' bucks. This is something I'd love to work on in a sprint sometime.