I use Ubuntu Linux for my development platform (and everything else). As such, I've had the luxury of a pretty good OS installed python development environment. I really don't know much about virtualenv. I do know its one of those tools that many people swear by with a bit of zeal. Some won't assist with a problem unless you first setup a virtualenv to eliminate variables. The thing is, it's never actually solved a problem of mine that I can think of by eliminating a variable. I have on a very few occasions found it useful to smash my way out of a horrible environment when I'm not under one of my trusty Ubuntu environments, but even then just to get up and running. So for me, it's a tool I don't know well that I will use only when I need to to meet someone else's requirements or to solve a problem quickly when finding the root cause isn't worth my time.

Plone 3, unfortunately (go Plone 4, GO!), still requires Python 2.4 and Ubuntu has, quite appropriately, recently dropped 2.4. While struggling to keep my Plone 3 buildouts running and with some RHEL deployments, I found myself reaching for virtualenv. The problems were a mess of path issues (PATH, PYTHONPATH, LD_LIBRARY_PATH, etc.). I don't know much about these issues and I don't want to. I prefer to delegate to my trusty OS and to buildout. In the end, though virtualenv helped somewhat, I still had to come to understand far more about that path issues than I feel I should.

It occurred to me, however, that if I'd been working on something other than Ubuntu all these years, I would have run into many more path and similar problems and I probably would have fallen in love with virtualenv. It also occurred to me, that in the process of learning to use virtualenv to help resolve these issues, I would probably have learned a lot more about all the path issues and would probably be fairly competent at comprehending such issues. I would probably also not be aware that I had gained such further familiarity and might attribute my new-found greater ease with my Python environments disproportionately to virtualenv.

This reminded me of my experience with zope.formlib and z3c.form. A while back I'd built a number of forms with zope.formlib/zope.app.form. I found it initially much more productive than Archetypes, mostly due to much more appropriate decoupling, but as time went on it seemed likely I would have to put at least as much time into understanding things I don't want to as I have had to do with Archetypes. Given I already have the requisite familiarity with Archetypes and can be quite productive in it, I limited my investment in zope.formlib. When z3c.form came around, I was very excited at the prospect that I might get much of the benefit of zope.fomrlib forms without having to know as many things I don't want to know. I'm now considerably more invested in z3c.form than I ever was in zope.formlib and I can safely say I need to know just about as much I don't feel I should have to as I did with zope.formlib.

I have noted that the advocates of z3c.form often seem to find the appropriate approach to a given problem to be very clear to them and take a lot of pride in being able to point to the right place in the docs. For my experience, however, I find that even after reading the right place in the docs and reviewing several other docs, I still have to do way too much UTSL to solve my problem. I also have the strong sense that after putting this much time into learning z3c.form, the obviousness of "The Right Way" will only come to me once I've had to learn nearly as much I shouldn't have to as I've had to do with Archetypes. I take from this that it may be another case of somewhat erroneously attributing the value of comprehension, understanding, and experience to the tool whose use lead to this familiarity largely incidentally.

(To be very clear, I don't want to discuss z3c.form here. I think investing in some successor to AT is very necessary at this point and we may not be able to afford waiting for something else. The documentation is also absolutely worth being proud of. I think most of the value of z3c.form will be in putting our collective weight behind something. I use z3c.form and will continue to do so.)

Now for the disturbing part, I know I've done this very thing many times but I can't yet list them. It's much easier to come to see a pattern like this when looking critically at or feeling frustrated with tools I'm just using but not heavily invested in than it is to do so with tools I'm building. I hope to temper my strategic decisions for adopting and suggesting tools with this critical awareness. I also hope to temper my own perspective of the value of tools I'm developing or becoming deeply invested in. I also hope our development community can learn to better ask whether the perceived value of a given tool is being unduly influenced by the incidental familiarity we've developed in building or using that tool.

The upgrade went pretty smoothly for my main site. I had some other, older, Plone portals where I've installed various add-ons in the past and for those I had to do some manual cleanup of some broken objects.

The unified folder implementation makes removing broken objects very difficult. The manage_delObjects(), _delObject, and _delOb methods are all broken before the folders have been migrated but the migration won't complete if there are broken objects in the folder. I had to do something like this where "idx" is the index in _objects of the broken object and 'foo' is the id:

>>> folder._objects = folder._objects[:idx]+folder._objects[idx+1:]
>>> delattr(folder, 'foo')
>>> import transaction
>>> transaction.commit()

On two of my portals I also had some regular TextIndex'es in the catalog, which I guess are no longer around. Those I was able to just delete in the "Indexes" tab of the ZMI for portal_catalog. Since these were core indexes like "Title" I used portal_setup to install just the catalog import step from the Plone base profile. Being a base profile this means running the import step cleared the indexes so I also had to update those catalogs.

After all that was done I just uninstalled the classic theme and Kupu and then installed and configured plone.app.caching. Everything seems to be working well.

Just to polish things off, I wanted to make sure I was getting the full benefit of Hanno's ExtensionClass/ZODB fix. The one thing I can't find is a clear migration procedure. As far as I understand the ZODB, the objects created before the fix would need to be written to the ZODB again to see the benefit from the fix. So I used the following at a "bin/instance debug" prompt:

>>> app.ZopeFindAndApply(app, search_sub=1, apply_func=lambda obj, path: setattr(obj, '_p_changed', True))
>>> import transaction
>>> transaction.commit()

That should do it, but if anyone knows how that's insufficient or knows a better way to accomplish the migration, please do leave a comment.

This is the first year of what I hope will be many Tahoe Sprints and was focused on Dexterity. I decided to start by trying to implement TTW field vocabularies for the plone.schemaeditor TTW Zope 3 schema editor. The plone.schemaeditor is a generalized approach to TTW schema editing and does not depend on Dexterity but is at the heart of Dexterity's TTW content type support. It had basic support for defining a few simple fields and adding support for user-selectable or user-entered vocabularies or sources seemed like the next step.

For those not familiar with Zope 3 schemata, fields that support choosing from a set of values can use either vocabularies or sources. Vocabularies are the older approach and are, roughly speaking, more suitable to simple sets or sequences of selectable values. Sources are a newer approach and are, roughly speaking, more suitable to complex choice behaviors such as searching for items to select, or choosing from very large sets of values where more conventional listing would not scale.

We discussed the various things one might want to do with specifying vocabularies or sources for fields. The simplest, and probably most common case, is where the user simply wants to enter the list of allowed values and have a simple selection widget presented when the field widget is rendered. For more complex sources, each source will have it's own implementation specific set of options for the user to configure when creating the fields and as such each source will need it's own add/edit form in the schema editor. We also discussed the possibility that users creating/editing schemata may want to re-use the same vocabulary or source across multiple fields, where those fields may themselves be in different schemata. Given all that we decided it was best to implement only simple vocabularies with a list of user-entered values on the field edit form and tackle the shared vocabulary and source support as a registry of vocabularies and sources that could be assigned to choice fields.

On the first day (well the first day not spent driving and socializing), I was able to get user-entered vocabularies working for single selection fields, labeled "Choice" for now in the schema editor. Early on the second day I finished user-entered vocabulary support for multiple-selection fields. If you'd like to take a look at it, fire up the dexterity development buildout and try it out.

The zc.relation package provides a very powerful library for establishing connections, read references and relationships, between objects. It also provides for walking and searching networks of connections. It's all very cool. It's also very generalized which is a good thing since it can be used in so many different ways. This does, however, make it very difficult to approach. It's my hope that as we find ways to integrate zc.relation into the Plone stack, that others will find clever ways to use zc.relation's powerful network traversing to do some unexpected things. So the combination of zc.relation and z3c.relationfield provides not only a more widely re-usable, loosely coupled replacement for the AT reference engine, but a much more powerful tool with capabilities far beyond what could ever be achieved with the AT reference engine.

One of the concerns which kept coming up was that getting rid of the AT reference engine meant getting rid of UIDs. With UUID use taking off and finding new and interesting uses it's important that we provide some sort of UUID system, but the AT reference engine is large and heavy and very much unnecessary to support UUIDs. UUIDs are very simple and should not be coupled to any particular framework or larger library when they can so easily be a very small library of their own easily integrated in lots of ways. So the question of how we want to support UUIDs and the question of what to do with the AT reference engine should no be conflated, they can and should be addressed separately.

Dexterity integrates the z3c.relationfield package to provide widgets for specifying zc.relation connections between content on z3c.form forms. This allows Dexterity content to establish relationships and references to both Dexterity and AT content. There is, however, no zc.relation field and widget for AT content so there's no way for AT content to establish relationships or references to Dexterity content.

David Brennan set out to build an AT field and widget that could make use of zc.relation relations and on the latter half of the second day I joined him for some pair programming. By lunch on the third day, we had working extensions of Products.Archetypes.Field.ReferenceField and Products.Archetypes.Widget.ReferenceWidget which used zc.relation behind the scenes.

Next up is an extension of Products.ATReferenceBrowserWidget which works with zc.relation. Once that is working, archetypes.schemaextender can be used with one which uses zc.relation. Finally, a migration can be to offer a ZCML file which would replace the ATCT relatedItems field implemented to migrate existing AT references to the zc.relation back-end.

Once that is completed archetypes.z3crelationfield can be used as a proving ground for the possibility of removing the reference engine from Archetypes itself. Once it's polished and the AT reference engine removed, the way will be paved to remove one of the hurdles which has made it so hard for newer content type frameworks to make an entrance on the Plone scene.

Firstly, an "Export" document action (ala "Print", "Send to", etc.) allows the collection's results to be exported to CSV where the columns are those specified in the collection's "Table Columns" fields. This merges (and deprecates) my collective.catalogexport package and duplicates SmartCSVExporter. One advantage of the collective.formcriteria version is that it will stream the results to the response. Since catalog results are lazy, this should be fairly efficient.

I've also tightened up the styling on the search form portlet so it's not quite as unwieldy. I'd still love to get some contributions here from someone better at CSS than I.

But the most significant piece of work has been to get the folder_contents form to work with collection results. The "Tabular Form" display layout is now available for collections.

• Use "Table Columns" fields for the folder contents form to control the folder contents columns
• Use a "Table Column Links" collection filed to specify which folder contents form columns should link to the item
• The "Copy", "Cut", "Rename", "Change State", and "Delete" folder contents form button now all work for objects listed by the collections even if they are in different folders

With all this in place, collections and formcriteria make a fairly powerful combination for managing Plone content. You can setup a collection with a search form portlet and use search queries to find all the objects you want to apply a workflow transition to, delete, or copy.

The amount of code duplication involved in writing a properly declaritive GenericSetup import/export handler has always bothered me. It seems like most of the duplication involves fairly manually walking the XML DOM to extract the data needed. I find this to be very tedious and error prone. I think implementing GS import/export handlers should have nothing to do with XML.

ZCML does a great job of allowing directive handlers to be simple Python functions that know nothing of XML. I wrote GSML to bring that separation of concerns to GS import handlers so I could stop writing DOM walkers.

Currently, GSML lives in collective.gsqi which is just my grab bag of patches to GS and portal_quickinstaller I use in my client work. Here's a doctest that covers writing an import handler with GSML. Here's are a groups import handler and a references import handler implemented with GSML.

The current implementation of GSML just wraps the ZCML configuration engine sufficiently to use it for GS import handlers. Since ZCML has never concerned itself with writing ZCML, however, GSML currently includes no support for export handlers. This is obviously the most important TODO.

So please try it out, take a look at it and let me know what you think. In particular, if anyone out there has any interest in the export stuff, either in contributing to it, sprinting on it, advising on it, or anything else, I'd really like to hear from you.

The old key will continue to be valid for some time, but i prefer all future correspondence to come to the new one. I would also like this new key to be re-integrated into the web of trust. This message is also available signed by both keys to certify the transition.

the old key was:

pub   1024D/DA3C0A60 2007-10-03
      Key fingerprint = 5A41 9205 15DA 73C7 9D04  9EAF 79E1 C766 DA3C 0A60

And the new key is:

pub   4096R/983EA5AB 2009-08-02
      Key fingerprint = A815 9EE0 8317 DE08 8FE8  4EA1 FC5B 12B8 983E A5AB

To fetch the full key, you can get it with:

wget -q -O- http://rpatterson.net/rpatterson.gpg | gpg --import -

Or, to fetch my new key from a public key server, you can simply do:

gpg --keyserver subkeys.pgp.net --recv-key 983EA5AB

If you already know my old key, you can now verify that the new key is signed by the old one:

gpg --check-sigs 983EA5AB

If you don't already know my old key, or you just want to be double extra paranoid, you can check the fingerprint against the one above:

gpg --fingerprint 983EA5AB

If you are satisfied that you've got the right key, and the UIDs match what you expect, I'd appreciate it if you would sign my key:

gpg --sign-key 983EA5AB

Lastly, if you could upload these signatures, i would appreciate it. You can either send me an e-mail with the new signatures (if you have a functional MTA on your system):

gpg --armor --export 983EA5AB | mail -s 'OpenPGP Signatures' rpatterson@rpatterson.net

Or you can just upload the signatures to a public keyserver directly:

gpg --keyserver subkeys.pgp.net --send-key 983EA5AB

Please let me know if there is any trouble, and sorry for the inconvenience.

Thanks!

The modifications were used to get the python-ldap package to properly authenticate/verify certificates when connecting via SSL/SASL/TLS. I found a Plone setup list post that provides most of the solution.

The right way to get certificates working is to have the underlying openldap installation properly configured to use the certificates. If that's configured properly, python-ldap will just work and no modifications are required to the Plone LDAP eggs.

The post contains the basic buildout configuration layout required to get things working which includes building openldap using CMMI and using a template to install an ldap.conf that tells the openldap build where to find the certificates. I ran into one issue during the openldap build for which some quick googling pointed me to an environment variable workaround. The post uses a buildout part to install the certificates into the buildout configuration, but since the ldap.conf template has to say where the certificates are anyways, I just point to them in the ldap.conf template at etc/ldap.conf.in:

TLS_CACERTDIR /usr/local/ssl/certs
TLS_REQCERT hard

Then we build python-ldap telling it to use the openldap build from the buildout. The final key is to make sure that the custom python-ldap egg is used and not one retrieved by buildout prior to building the custom egg. The way to achieve this is to make sure that the instance(s) eggs option does a valiable substitution from the python-ldap buildout part, thus forcing the right dependency ordering.

The end result is my ldap.cfg file:

[buildout]
extends = base.cfg
parts +=
    openldap-build
    ldap.conf
    python-ldap

[instance1]
eggs += ${python-ldap:egg}

[openldap-build]
recipe = zc.recipe.cmmi
url = ftp://ftp.openldap.org/pub/OpenLDAP/openldap-stable/openldap-stable-20090411.tgz
environment =
# Workaround for "error: storage size of ‘peercred’ isn’t known"
# http://www.openldap.org/lists/openldap-bugs/200808/msg00130.html
    CPPFLAGS=-D_GNU_SOURCE
extra_options =
    --with-sasl --with-tls --enable-slapd=no

[ldap.conf]
recipe = collective.recipe.template
depends = ${openldap-build:location}
input = etc/ldap.conf.in
output = ${openldap-build:location}/etc/openldap/ldap.conf

[python-ldap]
recipe = zc.recipe.egg:custom
egg = python-ldap
include-dirs = ${openldap-build:location}/include
library-dirs = ${openldap-build:location}/lib
rpath = ${openldap-build:location}/lib

Enjoy!

Update

The right way to solve my original problem concerning ConflictErrors when starting up multiple ZEO clients and subsequent TypeErrors, as pointed out by both Hanno Schlichting and Martijn Pieters, is to add the following to the configuration of all but the first ZEO client:

zope-conf-additional +=
    enable-product-installation False

Here's the genericized post in case someone needs supervisor staggering for some other reason.

[sleep]
recipe = zc.recipe.egg
eggs = zc.recipe.egg
initialization =
    import sys, time
    time.sleep(float(sys.argv[1]))
arguments = sys.argv[2], sys.argv[2:]
entry-points = sleep=os:execvp

[program:foo]
command = %(here)s/../bin/foo
startsecs = 30

[program:foo2]
command = %(here)s/../bin/sleep 30 %(here)s/../bin/foo2
startsecs = 60

[program:foo3]
command = %(here)s/../bin/sleep 60 %(here)s/../bin/foo3
startsecs = 90

[program:foo4]
command = %(here)s/../bin/sleep 90 %(here)s/../bin/foo4
startsecs = 120

If you're wondering what it does, it allows collections and their criteria to be used to generate search forms and search form portlets. It also includes a handful of useful collection views. Many have found it useful and I hope yet more will. :)

Thanks to Oxfam America, Jazkarta, and another client for sponsoring portions of the work!

In fact, I couldn't put it down. Despite running on only a few hours sleep, a massively overloaded brain, and AirCanada's genuinely respectable selection of on-demand movies, I kept working on the buildbot integration. But I'm getting ahead of myself.

Since my last post on the performance sprint, Andrew continued to alternate between poking at the write concurrency issues and doing functional benchmark comparisons for some of the work being done at the sprint to measure impact. I really want to resume working with Andrew on the write concurrency issue. In particular I want to document what of the various performance optimizations, experimental or otherwise, have any significant impact on the performance. I had a lot of fun working with Andrew so maybe we can do some remote sprinting.

Meanwhile, I know Tom was working on the documentation in the buildout and refining the "candidate" story. The "candidate" stuff is a buildout configuration file that is intended to make it easy for developers to do comparison benchmarks to measure the impact their work has on the baseline Plone performance. I'm very excited about this work as a part of making functional benchmarking accessible and it's definitely something we'll want to take advantage of as we poke at the write concurrency issues.

For my part, I further refined the funkload and zope.testing integration. I also added a zc.recipe.testrunner clone to support adding the funkload test runner to a buildout. My real focus on the last day and on the flight home, however, was trying to work towards buildbot integration. I started by adding a build-diffs console script that uses "fl-build-report --diff" to generates differential reports comparing the most recent benchmark report against the previous report and against any available reports from a day, a week, a month and a year ago. The goal is to use this script in a buildbot setup, then to check the diffs generated to raise an error or a warning when performance appears to be negatively impacted by a change. Buildbot could then also be a way to access the longer term (day, week, month, year) diffs for a more long-term view of performance changes.

This next step is the actual buildbot integration. All the funkload support is there and the buildout works as needed. Getting buildbot to detect status from the diffs and to provide links to the diff reports will require some support from a buildbot step. I began researching buildbot steps on the flights home, but haven't started work yet. Matt has a server ready and waiting when we finally do have a working buildbot setup so I'd really like to get this done.

Hopefully it won't be as long until I get to poke at the write concurrency issue and finishing the buildbot integration as it was until this post. :)

I think it's fair to say that Plone is a very successful open source project. On that point alone, it's an honor to serve a role in the continued success of the project. More than that, however, Plone has been successful for some time now. That success lies not just in the technology, but in the community as well, or if you prefer, the ecosystem.

What we have, then, is success that carries with it a history and a community that can't be abandoned without destroying that very success. That history and community also bring challenges with it. I myself find that my enjoyment of developing with Plone has kinda plateaued largely because I continue to have to deal with the code that is the legacy of that history.

The funny thing is that I don't find myself motivated as a crusader against that history wanting to rip it out wholesale. While this is a no-brainer in terms of the direction Plone development needs to take, I'm personally surprised by my motivations. I'm no longer so purely motivated as a developer. I'm genuinely excited to be a part of figuring out how to make Plone more fun and powerful for the whole community, users, themers, integrators, and developers alike.

How do we ditch old code without alienating large portions of the community when those portions of the community depend on their familiarity with the old code? How do we get developers excited about preserving continuity when we tend to prefer moving on? How do we make documentation sexy enough to attract and retain rock stars? How can we make preserving and improving performance as prestigious as adding new features?

I know I've changed because I can't imagine a more exciting set of questions to tackle. So yeah, I'm stoked to be a part of the FWT as we tackle exactly these issues in earnest. :)

I'm terribly behind on my blogging but I wanted to re-post this nice little tidbit from Planet Python for those who aren't on it already. The camelCase-mode for emacs allows you to do most word based operations over the separate words that make up a camel case word.

I've been needing something like this for some time. As an emacs junkie, I've gotten quite accustomed to being able to move point pretty much directly where I want it in just a few key strokes. Camel case words, however, can get quite long so correcting a typo or changing a plural in one of the constituent words can leave me reaching for my mouse of hammering away at C-f or C-b. Simply unacceptable! :)

So I downloaded camelCase.el to /usr/local/share/emacs/site-lisp and then added the following to my .emacs:

(require 'camelCase)
(add-hook 'find-file-hook 'camelCase-mode)

Ta da! CamelCaseGoodness!

I continued working with the load test benchmarking team yesterday. One of the more enjoyable aspects of our teams work is how natural and effective the division of labor has been. Tom and I worked on the funkload buildout and the Plone core load read-only tests. Andrew built on the read-only tests to produce a write-heavy load test. Ed and Russ have been working at least in part on different content profiles against which to run the different test scenarios.

Toward the end of the day, Tom and I moved onto working on making the funkload more generally usable to the wider Plone ecosystem. One of the first things I did after having a buildout that could run read-only load test benchmarks was to install and turn on CacheFu without a cache proxy. Then I ran the benchmarks again and had Funkload plot some pretty benchmark diff graphs. Tom started working on packaging this extension of the buildout as a sample so that add-on maintainers and integrators can see how to do the same for other add-ons. Then they can easily compare how their add-on affects base plone performance using funkload benchmark diffs. Funkload rocks! Meanwhile, I began work on making the funkload script invocations simpler and more familiar to those of us in the zope.testing world.

One goal here is to make load test benchmarking more accessible in general. Ideally, an integrator who is savvy enough to work with buildout, can use the collective.loadtesting buildout or extend it, record a new Funkload test using the recorder proxy. Then they can post the resulting test module and configuration with their problem report or question. Part of me shudders at the thought of encouraging broader access to benchmarking, especially since it's so easy to create unrepresentative benchmarks. I think, however, that drawing back the curtains on Plone performance to expose both the positive and the negative, even if messy, can be best in the end.

Meanwhile, the Andrew's write-heavy load tests reproduced the write-concurrency ZODB conflict bug that has recently been discussed on the lists. This is actually my big yin so I'm totally stoked to see some light being shed on this. The test scenario registers a new user, logs them in, goes to their member folder, adds a folder, adds a page to the new folder with lipsum field values, and logs out. The problems began to show themselves pretty heavily starting at about 5 concurrent users hitting one instance. After brainstorming with Lawrence, Andrew began generating load test diffs after experimenting with changes to try and isolate the write concurrency bug. First, Andrew looked into whether the response was being rendered before hitting a conflict error and thus being forced to render it again when it retries. The idea was that this could extend the duration of the transaction long enough to significantly increase conflicts. Archetypes does already, however, do a redirect after successful edit. We do have many more ideas to test out and now we have real measurements. The day ended with Andrew factoring out the member registration part of the test scenario to try and isolate the problem further.

Today Tom and I will likely focus on further polishing and documenting the buildout for release to the community. We'll probably also work on the buildbot configuration that Andreas provided. We want to package it to be run against Plone core development on a regular basis. It would be great to have a set of pages available to view the diff of performance for the last day of changes, the last week of changes, the last month of changes, etc..