Ross Patterson

Emacs tips: Navigate CamelCase - M-f, M-b, M-d, over camelCase!

ross — 2009-01-06T19:44:32Z

I'm terribly behind on my blogging but I wanted to re-post this nice little tidbit from Planet Python for those who aren't on it already. The camelCase-mode for emacs allows you to do most word based operations over the separate words that make up a camel case word.

I've been needing something like this for some time. As an emacs junkie, I've gotten quite accustomed to being able to move point pretty much directly where I want it in just a few key strokes. Camel case words, however, can get quite long so correcting a typo or changing a plural in one of the constituent words can leave me reaching for my mouse of hammering away at C-f or C-b. Simply unacceptable! :)

So I downloaded camelCase.el to /usr/local/share/emacs/site-lisp and then added the following to my .emacs:

(require 'camelCase)
(add-hook 'find-file-hook 'camelCase-mode)

Ta da! CamelCaseGoodness!

Load Balancing Accessibility - More progress on load test benchmarking

ross — 2008-12-13T06:13:38Z

Yesterday was another great day at the Plone Performance Sprint in Bristol, UK.

I continued working with the load test benchmarking team yesterday. One of the more enjoyable aspects of our teams work is how natural and effective the division of labor has been. Tom and I worked on the funkload buildout and the Plone core load read-only tests. Andrew built on the read-only tests to produce a write-heavy load test. Ed and Russ have been working at least in part on different content profiles against which to run the different test scenarios.

Toward the end of the day, Tom and I moved onto working on making the funkload more generally usable to the wider Plone ecosystem. One of the first things I did after having a buildout that could run read-only load test benchmarks was to install and turn on CacheFu without a cache proxy. Then I ran the benchmarks again and had Funkload plot some pretty benchmark diff graphs. Tom started working on packaging this extension of the buildout as a sample so that add-on maintainers and integrators can see how to do the same for other add-ons. Then they can easily compare how their add-on affects base plone performance using funkload benchmark diffs. Funkload rocks! Meanwhile, I began work on making the funkload script invocations simpler and more familiar to those of us in the zope.testing world.

One goal here is to make load test benchmarking more accessible in general. Ideally, an integrator who is savvy enough to work with buildout, can use the collective.loadtesting buildout or extend it, record a new Funkload test using the recorder proxy. Then they can post the resulting test module and configuration with their problem report or question. Part of me shudders at the thought of encouraging broader access to benchmarking, especially since it's so easy to create unrepresentative benchmarks. I think, however, that drawing back the curtains on Plone performance to expose both the positive and the negative, even if messy, can be best in the end.

Meanwhile, the Andrew's write-heavy load tests reproduced the write-concurrency ZODB conflict bug that has recently been discussed on the lists. This is actually my big yin so I'm totally stoked to see some light being shed on this. The test scenario registers a new user, logs them in, goes to their member folder, adds a folder, adds a page to the new folder with lipsum field values, and logs out. The problems began to show themselves pretty heavily starting at about 5 concurrent users hitting one instance. After brainstorming with Lawrence, Andrew began generating load test diffs after experimenting with changes to try and isolate the write concurrency bug. First, Andrew looked into whether the response was being rendered before hitting a conflict error and thus being forced to render it again when it retries. The idea was that this could extend the duration of the transaction long enough to significantly increase conflicts. Archetypes does already, however, do a redirect after successful edit. We do have many more ideas to test out and now we have real measurements. The day ended with Andrew factoring out the member registration part of the test scenario to try and isolate the problem further.

Today Tom and I will likely focus on further polishing and documenting the buildout for release to the community. We'll probably also work on the buildbot configuration that Andreas provided. We want to package it to be run against Plone core development on a regular basis. It would be great to have a set of pages available to view the diff of performance for the last day of changes, the last week of changes, the last month of changes, etc..

At the Plone Performance Sprint - In Bristol helping make Plone go faster

ross — 2008-12-12T08:19:06Z

I'm at the Plone Performance Sprint in Bristol, UK and I'm having a total blast.

First and foremost, it's a great group. I'm just having too much fun working in person with all these people I've heretofore only known through the interwebs. The topics brainstorming session yielded a lot of great ideas and potential directions. I kinda resent having to choose between the instrumentation and load testing topics. :)

Florian has a lot of nifty ideas about instrumenting various levels of the Plone stack to get meaningful performance data. This is sorely needed. Theory and guessing in discussions about Plone performance is all well and good, but as we all know, measure, don't guess. Florian's instrumentation effort stands to get us good measurements of things ranging from pickle retrieval on the ZODB level all the way up to viewlet rendering time in the UI. I'm definitely looking forward to using whatever they produce. I can't say much right now, but hopefully in the near future we'll all be hearing from Mr. Bent. :)

In the end I've decided to go with the load testing topic. I've been wanting good baseline metrics for Plone performance for some time. Every now and then, a Plone rock star does some profiling and finds some code and applies a two line change that increases performance by some ridiculous factor. While certainly not the rockstar's fault and not to denigrate the rockstar's contribution, but this should never happen. Something should have alerted us that a hotspot was introduced very shortly after it was introduced. Our hope is that having a basic set of load tests run by buildbot, we'll know when changes are made that impact performance. There are other goals you can read about on the wiki, but this is my primary goal. I hope to be a part of making work on Plone performance boringly predictable. Lets take the mystery out of it. :)

After the brainstorming and topic selection and such, we got a bit of a start on the load testing story. The first question was which tool to use, for which there were basically only two contenders: JMeter, and Funkload. I started out advocating for JMeter. I'd had a brief exposure to Funkload and had a bad experience with it, though I can't remember why any more. I built a very intricate load test suite with JMeter after that. It did everything I needed it to do, and the capacity to slice and dice the reports and graphs using the UI is great, but everything else sucks. The UI sucks. Using regexps for the things JMeter uses them for sucks. Using Java sucks. Still I advocated for it because it does what it says it will do quite admirably.

At this sprint, many were also under the impression that we should use JMeter, but there are also a handful of Funkload lovers. Through the subsequent discussion and experimentation, I think most, if not all, of us in the JMeter camp have been thoroughly converted. Now that I understand Funkload better, I see that I give up nothing I really need and I gain... well, Python!

One idea I'm not sure I'll have time to explore is integrating testbrowser and funkload. If I could make that work then I can write testbrowser doctests that can be run as load tests with full reporting options! /me swoons

Today we'll be getting started with the actual load tests. Good times!

New membrane and remember Maintainer - Rob Miller announced today that I'll be the new maintainer

ross — 2008-12-03T02:00:54Z

In a post to the remember list today, Rob Miller announced what we've been discussing for a while, I'm now the new maintainer for membrane and remember. I was stoked when he asked and since then we've had some great discussions so I'm even more stoked now. :)

I've subsequently posted a survey in hopes of getting a sense of what people are using membrane and remember for and what direction they'd like to see them take. It should only take a few seconds to respond to so please do if you have any interest in membrane or remember at all.

I'm looking forward to working with Rob and to help keep membrane and remember moving forward!

Evaluating Add-Ons - What is the risk of adding a given dependency?

ross — 2008-12-01T19:52:46Z

As often happens, a client asked me about adding a given package to their buildout. I just realized that I have a standard response to this so that might be something worth documenting for the wider community of admins and integrators. My criteria are as follows roughly in order of priority:

Good backing:

Find the author or maintainer on the plone.org/products page or on PyPI. Are they a company or individual that is widely known in the community?

Recent versions supported:

Does the add-on support recent versions of its own dependencies? For a Plone package today this means checking if it support Plone 3.

Final release:

Does the package have a final release that supports recent versions of its dependencies?

Released as an egg on a package index:

Is the add-on packaged as an egg and available on an easy_install accessible index such as PyPI?

Test coverage should, of course, be criteria #1. Sadly, we don't have a good enough test isolation story with ZTC and PTC, test failures in the client's buildout don't necessarily mean the add-on is broken. I know there's been talk about integrating test coverage data into plone.org/products but it's not here yet. The holy grail would be some sort of integration with distutils/setuptools. It would be great if I could do "python setup.py test --coverage" so that a subsequent "python setup.py register sdist upload" would incorporate the coverage data into the release meta-data such that it would be reported on the PyPI page. Certainly any such solution could be circumvented by the releaser, but I think that would be very rare, that it would more often encourage more maintainers to have good test coverage and this would be a huge win overall.

It's also interesting to note that I find recent version support more indicative of risk than final release status. This comes from experience. I've worked on too many projects of all sizes that had production sites running release candidates, betas, alphas, and even some development versions. One conclusion would be that this means that the software is of poor quality. I don't think that's it. I think it's that we, as a software ecosystem, are bad at releasing. Now eggs are certainly helping with this, but I think we also need to start an aggressive honor/shame (carrot/stick) campaign about releasing. Actually, maybe we should have an honor/shame campaign about add-ons in general. Perhaps a montly post to planet plone and the mailing list with a Hall of Fame and a Hall of Shame? :) Something that considers release status (how long has it been since that beta release? 3 months? Maybe you can release a final version?), test coverage, and meta-data cleanliness, etc.

collective.securitycleanup - GenericSetup handlers to restore Zope security to defaults

ross — 2008-12-01T18:25:30Z

WARNING: Backup your ZODB before using this package!

The Zope 2 security framework is very powerful and one of it's greatest strengths. A lot of it's power comes from it's flexibility. Exposing that power to site adminsitrators often ends up giving them enough rope to hang themselves with. This is exactly what the "Security" tab in the ZMI does.

In many cases, a site admin or consultant is faced with the daunting task of restoring all the security settings throughout the Zope object heirarchy in order to bring sanity and predictability back to the site. The collective.securitycleanup package provides GenericSetup handlers for restoring the role mappings and local roles back to their defaults. This handler can be used in combination with existing handlers to set role mappings and to re-apply workflow security settings to help start the process of security cleanup.

The clean up is performed on all ancestors including the Zope application root and by walking down the heirarchy to all descendants. This means all descendents of the context the handler is used on and all ancestors of the context including the root will be cleaned up. It will not clean up siblings or anything else that is not a direct ancestor to the context.

The clean up removes all permission settings stored on the instance which effectively restores them to code defaults. The clean up also removes all local roles except the "Owner" role for the user returned by OFS.interfasces.IOwned.getOwnerTuple() if already assigned.

Use of this tool will likely only ever be a starting point. So be sure to test thoroughly before deploying to your production server and backup your ZODB before using it.

Balancing DRY and Readability - Some complexity belongs in your editor/IDE

ross — 2008-11-25T15:13:19Z

I was writing a combination of setuphandlers and functional test fixtures recently when I found myself doing something noteworthy. I was converting both apparatus from a set of loops over content type names and prefixes to a few utility functions with lots of repetitious calls. Why was I violating DRY? Because I couldn't make sense of things! Oh yeah, DRY should never compromise readability. :)

It's definitely possible to take DRY too far leading to such slap-your-forehead moments as this. I think that setup code, whether for install or a test fixture, is particularly prone to this. I've both written a lot of setup code and read a lot of it in the wild where a bunch of complex structure is introduced in the name of DRY.

What I ended up doing was writing a small number of utility functions whose call signatures were intuitive and readable. Then I used a bunch of complex "M-x query-replace-regexp" commands in Emacs to convert the complicated structure into a simple, readable, flat, but nonetheless very repetitious function calls. Later when I needed additional set up, I used the same complex editor command to accomplish the task.

It occurred to me that what I was doing was taking unreadable complexity out of code where I, let alone someone else, couldn't remember or read what was going on, and i was moving it into my tools. My tools I use every day so I remember how to manage the complexity and I'm not condemning others to learn the complexity when I put it there. I think this makes a simple guideline for balancing DRY with readability. Some complexity belongs in your tool usage, not in the code.

Community Introspection - Same risks and rewards as for the individual

ross — 2008-11-22T03:15:54Z

One of the things I love about being a member of the Plone community, is our capacity for introspection. Like all things, however, it has its place and its limits. I've been contemplating how we might find those limits.

At the recent World Plone Day, I had the opportunity to hear Donna Snow's story in more detail. She was one of the "old school" skinners who was hurt by the Plone 3 changes. She has been assimilated. :) She's now a stronger advocate than ever. That's a great example of how the Plone communities' introspection is an asset. We didn't flinch, we incorporated.

OTOH, sometimes I think our community could have a bit more patience with the significant though perhaps unsexy transitions that the Plone stack is in the middle of. We're no longer the new kid on the block and so we lose some of our sex appeal. We're now a successful project whose greatest challenge now may well be to gracefully manage the history of that success while we continue to improve.

It may not be so scintillating to participate in keeping a successful project nimble as it was to participate in the rise of a shining new success. I know, however, that I at least will be very pleased with my time if I'm now a part of the subtler success of standing up well to new challenges and challengers.

I very much value our community's capacity to make constructive use of criticism and to be critical of ourselves. I just think it would help morale to also raise enthusiasm while we're doing so.

Nonprofit Software Development Summit - 3 brilliant days with cracked out LAMP folks

ross — 2008-11-21T03:45:34Z

I got back from the 2008 Nonprofit Software Development Summit in Oakland, CA last night. I had a total blast. It was simultaneously intense and relaxed which is an impressive achievement. It was also very surprising for someone who doesn't expect a tech conference to be intense. I generally pride myself on endless energy. The last day of the conference I didn't even have the energy for the final round of libations. I drove straight home and collapsed. :)

I don't want to offend other conferences or formats, but I hadn't realized how much I don't enjoy the traditional conference format. Now that I've experienced such an event where I was genuinely and organically engaged for 80-90% of the time, I recognize that I have always been very bored at other conferences. Nate tells me that bar camps or other events with a tendency to use the term "camp" have a similar format. Needless to say, I'll be keeping an eye out for such events.

This event didn't allow laptops at any of the sessions or main gatherings, and I didn't even miss it. I'd be dead of boredom at a traditional conference if my laptop were forbidden. I certainly didn't miss PowerPoint, which was also forbidden. This was also the first conference I presented at. It occurs to me how miserable it would have been to present to a room of downward facing eyes and tippity-tappity keyboards. When I thought about it further, I realized it's similarly miserable to sit in a session dealing with the conflict between boredom and the genuine desire to be engaged for both the presenter's sake and mine. The way this event handled this conflict was to limit presenters to 5-10 minutes of intro time after which it was all group discussion. It was very enjoyable and engaging both as an attendee and as a presenter. I assume all this is obvious to those who already enjoy this format. :)

I gave two presentation. The first was a contemplative session on Risks and Opportunities of Cloud Computing. I wanted to see how much traction my thoughts would get when it comes to adopting CC or PAAS while also building in structure to protect code, data, and vulnerable clients such as non-profit organizations. I found that most are still thinking about the question in terms of either "get over it and get on board" or "you'll get my app into the cloud when you pry it out of my cold, dead hands" or "you'll never find my app or data stuffed in my mattress". I want both protection from the risks and I want to take advantage of the opportunities. My hope is that open source frameworks that provide abstractions of core cloud services along with additional libraries offering solutions to common needs may be able to realize cloud apps that can be feasibly ported to other providers. These frameworks could then be a fulcrum for leverage to keep providers honest. This idea gained next to no traction with my attendees who wanted to go in other directions. This, in and of itself, is informative. If I'm wrong about using frameworks, or if I'm right and few are ready to move in that direction, either way there's a lot of work ahead.

For my second presentation I thought I'd play to my strengths: Best Practices for Plone Performance. Of course, I proposed this talk having never attended this conference before. The attendees of this conference were, I'd guess, 90% LAMP people which means the CMS world was dominated by Drupal. Joomla was a distant runner up. There were also a fair number of Ruby, and Rails, people there. The only people I met using Python, Zope, or Plone were Nate Aune and the good people from The Open Planning Project. As such, this talk wasn't overly attended. :) Just the same, a potential new Plone adopter was there and it was interesting to compare scaling wisdom between camps. From what I could glean, Plone has a much better scaling story in many senses. It's also worthy of note that nearly every time I mentioned Plone, the person I was talking to would mention ONE/Northwest. They have impressive recognition in the community.

Speed Geeking is a great for demonstrating technology and evangelizing or advocating all at once. This format allows a presenter to convey their enthusiasm for a project where the audience is consumers while eliminating much of the boredom that typically occurs when an audience is compelled to listen to one person talk. I found myself wishing that I could cover the very rich Plone feature set in such a limited 3-4 minute format. If I've noted one thing about what impresses people about Plone, it's how much they get OOTB. If you can get someone to pay attention long enough to cover the feature set, they're almost always surprised. I can't count the times I've heard, "I had no idea Plone could do so much." Unfortunately, you can't demo all these features in 3-4 minutes. I was thinking it would be good to have a slide show amongst the Plone marketing materials that could cover "What Plone can do" in 3-4 minutes. Anyone interested in working on that with me?

The LAMP people at the conference were all very open and there was a very pleasant absence of evangelism. I even noted that as the conference went on, everyone seemed to make a point of adding Plone to their list of token CMSes when speaking as they became more aware of our presence. All told, I think it was very productive for all sides and so much of that is due to the community of the event. Everyone there had gobs of energy, was genuinely open and friendly, and everyone seemed to be passionately committed to exciting and productive projects. I started as an attendee, came to see myself as a guest, and left feeling as a member. Quite an achievement indeed.

I'll blog more later about the sessions I attended.

Running Tests in Egg Buildouts - A quick note about a gotcha with plone.recipe.zope2instance.ctl

ross — 2008-11-15T04:25:58Z

Recently I made an egg of mine into a buildout for self-contained testing and I ran into a gotcha that took me too much time to figure out, so I thought I'd document it for others.

The plone.recipe.zope2instance.ctl test runner excludes the buildout root from the test paths. It gives the reason in the comments:

# XXX Somehow the buildout root gets on the sys.path. This causes
# weird problems, as packages like src.plone.portlets.plone.portlets
# are found by the testrunner

Since a ZopeSkel generated egg doesn't create a "src" directory to contain the actuall python packages, the egg root would need to be on the test path. When the egg is made into a buildout then all these factors collide to make the egg I want to test invisible to the test runner.

I worked around this by moving the actual python packages to a "src" directory. I always thought this was more proper anyways. Maybe ZopeSkel should be changed. At any rate, here's the setup.py additions that were necessary:

packages=find_packages('src', exclude=['ez_setup']),
package_dir = {'':'src'},

Hope this saves some trouble!

collective.catalogexport - Use ZCatalogs as export sources

ross — 2008-11-15T04:36:58Z

The data contained in tabular form in ZCatalogs is often exactly the data site admins frequently want to export into some other format, such as CSV. This package provides views for exporting the catalog data into various formats.

Currently, only exporting the whole catalog with all metadata/brains/columns as CSV is supported. I plan to add support for submiting arbitrary index queries and for controlling the metadata/brains/columns exported.

collective.slideshowfolder - Somtimes useful extensions to Products.slideshowfolder

ross — 2008-11-13T06:27:28Z

The SlideshowImage content type uses a reference to an existing normal image somewhere else in the site to act as a kind of link or alias. This allows for the creation of a folder as a slideshowfolder that displays images that are actually stored elsewhere.

collective.redirect - Administer redirects to internal or external URLs using Link like content

ross — 2008-11-12T04:55:41Z

Administer redirects to internal or external URLs using Link like content. Where Products.redirectiontool or plone.app.redirector only deal with redirecting to internal URLs internal to the portal, collective.redirect allows for redirecting to external URLs. The paths to redirect are administered using instances of the Redirect content type. The paths that are redirected are independent of the path of the Redirect instance for a couple of reasons.

Firstly, since the portal object is not a BTree based folder it will begin to behave poorly if too many objects are added to it. Allowing the redirected paths independent from the location of the Redirect instances allows for many redirects without putting too many objects in the portal root.

Secondly, having the paths independent of the Redirect instance locations allows users to create redirect for paths that they can't add content too. Keep in mind that this might be a bad thing for your site and can certainly be abused as a DOS attack of sorts.

If multiple redirects exist for the same path, the one with the more recent publication date will be preferred. Finally a redirect will never override an otherwise traversable URL. IOW, a redirect cannot override an actual content object, skin object, view, or anything else traversal. The redirect only occurs when a NotFound error would otherwise be raised.

My Borrowed World Plone Day - How I crashed Donna Snow's presentation

ross — 2008-11-07T22:05:55Z

I'm spending my World Plone Day at Donna Snow's presentations at Sunnyvale, CA. It's been a blast.

Donna started out describing her participation in the Plone community and her company, C Squared Enterprises (C2E). Then she moved onto a slideshow with a good overview of what Plone is and some of the best reasons for adopting it. Next she gave a live demonstration of a Plone 3.1.6 site and managed to convey a lot of the very feature rich set that Plone brings out of the box keeping everyone engaged the whole while. An excellent presentation.

Donna graciously invited me to make some contributions to the presentation from a developer's perspective. It gave me the chance to talk about some of the technical underpinnings of the Plone stack I most admire. The internationalization and translation capabilities seemed to get a lot of favor, big surprise there. :) I listed WSGI and Deliverance theming as some of the more likely future directions for Plone and was surprised how much interest that got. I definitely enjoyed participating.

Here are some of the questions asked:

What other CMS are there?
What is Plone's market share or adoption rate? Is it growing or shrinking? How does it compare with other CMS?
How is translation done?
Are live backups possible? Can they be done TTW?
Can we see the developers view of a template?
I heard Plone is slow?
What's this Zope 2 and Zope 3 stuff?
Can you make the calendar portlet start the week with Sunday?
What is the granularity of the security framework?
How can I integrate external database queries with Plone?

It was a first for me sitting in a room fielding questions from people who were doing their initial evaluation of Plone. While none of the questions were surprising, it was nonetheless very informative.

It was very generous of Donna and C2E to pay for the space and the materials and to donate all the time. I think it was worth it. Best quote of the day?

I'm a convert.

It looks like the second session is turning out to be more of an installfest. :)

Cloud Busting - What does it mean to be a good cloud citizen?

ross — 2008-10-23T05:43:52Z

I know I'm not alone, but I've been thinking a lot recently about what the cloud means for rich web applications like CMS's such as Plone.

Firstly, I find Google App Engine (GAE) much more inspiring than Amazon's Elastic Compute Cloud (EC2) when it comes to thinking about cloud computing. Kinda funny considering that EC2 actually has "cloud" in its name while GAE doesn't.

It seems to me that EC2 isn't really a cloud. For my money, EC2 is just a really big virtual hosting provider that has done a great job of limiting hosting costs by bringing efficiency to client relationships and expectations. They do this by simultaneously limiting their responsibility and clearly defining their remaining responsibility in accessible APIs. They exploit a very interesting and relatively new technology sweet spot to do this, virtualization. This has the potential to become the *foundation* for cloud computing,but is not cloud computing in itself. The strategy seems to be to provide the foundation, open up the technology race, and wait and see who succeeds in building good clouds on that foundation. I kinda like this strategy, but I do see some problems with it. Of course, mostly,I just don't like thinking about hardware, virtualized or no. :)

Firstly, computing in the cloud is definitely going to require changes from the code that operates in the cloud. EC2 enables application builders to construct large distributed web applications without ever making their code into good cloud citizens. That my not be a good thing. Similarly, various parties are building true cloud frameworks on top of EC2. It may be that an inferior cloud framework places fewer requirements on the hosted code, thus gaining more adopters,thus starving better cloud frameworks.

GAE seems to come at the cloud from the other direction. Rather than providing all options on something that isn't yet a cloud as EC2 does,GAE provides a true cloud but with very limited options. GAE too exploits a technology sweet spot, but a very mature and well established one, Python. The strategy there would seem to be to open up the technology race in the other direction inviting framework/application developers and GAE to come together over time. That is, framework/application developers will make their code better cloud citizens guided by the restrictions of a true cloud and GAE will become richer and less restrictive as Google sees where to put its resources in terms of extensions or flexibility. I've heard GAE called a platform for toy applications. It seems to me it's not so much a toy platform as a very serious platform in its adolescence.

Now, what about large, feature rich applications like CMS's such as Plone? The ideal large, feature rich application still has a lot of code, the code that implements and integrates all those features. As such, I'm going to set aside the question of whether or not a given large, feature rich application is *too* large, just for the moment. Such waste definitely becomes even more painful in the cloud and good cloud citizens should definitely continually evaluate their code waste even more vigilantly, it's just not what I want to consider right now.

I think specialization is going to be one of the most important aspects of cloud computing. In order for a cloud to be flexible and transparent while simultaneously achieving responsiveness and efficiency, applications are going to have to cooperate with the cloud so that requests can be handled by portions of the cloud that are already more or less familiar with handling those requests. Good cloud citizens will have to factor themselves such a way as to support this. Once code running on the cloud has factored itself into specializable requests, the cloud can be left to do the specialization as they see fit. The cloud provider is the party most interested to squeeze the most possible efficiency out of the cloud so the implementation of specialization should really be left to them. Of course, once again, mostly I just don't want to think about specialization, too close to the dreaded hardware. :)

AJAX to the rescue! It seems like AJAX first burst onto the scene to improve client side performance. If you replaced a portion of the DOM with the results of an XMLHttpRequest, you could avoid the page reload time cost incurred by the *browser* and the cost of the user experience disjoint incurred when their browser renders a new page. Using AJAX to compose a page from multiple requests is, however,potentially *more* valuable to specialize things on the *server* side. The win here is not so much the AJAX on the client side as is the capacity to break a page up into multiple specializable requests for the server side.

Specialization becomes most important with large, feature rich applications like CMS's. Without specialization, the application eventually requires too much in the way of resources to handle any given request and becomes a very bad cloud citizen.

Effective specialization also requires that applications be frugal about how much of their code needs to be loaded to serve a given request. This cuts down on memory consumption but perhaps more importantly, it cuts down on startup time when an portion of the cloud that is not yet specialized is called upon to serve a new request. It seems like any large application worth it's salt uses some sort of modular registry (plugins, components, whatever). I think it's vital that any such registry that hopes to be a good cloud citizen shouldn't load the code behind any given lookup until the lookup first occurs.

So lets talk about Plone. At the moment, Plone has a lot of work to do before it becomes a good cloud citizen, but it seems like it will be possible to get there. I will be blogging about this in more detail in the near future, but here's a sketch.

I wrote a small test that ran against an *existing* Plone trunk site and simply loaded the front page in a testbrowser. I confirmed that zope.testing coverage reporting does not report on modules that aren't imported as a part of the test run. Then I ran that test with a coverage report to get a sense of how much of the code that is imported for Plone startup is actually used to serve that page. Finally, I removed the coverage numbers for all Python standard modules and any third party dependencies in order to get a sense of the import waste in the Zope/CMF/Plone stack:

lines	cov%	lines uncovered	lines covered
167,752	40.00%	100,643.21	67,108.79

Firstly, 67k lines of code to serve up the front page seems like too much, but we're aware of that. The limbo between the more monolithic Zope 2 and the much more elegant successor Zope 3 is probably a significant part of that. I'm going to do some more detailed examination of the coverage report, but here is the CSV of the simple coverage numbers for those interested.

The thing I find most interesting is how much is imported code isn't used. I've been thinking about this for a while. While some of this is certainly import waste from unused portions of the more monolithic Zope 2 and Archetypes stacks, I think much of it may also come from ZCML and the Zope Component Architecture.

The ZCA gives us a wonderful registry for a very modular architecture but using the ZCA requires importing all the code for everything registered. Some significant advantage might be had by using a ZCA registry that didn't import the code until the first time that given lookup is performed.

There is an interesting exception, local site managers. Since local site managers store their registry in pickles, it may be that the code behind the component isn't loaded until the lookup unpickles the registration. I haven't confirmed this, but even if this isn't the case and the whole registry is unpickled at once, it might not be too hard to make each registration its own pickle. Marius mentioned an effort to reduce ZCML load time by pickling the configuration actions. Maybe it's possible to direct the ZCML actions at a local site manager instead which could then be persisted and used as the global site manager. Then loading the ZCML at startup would be unnecessary. In essence, this would mean *compiling* ZCML into a site manager pickle before deploying the applictaion to the cloud. This notion of compiling configuration for deployment brushing shoulders with pickles has inspired some other interesting notions I hope to blog more about soon.

As for AJAX, Viewlets seem like an ideal framework to exploit for AJAX composed request specialization. I might be feasible to replace viewlet renderers with ones that insert JavaScript into the containing page which will load the viewlets in separate, specializable requests. Someone's already tried it.

As for Plone on GAE, there are other, possibly larger hurdles, not the least of which is Zope's use of C extensions. I know work has already been done to lessen dependency on Acquisition wrappers, but I don't know how close that brings us to eliminating the C extensions there. As for persistent, I don't know if what is done in persistent.cPersistence can be done in python but I have heard noises that the effort to port ZODB to Jython may yield fruit here. It also seems like it might be possible to use some of the ORM libraries out there to implement a python only version of BTrees that uses tables on the backend. At any rate, all of this really is pie in the sky at this point, but I have a 10% Pie in the Sky Manifesto and an awful lot of good things have started with such thinking. I'll be blogging on this more later.

At this point I'm more interested in what computing in the cloud means in the more general senses detailed above but I'd love to hear any thoughts on any of this.