Ross Patterson

Testbrowser Redirects with Fragments - Using zope.testbrowser when a redirect URL includes a fragment

ross — 2008-08-09T00:33:59Z

I was trying to add functional test coverage for commenting on content when I kept getting 404 NotFound errors. I tried it in a real browser and everything worked just fine. After a little googling, I found an answer in a post on the zope3-dev mailing list:

Note that the fragment identifier is *never* going to be passed to the server by a "real" browser: instead, the browser strips of the fragment part, submits the remainder of the URL to the server, and then does a search for the appropriate '<a name="bar">' element *on the client side*.

I'm not sure how this is pertinent to the problem you have found, but Zope will never traverse '#bar' or '%23bar' in the real world.

It seems like maybe testbrowser should not send fragments to the underlying mechanize browser. But failing that, for now I just add a note to my tests about why we expect a 404, NotFound error in our tests that we won't see in a real browser.

Hope this saves someone else the search time. :)

Re-using SSH Connections - Automatic master SSH connections for sharing and multiplexing

ross — 2008-07-26T07:25:56Z

Yet again I've discovered that somethign I've been dreaming of forever already existed for just about that long. :)

I use SSH for so many things these days that the startup time for connections really begins to add up. Enter OpenSSH connection sharing. See that blog post for details. Also see this thread on the SVN development list for details of a workaround for an svn+ssh issue.

I was still left wanting, however. The lion's share of the time I want to save re-using SSH connections is for svn+ssh. The SVN workaround requires that I think enough ahead to start the master for every host I connect to. I connect to a lot of hosts and they're constantly changing so I wanted something I didn't have to think about. Also mildly annoying is the fact that the first SSH session to a host would become the master and couldn't be closed or killed without killing all slaves and, of course, killing the master.

I wrote a wrapper for SSH that would start the master in the background if need be and then run the ssh command as a normal slave. This solution allows the individual connection to close or be killed without affecting the master. It also obviates the need for the svn+ssh workaround. Enjoy:

#!/bin/sh
## Wrapper script for ssh that backgrounds the control master if one
## isn't already started.

## the real ssh
SSH="/usr/bin/ssh"

## optstring assembled from `man ssh`
optstring="+1246AaCfgKkMNnqsTtVvXxYb:c:D:e:F:i:L:l:m:O:o:p:R:S:w:"
## get ssh options
opts=`getopt -- "$optstring" "$@"`

## the non-option args follow "--"
## convert to an array of IFS separated strings
args=(${opts#*--})

## the host is the first non-option arg
## use eval to process the quoted strings returned by getopt
host=`eval echo ${args[0]}`

## if the master isn't running, start it in the background
$SSH -q -O check $host 2>/dev/null || $SSH -MNf $host

## replace ourselves with the reall ssh call
exec $SSH "$@"

Spaces in User Names and psvn.el - Handling SVN user names with spaces in emacs' svn-status.

ross — 2008-07-25T17:43:50Z

I recently started on a project using svnserve's --tunnel-user option where many of the committers have user names with spaces. Much to my chagrin, my beloved svn-status in emacs choked on those user names un able to tell where the user name ended and the filename began.

Then I found a comment in the current version of psvn.el detailing how to work around this. Just be sure you have the current version on your load-path and add something like the following to your ~/.emacs:

(setq svn-user-names-including-blanks '(\"feng shui\" \"mister blank\"))
(add-hook 'svn-pre-parse-status-hook 'svn-status-parse-fixup-user-names-including-blanks)

Adding testbrowser Form Inputs - Mock dynamic form inputs with zope.testbrowser

ross — 2008-06-01T23:02:14Z

When writing functional tests that cover forms with elements that are dynamically manipulated from JavaScript, such as the ReferenceBrowserWidget, I often simply didn't cover those elements becuase it simply wasn't worth it to add Selenium or zc.testbrowser.real as a testing dependency for what are usually very small pieces of what is being tested.

What I've always wanted was a way to reproduce the effects of the dynamic manipulation in the testbrowser. When I was recently bitten by a bug that I didn't catch for lack of functional coverage of a ReferenceField, I dug into the ClientForm package that zope.testbrowser uses and found the HTMLForm.new_control() method. This is the method that ClientForm uses to construct the controls that zope.testbrowser provides lookup for.

Since Firefox allows you to inspect the DOM either by selecting the elements and using "View selection source" or with Web Developer's DOM inspector, you can ferret out what the dynamic manipulation created and use that information to construct the HTMLForm.new_control() call that will reproduce that. The following example would add the foo object to the related items on another object's edit form.

>>> form = browser.getForm(name="edit_form")
>>> form.mech_form.new_control(
...     type='checkbox',
...     name='relatedItems:list',
...     attrs=dict(checked='checked', value=foo.UID()))

See the ClientForm source for more details.

BTW, it seems the the ReferenceBrowserWidget includes an empty string on submission of the form. This empty string case seems to be handled in the field's set() method and not in the widget where it seems like it should be. Not knowing this, I implemented a custom mutator with a bug that I didn't catch because I didn't have test coverage on the form submission. Now I do. :)

Aerobie AeroPress - Who knew the best coffee maker would come from a toy company?

ross — 2008-05-25T03:07:13Z

I resisted it for some time. Becoming a coffee snob, that is. But it's happened and it's time I acknowledged it.

One of my favorite baristas made the most peculiar recommendation for a coffee maker for home, the Aerobie Aeropress. He warned me not to judge the book by its cover and he was right. The only things that make better coffee are ~$10k espresso machines or the ~$10k Clover. The AeroPress is only ~$35 and I don't need a barista's skill to make great coffee at home. Wow, I sound like an ad. Ok, but it's worth it just this once. I keep finding myself wanting to pack it when I travel. It's rediculous.

Seriously, if you like coffee at all, try it. Pay no attention to the cheesy marketing, neither theirs nor mine. :)

Buildout - One Zope2 Build - I can share one Zope2 build between all buildouts

ross — 2008-05-25T01:33:56Z

It's possible to share one Zope2 buildout between buildouts without modifying the buildouts' config files. It's documented on the PyPI page of plone.recipe.zope2instance.

Updated: Somehow I'd missed this for sometime. I think I was looking at the docs for an old egg.

collective.formcriteria - User enterable search criteria forms for collections/topic/smart folders.

ross — 2008-06-30T04:59:45Z

This package provides new criteria types based on the
ATContentTypes.criteria types that are used to create a form at the
top of the topic view. Users can use the form to submit criteria to
supplement any search criteria in the topic. Values entered on the
criteria tab for the topic become the default values on the form.

Also provided is an alternative display layout that uses the folder
contents table and can still display the search form viewlet.

See the PyPI page for more details.

collective.testcaselayer - Use test cases as zope.testing layers

ross — 2008-05-24T06:22:41Z

The support for layers provided by zope.testing helps to lessen the amount of time consumed during test driven development by sharing expensive test fixtures, such as is often requires for functional test. This package provides several well tested facilities to make writing and using layers faster and easier.

Layer authors often end up reproducing the functionality provided by their test case classes since the same functionality is needed to perform layer set up or tear down. The collective.testcaselayer.ztc, collective.testcaselayer.ctc, and collective.testcaselayer.ptc modules provide layer base classes that mix in the test case functionality from ZopeTestCase, CMFTestCase, and PloneTestCase, respectively. See the collective.testcaselayer.ztc, and collective.testcaselayer.ptc sections below (or ztc.txt and ptc.txt if reading this in the source) for more details. These layer base classes also include the layer base class support from collective.testcaselayer.layer and the sandboxed ZODB layer support from collective.testcaselayer.sandbox described below. Additionally, these modules allow for using the test case fixtures as layers themselves.

While class objects can be used as layers, as opposed to instances of classes, doing so means that it is not possible for a layer to subclass another layer *just* to re-use functionality without also depending on that layer being set up as well. See the collective.testcaselayer.layer section below (or layer.txt if reading this in the source) for more details.

The DemoStorage included with the ZODB provides a way to "nest" ZODB stores such that all writes will go to the DemoStorage while reads will be taken from the base storage if not available from the DemoStorage. The collective.testcaselayer.sandbox module uses this feature to associate a DemoStorage with each sandboxed layer to which set up changes are committed and restore the base storage on tear down. Thus sibling layers that write to the ZODB can be isolated from each other. See the collective.testcaselayer.sandbox section below (or sandbox.txt if reading this in the source) for more details.

See the PyPI page for more details.

Products.PDBDebugMode Egg - I've eggified my post_mortem debugging tool

ross — 2008-05-16T01:01:03Z

Just a quick heads up that I've eggified Products.PDBDebugMode so it's a lot easier to use in buildouts. It can be as simple as:

[buildout]
eggs =
    ...
    Products.PDBDebugMode

If you want some isolation, then you can use zc.recipe.egg in a separate part:

[instance]
...
eggs = 
    ...
    Products.PDBDebugMode

Update: Simplified per Martin's comment.

Debugging with MockMailHost - Using the mock MailHost on a real instance

ross — 2008-05-12T18:16:52Z

I often find the need to debug user registration or other functionality that involves sending mail through the MailHost where I either don't want the mail to go out, or I want to intercept it for examination or both. Using "zopectl debug" I replace the MailHost with a MockMailHost as follows::

>>> app.Plone._delObject('MailHost')
>>> from Products.PasswordResetTool.tests import utils
>>> mail_host = utils.MockMailHost('MailHost')
>>> app.Plone._setObject('MailHost', mail_host)
'MailHost'
>>> app.Plone.MailHost
<MockMailHost at /Plone/MailHost>
>>> import transaction
>>> transaction.commit()

Then using PDBDebugMode or "zopectl debug" again, the messages sent through the MailHost can be examined as follows::

(Pdb) print self.MailHost.messages[-1]

I have long intended to build a package with a GenericSetup profile that will do this for me, but I wanted to get this out there for now. Enjoy!

rpatterson.mailsync - OfflineIMAP, mswatch, and Gnus integration for realtime mail

ross — 2008-06-30T05:00:24Z

Like many others I've been frustrated with the options that are available for email clients. I recently had a bit of break through and packaged the reusable parts of my mail setup for consumption by others. See the backstory for all the gory details of the hisotry. I've factored everything with an eye towards extensibility so if there's enough interest, I'll start a project and a public repo somewhere and put this out there. See the TODO list if your interested in any of the ideas I have for improvements.

To try it out, just see the README on the pypi page. Here's a brief description.

This package provides some scripts that wrap parts of mswatch and
OfflineIMAP and integrate with Gnus to provide a local maildir that
is synchronized with a remote maildir as changes occur, instead of
polling on a regular basis. This provides for near instant delivery
of new mail while also reducing resource utilization. Integration
with the Emacs mail and newsreader, Gnus, is also provided in such a
way that your single threaded Emacs process is blocked much less as
changes occur to the maildirs.

z3c.persistentfactory - Persistent instance method ZCA factories

ross — 2008-04-09T21:10:48Z

Homepage: http://pypi.python.org/pypi/z3c.persistentfactory

The ZCA and the ZODB are a good combination where components require persistent state. ZCA factories or handlers typically retrieve any persistent state required from the persistent objects being adapted. If the persistent state required is not specific to the objects being adapted, a common solution is to register a persistent utility which is then looked up in the factory or handler. The persistent utility approach requires, however, that the one appropriate utility is looked up which requires support in the ZCA registrations either in the interface provided or the utility name.

In some cases, however, it is more consistent with the object oriented semantics of Python and the ZCA to think of the factory or handler as an instance method of a persistent object. With this approach the non-context specific persistent state can be accessed on self.

Instance Method Event Handler

One example where this may be useful is where some non-context persistent state is tightly coupled to some event handlers in such a way where instance methods are better semantics.

The Baz class uses the decorator in the python code. Note that the factory decorator must come before the declaration decorators so that it will be run last and will reflect the declarations.

>>> from z3c.persistentfactory import testing
>>> baz = testing.Baz()

>>> from zope import component
>>> component.provideHandler(factory=baz.factory)

The method adapts IFoo, so create an object providing IFoo to be used as the event.

>>> component.adaptedBy(baz.factory)
(<InterfaceClass z3c.persistentfactory.testing.IFoo>,)

>>> from zope import interface
>>> foo = testing.Foo()
>>> interface.alsoProvides(foo, testing.IFoo)

When the event is notified, the method is called with the event as an argument.

>>> import zope.event
>>> zope.event.notify(foo)
Called <bound method Baz.factory of
<z3c.persistentfactory.testing.Baz object at ...>>
  args: (<z3c.persistentfactory.testing.Foo object at ...>,)
  kwargs: {}

Instance Method Adapter Factory

Another example is where an adapter factory needs to look up persistent state specific to the objects being adapted but where that state can't be stored on the adapted objects them selves. The component storing the shared persistent state can register one of it's instance methods as the adapter factory which will look up the necessary persistent state on self.

>>> component.provideAdapter(factory=baz.factory)

The method implements IBar.

>>> tuple(interface.implementedBy(baz.factory))
(<InterfaceClass z3c.persistentfactory.testing.IBar>,)

When the adapter is looked up, the metod is called with the object to be adapted as an argument.

>>> result = component.getAdapter(foo, testing.IBar)
Called <bound method Baz.factory of
<z3c.persistentfactory.testing.Baz object at ...>>
  args: (<z3c.persistentfactory.testing.Foo object at ...>,)
  kwargs: {}
>>> result
(<bound method Baz.factory of
 <z3c.persistentfactory.testing.Baz object at ...>>,
 (<z3c.persistentfactory.testing.Foo object at ...>,), {})

Hotel internet on Ubuntu - TCP window scaling and broken routers

ross — 2008-02-11T06:22:13Z

I've run into odd networking problems on certain networks, generally hotel networks but also some cafe networks. The problem afflicts both wireless and wired connections. I'm able to
get an IP address from the DHCP server most of the time, and I can even establish an SSH connection to my server, but all other traffic is hampered by continuous timeouts. Oh, and the really annoying thing is that it only affects Linux, not Windows or Mac.

In my most recent bout with this problem I finally found an article that describes the problem and a workaround. According to the article, the problem stems from broken routers that are mishandling a TCP option. See the article for the details and for the workaround.

There's one bit that troubles me:

If, instead, the situation is left as it is, pressure on the router manufacturers should get the problem fixed relatively quickly. It has been a few years, now, that Linux has a strong enough presence in the networking world that it can get away with taking this sort of position.

I see and agree with the point where Linux is a server OS. In this case, however, Linux is a desktop and laptop OS and having a setting that causes problems that Windows and Mac users don't encounter makes us look bad against the competition. It really doesn't matter if the fault is the routers. Maybe the TCP window scaling option should be disabled by Ubuntu for desktop installs?

Is python-openid 2.0.2 broken - an upgrade broke openid login

ross — 2008-02-11T06:22:13Z

I want to document my adventures getting openid login working on my Plone 3.0 site. My site is using the plone-site package from Debian unstable which uses the python-openid 2.0.2 package. When visitors reported openid login wasn't working, I started by trying to reproduce the problem on my Ubuntu laptop. I found that the problem didn't occur if version 2.0.0 of the python-openid was installed. This was true in both debian and ubuntu.

I worked around this issue on my server running Debian by installing the python-openid 2.0.0 package from Ubuntu Gutsy, which is not an approach for the faint of Debian heart and involves a little /etc/apt/preference and /etc/apt/sources.list hackery.

I don't know if the problem occurs with a vanilla python-openid from source because I can't find a distribution. I also didn't dig any deeper to see if the fault for this lies in Plone or python-openid.

At any rate, openid login and commenting is now available on this site. Now you know, and knowing is half the battle! :)

ZopeTestCase and packages - Please put ZopeTestCase out of my misery

ross — 2008-02-11T06:22:13Z

In this wonderful world between Zope 2 and 3, called limbo... I mean Five, there's a fun little trap concerning ZCML, packages as products and ZopeTestCase. I wanted to document it somewhere so here it is.

If you're using packages as products, then to install the package as a product for tests, you'll need the ZCML of the package to be loaded. If the package happens to be included in the configure.zcml of a product,then you'll need Five's loadProducts to have been run in order to install the package as a product. With ZopeTestCase, in order for a product to be available for Five's loadProducts, installProduct must be called before loadProducts is run. Five's loadProducts is usually run in the instance's site.zcml which for Plone tests is usually loaded by the ZCML layer.

The end result is that all installProduct calls must be made *before* loadProducts is run which in turn must be run before any installPackage calls can be made if you do any including of packages as products in product configure.zcml files.

Admittedly, this problem goes away if you don't include the ZCML for packages as products inside a product's ZCML.

If anyone is interested here's an example of using layers to work around this:

from Testing import ZopeTestCase
from Products.PloneTestCase import layer

ADDONS = [
    # Your product dependencies here
    ]

class InstallProducts(object):
    """Install products before the ZCML layer so that their ZCML will
    be loaded"""

    @classmethod
    def setUp(cls):
        for dep in ADDONS:
            if ZopeTestCase.hasProduct(dep):
                ZopeTestCase.installProduct(dep)
    
    @classmethod
    def tearDown(cls):
        pass

    @classmethod
    def testSetUp(cls):
        pass

    @classmethod
    def testTearDown(cls):
        pass

class InstallPackages(InstallProducts, layer.ZCML):
    """Install packages after the ZCML layer so that products will
    have had a chance to include the package ZCML"""

    @classmethod
    def setUp(cls):
        for dep in ADDONS:
            if ZopeTestCase.hasPackage(dep):
                ZopeTestCase.installPackage(dep)
    
    @classmethod
    def tearDown(cls):
        pass

    @classmethod
    def testSetUp(cls):
        pass

    @classmethod
    def testTearDown(cls):
        pass