Python Web Apps on IIS

Can Plone work better on Windows? I have a working proof-of-concept with Web Platform Installer, Web Matrix and IIS Express that may improve Plone’s installation story. Help me get it out the door.

Please don’t get me wrong, I hate Microsoft and Windows as much as the next OSS or Python web developer. I just feel it’s important to be honest with ourselves about why they’re bad. I grow a little tired of all the griping sessions whenever the sorry state of Plone’s Windows installer support is discussed. I think there’s plenty to gripe about and there’s plenty that Microsoft does poorly but lets also be honest with ourselves. If Microsoft didn’t do some things really well they wouldn’t be a problem for us and we wouldn’t even be discussing whether to support them as a platform.

At any rate, the discussions about the Windows installer at the 2012 Cioppino Sprint were as frustrating and disappointing as ever. When everyone was through griping, however, I couldn’t find anyone who at the end of that discussion thought we could afford strategically to drop the Windows installer or leave it to a 3rd party.

I’ve started putting Microsoft’s Web Platform Installer, Web Matrix, and IIS Express together for Plone 4.2 and Python 2.7 and have a proof-of-concept working on my Windows VM that actually gives us Open Source all the way up to IIS, see the screenshot above. Plone is actually really snappy running this way even on my very slow Windows VM.

Below I document what I’ve learned as I proceeded and my final findings. If you read nothing else please read the Help Needed! section and let me know if you or anyone else can help me get this to a place where others can try it out. I’ll be sprinting on this at the post-PSE sprints so anyone who has IIS or Windows installer experience, I’d love to sprint with you on this.

Help Needed!

Primarily, I need help with things that I can’t do with just Web Deploy and WebPI. For example, the way IIS does FastCGI means that a configuration change has to be made to the global IIS config for each FastCGI app. IOW, it can’t be done in the app/deployment-local web.config file. I know the shell command to run to make the change, but I don’t know how to package that properly so that it runs with escalated privileges when necessary.

Similarly, we need to figure out how to handle ZEO in the way that is closest to “correct” for IIS. Should we just install an autostart service? IIS have a database provisioning and control framework. Can that be adapted to also manage ZODB databases and control running ZEO processes? Or should we just wrap it up in such a way that ZEO is started the first, and only the first time, IIS launches the IIS process and shuts it down when stopping the app? If so, how?

Since we’re using WSGI via FastCGI using Flup, we’re dependent on Zope’s WSGI server. Unfortunately it lacks the publication hooks used by things like plone.app.theming and plone.app.caching. This is really a bug in the Zope2 WSGI publisher and as such affects all WSGI deployments, not just Windows. @Hanno and @davisagli think they’ll be able to get to this on in the next few days.

Finally, if I’m wrong about any of the technical stuff in here, I’d love to hear about it. The documentation is crap for all this stuff and it has been way too hard to figure it all out, so I’d love any leg up I can get.

Supporting Windows

One thing that came out of the discussion, that I thought was interesting, was the sentiment that if we were going to support Windows we shouldn’t do it while telling Windows users to take a leap at the same time. IOW, we say we support windows out of one side of our mouths, and blame users for choosing windows out the other side of our mouths when they encounter fundamental problems or find them selves encountering an unfamiliar learning curve. I see two things behind such problems: lack of integration with typical Windows tool chains, and differences in the documentation.

The differences in the documentation largely come from the fact that the current windows installer isn’t based off the unified installer so the buildout/package layout is different than what’s in most of our docs. To that end, I started looking into what it would take to build a Windows installer from the unified installer.

The lack of integration with typical Windows tool chains is a much bigger issue and much of that issue is a problem for most Python web applications, not just Plone. My research did, however, turn up some very promising new tools from Microsoft that we may be able to use to provide a better experience for Windows Plone developers and Windows Plone deployments.

There are also two major audiences that tend to make use of Plone’s Windows support: developers and deployments. Steve McMahon believes that most of the Windows downloads are by developers looking to get started doing Plone integrations or custom Plone development. Another target the installer has been used for, and this is more of what I’ve seen in my experience as a developer, is as a basis for production Plone deployments running on Windows servers. I definitely trust Steve’s sense more than my own, but the good news is that this new Windows tool chain seeks to provide a nicely integrated experience from web app development through to web app deployment.

So for the latter half of the sprint and ever since then, I’ve been obsessed with the Web Platform Installer (WebPI or WPI), IISExpress, and Web Matrix tool chain. The WebPI is actually a fairly open framework for describing web frameworks and web apps including dependencies and arbitrary installation commands in an extended atom feed. Helicon uses this to provide a Django install story nicely integrated with this Windows tool chain. Using IISExpress and Web Matrix also allows developers to work in a local environment isolated to their user directory without needing to have the full IIS ($$$) installed.

Notes and Findings

Below is a loosely organized grab-bag of notes and findings I recorded while working on this. I publish it here only for reference.

Hosting

  • ISAPI-WSGI doesn’t support IISExpress

    The ISAPI-WSGI OSS project is used by a number of Python WSGI projects to support IIS. ISAPI-WSGI depends on py2win32 which in turn depends on IIS 6 or the IIS 7 plugin providing IIS 6 compatibility. The IIS 6 compatibility plugin isn’t supported on IISExpress.

  • IIS only supports FastCGI

    This is how the WebPI and Web Matrix tool chain support PHP apps or other apps, such as plone, that need to have long running separate processes to run efficiently. This FCGI support is also restricted to using Windows named pipes. IOW, no TCP sockets to a separately running FCGI server process.

  • No current FCGI to WSGI gateway works with Windows

    The FastCGI spec calls for the STDIN_FILENO to refer to a socket which is then used for two-way communication between the server and the process handling requests. Naturally, Microsoft has embraced and extended this standard in IIS such that instead of a single socket it uses two Windows Named Pipes one each for receiving and sending. IIS may also support TCP sockets behind the scenes. This means that anything that expects to use normal sockets for FCGI, like flup won’t work with IIS. I can find no other OSS FCGI to WSGI gateway that supports Windows named pipes.

  • IIRF doesn’t support IISExpress

    We might be able to use Ionic’s Isapi Rewrite Filter to proxy IIS requests through to a separately running Python process. This is less then ideal since it may make the Web Matrix experience less integrated and requiring more un-Windows-like knowledge. IIRF doesn’t support IISExpress, at any rate, though it may be possible to manually install it into IISExpress.

  • Helicon Zoo Module

    I suppose the lack of other working options is exactly why Helicon Tech built it’s own solution for this. I prefer to have OSS all the way up to IIS itself, but that’s difficult when you play in the Microsoft sandbox. At least it looks like Helicon has paid some attention to performance. Furthermore, having company support may yield better long term maintenance for IIS support than an OSS project in the Microsoft universe. That still doesn’t mean I like it.

    Part of the Helicon Zoo Module is a zoofcgi.py script which is their own FCGI-WSGI gateway that seems to use STDIN_FILENO instead of sockets to do the FCGI communication. IOW, Helicon has written a totally new FCGI-WSGI gateway that works with IIS’s broken FCGI implementation. In their implementation I see a lot that looks familiar from flup.

  • Modifying zoofcgi.py to run Plone

    Unfortunately, Helicon’s zoofcgi.py only supports a Django WSGI app or a example wsgi app, with no way that I saw to use it to run an arbitrary WSGI app. Replace the run_example_app() function with the following to enable loading of an arbitrary WSGI app from a Paste *.ini file:

    from paste.script.util.logging_config import fileConfig
from paste.deploy import loadapp
def run_example_app():
    config = os.environ.get('WSGI_CONFIG_FILE')
    if config:
        config = os.path.abspath( config )
        fileConfig(config)
        application = loadapp('config:%s'%(config,))
    else:
        application = example_application
    if __debug__: logging.info('run_fcgi: STARTED')
    FCGIServer(application).run()
    if __debug__: logging.info('run_fcgi: EXITED')

    In ~My DocumentsIISExpressconfigapplicationhost.config change <engine name=”python.2.7.pipe”… to:

    <engine name="python.2.7.pipe"
        fullPath="%SystemDrive%\Plone42\zeocluster\bin\zopeskelpy.exe"
        arguments="%SystemDrive%\ZooExpress\Workers\python\zoofcgi.py"
        transport="NamedPipe"
        protocol="fastcgi" />

    web.config in the site root:

    <?xml version="1.0" encoding="UTF-8"?>
<configuration>
  <system.webServer>
    <heliconZoo>
      <application name="wsgi.project" >
        <environmentVariables>
          <add name="WSGI_CONFIG_FILE" value="%APPL_PHYSICAL_PATH%\wsgi.ini" />
  </environmentVariables>
      </application>
    </heliconZoo>
    <handlers>
      <add name="wsgi.project"
           scriptProcessor="python.2.7.pipe"
           path="*"
           verb="*"
           modules="HeliconZoo_x86" />
    </handlers>
  </system.webServer>
</configuration>

    It should be possible to re-use zoofcgi.py without modifying it by making it importable, IOW putting it’s dir on sys.path. Ideally, I’d like to see instead is a library that can wrap the named pipes provided by IIS such that they emulate a normal socket.

  • iiswsgi and filesocket

    I implemented a crude socket-like implementation called filesocket that wraps two open files and acts like a socket. Then I copied and pasted all the bits from flup that depend too rigidly on actual socket or on things not available in windows and adapted them as needed. Finally, I wrapped all that up with the necessary paste.deploy bits so that IIS can launch Plone as a WSGI app, after adding bits from the WSGI buildout to the unified installer, from a Paste ini file, using the paster config without a server since we’re interfacing directly with IIS. Finally, add the following to configuration/system.webServer/fastCgi in ~\My Documents\IISExpress\config\applicationhost.config:

    <application fullPath="C:\Python27\python.exe"
           arguments="-u C:\Plone42\zeocluster\bin\iiswsgi-script.py -c C:\Plone42\zeocluster\production.ini"
           monitorChangesTo="C:\Plone42\zeocluster\production.ini"
           maxInstances="1">
</application>

    This is one of the things I need help with. This can be done using the IIS appcmd.exe program but I need to know how to do this during the Web PI install process.

    It works and now we have Open Source all the way up to IIS!

  • Zope namespace URLs

    Zope has a lot of special URL structures like ++resource++foo.css but IIS chokes on these. Add the following to web.config:

    <system.webServer>
  <security>
    <requestFiltering
        allowDoubleEscaping="true">
    </requestFiltering>
  </security>
...

  • WebMatrix not launching instance until it’s been run in the foreground

    I’m noticing that the Plone instance keeps restarting. Before I fixed the pluses in URL issue, it at least eventually got running stably after several restarts, but now it’s restarting much more frequently. I thought it was caused by putting the resource registries in development mode because it means more requests and the number of requests that IIS lets build up before restarting the FCGI process is reached. When Plone is warming up, almost all of these request pile up. But I’ve also seen it be perfectly stable with the resource registries development mode on so I’ll need to catch this in the act again to debug.

    Since then, I think the problem is that for some reason it needs to be started in foreground mode before WebMatrix/IISExpress can successfully start it.

  • How to restart in debug-mode? Logging?

    What is the best way to make debugging information accessible to the MS toolchain? What’s the best way give WebMatrix developers and IIS admins to restart the instance in foreground-mode and/or debug-mode? What about giving them more integrated access to the logs? Should the default buildout configuration of the unified installer for Windows use event log handlers?

    Currently, the error reporting out of WebMatrix/IISExpress when launching the instance or anything else fails is miserable. Is there something we can do to better integrate with it’s trace logs such that more Zope/Plone/Console/UNIXy specific output is reflected somewhere?

  • Fixed two Windows bugs

    Docutils 0.7 conflicts with PIL causing:

    AccessInit: hash collision: 3 for both 1 and 1

    Upgrading to Docutils 0.9 fixes this.

    Also contributed a fix to Zope2 that addresses many of the stale lock file problems on windows.

Packaging

  • Creating a Web Deploy Package

    Microsoft provides some docs for creating Web Deploy packages. It may also be possible to use the msdeploy tool to make a package in a more automated way.

    More helpful pages about making web deploy packages:

  • Bootstrapping the Unified Installer

    The first time building the Web Deploy package based on the Unified Installer, some things need to be installed and configured that can’t really be automated. Some of these steps shouldn’t be necessary in the long run, since it should be possible to use the existing Web PI feed to install the dependencies now that the feed is working. Much of this should also be added to the Unified Installer in some sort of platform specific way. But if the Web PI feed ever needs to be created anew, or maybe when switching Python versions, it may be necessary to do the same things I had to do on my Windows VM to begin creating the Web PI feed and Plone Web Deploy package. I’m documenting all that in a README in the Unified Installer.

  • Non-buildout root Web Deploy Package?

    For my proof-of-concept, I manually created a web.config file in the zeocluster folder of the buildout created by the Unified Installer and then manually added that as a “site” in WebMatrix. Eventually, we need to figure out how to make the Unified Installer buildout root also be the root of the site for WebMatrix/IISExpress/IIS or how to have a web deploy package where the root of the installed site is a subdirectory of the web deploy package.

  • Deploying to a path with spaces

    I ran into the following error when the buildout was at a location with spaces in the path:

    WindowsError: [Error 87] The parameter is incorrect

    To try to narrow down the issue, I used buildout to create a debug-mode only script with the following part:

    [debug]
recipe = zc.recipe.egg
eggs = ${instance:eggs}
entry-points = debug=Zope2.Startup.run:run
initialization =
    import sys
    sys.argv.extend(["-C", r"${instance:location}\etc\zope.conf",
                     "-X", "debug-mode=on"])

    The script this produces works just fine, so the problem is in the plone.recipe.zope2instance recipe.

  • Installing FastCGI application, running bootstrap and buildout

    Using <runCommand> in manifest.xml is possible in theory:

    I tried adding doing this and then just adding a site from a folder in WebMatrix, nothing. But it may be that WebMatrix doesn’t do anything with manifest.xml when adding a site that way so I’ll have to try it by adding it to the WebPI feed. May also want to experiment with manually using msdeploy.exe.

Installing

  • Writing the Web Platform Installer Atom Feed

    Microsoft provides a reference to what elements and attributes it adds to the ATOM namespace.

  • bdist_wininst Python Installers aren’t silent

    Windows MSI installers seem to support a /verysilent flag. Unfortunatley, the Python distutils support for building Windows installers has no silent option. A workaround may be to use Web PI’s support for arbitrary installer commands to extract the installer without running it. It may also work to convert the wininst packages into MSI packages.

    In short, we need MSI’s for PIL, and pywin32, and binary windows eggs for lxml.

    Researching free software to build MSI’s. We need custom actions support, so Advanced Installer won’t do.

    It turns out that easy_install can make an egg out of a bdist_wininst exe installer. So we can now move all those dependencies into buildout by using find-links that point to the bdist_wininst exe downloads. This is how I’m getting pywin32 now.

  • Get the SHA1 Hash of the Web Deploy Package

    Microsoft provides a tool for generating this hash. I’m not sure if using this tool is strictly necessary or if there may be a way to get the msdeploy tool to do this as a part of a more automated packaging process.

  • Web App Gallery

    Microsoft actually has a web application gallery that they say applications can submit their application to. If approved, these applications would be available in Web PI without requiring the user to enter a custom feed.

Plone Sprints East   Sea Sprint 2012 Report-Out

Comments

comments powered by Disqus