A Twisted Web Tutorial

Twisted Web -- The Tutorial

Welcome
Gimmick -- Buffy quotes

Sweet (Once More With Feeling, season 6) -- Showtime

Twisted Web

Web server, using Twisted
- Like Apache, Zope...
Serve static files
Run CGIs
Other uses...

Giles (I Robot -- You Jane, season 1) -- There's a demon in the internet

Short Example: Putting a Server Up

Here's all you need to know to bring up a server

% mktap --uid=33 --gid=33 web --path=/var/www/htdocs --port=80
% sudo twistd -f web.tap

The rest of the talk will explain what that means...
...and how to do more complicated things

Buffy (Once More, With Feeling, season 6) -- I've got a theory. It doesn't matter.

Setup and Configuration Utilities

mktap
twistd
websetroot
- Won't be covered, unless there is time

Xander (The Harvest, season 1) -- crosses, garlic, stake through the heart

Digression: What are TAPs

Pickled 'application configuration'
Object which contains all the information about application
The canonical way to represent configurations in Twisted
Machine editable

Master (The Wish, season 3) -- Behold the technical wonder

mktap

General usage
Flexibility and Power

Buffy (Bad Eggs, season 2) -- I'm gonna need a *big* weapon

mktap web: Common Useful Options

--path: serve from given path
--port: listen on given port
--user: serve from users' directories and personal servers
--logfile: log to NCSA compatible logfile given
--processor: add a special processor for a given extension

Buffy (Bad Eggs, season 2) -- That's probably not gonna be the winning argument, is it?

twistd

Start a Twisted Application
Loads an instance of twisted.internet.app.Application from a file
Daemonizes, binds to appropriate ports, and starts the Twisted mainloop

Giles (Teacher's Pet, season 1) -- That's all he said? Fork Guy?

What's a Resource?

Everything represented as twisted.web.resource.Resource
Important interface:
- getChild()
- render()

Sean (Go Fish, season 3) -- You're soakin' in it, bud.

Resource Examples

Files
- MIME
- Processors
Others
- Virtual hosts
- User directories

Xander (As You Were, season 6) -- We have friends, family and demons

Web Development

Processors
- Inherited from resource.Resource
- Interpret files as code rather than data
Default processors
- .php -- default PHP
- .cgi -- Common Gateway Interface
- .rpy -- Correct way, Python scripting
- .trp -- Resource pickles
- ...more
You can also write your own

Xander (Family, season 5) -- That was a tangled web

Custom Processor

A custom processor to handle Perl CGIs (in a module called PerlScript)

from twisted.web import static, twcgi

class PerlScript(twcgi.FilteredScript):
    filter = '/usr/bin/perl' # Points to the perl parser

Use:
- mktap web --path=/home/nafai/public_html --processor=.pl=PerlScript.PerlScript

Tara (Family, season 4) -- There was the front of a camel

Resource Scripting

Subclass resource.Resource
Write a render(self, request) method
- Return string for immediate response
- Return NOT_DONE_YET and write to request
Create an .rpy file that sets 'resource' to an instance

Tara (Once More, With Feeling, season 6) -- You make me complete

.rpy example

from twisted.web import resource as resourcelib

class MyGreatResource(resourcelib.Resource):
    def render(self, request):
        return "<html>foo</html>"

resource = MyGreatResource()

Willow (Welcome to the Hellmouth, season 1) -- It's probably easy for you.

Alternative Configuration Formats

xml
source
Python

Ben (The Gift, season 5) -- I wish there was another way

Alternative Configuration Formats -- Python

Write manually
Just uses twisted.web API
Possible to do anything
- Write loops
- Read other files
- (not recommended) Define functions or classes

Buffy (The I In Team, season 4) -- But I've learned that it pays to be flexible in life.

Python Configuration Example

Create application
Make it listen on port 80 for web requests...
...which should be served from /var/www/htdocs

from twisted.internet import app
from twisted.web import static, server

application = app.Application('web')
application.listenTCP(80,
              server.Site(static.File("/var/www/htdocs")))

Willow (The Pack, season 1) -- It's simple, really.

Bannerfish -- A Case Study in Deployment

Serves banner ads
Has algorithms to maintain randomness and fairness
http://itamarst.org/software/bannerfish/
But how to deploy?

Xander (Halloween, season 2) -- Let's move out.

Bannerfish -- Standalone tap

mktap bannerfish
twistd -f bannerfish.tap
For full list of options
- mktap --help bannerfish

Ethan (Halloween, season 2) -- Don't wish to blow my own trumpet, but --

Bannerfish -- Standalone tap (behind reverse proxy)

mktap bannerfish --port 81 --proxyhost=example.com
twistd -f bannerfish.tap
Now can work on internal server behind firewall
If main server is Twisted Web, following Resource script will serve from bannerfish

resource = proxy.ReverseProxyResource('localhost', 81, '/')

Buffy (Halloween, season 2) -- You're sweet. A terrible liar, but sweet.

Bannerfish -- Standalone Python

from twisted.internet import app
from twisted.cred import authorizer
from twisted.web import server
from bannerfish import service

application = app.Application("bannerfish")
auth = authorizer.DefaultAuthorizer(app)
svc = service.BannerService('/var/bannerfish',
                            "bannerfish", application, auth)
site = server.Site(svc.buildResource(None, None))
application.listenTCP(80, site)

Spike (Halloween, season 2) -- Shaking. Terrified. Alone. Lost little lamb.

Bannerfish -- /etc/twisted-web/local.d Drop In

from twisted.cred import authorizer
from bannerfish import service

auth = authorizer.DefaultAuthorizer(app)
svc = service.BannerService('/var/bannerfish',
                       "bannerfish", application, auth)
resource = svc.buildResource(None, None)
default.addChild("bannerfish", resource)

Cordelia (Halloween, season 2) -- Well, I guess you better get them back to their parents.

Bannerfish -- Resource Script

from twisted.cred import authorizer
from twisted.internet import app
from bannerfish import service

application = registry.getComponent(app.Application)
auth = authorizer.DefaultAuthorizer(application)
svc = service.BannerService('/var/bannerfish',
                           "bannerfish", application, auth)
resource = svc.buildResource(None, None)

But see later, about registry

Xander (Innocence, season 2) -- They like to see the big guns.

Bannerfish -- Distributed (Slave)

from twisted.internet import application
from twisted.cred import authorizer
from twisted.web import server
from bannerfish import service
application = app.Application("bannerfish")
auth = authorizer.DefaultAuthorizer(application)
svc = service.BannerService('/var/bannerfish',
                         "bannerfish", application, auth)
site = server.Site(svc.buildResource(None, None))
fact = pb.BrokerFactory(site)
site = server.Site(root)
application.listenUNIX('/var/run/bannerfish', fact)

Bannerfish -- Distributed (Master, Resource Script)

from twisted.web import distrib

resource = distrib.ResourceSubscription('unix',
                        '/var/run/bannerfish')

Oz (Innocence, season 2) -- So, do you guys steal weapons from the Army a lot?

Bannerfish -- Other options

Mix and match possible
Can serve same content multiple ways simultaneously
Might be useful as a way to serve same ads different ways
...or serve ads from several bannerfish servers...
...each deployed differently.

Buffy (Tabula Rasa, season 6) -- I'm like a superhero or something

Bannerfish -- Conclusions

What are the tradeoffs?
Everything works in the simple cases
Not enough complicated cases to have data
Luckily, easy to move between them
Motto -- move deployment choices as late as possible

Giles (Killed By Death, season 2) -- Simple enough, but, but

Questions?

Vampire Willow (Dopplegangland, season 3): Questions? Comments?

Bonus Slides

Xander (The Dark Age, season 2) -- A bonus day of class plus Cordelia.

Python Configuration -- Hints

Working with persistence
Processors
Indices
Virtual Hosts

Buffy (Phases, season 2) -- Have you dropped any hints?

Python Configuration -- Persistence

Don't define functions or classes
Don't modify class attributes

Spike (Once More, With Feeling) -- Let me rest in peace

Python Configuration -- Processors


from twisted.internet import app
from twisted.web import static, server
from twisted.web import twcgi

root = static.File("/var/www")
root.processors = {".cgi": twcgi.CGIScript}
application = app.Application('web')
application.listenTCP(80, server.Site(root))

Manny (Doublemeat Palace, season 6) -- It's a meat process

Python Configuration -- Indices


root = static.File("/var/www")
root.indices = ['index.rpy', 'index.html']

Willow (Buffy vs. Dracula, season 5) -- Labelling your amulets and indexing your diaries

Python Configuration -- Virtual Hosts


from twisted.web import vhost
default = static.File("/var/www")
foo = static.File("/var/foo")
root = vhost.NamedVirtualHost(default)
root.addHost('foo.com', foo)

Fritz (I Robot, You Jane, season 1) -- The only reality is virtual.

Python Configuration -- uber example


from twisted.internet import app
from twisted.web import static, server, vhost, script

default = static.File("/var/www")
default.processors = {".rpy", script.ResourceScript}
root = vhost.NamedVirtualHost(default
foo = static.File("/var/foo")
foo.indices = ['index.xhtml', 'index.html']
root.addHost('foo.com', foo)
site = server.Site(root)
application = app.Application('web')
application.listenTCP(80, site, interface='127.0.0.1')

Buffy (Potential, season 7) -- It was putting a lot of stock in that uber-vamp

Python Configuration -- Splitting With Reverse Proxy

Original use case - SVN

from twisted.web import proxy

root.putChild('foo',
     proxy.ReverseProxyResource('localhost',
                                81, '/foo/'))

Buffy (Once More, With Feeling, season 6 -- So I will walk through the fire

mktap examples

mktap web
mktap web --path=/var/www --logfile=/var/log/twistedweb.log
mktap web --port=80 --path=/var/www --mime-type=text/plain
mktap web --path=/home/nafai/public_html --processor=.pl=PerlProcessor.PerlProcessor --index=index.pl

Anya (I Was Made to Love You, season 4) -- You can also see the website I designed for the magic shop

mktap examples (cont'd)

mktap web --users
mktap web --ignore-ext=.cgi
mktap web --index=index.cgi --index=index.rpy --index=index.html

Buffy (Once More, With Feeling, season 6) -- All the twists and bends

mktap examples (alternate formats)

mktap --type=source web
mktap --type=xml web

Tara (Seeing Red, season 6) -- It isn't written in any ancient language we could identify.

mktap examples (setting uid)

mktap --uid=33 web
mktap --gid=33 web
Uid/Gid of www-data on Debian systems
Not possible to use username
More about this later

Buffy (Who Are You?, season 4) -- I would be Buffy

twistd examples

twistd -f web.tap -l /var/log/twisted.log
twistd -f web.tap --pidfile /var/run/web.pid
twistd -x web.tax
- For mktap --type=xml
twistd -s web.tas
- For mktap --type=source

Xander (Teacher's Pet, season 1) -- How come *that* never came up?

Shutting down twistd

On Unix (in general):
- kill `cat twistd.pid`
On Windows:
- Cannot daemonize on Windows, so just run twistd in a command prompt
- Switch to the command prompt, and press Control-C

Buffy (Prophecy Girl, season 1) -- I don't wanna die.

Shutdown TAPs

Since TAPs store persistent data for an application, a 'shutdown' TAP is created on twistd shutdown
You'll often want to start your Twisted application on subsequent runs with the shutdown TAP

Headstone (The Gift, season 5) -- She saved the world. A lot.

twistd and security

When twistd is run as root, it will shed root privileges for the uid and gid of either the user that created the TAP or those specified on the mktap commandline.

Buffy (Dopplegangland, season 3) -- I think it's good to be reliable

Resource Call Examples

/foo/bar/baz gets converted to:

site.getChild('foo', request
   ).getChild('bar', request
   ).getChild('baz', request
   ).render(request)

Willow/Tara (Afterlife, Part 2, season 6) -- Child of words, hear thy makers

Resource Call Examples (cont'd)

/foo/bar/baz/ gets converted to:

site.getChild('foo', request
   ).getChild('bar', request
   ).getChild('baz', request
   ).getChild('', request
   ).render(request)

Buffy (Gone, season 6) -- Stop trying to see me.

Distributed Servers -- Theory

Master is a resource
Slave is a server
Same server can have both master and slave parts

Anya (Once More, With Feeling, season 6) -- I've got a theory, it could be bunnies

Distributed Servers -- Manually

from twisted.internet import app, protocol
from twisted.web import server, distrib, static
from twisted.spread import pb

application = app.Application("silly-web")
# The "master" server
site = server.Site(distrib.ResourceSubscription('unix', '.rp'))
application.listenTCP(19988, site)
# The "slave" server
fact = pb.BrokerFactory(distrib.ResourcePublisher(
           server.Site(static.File('static'))))
application.listenUNIX('./.rp', fact)

Buffy (Some Assembly Required, season 2) -- Men dig up the corpses and the women have the babies.

Distributed Servers -- Manual (cont'd)

First Server

from twisted.internet import app, protocol
from twisted.web import server, distrib, static, vhost
from twisted.spread import pb

application = app.Application("ping-web")

default = static.File("/var/www/foo")
root = vhost.NamedVirtualHost(default)
root.addVhost("foo.com", default)
bar = distrib.ResourceSubscription('unix', '.bar')
root.addVhost("bar.com", bar)

fact = pb.BrokerFactory(static.Site(default))
site = server.Site(root)
application.listenTCP(19988, site)
application.listenUNIX('./.foo', fact)

Buffy (Welcome to the Hellmouth, season 1) -- Now, we can do this the hard way, or...

Distributed Servers -- Manual (cont'd 2)

Second Server

from twisted.internet import app, protocol
from twisted.web import server, distrib, static, vhost
from twisted.spread import pb

application = app.Application("pong-web")

foo = distrib.ResourceSubscription('unix', '.foo')
root = vhost.NamedVirtualHost(foo)
root.addVhost("foo.com", foo)
bar = static.File("/var/www/bar")
root.addVhost("bar.com", bar)

fact = pb.BrokerFactory(static.Site(bar))
site = server.Site(root)
application.listenTCP(19989, site)
application.listenUNIX('./.bar', fact)

Buffy (Welcome to the Hellmouth, season 1) -- ...well, actually there's just the hard way.

Distributed Servers -- User Directory

A resource
Child that looks like 'moshez' -- ~moshez/public_html
Child that looks like 'moshez.twistd' -- moshez's personal server

Master (The Wish, season 3) -- Mass production!

Distributed Servers -- User Directory Server

With mktap: mktap web --user
With Python configuration

from twisted.internet import app
from twisted.web import static, server, distrib

root = static.File("/var/www")
root.putChild("users", distrib.UserDirectory())
site = server.Site(root)
application = app.Application('web')
application.listenTCP(80, site)

Richard (Reptile Boy, season 2) -- In his name.

Distributed Servers -- Personal Servers

With mktap: mktap web --personal ...
With Python configuration

from twisted.internet import app
from twisted.web import static, server, distrib
from twisted.spread import pb

root = static.File("/home/moshez/twistd")
site = server.Site(root)

fact = pb.BrokerFactory(distrib.ResourcePublisher(site))
application.listenUNIX('/home/moshez/.twisted-web-pb', fact)

Giles (Bargaining, season 6) -- It's my personal collection

Debian Configuration

Inside twisted-web package
Goal -- look like other web servers to users
Goal -- interoperate easily
Goal -- allow users to avoid modifying files

Buffy (Bad Girls, season 3) -- We can help each other.

Debian Configuration -- Usage

Changing port -- edit /etc/twisted-web/ports
Want to use behind reverse proxy? Use rptwisted
Change anything else -- drop files in /etc/twisted-web/local.d

Faith (Home Coming, season 3) -- we'll use 'em

Debian Configuration -- Drop In Examples

from twisted.web import static
import os

vhostDir = '/var/www/vhost/'

for file in os.listdir(vhostDir):
    root.addHost(file, static.File(os.path.join(vhostDir, file)))

Buffy (The Freshman, season 4) -- I just thought I'd drop in

Debian Configuration -- Drop In Examples (cont'd)

from twisted.web import script, static

default.processors['.rpy'] = script.ResourceScript
default.ignoreExt('rpy')

Riley (As You Were, season 6) -- Sorry to just drop in on you

Debian Configuration -- Drop In Examples (cont'd 2)

from twisted.web import vhost

default.putChild('vhost', vhost.VHostMonsterResource())

Sam (As You Were, season 6) -- a hairy night drop into hostile territory

twistedmatrix.com Configuration

Some highlights

...
indexNames = ['index', 'index.html', 'index.xhtml', 'index.rpy','index.cgi']
...
root.putChild('mailman', twcgi.CGIDirectory('/usr/lib/cgi-bin'))
root.putChild('users', distrib.UserDirectory())
root.putChild('cgi-bin', twcgi.CGIDirectory('/usr/lib/cgi-bin'))
root.putChild('doc', static.File('/usr/share/doc'))
...
uid = pwd.getpwnam('www-data')[2]
gid = grp.getgrnam('www-data')[2]
...
top = rewrite.RewriterResource(root, rewrite.tildeToUsers)
...
application = app.Application("web", uid=uid, gid=gid)

Xander (The Witch, season 1) -- May all lesser cretins bow before me.

Apache vs. Twisted Web

Apache is faster
Apache -- Threads/processes model
Twisted -- async model
Apache -- has C security holes (buffer overflows)
Twisted -- easy to set up
Twisted -- built in Python programmability

Willow (Buffy vs. Dracular, season 5) -- I think we've just put our finger on why we're the sidekicks

Apache/Twisted Web Integration

Use both!
Apache's reverse proxy works well
Easy to have a site which is partially managed by Apache
Documentation has examples of configurations

Xander (What's My Line, season 2) -- Angel's our friend! Except I don't like him.

Zope vs. Twisted Web

Zope -- fully editable through the web
Zope -- uses ZODB, not file system
Twisted -- can integrate with other protocols easily
Twisted -- extension code has much less overhead

Willow (Dopplegangland, season 3) -- Competition is natural and healthy

Zope/Twisted Web Integration

Possible to use Twisted as Zope's network layer
Hackish with Zope2
Easier with Zope3

Snyder (Dopplegangland, season 3) -- It's a perfect match.

Zope/Twisted Web Integration (cont'd)

Less direct -- use Apache
Reverse proxy parts to Zope
Reverse proxy parts to Twisted Web

Wesley (Dopplegangland, season 3) -- Still a little sloppy, though

Applications Appropriate for Twisted Web

Webmail
Blogs
Web/other protocol chat systems

Sweet (Once More, With Feeling) -- Why don't you come and play?

Behind Reverse Proxy

Sometimes, we want Twisted to pretend to be another host/port
Reverse proxies, NATs, etc.
Reverse proxy to /vhost/http/<host:port>/
Make sure root has a child called vhost of type twisted.web.vhost.VirtualHostingMonster

Jenny (I Robot -- You Jane, season 1) -- The divine exists in cyberspace

Rewrite Rules

Change a URL to another
Useful for different treatment from outside resources
Wraps a resource

Spike (What's My Line, season 2) -- Read it again.

Rewrite Rules -- Example


root = static.File("/var/www")
root.putChild("users", distrib.UserDirectory())
root = rewrite.RewriterResource(root, rewrite.tildeToUsers)

Now, /~moshez/ works

Spike (What's My Line, season 2) -- I think it's just enough kill.

websetroot

Used to change what the root of the server points to
Set it to a Resource contained either in a Python source file or a Pickle file

Manny (DoubleMeat Palace, season 6) -- We have a lot of turnover here

Sample websetroot command lines

websetroot -p 80 -f web.tap --script rootResource.py
websetroot -p 8080 -f web.tap --pickle rootPickle

Manny (DoubleMeat Palace, season 6) -- You can toss it

init.d

pidfile
chdir
chroot?
No smooth reloading
Persistence

Bob (Zeppo, season 3) -- He hasn't been initiated.

Special Bonus - How to Configure <user>.example.com

import pwd, os
from twisted.web import resource, error, distrib
from twisted.protocols import http

class UserNameVirtualHost(resource.Resource):

    def __init__(self, default, tail):
        resource.Resource.__init__(self)
        self.default = default
        self.tail = tail
        self.users = {}

    def _getResourceForRequest(self, request):
        host=request.getHeader('host')
        if host.endswith(tail):
            username = host[:-len(tail)]
        else:
            username = default
        if self.users.has_key(username):
            return self.users[username]
        try:
            (pw_name, pw_passwd, pw_uid, pw_gid, pw_gecos, pw_dir,
             pw_shell) = pwd.getpwnam(username)
        except KeyError:
            return error.ErrorPage(http.NOT_FOUND,
                                   "No Such User",
                          "The user %s was not found on this system." %
                                   repr(username))
        twistdsock = os.path.join(pw_dir, ".twistd-web-pb")
        rs = distrib.ResourceSubscription('unix',twistdsock)
        self.users[username] = rs
        return rs

    def render(self, request):
        resrc = self._getResourceForRequest(request)
        return resrc.render(request)

    def getChild(self, path, request):
        resrc = self._getResourceForRequest(request)
        request.path=request.path[:-1]
        request.postpath=request.uri.split('/')[1:]
        print request, request.path, request.postpath
        return resrc.getChildForRequest(request)

Morgan (The Puppet Show, season 1) -- Weird? What d'you mean?

Special Bonus - How to Configure <user>.example.com (cont'd)

Put above in a module (say, uservhost)
Use following configuration file

from twisted.internet import app
from twisted.web import server
import uservhost

root = UserNameVirtualHost("www", "example.com")
site = server.Site(root)
application = app.Application('web')
application.listenTCP(80, site)

Snyder (The Puppet Show, season 1) -- You need to integrate into this school, people.

Using the Twisted Registry

Use especially in Resource Scripts
Save persistent information
Uses Twisted's Componentized to be extensible

Angel (Helpless, season 3) -- I wanted to keep it safe

Using the Twisted Registry -- example


from twisted.web import distrib

resource = registry.getComponent(distrib.UserDirectory)
if not resource:
    resource = distrib.UserDirectory()
    registry.setComponent(distrib.UserDirectory, resource)

Paul (The Freshman, season 4) -- Do you know where they're distributing the [...] applications?

Using the Twisted Registry -- problems

In most cases -- need to write a custom class
Saves data in-memory
Won't work as expected unless -shutdown taps are used

Anya (Once More, With Feeling, season 6) -- The only trouble is [pause] I'll never tell.

Alternative Configuration Formats -- XML

Can be generated from mktap
Editable with any XML editor
Easy to do easy things
- Change a port
Nontrivial to do hard things
- Bind to specific IP

Buffy (Once More, With Feeling, season 6) -- To fit in in this glittering world.

Alternative Configuration Formats -- XML -- example

<?xml version="1.0"?>

<instance class="twisted.internet.app.Application" reference="1">
  <dictionary>
...
    <string role="key" value="tcpPorts" />
    <list>
      <tuple>
        <int value="80" />
        <instance class="twisted.web.server.Site">
          <dictionary>
...
            <string role="key" value="resource" />
            <instance class="twisted.web.static.File">
              <dictionary>
...
                <string role="key" value="path" />
                <string value="/var/www" />
...
              </dictionary>
            </instance>
...
          </dictionary>
        </instance>
...
      </tuple>
    </list>
...
  </dictionary>
</instance>

Natalie (Teacher's Pet, season 1) -- There's nothing ugly about these creatures

Alternative Configuration Formats -- Source

Can be generated from mktap
Editable with any Python source editor
Easy to do easy things
- Change a port
Nontrivial to do hard things
- Bind to specific IP

Willow/Giles/Xander (Primeval, season 4) -- You could never hope to grasp the source

Alternative Configuration Formats -- Source -- Example

app=Ref(1,
  Instance('twisted.internet.app.Application',{
...
      'tcpPorts':[
        (
          80,
          Instance('twisted.web.server.Site',
...
            resource=Instance('twisted.web.static.File',{
...
                'path':'/var/www',
...
          ),
        ],
...
      }))

Tara (Family, season 5) -- You learn her source, and, uh we'll introduce her to her insect reflection

Twisted Web - Beginnings


<glyphAtWork> the http server was so we could say "Web!" if we ever did
              a freshmeat announcement
<glyphAtWork> this makes people excited

Turned out he was right

Dawn (Get It Done, season 7) -- I think it's an origin myth.

Woven Overview

HTML templates
Model/View/Controller architecture
Integrated with deferred
Classical systems work badly with async
More -- beyond scope of this tutorial

Razor (Bargaining, season 6) -- A pretty toy