A Twisted Web Tutorial
Twisted Web -- The Tutorial
- Welcome
- Gimmick -- Buffy quotes
Sweet (Once More With Feeling, season 6) -- Showtime
Twisted Web
- Web server, using Twisted
- Serve static files
- Run CGIs
- Other uses...
Giles (I Robot -- You Jane, season 1) -- There's a demon in the internet
Short Example: Putting a Server Up
- Here's all you need to know to bring up a server
% mktap --uid=33 --gid=33 web --path=/var/www/htdocs --port=80
% sudo twistd -f web.tap
- The rest of the talk will explain what that means...
- ...and how to do more complicated things
Buffy (Once More, With Feeling, season 6) -- I've got a theory. It doesn't matter.
Setup and Configuration Utilities
- mktap
- twistd
- websetroot
- Won't be covered, unless there is time
Xander (The Harvest, season 1) -- crosses, garlic, stake through the heart
Digression: What are TAPs
- Pickled 'application configuration'
- Object which contains all the information about application
- The canonical way to represent configurations in Twisted
- Machine editable
Master (The Wish, season 3) -- Behold the technical wonder
mktap
- General usage
- Flexibility and Power
Buffy (Bad Eggs, season 2) -- I'm gonna need a *big* weapon
mktap web: Common Useful Options
- --path: serve from given path
- --port: listen on given port
- --user: serve from users' directories and personal servers
- --logfile: log to NCSA compatible logfile given
- --processor: add a special processor for a given extension
Buffy (Bad Eggs, season 2) -- That's probably not gonna be the winning argument, is it?
twistd
- Start a Twisted Application
- Loads an instance of twisted.internet.app.Application from a file
- Daemonizes, binds to appropriate ports, and starts the Twisted mainloop
Giles (Teacher's Pet, season 1) -- That's all he said? Fork Guy?
What's a Resource?
- Everything represented as twisted.web.resource.Resource
- Important interface:
Sean (Go Fish, season 3) -- You're soakin' in it, bud.
Resource Examples
- Files
- Others
- Virtual hosts
- User directories
Xander (As You Were, season 6) -- We have friends, family and demons
Web Development
- Processors
- Inherited from resource.Resource
- Interpret files as code rather than data
- Default processors
- .php -- default PHP
- .cgi -- Common Gateway Interface
- .rpy -- Correct way, Python scripting
- .trp -- Resource pickles
- ...more
- You can also write your own
Xander (Family, season 5) -- That was a tangled web
Custom Processor
- A custom processor to handle Perl CGIs (in a module called PerlScript)
from twisted.web import static, twcgi
class PerlScript(twcgi.FilteredScript):
filter = '/usr/bin/perl' # Points to the perl parser
- Use:
- mktap web --path=/home/nafai/public_html --processor=.pl=PerlScript.PerlScript
Tara (Family, season 4) -- There was the front of a camel
Resource Scripting
- Subclass resource.Resource
- Write a render(self, request) method
- Return string for immediate response
- Return NOT_DONE_YET and write to request
- Create an .rpy file that sets 'resource' to an instance
Tara (Once More, With Feeling, season 6) -- You make me complete
.rpy example
from twisted.web import resource as resourcelib
class MyGreatResource(resourcelib.Resource):
def render(self, request):
return "<html>foo</html>"
resource = MyGreatResource()
Willow (Welcome to the Hellmouth, season 1) -- It's probably easy for you.
Alternative Configuration Formats
Ben (The Gift, season 5) -- I wish there was another way
Alternative Configuration Formats -- Python
- Write manually
- Just uses twisted.web API
- Possible to do anything
- Write loops
- Read other files
- (not recommended) Define functions or classes
Buffy (The I In Team, season 4) -- But I've learned that it pays to be flexible in life.
Python Configuration Example
- Create application
- Make it listen on port 80 for web requests...
- ...which should be served from /var/www/htdocs
from twisted.internet import app
from twisted.web import static, server
application = app.Application('web')
application.listenTCP(80,
server.Site(static.File("/var/www/htdocs")))
Willow (The Pack, season 1) -- It's simple, really.
Bannerfish -- A Case Study in Deployment
Xander (Halloween, season 2) -- Let's move out.
Bannerfish -- Standalone tap
- mktap bannerfish
- twistd -f bannerfish.tap
- For full list of options
Ethan (Halloween, season 2) -- Don't wish to blow my own trumpet, but --
Bannerfish -- Standalone tap (behind reverse proxy)
- mktap bannerfish --port 81 --proxyhost=example.com
- twistd -f bannerfish.tap
- Now can work on internal server behind firewall
- If main server is Twisted Web, following Resource script will serve from bannerfish
resource = proxy.ReverseProxyResource('localhost', 81, '/')
Buffy (Halloween, season 2) -- You're sweet. A terrible liar, but sweet.
Bannerfish -- Standalone Python
from twisted.internet import app
from twisted.cred import authorizer
from twisted.web import server
from bannerfish import service
application = app.Application("bannerfish")
auth = authorizer.DefaultAuthorizer(app)
svc = service.BannerService('/var/bannerfish',
"bannerfish", application, auth)
site = server.Site(svc.buildResource(None, None))
application.listenTCP(80, site)
Spike (Halloween, season 2) -- Shaking. Terrified. Alone. Lost little lamb.
Bannerfish -- /etc/twisted-web/local.d Drop In
from twisted.cred import authorizer
from bannerfish import service
auth = authorizer.DefaultAuthorizer(app)
svc = service.BannerService('/var/bannerfish',
"bannerfish", application, auth)
resource = svc.buildResource(None, None)
default.addChild("bannerfish", resource)
Cordelia (Halloween, season 2) -- Well, I guess you better get them back to their parents.
Bannerfish -- Resource Script
from twisted.cred import authorizer
from twisted.internet import app
from bannerfish import service
application = registry.getComponent(app.Application)
auth = authorizer.DefaultAuthorizer(application)
svc = service.BannerService('/var/bannerfish',
"bannerfish", application, auth)
resource = svc.buildResource(None, None)
- But see later, about registry
Xander (Innocence, season 2) -- They like to see the big guns.
Bannerfish -- Distributed (Slave)
from twisted.internet import application
from twisted.cred import authorizer
from twisted.web import server
from bannerfish import service
application = app.Application("bannerfish")
auth = authorizer.DefaultAuthorizer(application)
svc = service.BannerService('/var/bannerfish',
"bannerfish", application, auth)
site = server.Site(svc.buildResource(None, None))
fact = pb.BrokerFactory(site)
site = server.Site(root)
application.listenUNIX('/var/run/bannerfish', fact)
Bannerfish -- Distributed (Master, Resource Script)
from twisted.web import distrib
resource = distrib.ResourceSubscription('unix',
'/var/run/bannerfish')
Oz (Innocence, season 2) -- So, do you guys steal weapons from the Army a lot?
Bannerfish -- Other options
- Mix and match possible
- Can serve same content multiple ways simultaneously
- Might be useful as a way to serve same ads different ways
- ...or serve ads from several bannerfish servers...
- ...each deployed differently.
Buffy (Tabula Rasa, season 6) -- I'm like a superhero or something
Bannerfish -- Conclusions
- What are the tradeoffs?
- Everything works in the simple cases
- Not enough complicated cases to have data
- Luckily, easy to move between them
- Motto -- move deployment choices as late as possible
Giles (Killed By Death, season 2) -- Simple enough, but, but
Further Reading
- Short overview -- doc/howto/web-overview.html
- In depth review -- doc/howto/using-twistedweb.html
- Using databases -- doc/howto/enterprise.html
- Deferred execution -- doc/howto/deferred.html
- Resource script examples -- doc/examples/*.rpy.py
Giles (I Was Made to Love You, Season 5) -- There's an enormous amount of research we should do before -- no I'm lying
Questions?
Vampire Willow (Dopplegangland, season 3): Questions? Comments?
Bonus Slides
Xander (The Dark Age, season 2) -- A bonus day of class plus Cordelia.
Python Configuration -- Hints
- Working with persistence
- Processors
- Indices
- Virtual Hosts
Buffy (Phases, season 2) -- Have you dropped any hints?
Python Configuration -- Persistence
- Don't define functions or classes
- Don't modify class attributes
Spike (Once More, With Feeling) -- Let me rest in peace
Python Configuration -- Processors
from twisted.internet import app
from twisted.web import static, server
from twisted.web import twcgi
root = static.File("/var/www")
root.processors = {".cgi": twcgi.CGIScript}
application = app.Application('web')
application.listenTCP(80, server.Site(root))
Manny (Doublemeat Palace, season 6) -- It's a meat process
Python Configuration -- Indices
root = static.File("/var/www")
root.indices = ['index.rpy', 'index.html']
Willow (Buffy vs. Dracula, season 5) -- Labelling your amulets and indexing your diaries
Python Configuration -- Virtual Hosts
from twisted.web import vhost
default = static.File("/var/www")
foo = static.File("/var/foo")
root = vhost.NamedVirtualHost(default)
root.addHost('foo.com', foo)
Fritz (I Robot, You Jane, season 1) -- The only reality is virtual.
Python Configuration -- uber example
from twisted.internet import app
from twisted.web import static, server, vhost, script
default = static.File("/var/www")
default.processors = {".rpy", script.ResourceScript}
root = vhost.NamedVirtualHost(default
foo = static.File("/var/foo")
foo.indices = ['index.xhtml', 'index.html']
root.addHost('foo.com', foo)
site = server.Site(root)
application = app.Application('web')
application.listenTCP(80, site, interface='127.0.0.1')
Buffy (Potential, season 7) -- It was putting a lot of stock in that uber-vamp
Python Configuration -- Splitting With Reverse Proxy
from twisted.web import proxy
root.putChild('foo',
proxy.ReverseProxyResource('localhost',
81, '/foo/'))
Buffy (Once More, With Feeling, season 6 -- So I will walk through the fire
mktap examples
- mktap web
- mktap web --path=/var/www --logfile=/var/log/twistedweb.log
- mktap web --port=80 --path=/var/www --mime-type=text/plain
- mktap web --path=/home/nafai/public_html --processor=.pl=PerlProcessor.PerlProcessor --index=index.pl
Anya (I Was Made to Love You, season 4) -- You can also see the website I designed for the magic shop
mktap examples (cont'd)
- mktap web --users
- mktap web --ignore-ext=.cgi
- mktap web --index=index.cgi --index=index.rpy --index=index.html
Buffy (Once More, With Feeling, season 6) -- All the twists and bends
mktap examples (alternate formats)
- mktap --type=source web
- mktap --type=xml web
Tara (Seeing Red, season 6) -- It isn't written in any ancient language we could identify.
mktap examples (setting uid)
- mktap --uid=33 web
- mktap --gid=33 web
- Uid/Gid of www-data on Debian systems
- Not possible to use username
- More about this later
Buffy (Who Are You?, season 4) -- I would be Buffy
twistd examples
- twistd -f web.tap -l /var/log/twisted.log
- twistd -f web.tap --pidfile /var/run/web.pid
- twistd -x web.tax
- twistd -s web.tas
Xander (Teacher's Pet, season 1) -- How come *that* never came up?
Shutting down twistd
- On Unix (in general):
- On Windows:
- Cannot daemonize on Windows, so just run twistd in a command prompt
- Switch to the command prompt, and press Control-C
Buffy (Prophecy Girl, season 1) -- I don't wanna die.
Shutdown TAPs
- Since TAPs store persistent data for an application, a 'shutdown' TAP is created on twistd shutdown
- You'll often want to start your Twisted application on subsequent runs with the shutdown TAP
Headstone (The Gift, season 5) -- She saved the world. A lot.
twistd and security
- When twistd is run as root, it will shed root privileges for the uid and gid of either the user that created the TAP or those specified on the mktap commandline.
Buffy (Dopplegangland, season 3) -- I think it's good to be reliable
Resource Call Examples
- /foo/bar/baz gets converted to:
site.getChild('foo', request
).getChild('bar', request
).getChild('baz', request
).render(request)
Willow/Tara (Afterlife, Part 2, season 6) -- Child of words, hear thy makers
Resource Call Examples (cont'd)
- /foo/bar/baz/ gets converted to:
site.getChild('foo', request
).getChild('bar', request
).getChild('baz', request
).getChild('', request
).render(request)
Buffy (Gone, season 6) -- Stop trying to see me.
Distributed Servers -- Theory
- Master is a resource
- Slave is a server
- Same server can have both master and slave parts
Anya (Once More, With Feeling, season 6) -- I've got a theory, it could be bunnies
Distributed Servers -- Manually
from twisted.internet import app, protocol
from twisted.web import server, distrib, static
from twisted.spread import pb
application = app.Application("silly-web")
# The "master" server
site = server.Site(distrib.ResourceSubscription('unix', '.rp'))
application.listenTCP(19988, site)
# The "slave" server
fact = pb.BrokerFactory(distrib.ResourcePublisher(
server.Site(static.File('static'))))
application.listenUNIX('./.rp', fact)
Buffy (Some Assembly Required, season 2) -- Men dig up the corpses and the women have the babies.
Distributed Servers -- Manual (cont'd)
from twisted.internet import app, protocol
from twisted.web import server, distrib, static, vhost
from twisted.spread import pb
application = app.Application("ping-web")
default = static.File("/var/www/foo")
root = vhost.NamedVirtualHost(default)
root.addVhost("foo.com", default)
bar = distrib.ResourceSubscription('unix', '.bar')
root.addVhost("bar.com", bar)
fact = pb.BrokerFactory(static.Site(default))
site = server.Site(root)
application.listenTCP(19988, site)
application.listenUNIX('./.foo', fact)
Buffy (Welcome to the Hellmouth, season 1) -- Now, we can do this the hard way, or...
Distributed Servers -- Manual (cont'd 2)
from twisted.internet import app, protocol
from twisted.web import server, distrib, static, vhost
from twisted.spread import pb
application = app.Application("pong-web")
foo = distrib.ResourceSubscription('unix', '.foo')
root = vhost.NamedVirtualHost(foo)
root.addVhost("foo.com", foo)
bar = static.File("/var/www/bar")
root.addVhost("bar.com", bar)
fact = pb.BrokerFactory(static.Site(bar))
site = server.Site(root)
application.listenTCP(19989, site)
application.listenUNIX('./.bar', fact)
Buffy (Welcome to the Hellmouth, season 1) -- ...well, actually there's just the hard way.
Distributed Servers -- User Directory
- A resource
- Child that looks like 'moshez' -- ~moshez/public_html
- Child that looks like 'moshez.twistd' -- moshez's personal server
Master (The Wish, season 3) -- Mass production!
Distributed Servers -- User Directory Server
- With mktap: mktap web --user
- With Python configuration
from twisted.internet import app
from twisted.web import static, server, distrib
root = static.File("/var/www")
root.putChild("users", distrib.UserDirectory())
site = server.Site(root)
application = app.Application('web')
application.listenTCP(80, site)
Richard (Reptile Boy, season 2) -- In his name.
Distributed Servers -- Personal Servers
- With mktap: mktap web --personal ...
- With Python configuration
from twisted.internet import app
from twisted.web import static, server, distrib
from twisted.spread import pb
root = static.File("/home/moshez/twistd")
site = server.Site(root)
fact = pb.BrokerFactory(distrib.ResourcePublisher(site))
application.listenUNIX('/home/moshez/.twisted-web-pb', fact)
Giles (Bargaining, season 6) -- It's my personal collection
Debian Configuration
- Inside twisted-web package
- Goal -- look like other web servers to users
- Goal -- interoperate easily
- Goal -- allow users to avoid modifying files
Buffy (Bad Girls, season 3) -- We can help each other.
Debian Configuration -- Usage
- Changing port -- edit /etc/twisted-web/ports
- Want to use behind reverse proxy? Use rptwisted
- Change anything else -- drop files in /etc/twisted-web/local.d
Faith (Home Coming, season 3) -- we'll use 'em
Debian Configuration -- Drop In Examples
from twisted.web import static
import os
vhostDir = '/var/www/vhost/'
for file in os.listdir(vhostDir):
root.addHost(file, static.File(os.path.join(vhostDir, file)))
Buffy (The Freshman, season 4) -- I just thought I'd drop in
Debian Configuration -- Drop In Examples (cont'd)
from twisted.web import script, static
default.processors['.rpy'] = script.ResourceScript
default.ignoreExt('rpy')
Riley (As You Were, season 6) -- Sorry to just drop in on you
Debian Configuration -- Drop In Examples (cont'd 2)
from twisted.web import vhost
default.putChild('vhost', vhost.VHostMonsterResource())
Sam (As You Were, season 6) -- a hairy night drop into hostile territory
twistedmatrix.com Configuration
...
indexNames = ['index', 'index.html', 'index.xhtml', 'index.rpy','index.cgi']
...
root.putChild('mailman', twcgi.CGIDirectory('/usr/lib/cgi-bin'))
root.putChild('users', distrib.UserDirectory())
root.putChild('cgi-bin', twcgi.CGIDirectory('/usr/lib/cgi-bin'))
root.putChild('doc', static.File('/usr/share/doc'))
...
uid = pwd.getpwnam('www-data')[2]
gid = grp.getgrnam('www-data')[2]
...
top = rewrite.RewriterResource(root, rewrite.tildeToUsers)
...
application = app.Application("web", uid=uid, gid=gid)
Xander (The Witch, season 1) -- May all lesser cretins bow before me.
Apache vs. Twisted Web
- Apache is faster
- Apache -- Threads/processes model
- Twisted -- async model
- Apache -- has C security holes (buffer overflows)
- Twisted -- easy to set up
- Twisted -- built in Python programmability
Willow (Buffy vs. Dracular, season 5) -- I think we've just put our finger on why we're the sidekicks
Apache/Twisted Web Integration
- Use both!
- Apache's reverse proxy works well
- Easy to have a site which is partially managed by Apache
- Documentation has examples of configurations
Xander (What's My Line, season 2) -- Angel's our friend! Except I don't like him.
Zope vs. Twisted Web
- Zope -- fully editable through the web
- Zope -- uses ZODB, not file system
- Twisted -- can integrate with other protocols easily
- Twisted -- extension code has much less overhead
Willow (Dopplegangland, season 3) -- Competition is natural and healthy
Zope/Twisted Web Integration
- Possible to use Twisted as Zope's network layer
- Hackish with Zope2
- Easier with Zope3
Snyder (Dopplegangland, season 3) -- It's a perfect match.
Zope/Twisted Web Integration (cont'd)
- Less direct -- use Apache
- Reverse proxy parts to Zope
- Reverse proxy parts to Twisted Web
Wesley (Dopplegangland, season 3) -- Still a little sloppy, though
Applications Appropriate for Twisted Web
- Webmail
- Blogs
- Web/other protocol chat systems
Sweet (Once More, With Feeling) -- Why don't you come and play?
Behind Reverse Proxy
- Sometimes, we want Twisted to pretend to be another host/port
- Reverse proxies, NATs, etc.
- Reverse proxy to /vhost/http/<host:port>/
- Make sure root has a child called vhost of type twisted.web.vhost.VirtualHostingMonster
Jenny (I Robot -- You Jane, season 1) -- The divine exists in cyberspace
Rewrite Rules
- Change a URL to another
- Useful for different treatment from outside resources
- Wraps a resource
Spike (What's My Line, season 2) -- Read it again.
Rewrite Rules -- Example
root = static.File("/var/www")
root.putChild("users", distrib.UserDirectory())
root = rewrite.RewriterResource(root, rewrite.tildeToUsers)
Spike (What's My Line, season 2) -- I think it's just enough kill.
websetroot
- Used to change what the root of the server points to
- Set it to a Resource contained either in a Python source file or a Pickle file
Manny (DoubleMeat Palace, season 6) -- We have a lot of turnover here
Sample websetroot command lines
- websetroot -p 80 -f web.tap --script rootResource.py
- websetroot -p 8080 -f web.tap --pickle rootPickle
Manny (DoubleMeat Palace, season 6) -- You can toss it
init.d
- pidfile
- chdir
- chroot?
- No smooth reloading
- Persistence
Bob (Zeppo, season 3) -- He hasn't been initiated.
Special Bonus - How to Configure <user>.example.com
import pwd, os
from twisted.web import resource, error, distrib
from twisted.protocols import http
class UserNameVirtualHost(resource.Resource):
def __init__(self, default, tail):
resource.Resource.__init__(self)
self.default = default
self.tail = tail
self.users = {}
def _getResourceForRequest(self, request):
host=request.getHeader('host')
if host.endswith(tail):
username = host[:-len(tail)]
else:
username = default
if self.users.has_key(username):
return self.users[username]
try:
(pw_name, pw_passwd, pw_uid, pw_gid, pw_gecos, pw_dir,
pw_shell) = pwd.getpwnam(username)
except KeyError:
return error.ErrorPage(http.NOT_FOUND,
"No Such User",
"The user %s was not found on this system." %
repr(username))
twistdsock = os.path.join(pw_dir, ".twistd-web-pb")
rs = distrib.ResourceSubscription('unix',twistdsock)
self.users[username] = rs
return rs
def render(self, request):
resrc = self._getResourceForRequest(request)
return resrc.render(request)
def getChild(self, path, request):
resrc = self._getResourceForRequest(request)
request.path=request.path[:-1]
request.postpath=request.uri.split('/')[1:]
print request, request.path, request.postpath
return resrc.getChildForRequest(request)
Morgan (The Puppet Show, season 1) -- Weird? What d'you mean?
Special Bonus - How to Configure <user>.example.com (cont'd)
- Put above in a module (say, uservhost)
- Use following configuration file
from twisted.internet import app
from twisted.web import server
import uservhost
root = UserNameVirtualHost("www", "example.com")
site = server.Site(root)
application = app.Application('web')
application.listenTCP(80, site)
Snyder (The Puppet Show, season 1) -- You need to integrate into this school, people.
Using the Twisted Registry
- Use especially in Resource Scripts
- Save persistent information
- Uses Twisted's Componentized to be extensible
Angel (Helpless, season 3) -- I wanted to keep it safe
Using the Twisted Registry -- example
from twisted.web import distrib
resource = registry.getComponent(distrib.UserDirectory)
if not resource:
resource = distrib.UserDirectory()
registry.setComponent(distrib.UserDirectory, resource)
Paul (The Freshman, season 4) -- Do you know where they're distributing the [...] applications?
Using the Twisted Registry -- problems
- In most cases -- need to write a custom class
- Saves data in-memory
- Won't work as expected unless -shutdown taps are used
Anya (Once More, With Feeling, season 6) -- The only trouble is [pause] I'll never tell.
Alternative Configuration Formats -- XML
- Can be generated from mktap
- Editable with any XML editor
- Easy to do easy things
- Nontrivial to do hard things
Buffy (Once More, With Feeling, season 6) -- To fit in in this glittering world.
Alternative Configuration Formats -- XML -- example
<?xml version="1.0"?>
<instance class="twisted.internet.app.Application" reference="1">
<dictionary>
...
<string role="key" value="tcpPorts" />
<list>
<tuple>
<int value="80" />
<instance class="twisted.web.server.Site">
<dictionary>
...
<string role="key" value="resource" />
<instance class="twisted.web.static.File">
<dictionary>
...
<string role="key" value="path" />
<string value="/var/www" />
...
</dictionary>
</instance>
...
</dictionary>
</instance>
...
</tuple>
</list>
...
</dictionary>
</instance>
Natalie (Teacher's Pet, season 1) -- There's nothing ugly about these creatures
Alternative Configuration Formats -- Source
- Can be generated from mktap
- Editable with any Python source editor
- Easy to do easy things
- Nontrivial to do hard things
Willow/Giles/Xander (Primeval, season 4) -- You could never hope to grasp the source
Alternative Configuration Formats -- Source -- Example
app=Ref(1,
Instance('twisted.internet.app.Application',{
...
'tcpPorts':[
(
80,
Instance('twisted.web.server.Site',
...
resource=Instance('twisted.web.static.File',{
...
'path':'/var/www',
...
),
],
...
}))
Tara (Family, season 5) -- You learn her source, and, uh we'll introduce her to her insect reflection
Twisted Web - Beginnings
<glyphAtWork> the http server was so we could say "Web!" if we ever did
a freshmeat announcement
<glyphAtWork> this makes people excited
Dawn (Get It Done, season 7) -- I think it's an origin myth.
Woven Overview
- HTML templates
- Model/View/Controller architecture
- Integrated with deferred
- Classical systems work badly with async
- More -- beyond scope of this tutorial
Razor (Bargaining, season 6) -- A pretty toy