<html><head><title>Twisted Tutorial</title></head> <body> <h1>Twisted Tutorial</h1> <h2>Twisted -- The Tutorial</h2><ul> <li>Welcome</li> <li>Gimmick -- Charmed quotes</li> </ul> <hr /> <em>Prue (Something Wicca This Way Comes, season 1) -- Piper, the girl has no vision, no sense of the future.</em> <h2>Twisted -- Networking For Python</h2><ul> <li>Handles the icky socket stuff</li> <li>Handles the icky select stuff</li> <li>No threads, no blocking</li> </ul> <hr /> <em>Leo (Bite Me, season 4) -- As far as I know they're apart of a whole different network now.</em> <h2>Finger</h2><ul> <li>Send username</li> <li>Get back some stuff about user</li> <li>Will only implement subset of protocol here</li> </ul> <hr /> <em>Natalie (Blinded By the Whitelighter) -- I'll assume a demon attacked your finger</em> <h2>Finger - Protocol code</h2> <pre class="python"> from twisted.protocols import basic class FingerClient(basic.LineReceiver): # This will be called when the connection is made def connectionMade(self): self.sendLine(self.factory.user) # This will be called when the server sends us a line. # IMPORTANT: line *without "\n" at end. # Yes, this means empty line does not mean EOF def lineReceived(self, line): print line # This will be called when the connection is terminated def connectionLost(self, _): print "-"*40 </pre> <hr /> <em>Phoebe (Blind Sided, season 1) -- Standard dating protocol.</em> <h2>Finger - client factory</h2><ul> <li>Keep configuration information</li> <li>In this case, just the username</li></ul> <pre class="python"> from twisted.internet import protocol class FingerFactory(protocol.ClientFactory): protocol = FingerProtocol def __init__(self, user): self.user = user def clientConnectionFailed(self, _, reason): print "error", reason.value </pre> <hr /> <em>Jack (Ms. Hellfire, season 2) -- Well, they'd better be a rich client</em> <h2>Finger - tying it all together</h2><ul> <li>Actually run above code</li> <li>Use reactors</li></ul> <pre class="python"> from twisted.internet import reactor import sys user, host = sys.argv[1].split('@') port = 79 reactor.connectTCP(host, port, FingerFactory(port)) reactor.run() </pre> <hr /> <em>Prue/Phoebe/Piper (Something Wicca This Way Comes, season 1) -- The power of three will set us free</em> <h2>Finger - a bug</h2><ul> <li>Succeed or fail, program doesn't exit</li> <li>Reactor continues in a loop</li> <li>Takes almost no CPU time...</li> <li>...but still wrong behaviour</li> </ul> <hr /> <em>Leo (Trial By Magic, season 4) -- Demons you can handle but not rats?</em> <h2>Digression - Deferreds</h2><ul> <li>In order to be more flexible, we want callbacks</li> <li>Common callbacks are too weak</li> <li>We used 'deferreds' as an abstraction for callbacks</li> </ul> <hr /> <em>Piper (Morality Bites, season 2) -- Talk about it later.</em> <h2>Finger - reimplementing correctly</h2> <pre class="python"> from twisted.protocols import basic from twisted.internet import protocol, defer import sys class FingerClient(basic.LineReceiver): def connectionMade(self): self.transport.write(self.factory.user+"\n") def lineReceived(self, line): self.factory.gotLine(line) </pre> <h2>Finger - reimplementing correctly (cont'd)</h2> <pre class="python"> class FingerFactory(protocol.ClientFactory): protocol = FingerProtocol def __init__(self, user): self.user, self.d = user, defer.Deferred() def gotLine(self, line): print line def clientConnectionLost(self, _, why): self.d.callback(None) def clientConnectionFailed(self, _, why): self.d.errback(why) </pre> <h2>Finger - reimplementing correctly (cont'd 2)</h2> <pre class="python"> if __name__ == '__main__': from twisted.internet import reactor from twisted.python import util user, host = sys.argv[1].split('@') f = FingerFactory(user) port = 79 reactor.connectTCP(host, port, FingerFactory(port)) f.d.addCallback(lambda _: reactor.stop()) f.d.addErrback(lambda _: (util.println("could not connect"), reactor.stop())) reactor.run() </pre> <hr /> <em>Phoebe (Charmed and Dangerous, season 4) -- That's what we were missing.</em> <h2>Servers</h2><ul> <li>Servers are actually easier</li> <li>Servers meant to wait for events</li> <li>Most of concepts similar to clients</li> </ul> <hr /> <em>Genie (Be Careful What You Witch For, season 2) -- All I know is that you rubbed and now I serve.</em> <h2>Finger - protocol</h2> <pre class="python"> class FingerServer(basic.LineReceiver): def lineReceived(self, line): self.transport.write(self.factory.getUser(line)) self.transport.loseConnection() </pre> <hr /> <em>Secretary (The Painted World, season 2) -- Well, you won't have any trouble with this if you figured that out.</em> <h2>Finger - factory</h2> <pre class="python"> class FingerServerFactory(protocol.Factory): protocol = FingerServer def __init__(self): self.users = {} self.message = "No such user\n" def getUser(self, name): return self.users.get(name, self.message) def setUser(self, user, status): self.users[user] = status </pre> <hr /> <em>Prue (The Demon Who Came In From the Cole, season 3) -- Okay, so who are they?</em> <h2>Finger - glue</h2> <pre class="python"> factory = FingerServerFactory() factory.setUser("moshez", "Online - Sitting at computer\n") factory.setUser("spiv", "Offline - Surfing the waves\n") reactor.listenTCP(79, factory) </pre> <hr /> <em>Prue (All Halliwell's Eve, season 3) -- Put it all together, it may just work.</em> <h2>Finger Server - problem</h2><ul> <li>What if server has to actually work to find user's status?</li> <li>For example, read status from a website</li> <li>API forces us to block -- not good</li> </ul> <hr /> <em>Piper (All Halliwell's Eve, season 3) -- We've got big problems, a little time and a little magic.</em> <h2>Finger server -- new protocol</h2> <pre class="python"> class FingerServer(basic.LineReceiver): def lineReceived(self, line): d = self.factory.getUser(line) d.addCallback(self.writeResponse) d.addErrback(self.writeError) def writeResponse(self, response): self.transport.write(response) self.transport.loseConnection() def writeError(self, error): self.transport.write("Server error -- try later\n") self.transport.loseConnection() </pre> <hr /> <em>Piper (Ex Libris, season 2) -- We'll worry about it later.</em> <h2>Finger - factory</h2> <pre class="python"> class FingerServerFactory(protocol.Factory): protocol = FingerServer def __init__(self): self.users = {} self.message = "No such user\n" def getUser(self, name): return defer.succeed(self.users.get(name, self.message)) def setUser(self, user, status): self.users[user] = status </pre> <hr /> <em>Piper/Zen Master (Enter the Demon, season 4) -- It is a different realm down there with new rules.</em> <h2>Finger - web factory</h2> <pre class="python"> from twisted.web import client class FingerWebFactory(protocol.Factory): protocol = FingerServer def getUser(self, name): url = "http://example.com/~%s/online" % name d = client.getPage(url) d.addErrback(lambda _: "No such user\n") return d </pre> <hr /> <em>Applicant #3 (The Painted World, season 2) -- in this day and age, who can't write in the HTML numeric languages, right?</em> <h2>Application</h2><ul> <li>The Twisted way of configuration files</li> <li>Decouple configuration from running</li></ul> <h2>Application (Example)</h2> <pre class="python"> # File: finger.tpy from twisted.internet import app import fingerserver factory = fingerserver.FingerServerFactory() factory.setUser("moshez", "Online - Sitting at computer\n") factory.setUser("spiv", "Offline - Surfing the waves\n") application = app.Application("finger") application.listenTCP(79, factory) </pre> <hr /> <em>Paige (Hell Hath No Fury, season 4) -- I am taking full responsibility for being late with the application.</em> <h2>twistd</h2><ul> <li>TWISTed Daemonizer</li> <li>Daemonizes Twisted servers</li> <li>Takes care of log files, PID files, etc.</li> <li>twistd -y finger.tpy</li> </ul> <hr /> <em>Phoebe (Sleuthing With the Enemy, season 3) -- Was it some sick twisted demonic thrill?</em> <h2>twistd examples</h2><ul> <li>twistd -y finger.tpy -l /var/finger/log</li> <li>twistd -y finger.tpy --pidfile /var/run/finger.pid</li> <li>twistd -y finger.tpy --chroot /var/run</li> </ul> <hr /> <em>Professor Whittlessy (Is There a Woogy In the House?, season 1) -- I use your house as an example</em> <h2>Writing Plugins</h2><ul> <li>Automatically create application configurations</li> <li>Accessible via commandline or GUI</li></ul> <h2>Writing Plugins (Example)</h2> <pre class="python"> # File finger/tap.py from twisted.python import usage class Options(usage.Options): synopsis = "Usage: mktap finger [options]" optParameters = [["port", "p", 6666,"Set the port number."]] longdesc = 'Finger Server' users = () def opt_user(self, user): if not '=' in user: status = "Online" else: user, status = user.split('=', 1) self.users += ((user, status+"\n"),) </pre> <h2>Writing Plugins (Example cont'd)</h2> <pre class="python"> def updateApplication(app, config): f = FingerFactory() for (user, status) in config.users: f.setUser(user, status) app.listenTCP(int(config.opts['port']), s) </pre> <hr /> <em>Paige (Bite Me, season 4) -- They won't join us willingly.</em> <h2>Writing Plugins (Example cont'd 2)</h2> <pre class="python"> # File finger/plugins.tml register("Finger", "finger.tap", description="Finger Server", type='tap', tapname="finger") </pre> <hr /> <em>Queen (Bite Me, season 4) -- That's what families are for.</em> <h2>Using mktap</h2><ul> <li>mktap finger --user moshez --user spiv=Offline</li> <li>twistd -f finger.tap</li> </ul> <hr /> <em>Piper (Charmed and Dangerous, season 4) -- We'll use potions instead.</em> <h2>Delayed execution</h2><ul> <li>Basic interface: reactor.callLater(<time>, <function>, [<arg>, [<arg> ...]])</li> <li>reactor.callLater(10, reactor.stop)</li> <li>reactor.callLater(5, util.println, 'hello', 'world')</li> </ul> <hr /> <em>Cole (Enter the Demon, season 4) -- I know, but not right now.</em> <h2>callLater(0,) -- An idiom</h2><ul> <li>Use to set up a call in next iteration of loop</li> <li>Can be used in algorithm-heavy code to let other code run</li></ul> <pre class="python"> def calculateFact(cur, acc=1, d=None): d = d or defer.Deferred() if cur<=1: d.callback(acc) else: reactor.callLater(0, calculateFact, acc*cur, cur-1, d) calculateFact(10 ).addCallback(lambda n: (util.println(n), reactor.stop())) reactor.run() </pre> <hr /> <em>Piper (Lost and Bound, season 4) -- Someone, I won't say who, has the insane notion</em> <h2>UNIX Domain Sockets</h2><ul> <li>Servers<ul><li>reactor.listenUNIX('/var/run/finger.sock', FingerWebFactory())</li> </ul></li> <li>Clients<ul><li>reactor.connectUNIX('/var/run/finger.sock', FingerFactory())</li> </ul></li> </ul> <hr /> <em>Kate (Once Upon a Time, season 3) -- Fairies don't talk the same way people do.</em> <h2>SSL Servers</h2> <pre class="python"> from OpenSSL import SSL class ServerContextFactory: def getContext(self): ctx = SSL.Context(SSL.SSLv23_METHOD) ctx.use_certificate_file('server.pem') ctx.use_privatekey_file('server.pem') return ctx reactor.listenSSL(111, FingerWebFactory(), ServerContextFactory()) </pre> <h2>SSL Clients</h2> <ul> <li>from twisted.internet import ssl</li> <li>reactor.connectSSL(111, 'localhost', FingerFactory(), ssl.ClientContextFactory())</li> </ul> <hr /> <em>Natalie (Blinded By the Whitelighter, season 3) -- I mean, in private if you wouldn't mind</em> <h2>Running Processes</h2><ul> <li>A process has two outputs: stdout and stderr</li> <li>Protocol to interface with it is different</li></ul> <pre class="python"> class Advertizer(protocol.ProcessProtocol): def outReceived(self, data): print "out", `data` def errReceived(self, data): print "error", `data` def processEnded(self, reason): print "ended", reason reactor.spawnProcess(Advertizer(), "echo", ["echo", "hello", "world"]) </pre> <hr /> <em>Prue (Coyote Piper, season 3) -- You have to know that you can talk to me</em> <h2>Further Reading</h2><ul> <li><a href="http://twistedmatrix.com/documents/">Twisted Docs</a></li> </ul> <hr /> <em>Phoebe (Animal Pragmatism, season 2) -- Ooh, the girls in school are reading this.</em> <h2>Questions?</h2> <em>Piper (Something Wicca This Way Comes, season 1) -- Tell me that's not our old spirit board?</em> <h2>Bonus Slides</h2> <em>Prue (Sleuthing With the Enemy, season 3) -- All right, you start talking or we start the bonus round.</em> <h2>Perspective Broker</h2><ul> <li>Meant to be worked async</li> <li>Can transfer references or copies</li> <li>Secure (no pickles or other remote execution mechanisms)</li> <li>Lightweight (bandwidth and CPU)</li> <li>Translucent</li> </ul> <hr /> <em>Paige (Charmed Again, season 4) -- I guess I just kind of feel - connected somehow.</em> <h2>PB Remote Control Finger (Server)</h2> <pre class="python"> from twisted.spread import pb class FingerSetter(pb.Root): def __init__(self, ff): self.ff = ff def remote_setUser(self, name, status): self.ff.setUser(name, status+"\n") ff = FingerServerFactory() setter = FingerSetter(ff) reactor.listenUNIX("/var/run/finger.control", pb.BrokerFactory(setter)) </pre> <hr /> <em>Piper (Be Careful What You Witch For, season 2) -- Okay, you think you can control the power this time?</em> <h2>PB Remote Control Finger (Client)</h2> <pre class="python"> from twisted.spread import pb from twisted.internet import reactor import sys def failed(reason): print "failed:", reason.value;reactor.stop() pb.getObjectAt("unix", "/var/run/finger.control", 30 ).addCallback(lambda o: o.callRemote("setUser", *sys.argv[1:3], ).addCallbacks(lambda _: reactor.stop(), failed) reactor.run() </pre> <hr /> <em>Leo (Be Careful What You Witch For, season 2) -- How about you just keep your arms down until you learn how to work the controls.</em> <h2>Perspective Broker (Trick)</h2><ul> <li>Add to the application something which will call reactor.stop()</li> <li>Portable (works on Windows)</li> <li>Gets around OS security limitations</li> <li>Need to add application-level security</li> <li>The docs have the answers (see 'cred')</li> </ul> <hr /> <em>Piper (Lost and Bound, season 4) -- They're not good or bad by themselves, it's how we use them</em> <h2>Perspective Broker (Authentication)</h2><ul> <li>pb.cred</li> <li>Perspectives</li> <li>Can get remote user with every call<ul><li>Inherit from pb.Perpsective</li> <li>Call methods perspective_<name>(self, remoteUser, ...)</li> </ul></li> </ul> <hr /> <em>Piper (She's a Man, Baby, a Man!, season 2) -- Okey-Dokey. I get the point.</em> <h2>Perspective Broker - About Large Data Streams</h2> <ul> <li>Sending large (>640kb) strings is impossible -- feature, not bug.</li> <li>It stops DoSes</li> <li>Nobody would ever need...<ul><li>JokeTooOldError</li></ul></li> <li>Use twisted.spread.utils.Pager -- sends the data in managable chunks.</li> </ul> <hr /> <em>Piper (Womb Raider, season 4) -- Oral tradition tales of a giant whose body served as a portal to other dimensions.</em> <h2>Producers and Consumers</h2><ul> <li>Use for things like sending a big file</li> <li>A good alternative to manually reactor.callLater(0,)-ing</li> <li>See twisted.internet.interfaces.{IProducer,IConsumer}</li> </ul> <hr /> <em>Phoebe (Black as Cole, season 4) -- Apparently he feeds on the remains of other demons' victims.</em> <h2>Threads (callInThread)</h2><ul> <li>Use for long running calculations</li> <li>Use for blocking calls you can't do without</li> <li>deferred = reactor.callInThread(function, arg, arg)</li> </ul> <hr /> <em>Piper (The Painted World, season 2) -- There will be consequences. There always are.</em> <h2>Threads (callFromThread)</h2><ul> <li>Use from a function running in a different thread</li> <li>Always thread safe</li> <li>Interface to non-thread-safe APIs</li> <li>reactor.callFromThread(protocol.transport.write, s)</li> </ul> <hr /> <em>Phoebe (Witch Trial, season 2) -- Maybe it's still in the house. Just on different plane.</em> <h2>Using ApplicationService</h2><ul> <li>Keep useful data...</li> <li>...or useful volatile objects</li> <li>Support start/stop notification</li> <li>Example: process monitor</li> </ul> <hr /> <em>Phoebe (Marry Go Round, season 4) -- Yeah, that's just in case you need psychic services.</em> <h2>Playing With Persistence</h2><ul> <li>Shutdown taps are useful</li> <li>Even if you use twistd -y</li> <li>So remember<ul><li>Classes belong in modules</li> <li>Functions belong in modules</li> <li>Modifying class attributes should be avoided</li> </ul></li> </ul> <hr /> <em>Cole (Marry Go Round, season 4) -- That Lazerus demon is a time bomb waiting to explode</em> </body></html>