Intro to Twisted and Event Driven Programming – a simple chat server

This is a companion to a simple tutorial I gave at the OCPython Meetup in Irvine on October 6th, 2015.

What is Event-Driven programming?

Twisted is an asynchronous event-driven communication library that implements the Reactor Pattern. What this means is that twisted takes all of the asynchronous events coming from things like TCP/UDP streams, Serial lines, etc, and turns them into discrete ‘events’. The reactor then ‘reacts’ to these events one by one by calling your code.

For example, instead of writing code like this:

you would set up an event listener  like this:

This inversion of control makes it simple and straightforward to write certain types of servers. Also, since you only ever deal with one event at a time you never have to worry about threads or race conditions.

Things event-driven programming is good at:

  • I/O bound applications (Where you spend most of your time ‘waiting’ for other things)
  • Applications with heavy resource sharing
  • High throughput but low processing overhead applications (i.e. get in and get out fast)

Things event-driven programming is bad at (and all the ways twisted tries to mitigate these negatives):

  • Interfacing with blocking code (Twisted has easy mapping between deferred and threads with deferToThread() and blockingCallFromThread())
  • Long procedures a.k.a. ‘Callback Hell’ (Twisted’s @inlineCallbacks uses generators to make async code that looks and smells like synchronous procedural code.)
  • Processor intensive applications. Since the reactor can only ‘react’ to one event at a time, any cpu-hogging code effectively blocks the entire server. (Twisted’s deferToThread() can offload CPU intensive functions to another thread, but it has its limitations)
  • Parallel code. The reactor can only ‘react’ to one event at a time, and this can not be parrallelized. (Twisted offers no mitigation here. If your code is embarrassingly parallel, you’re better off going with threads)


Now let’s talk about Twisted.

The main objects in twisted are: The Reactor, Transports, Protocols, Factories, Deferreds.

  • The Reactor  is the heart of twisted. It listens on ports, manages events, and pulls the whole system together. There should only ever be one in your application, so treat it like a singleton.
  • Transports  are the physical wire (or effective equivalent abstraction) that carry bytes. E.g.: TCP, UDP, Serial, Modbus, etc.  Protocols use transports to send and receive events (e.g.: messages/packets/data). This de-coupling is important because it means that the same protocol code can work with any transport it is bound with.
  •  Protocols  are application specific objects that represent a single connection. In the below chat-server examples, each client connection will have an associated Protocol object to manage that specific connection.
  • Factories create new protocols dynamically. Factories are attached to Transports through the Reactor, and when a new connection is established, the Factory creates the appropriate protocol object to manage that connection in its buildProtocol() method.
  • Deferreds are a way to schedule asynchronous callbacks without blocking. They are like javascript’s Promises, or Scala’s Futures. I won’t go into them in this post, but they are such an integral part of Twisted they had to be mentioned

See Twisted’s excellent getting started guide for more details

Chat Server

Here is the first version of the chat server. We create a custom protocol that inherits from LineReceiver. LineReceiver is a helper Protocol class that takes a stream of bytes from a transport, and calls lineReceived() when it hits a newline terminator. (\n by default, but can be customized to any byte-value)

The CharServerFactory is very simple. When a new protocol is requested by the reactor, it constructs it and returns it.

The second to last line instructs the reactor to listen on port 1234 for tcp connections, and to ask a ChatServerFactory for protocols when it gets a new connection. The last line actually runs the reactor. The reactor takes control of the main thread and this function call will not return.

Test it out! Run the script and open up Telnet (‘telnet localhost 1234’ on the command line) or launch PuTTY and connect to localhost:1234 .

Ok so that version isn’t really a chat server. It just prints the lines that were sent to it. Let’s add a feature so we know which of our connected clients typed each line.

How can we get the info of the connected client? Ask the transport of course! Each protocol gets a reference to its transport layer in the self.transport member variable.

Alright, so the server is now printing each line it gets sent, but we want each connected client to get the messages, not just have it printed on the console by the server.  LineReceiver has a helper method called sendLine() that sends a string down its associated transport. All we need is a reference to the protocol we want to send the line to.

Since the Factory constructs each protocol as it gets new connections, the Factory can keep track of them in a list, and can pass a reference to itself to each protocols constructor.

Hey! This is starting to actually look like a chat server now. Let’s add some more features like a MOTD, user enter/leave notifications and an easy way for clients to exit by typing /exit.

All protocols have a connectionMade() and connectionLost() class that get called when connections are established and destroyed. I refactored out the logic to send a line to all clients because it was called in multiple places.