Bit Fields in Python

binary data

<Shameless Plug>

We use home-grown tool at Promenade called Parlay that allows us to write python scripts to talk to embedded devices, script behaviors, unit test hardware and simulate hardware we don’t have yet. It’s a super easy to use solution, written in Python and Javascript that can get non-embedded programmers (like me) up and running on hardware fast, and allows non-programmers (like scientists or testers) to interact and script behaviors without having to take any programming courses.  It’s dual licensed GPL and Commercial, so feel free to clone GPL version on github.

</Shameless Plug>

So that means we’re interfacing Python code with packed binary protocols over RS-232, CAN, GPIB, etc all the time.

Python’s struct library makes interfacing with binary protocols a snap.  For instance, if there is a struct like this in C (and it’s packed on a little endian machine before being sent over a serial line)

then we can read it from a binary buffer in python like this

and we can write to a binary buffer like this

struct.pack and struct.unpack make communicating over a binary protocol a snap. They are some of my favorite examples of how Python makes tedious tasks simple, easy, and readable.

What about Bitfields?

struct.pack doesn’t work well with a C struct that uses bitfields though. For example, the following C struct only takes up 32 bits (size of unsigned int). 4 bits for x, 3 bits for y, 5 bits for z and 20 bits for c.

The struct library  will let us get a 32 bit int out of the buffer, but doesn’t give you the ability to break apart arbitrary bits. So we’re stuck with an ugly solution like this

Yuck!  That’s verbose, prone to math errors and a pain to maintain if there are any changes to the struct. Seriously, even this toy example took multiple attempts and a pad of paper to get right. You have to mentally keep track of endianess, convert the bit masks from binary. There is way too much chance for human error here.

CTypes to the Rescue

Time to bring in my good friend ctypes.  CTypes is used when you want to interface Python with a C library. It’s typically used when trying to leverage  C code that is already written for legacy or performance reasons.

It turns out that CTypes can be used to make our life a lot easier, even when talking over a remote protocol like Serial. Check out this example:

 

Looks an awful lot like the struct we defined in our C code doesn’t it? That’s because that’s exactly what it is!

PacketBits inherits from  ctypes.LittleEndianStructure, which means it will be packed into a little endian structure. Each field has 3 arguments (name, ctypes type, bit-length) just like in a C struct.

The class Packet is a union between the bit field struct, and a simple 32 bit int, so we can easily pack the full structure to and from struct.pack and struct.unpack for transport.

For example

 

That’s all there is to it.  ctypes.struct is easy to use, easy to maintain and best of all makes the code look pythonic.

Intro to Twisted and Event Driven Programming – a simple chat server

This is a companion to a simple tutorial I gave at the OCPython Meetup in Irvine on October 6th, 2015.

What is Event-Driven programming?

Twisted is an asynchronous event-driven communication library that implements the Reactor Pattern. What this means is that twisted takes all of the asynchronous events coming from things like TCP/UDP streams, Serial lines, etc, and turns them into discrete ‘events’. The reactor then ‘reacts’ to these events one by one by calling your code.

For example, instead of writing code like this:

you would set up an event listener  like this:

This inversion of control makes it simple and straightforward to write certain types of servers. Also, since you only ever deal with one event at a time you never have to worry about threads or race conditions.

Things event-driven programming is good at:

  • I/O bound applications (Where you spend most of your time ‘waiting’ for other things)
  • Applications with heavy resource sharing
  • High throughput but low processing overhead applications (i.e. get in and get out fast)

Things event-driven programming is bad at (and all the ways twisted tries to mitigate these negatives):

  • Interfacing with blocking code (Twisted has easy mapping between deferred and threads with deferToThread() and blockingCallFromThread())
  • Long procedures a.k.a. ‘Callback Hell’ (Twisted’s @inlineCallbacks uses generators to make async code that looks and smells like synchronous procedural code.)
  • Processor intensive applications. Since the reactor can only ‘react’ to one event at a time, any cpu-hogging code effectively blocks the entire server. (Twisted’s deferToThread() can offload CPU intensive functions to another thread, but it has its limitations)
  • Parallel code. The reactor can only ‘react’ to one event at a time, and this can not be parrallelized. (Twisted offers no mitigation here. If your code is embarrassingly parallel, you’re better off going with threads)

 

Now let’s talk about Twisted.

The main objects in twisted are: The Reactor, Transports, Protocols, Factories, Deferreds.

  • The Reactor  is the heart of twisted. It listens on ports, manages events, and pulls the whole system together. There should only ever be one in your application, so treat it like a singleton.
  • Transports  are the physical wire (or effective equivalent abstraction) that carry bytes. E.g.: TCP, UDP, Serial, Modbus, etc.  Protocols use transports to send and receive events (e.g.: messages/packets/data). This de-coupling is important because it means that the same protocol code can work with any transport it is bound with.
  •  Protocols  are application specific objects that represent a single connection. In the below chat-server examples, each client connection will have an associated Protocol object to manage that specific connection.
  • Factories create new protocols dynamically. Factories are attached to Transports through the Reactor, and when a new connection is established, the Factory creates the appropriate protocol object to manage that connection in its buildProtocol() method.
  • Deferreds are a way to schedule asynchronous callbacks without blocking. They are like javascript’s Promises, or Scala’s Futures. I won’t go into them in this post, but they are such an integral part of Twisted they had to be mentioned

See Twisted’s excellent getting started guide for more details

Chat Server

Here is the first version of the chat server. We create a custom protocol that inherits from LineReceiver. LineReceiver is a helper Protocol class that takes a stream of bytes from a transport, and calls lineReceived() when it hits a newline terminator. (\n by default, but can be customized to any byte-value)

The CharServerFactory is very simple. When a new protocol is requested by the reactor, it constructs it and returns it.

The second to last line instructs the reactor to listen on port 1234 for tcp connections, and to ask a ChatServerFactory for protocols when it gets a new connection. The last line actually runs the reactor. The reactor takes control of the main thread and this function call will not return.

Test it out! Run the script and open up Telnet (‘telnet localhost 1234’ on the command line) or launch PuTTY and connect to localhost:1234 .

Ok so that version isn’t really a chat server. It just prints the lines that were sent to it. Let’s add a feature so we know which of our connected clients typed each line.

How can we get the info of the connected client? Ask the transport of course! Each protocol gets a reference to its transport layer in the self.transport member variable.

Alright, so the server is now printing each line it gets sent, but we want each connected client to get the messages, not just have it printed on the console by the server.  LineReceiver has a helper method called sendLine() that sends a string down its associated transport. All we need is a reference to the protocol we want to send the line to.

Since the Factory constructs each protocol as it gets new connections, the Factory can keep track of them in a list, and can pass a reference to itself to each protocols constructor.

Hey! This is starting to actually look like a chat server now. Let’s add some more features like a MOTD, user enter/leave notifications and an easy way for clients to exit by typing /exit.

All protocols have a connectionMade() and connectionLost() class that get called when connections are established and destroyed. I refactored out the logic to send a line to all clients because it was called in multiple places.