Bit Fields in Python

binary data

<Shameless Plug>

We use home-grown tool at Promenade called Parlay that allows us to write python scripts to talk to embedded devices, script behaviors, unit test hardware and simulate hardware we don’t have yet. It’s a super easy to use solution, written in Python and Javascript that can get non-embedded programmers (like me) up and running on hardware fast, and allows non-programmers (like scientists or testers) to interact and script behaviors without having to take any programming courses.  It’s dual licensed GPL and Commercial, so feel free to clone GPL version on github.

</Shameless Plug>

So that means we’re interfacing Python code with packed binary protocols over RS-232, CAN, GPIB, etc all the time.

Python’s struct library makes interfacing with binary protocols a snap.  For instance, if there is a struct like this in C (and it’s packed on a little endian machine before being sent over a serial line)

then we can read it from a binary buffer in python like this

and we can write to a binary buffer like this

struct.pack and struct.unpack make communicating over a binary protocol a snap. They are some of my favorite examples of how Python makes tedious tasks simple, easy, and readable.

What about Bitfields?

struct.pack doesn’t work well with a C struct that uses bitfields though. For example, the following C struct only takes up 32 bits (size of unsigned int). 4 bits for x, 3 bits for y, 5 bits for z and 20 bits for c.

The struct library  will let us get a 32 bit int out of the buffer, but doesn’t give you the ability to break apart arbitrary bits. So we’re stuck with an ugly solution like this

Yuck!  That’s verbose, prone to math errors and a pain to maintain if there are any changes to the struct. Seriously, even this toy example took multiple attempts and a pad of paper to get right. You have to mentally keep track of endianess, convert the bit masks from binary. There is way too much chance for human error here.

CTypes to the Rescue

Time to bring in my good friend ctypes.  CTypes is used when you want to interface Python with a C library. It’s typically used when trying to leverage  C code that is already written for legacy or performance reasons.

It turns out that CTypes can be used to make our life a lot easier, even when talking over a remote protocol like Serial. Check out this example:

 

Looks an awful lot like the struct we defined in our C code doesn’t it? That’s because that’s exactly what it is!

PacketBits inherits from  ctypes.LittleEndianStructure, which means it will be packed into a little endian structure. Each field has 3 arguments (name, ctypes type, bit-length) just like in a C struct.

The class Packet is a union between the bit field struct, and a simple 32 bit int, so we can easily pack the full structure to and from struct.pack and struct.unpack for transport.

For example

 

That’s all there is to it.  ctypes.struct is easy to use, easy to maintain and best of all makes the code look pythonic.

Leave a Reply

2 comments

  1. daniel L says:

    Could you show me the source code how to define buf ? Thanks

    • daniel L says:

      I just checked your github repository and found following snippet . I guess that’s what I am looking for:

      # Using byte array so unstuff can use numbers instead of strings
      buf = bytearray()