<Shameless Plug>
We use home-grown tool at Promenade called Parlay that allows us to write python scripts to talk to embedded devices, script behaviors, unit test hardware and simulate hardware we don’t have yet. It’s a super easy to use solution, written in Python and Javascript that can get non-embedded programmers (like me) up and running on hardware fast, and allows non-programmers (like scientists or testers) to interact and script behaviors without having to take any programming courses. It’s dual licensed GPL and Commercial, so feel free to clone GPL version on github.
</Shameless Plug>
So that means we’re interfacing Python code with packed binary protocols over RS-232, CAN, GPIB, etc all the time.
Python’s struct library makes interfacing with binary protocols a snap. For instance, if there is a struct like this in C (and it’s packed on a little endian machine before being sent over a serial line)
1 2 3 4 5 6 |
struct { int x; short y; char z; } |
then we can read it from a binary buffer in python like this
1 2 3 |
# buf is the buffer of bytes # < = little endian, i = int, u = short, c = char x,y,z = struct.unpack("<iuc", buf) |
and we can write to a binary buffer like this
1 2 3 |
# buf is a binary buffer # < = little endian, i = int, u = short, c = char buf = struct.pack("<iuc", x, y, z) |
struct.pack and struct.unpack make communicating over a binary protocol a snap. They are some of my favorite examples of how Python makes tedious tasks simple, easy, and readable.
What about Bitfields?
struct.pack doesn’t work well with a C struct that uses bitfields though. For example, the following C struct only takes up 32 bits (size of unsigned int). 4 bits for x, 3 bits for y, 5 bits for z and 20 bits for c.
1 2 3 4 5 6 7 |
struct { unsigned int x : 4 unsigned int y : 3 unsigned int z : 5 unsigned int c : 20 } |
The struct library will let us get a 32 bit int out of the buffer, but doesn’t give you the ability to break apart arbitrary bits. So we’re stuck with an ugly solution like this
1 2 3 4 5 6 7 8 9 10 |
# buf is the binary buffer i = struct.unpack("<i", buf) x = i & 0xf # 4 bits i = i >> 4 y = i & 0x7 # 3 bits i = i >> 3 z = i & 0x1f # 5 bits i = i >> 5 c = i & 0xfffff # 20 bits |
Yuck! That’s verbose, prone to math errors and a pain to maintain if there are any changes to the struct. Seriously, even this toy example took multiple attempts and a pad of paper to get right. You have to mentally keep track of endianess, convert the bit masks from binary. There is way too much chance for human error here.
CTypes to the Rescue
Time to bring in my good friend ctypes. CTypes is used when you want to interface Python with a C library. It’s typically used when trying to leverage C code that is already written for legacy or performance reasons.
It turns out that CTypes can be used to make our life a lot easier, even when talking over a remote protocol like Serial. Check out this example:
1 2 3 4 5 6 7 8 9 10 11 12 13 |
import ctypes class PacketBits(ctypes.LittleEndianStructure): _fields_ = [ ("x", ctypes.c_uint32, 4), ("y", ctypes.c_uint32, 3), ("z", ctypes.c_uint32, 5), ("c", ctypes.c_uint32, 20), ] class Packet(ctypes.Union): _fields_ = [("bits", PacketBits), ("binary_data", ctypes.c_uint32)] |
Looks an awful lot like the struct we defined in our C code doesn’t it? That’s because that’s exactly what it is!
PacketBits inherits from ctypes.LittleEndianStructure, which means it will be packed into a little endian structure. Each field has 3 arguments (name, ctypes type, bit-length) just like in a C struct.
The class Packet is a union between the bit field struct, and a simple 32 bit int, so we can easily pack the full structure to and from struct.pack and struct.unpack for transport.
For example
1 2 3 4 5 6 7 8 9 10 11 |
packet = Packet() packet.binary_data = struct.unpack("<i", buf) # Thats it. We can now interact with the fields in packet.bits print packet.bits.x, packet.bits.y, packet.bits.z ... # make a few changes packet.bits.x = 6 packet.bits.c = 26 # pack it up for sending down a serial line buf = struct.pack("<i", packet.binary_data) |
That’s all there is to it. ctypes.struct is easy to use, easy to maintain and best of all makes the code look pythonic.