glibc, yocto, and cross compiling

(There is a TL;DR for those of you just interested in the fix at the bottom)

I have a love/hate relationship with yocto. It promises to make building custom embedded linux variants as easy as ordering from a menu, and it seems to get me 90% there every time. The problem is with the last 10%. Hob, the UI, is constantly crashing, so I have to default to the command line interface. It uses absolute paths EVERYWHERE, making portability an problem, and debugging issues is a nightmare. Finding where recipes are, and which layers possibly amended or patched them is like trying to find a vaguely worded meta-needle in a haystack (grep is always your friend in situations like this).

My latest issue was when I tried to cross compile an Application built with Qt and Boost for an iMX6 board. The meta-qt5 layer seems too be mature enough that it built the sysroot and QT Creator environment without a hitch (minus the fact that I still had to patch it to explicitly choose ALSA over gstreamer).  Cross compiling was a different experience. The compiling itself worked fine, but the linker complained it couldn’t find /lib/libpthread.so.0 . I checked the sysroot, and sure enough it had a /lib/libpthread.so.0 .  I spent the next hour banging my head on the keyboard and yelling “ITS RIGHT THERE!!!” at the screen before I realized, “Wait a minute, is it really looking for /lib/libpthread.so.0″ (Emphasis on that first slash aka root on the root filesystem). Sure enough, if I copied libpthread.so.0 from my sysroot’s /lib to my host’s /lib, it compiled fine! “Great, more absolute paths – thanks yocto”, I thought.

Well, after digging a littler deeper it turns out this isn’t entirely yocto’s fault. It’s glibc’s. If you look at the libpthread.so.0 made by glibc, you’ll see that it  hard codes the path it gives to the linker

$SYSROOT/usr/lib/libpthread.so

I wasn’t sure about why it did this, until I read some snarky comments in the makefile of uClibc:

The amount of time saved by this optimization is actually too small to
measure. The linker just had to search the library path to find the
linker script, so the dentries are cache hot if it has to search the
same path again.  But it’s what glibc does, so we do it too.

Seems about as good a reason as any I guess. Hopefully this post helps someone avoid a couple hours of debugging and searching.

TL;DR: If you’re trying to cross compile an application and get a linker error like:

ld: cannot find /lib/libpthread.so.0

Then edit the following files:

/sysroots/${BOARD_NAME}/usr/lib/libpthread.so
/sysroots/${BOARD_NAME}/usr/lib/libc.so

and make the absolute paths relative paths so they look like this:

libpthread.so

libc.so

 

Leave a Reply

One comment

  1. Thanks for the hint, I spend some time to find what was the issue when I tried to create a meta-toolchain