Cross compiling inside docker

Using docker for cross compilation
Published on August 2, 2018 under the tag docker, qemu, crooss-compilation, virtualization

Recently I needed to compile binaries for arm64 cpu. The fastest way to do this is to build cross compilation toolchain and to run x86 gcc that will generate arm64 executables.

Another way to do this is to use qemu’s user emulation. This feature of qemu allows to run executables that were compiled for different architecture on the x86 host by emulating the architecture and translating the system calls abi between the different architectures.

We can start by getting docker image with aarch64 centos file system.

docker pull aarch64/alpine

If we try to run our image we will get execution format error

docker run -it aarch64/alpine /bin/sh
standard_init_linux.go:195: exec user process caused "exec format error"

This is since we are trying to run aarch64 executable on x86 machine. By looking inside the image root file system we can see that the binary is an ARM aarch64 binary.

$ sudo file `d inspect aarch64/alpine -f '{{.GraphDriver.Data.RootDir}}'`/bin/busybox
/docker/overlay/edfece616e612e0aee33cfa5d0efe2b23c563d46ea232b2938d0e9c39a61273c/root/bin/busybox: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, stripped

To help us in running these arm64 binaries we can use the qemu’s user emulation. Firs we are going to compile static version of qemu’s user emulation for arm64 architecture.

We can use the following docker file to build the statically linked qemu inside docker instance. Edit Dockerfile:

FROM centos:7

RUN set -x \
    && yum groupinstall -y development \
    && yum install -y yum install wget glibc-static zlib-static pcre-static glib2-static

RUN set -x \
    && mkdir /data \
    && cd /data \
    && wget https://github.com/qemu/qemu/archive/v2.12.0.tar.gz \
    && tar xvf *.tar.gz \
    && cd qemu-2.12.0 \
    && ./configure --target-list=aarch64-linux-user --enable-linux-user --disable-system --disable-tools --static \
    && make -j4

Then we can copy the executable from it using

docker build -t build-env .
docker run --rm --name build-env-instance -dt build-env /bin/bash
docker cp build-env-instance:/data/qemu-2.12.0/aarch64-linux-user/qemu-aarch64 .
docker stop build-env-instance

Now we have qemu-aarch64 in our current directory. Let’s use it to build docker instance that can run aarch64 executables. Edit Dockerfile.aarch64

FROM aarch64/alpine
ADD qemu-aarch64 /

Let’s build it and try to run it.

docker build -t my-aarch64-env -f Dockerfile.aarch64 .
docker run --rm -it my-aarch64-env /bin/sh

But we are still getting:

standard_init_linux.go:195: exec user process caused "exec format error"

The last step is to instruct our kernel to apply the qemu-aarch64 interpreter each time we running arm64 elf file. This can be done using

# if /proc/sys/fs/binfmt_misc is not mounted
sudo mount binfmt_misc -t binfmt_misc /proc/sys/fs/binfmt_misc
sudo bash -c "echo ':aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00:\xff\xff\xff\xff\xff\xff\xff\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfe\xff\xff\xff:/qemu-aarch64:' > /proc/sys/fs/binfmt_misc/register"

And now we can run and compile arm64 code on x86

docker run --rm -it my-aarch64-env /bin/sh
$ uname -m
aarch64
$ apk update -q
$ apk add -q file
$ file /bin/busybox
/bin/busybox: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, stripped
$ apk add gcc libc-dev -q
$ echo 'int main() { printf("hello world\n"); }' > 1.c
$ gcc 1.c
1.c: In function 'main':
1.c:1:14: warning: implicit declaration of function 'printf' [-Wimplicit-function-declaration]
 int main() { printf("hello world\n"); }
              ^~~~~~
1.c:1:14: warning: incompatible implicit declaration of built-in function 'printf'
1.c:1:14: note: include '<stdio.h>' or provide a declaration of 'printf'
$ ./a.out
hello world
$ file a.out
a.out: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, not stripped
$

If we run ps on our host we can see that the /bin/sh in the container is actually executed using the qemu-aarch64 interpreter

$ ps -ef |grep qemu |grep -v grep
root     16389 16362  0 12:14 pts/0    00:00:00 /qemu-aarch64 /bin/sh