Build farm

ccache

ccache caches the files produced from each single file compilation and directly reuse the result when the hashed compilation context is the same. Currently works with GCC, Clang, MSVC.

sudo apt install ccache
# Specify where the cache files should be stored
export CCACHE_DIR="$HOME/Projects/myproject/ccache"

# ccache will read the config file in
1) $CCACHE_DIR/ccache.conf  # First priority
2) /etc/ccache.conf         # Second priority, system default
# Set the cache folder size in config file by:
max_size = 10.0G

# Example of using ccache in your build by prepending it to compiler
ccache g++ main.cpp
# You may also modify PATH to use compiler soft link overriden by ccache
export PATH="/usr/lib/ccache:$PATH" # Debian/Ubuntu
export PATH="/usr/lib/ccache:$PATH" # Fedora/CentOS/RHEL

# Force purge all caches
ccache -C

Distcc farm

distcc allows compiler to distribute compilation jobs to other hosts, therefore making use of computating power of idle machines. You may use docker/multipass to create isolated distcc server on powerful desktop to aid low power machine compilation.

Setup

sudo apt update
sudo apt install build-essential
sudo apt install distcc
# For client, very useful for visualizing work distribution. You can also use distccmon-text.
sudo apt install distccmon-gnome

### For server sharing their compuational power

sudo vim /etc/default/distcc
# STARTDISTCC="true"
# ALLOWEDNETS="10.74.100.0/24"  # Whitelist of accepted client, details are given in the file comment
# LISTENER="10.74.100.244"      # Have to manually set own ip
# JOBS="2"                      # Choose wisely to avoid lagging the server normal operation
# ZEROCONF="true"               # Allow client to use hostname(without .local) to specify the server
sudo systemctl restart distcc.service
sudo systemctl status distcc.service
distcc --show-hosts # Show the list of configured hosts
distcc -j           # Show the number of available job quota, can use this to set make job count
distccmon-gnome
distccmon-text


### For client using the computation power, set the variable

# Here are some examples:
# By default distcc will use hosts in /etc/distcc/hosts, override by DISTCC_HOSTS
# Earlier host always get priority, so set localhost or closer/faster server first
# This is especially significant in single thread operations such as ./configure
DISTCC_HOSTS="distcc01 distcc02"
DISTCC_HOSTS="distcc01/2 distcc02/2" # Specify max job for each server

# Configure the wrapped compiler in this format
CC="distcc gcc"
CXX="distcc g++"

# There are two sensible options to further improve performance
# 1) Use ccache together
CC="ccache distcc gcc" CXX="ccache distcc g++"
# 2) Use pump mode, currently the fastest option
TODO
# 3) Use ccache and pump mode together
# it hurts performance, just go for either 1) or 2)



could not get +zeroconf automatic LAN server discovery working

CMake build simple example

# There are two distcc servers: distcc01 and distcc02
sudo apt install libssl-dev build-essential gcc g++
wget https://github.com/Kitware/CMake/releases/download/v3.27.1/cmake-3.27.1.tar.gz
tar xvzf cmake-3.27.1.tar.gz
cd cmake-3.27.1
DISTCC_HOSTS="localhost distcc01 distcc02" CC="distcc gcc" CXX="distcc g++" ./bootstrap --parallel=4
DISTCC_HOSTS="distcc01 distcc02" CC="distcc gcc" CXX="distcc g++" make -j4

Cross-compiling

Find the client compiler info by running on client machine $client> gcc -v
Follow the specs and guess which compiler you should get.
Here is the one where I guess to be matching my system:

$server> sudo apt install gcc-9-aarch64-linux-gnu
$server> sudo apt install g++-9-aarch64-linux-gnu

For this installation, you should get aarch64-linux-gnu-gcc-9 and aarch64-linux-gnu-g++-9

Check if their specs match the one on client

$server> aarch64-linux-gnu-gcc-9 -v
$server> aarch64-linux-gnu-g++-9 -v

Check that they are listed in /usr/lib/distcc
distccd is using a whitelist, only compilers listed in that folder is allowed to run on server.
If new compiler does not show up there, check your apt installation first,
then resolve to build your own soft link if your installation is really alright
Use distcc with full compiler name on client side
```
$client> CC="distcc aarch64-linux-gnu-gcc-9"
$client> CXX="distcc aarch64-linux-gnu-g++-9"
```
Note that if you want to compile using client as well, the client need to have that compiler as well.
Client side probably already have that soft link if your compiler guess is correct.
Usually client side have a soft link chain of gcc -> gcc-9 -> aarch64-linux-gnu-gcc-9 in this case.

Faster compilation

# Your day saver
ccache
distcc / icecream

Benchmark on x86 desktop and pi

Compiling the powder toy release build v97.0.352 on rpi4b
There are two available distcc servers
| Name | CPU Model | CPU freq | Offered physical cores | |——–|————————————-|———-|————————| | x86cc | Intel i7-9700 | 4.70 GHz | 6 | | xavier | Nvidia ARMv8 Processor aarch64 8C8T | 2.27 GHz | 8 |

Here are the benchmark results:
| Name | Round | DISTCC_HOSTS | job count | Real time | |—————|——-|———————–|———–|————| | vanilla | NA | NA | 4 | 10m48.169s | | ccache | 2 | NA | 4 | 0m8.726s | | distcc | NA | x86cc/6 | 6 | 1m23.241s | | distcc | NA | x86cc/12 | 12 | 1m6.245s | | distcc | NA | x86cc/12 localhost/2 | 14 | 1m16.037s | | distcc | NA | xavier/16 localhost/2 | 18 | 2m38.676s | | distcc | NA | x86cc/12 xavier/16 | 28 | 1m18.603s | | distcc | NA | x86cc/12 xavier/8 | 20 | 1m19.018s | | ccache+distcc | 1 | x86cc/12 | 12 | 1m38.261s | | ccache+distcc | 2 | x86cc/12 | 12 | 0m8.993s |

It is observed that the bottleneck is local CPU/disk speed in preprocessing the files,
which could not be done by distcc (pump mode seems to counter this)
and the localhost could not keep up with number of distcc server jobs.
In such case, the distcc server CPU is not saturated and CPU usage is limited.

ROS faster compilation

This is useful especially when you are using rpi, which has limited computational power
and ROS compilation time is long in default config.

# Current best: ccache + distcc

# Setup
# just like your usual ccache distcc setup
export DISTCC_HOSTS="ccserver/12"
export CC="ccache distcc aarch64-linux-gnu-gcc-9"
export CXX="ccache distcc aarch64-linux-gnu-g++-9"
# If your previous build is not using ccache&distcc, make sure you remove all old CMake caches:
# rm -r ~/catkin_ws/devel/ ~/catkin_ws/build/
catkin_make -j 12