Building and Booting a Custom Linux Kernel: A Real-World Journey.

Building and Booting a Custom Linux Kernel: A Real-World Journey

Building a custom Linux kernel from source is an exciting but challenging task. This post documents my step-by-step process of downloading, customizing, compiling, installing, and booting a custom Linux kernel on a Debian cloud server. It also covers the problems I encountered along the way and how I solved them.

Environment Setup

I started with a Debian 12 server hosted on DigitalOcean. The machine is headless with SSH-only access, running the stock kernel 6.1.0-26-amd64. My goal was to:

  • Download Linux kernel 6.1 source code
  • Add a custom printk log message during boot
  • Compile and install the custom kernel alongside existing ones
  • Boot into my kernel and verify the changes

I installed the essential build tools:

sudo apt update
sudo apt install -y build-essential libncurses-dev bison flex libssl-dev libelf-dev bc git

Downloading and Preparing Kernel Source

I cloned the stable Linux kernel 6.1 source with:

git clone --depth=1 --branch v6.1 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git linux-6.1-custom
cd linux-6.1-custom

To uniquely identify my kernel build, I modified the Makefile to append a custom suffix:

EXTRAVERSION = -custom

Adding a Custom Boot Log

I wanted a clear indication that my kernel was running. I added the following line inside the start_kernel() function in init/main.c:

pr_info("Custom Kernel Loaded Successfully\n");

Important: This line must be inside a function body. Placing it outside caused cryptic compiler errors because kernel logging macros rely on specific macro expansions that can’t be used at global scope.

Kernel Configuration

To preserve existing settings, I copied the current config and updated it:

cp /boot/config-$(uname -r) .config
yes "" | make olddefconfig

A warning about ANDROID_BINDER_IPC=m appeared but was automatically handled by the build system.

Building the Kernel: Issues and Solutions

1. Kernel Image (bzImage) Not Generated

Initially, running:

make -j$(nproc)

only built kernel modules (.ko files). The actual kernel image arch/x86/boot/bzImage was missing. This caused make install to fail.

Solution: Explicitly build the kernel image with:

make bzImage -j$(nproc)

2. BTF Debug Info Build Failure

The build failed at the vmlinux stage with:

BTF: .tmp_vmlinux.btf: pahole (pahole) is not available
Failed to generate BTF for vmlinux
Try to disable CONFIG_DEBUG_INFO_BTF

The build system needs the external tool pahole to generate BPF Type Format debug info.

Solution Options:

  • Disable BTF debug info:

    scripts/config --disable CONFIG_DEBUG_INFO_BTF
make olddefconfig

  • Or install pahole:

    sudo apt install dwarves

For a quicker build, I chose to disable BTF.

Installing and Configuring GRUB

After successfully building the kernel and modules:

sudo make modules_install
sudo make install
sudo update-grub

GRUB detected my kernel as 6.1.0-custom+.

Booting the Custom Kernel

Despite installation, the system booted into the default kernel:

uname -r
6.1.0-37-amd64

I checked available GRUB entries:

grep "menuentry '" /boot/grub/grub.cfg | grep custom

Output showed:

Debian GNU/Linux, with Linux 6.1.0-custom+

To boot my kernel by default, I updated /etc/default/grub:

sudo sed -i 's|^GRUB_DEFAULT=.*|GRUB_DEFAULT="Advanced options for Debian GNU/Linux>Debian GNU/Linux, with Linux 6.1.0-custom+"|' /etc/default/grub
sudo update-grub

Rebooting this time booted into my custom kernel.

Verification

After reboot:

uname -r
# Output: 6.1.0-custom+

Checking the kernel log:

dmesg | grep "Custom Kernel Loaded"
# Output: Custom Kernel Loaded Successfully

Lessons Learned

  • Kernel logging macros (pr_info, printk) must be inside functions. Placing them outside causes compilation errors.
  • Explicitly building bzImage is necessary if make alone only compiles modules.
  • BTF debug info requires pahole. If missing, disable CONFIG_DEBUG_INFO_BTF or install pahole.
  • GRUB may default to older kernels. Update GRUB_DEFAULT to boot your custom kernel automatically.
  • On headless/cloud servers, configuring GRUB default is critical since interactive boot selection isn’t possible.

Conclusion

Building and booting a custom Linux kernel can be tricky but rewarding. Careful step-by-step work and troubleshooting will lead to success, deepening your understanding of Linux internals. This experience is invaluable for embedded development, kernel hacking, or custom OS work.

If you’d like to automate this process or have questions, feel free to reach out!

Happy hacking!