AAOS - Build a Kernel Module

Tags: automotive android kernel

Vendor modules are not included in the kernel. They are built separately, then included into the system based on the vendor configurations.

For general Android Kernel, refer to https://source.android.com/docs/setup/build/building-kernels.
For Cuttlefish Device, refer to https://source.android.com/docs/devices/cuttlefish/kernel-dev.

Prebuilt Kernel

The AOSP tree contains only prebuilt kernel binaries. When and AVD image is built, it copies the prebuilt kernel image to the output folder.

Start from the main makefile:

mgrep "aosp_cf_x86_64_auto"

./device/google/cuttlefish/AndroidProducts.mk:32: aosp_cf_x86_64_auto:$(LOCAL_DIR)/vsoc_x86_64_only/auto/aosp_cf.mk \

This makefile inherits from other make files:

$(call inherit-product, $(SRC_TARGET_DIR)/product/core_64_bit_only.mk)
$(call inherit-product, packages/services/Car/car_product/build/car_generic_system.mk)
$(call inherit-product, packages/services/Car/car_product/build/car_system_ext.mk)
$(call inherit-product, packages/services/Car/car_product/build/car_product.mk)
$(call inherit-product, device/google/cuttlefish/shared/auto/device_vendor.mk)
$(call inherit-product, device/google/cuttlefish/vsoc_x86_64/kernel.mk)
$(call inherit-product, device/google/cuttlefish/vsoc_x86_64/bootloader.mk)

Kernel is defined in the makefile:

TARGET_KERNEL_USE ?= 6.1
TARGET_KERNEL_ARCH ?= $(TARGET_ARCH)
SYSTEM_DLKM_SRC ?= kernel/prebuilts/$(TARGET_KERNEL_USE)/$(TARGET_KERNEL_ARCH)
TARGET_KERNEL_PATH ?= $(SYSTEM_DLKM_SRC)/kernel-$(TARGET_KERNEL_USE)
KERNEL_MODULES_PATH ?= kernel/prebuilts/common-modules/virtual-device/$(TARGET_KERNEL_USE)/$(subst _,-,$(TARGET_KERNEL_ARCH))
PRODUCT_COPY_FILES += $(TARGET_KERNEL_PATH):kernel

The variable TARGET_KERNEL_PATH is finally expanded to kernel/prebuilts/6.1/x86_64/kernel-6.1:

file kernel/prebuilts/6.1/x86_64/kernel-6.1

Linux kernel x86 boot executable bzImage, version 6.1.68-android14-11-ga7f647f49daf-ab11484633 (build-user@build-host) #1 SMP PREEMPT Wed Feb 21 00:52:15 UTC 2024, RO-rootFS, swap_dev 0X11, Normal VGA

Compile Kernel

Android Kernel is hosted in a different repo from the AOSP project. Check the branches and their build systems at https://source.android.com/docs/setup/reference/bazel-support.

Android 13 introduced building kernels with Bazel (kleaf) - it is super fast
Older Android versions use build.sh, which is quite slow
From Android 11, Vendor modules are recommended to built separately in common-modules

For Cuttlefish Device, refer to https://source.android.com/docs/devices/cuttlefish/kernel-dev.

Cuttlefish supports the following kernel manifests on aosp-main repo, branches: common-android14-6.1, common-android14-5.15, common-android-mainline

Define macros:

export AOSP_HOME="$HOME/aosp"
export AOSP_KERNEL_BRANCH="common-android14-6.1"

Initialize the repo:

mkdir -p ${AOSP_HOME}/${AOSP_KERNEL_BRANCH}
cd ${AOSP_HOME}/${AOSP_KERNEL_BRANCH}
repo init --depth=1 -u https://android.googlesource.com/kernel/manifest -b ${AOSP_KERNEL_BRANCH}

Download source code:

repo sync -c -j$(nproc)

Cuttlefish Device has its kernel build named virtual_device_x86_64 which is based on kernel_x86_64:

kernel_build(
    name = "virtual_device_x86_64",
    srcs = [":virtual_device_x86_64_common_sources"],
    outs = [],
    arch = "x86_64",
    base_kernel = "//common:kernel_x86_64",
    build_config = "build.config.virtual_device.x86_64",
    make_goals = [
        "modules",
    ],
    module_outs = _VIRT_COMMON_MODULES + [
        # keep sorted
        "dummy-cpufreq.ko",
        "nd_virtio.ko",
        "test_meminit.ko",
        "virtio_pmem.ko",
    ],
)

Build Virtual Device Kernel and Modules:

tools/bazel run //common-modules/virtual-device:virtual_device_x86_64_dist

[dist] INFO: Copying to /home/trongvq/aosp/kernel-common-android14-6.1/out/virtual_device_x86_64/dist

All modules will be included in InitRamFS.

Check the Kernel image:

file out/virtual_device_x86_64/dist/bzImage

Linux kernel x86 boot executable bzImage, version 6.1.99-android14-11-maybe-dirty (build-user@build-host) #1 SMP PREEMPT Thu Jan  1 00:00:00 UTC 1970, RO-rootFS, swap_dev 0X11, Normal VGA

Run with new Kernel

Cuttlefish Device

It is easy to tell Cuttlefish to use new kernel and initramfs (included all virtual kernel modules):

export AOSP_HOME="$HOME/aosp"
export AOSP_KERNEL_BRANCH="common-android14-6.1"
launch_cvd --gpu_mode=gfxstream --resume=false -kernel_path=${AOSP_HOME}/${AOSP_KERNEL_BRANCH}/out/virtual_device_x86_64/dist/bzImage -initramfs_path=${AOSP_HOME}/${AOSP_KERNEL_BRANCH}/out/virtual_device_x86_64/dist/initramfs.img

Cuttlefish

Android Virtual Device

This needs to be rebuilt to run with new kernel:

TARGET_KERNEL_USE ?= 6.1
KERNEL_ARTIFACTS_PATH := /home/trongvq/aosp/common-android14-6.1/out/virtual_device_x86_64/dist
VIRTUAL_DEVICE_KERNEL_MODULES_PATH := $(KERNEL_ARTIFACTS_PATH)
EMULATOR_KERNEL_FILE := $(KERNEL_ARTIFACTS_PATH)/bzImage

then run emulator as usual.

Emulator

Compilation may fail due to a missing kernel module, remove it from makefile and build again.

Kernel Modules

Modules are pieces of code that can be loaded and unloaded into the kernel upon demand. HAL implementations are packaged into modules and loaded by the Android system at the appropriate time. A kernel module can be a device driver that handles or manages a hardware

As part of the module kernel requirements introduced in Android 8.0, all system-on-chip (SoC) kernels must support loadable kernel modules. To support loadable kernel modules, android-base.cfg in all common kernels includes the following kernel-config options:

CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
CONFIG_MODVERSIONS=y

All device kernels must enable these options. Kernel modules should also support unloading and reloading whenever possible.

Refer to: https://android.googlesource.com/kernel/build/+/refs/heads/main/kleaf/.

Below command will run a menu to configure internal modules in kernel itself. External/Vendor modules do not listed in Kernel Menu Config.

tools/bazel run //common:kernel_x86_64_config -- menuconfig

Vendor Modules

Refer to: https://android.googlesource.com/kernel/build/+/refs/heads/main/kleaf/docs/ddk/main.md.

From Android 11, Vendor Modules are recommended to built separately. One part of Kleaf is the Driver Development Kit (DDK) which is used to build external modules.

Here is a simple vendor module structure:

Directorycommon-android14-6.1
- Directoryvendor
  - Directorytrongvq # vendor name
    BUILD.bazel # build file
    Directoryhello_world # vendor module
    hello_world.c

Build a Module

Create a folder for vendor packages:
```
mkdir vendor/trongvq
```

Create a folder for venfor module and its source code:

mkdir vendor/trongvq/hello_world

#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>

static int hello_world_init(void) {
    printk(KERN_ALERT "trongvq: Hello World!\n");
    return 0;
}

static void hello_world_exit(void) {
    printk(KERN_ALERT "trongvq: Goodbye World!\n");
}

module_init(hello_world_init);
module_exit(hello_world_exit);

MODULE_LICENSE("GPL");

Create a ddk_module so system will know that there is a module there,
this module is built for kernel named virtual_device_x86_64:

load("//build/kernel/kleaf:kernel.bzl", "ddk_module")

filegroup(
    name = "hello_world_sources",
    srcs = glob([
        "hello_world/*.c",
    ]),
)

ddk_module(
    name = "hello_world",
    srcs = [":hello_world_sources"],
    out = "hello_world.ko",
    kernel_build = "//common-modules/virtual-device:virtual_device_x86_64",
    deps = [
        "//common:all_headers_x86_64",
    ],
)

Compile the vendor module:

tools/bazel build //vendor/trongvq:hello_world

Output: bazel-bin/vendor/trongvq/hello_world/hello_world.ko

Load and test module on device:

adb push bazel-bin/vendor/trongvq/hello_world/hello_world.ko /data/local/tmp/

adb root
adb shell insmod /data/local/tmp/hello_world.ko
adb shell dmesg | grep trongvq

trongvq: Hello World!

adb shell rmmod /data/local/tmp/hello_world.ko
adb shell dmesg | grep trongvq

trongvq: Hello World!
trongvq: Goodbye World!

Integrate Module to Kernel

Create a kernel_module_group and make it visible as a package __pkg__
for the target kernel build //common-modules/virtual-device:

load("//build/kernel/kleaf:kernel.bzl", "ddk_module")
load("//build/kernel/kleaf:kernel.bzl", "kernel_module_group")

filegroup(
    name = "hello_world_sources",
    srcs = glob([
        "hello_world/*.c",
    ]),
)

ddk_module(
    name = "hello_world",
    srcs = [":hello_world_sources"],
    out = "hello_world.ko",
    kernel_build = "//common-modules/virtual-device:virtual_device_x86_64",
    deps = [
        "//common:all_headers_x86_64",
    ],
)

kernel_module_group(
    name = "kernel_modules",
    srcs = [
        ":hello_world"
    ],
    visibility = ["//common-modules/virtual-device:__pkg__"],
)

In the target kernel build, make kernel_build visible for vendor packages,
and under a kernel_modules_install where it lists kernel modules,
add the vendor kernel_module_group into the list:

kernel_build(
    name = "virtual_device_x86_64",
    ...
    visibility = [
        "//vendor/trongvq:__pkg__",
    ],
)

kernel_modules_install(
    name = "virtual_device_x86_64_modules_install",
    kernel_build = ":virtual_device_x86_64",
    kernel_modules = [
        ":virtual_device_x86_64_external_modules",
        "//vendor/trongvq:kernel_modules"
    ],
)

Compile the target kernel again:
```
tools/bazel run //common-modules/virtual-device:virtual_device_x86_64_dist
```
Now, the vendor module should be copied into out/virtual_device_x86_64/dist/hello_world.ko

Load Cuttlefish again:

export AOSP_HOME="$HOME/aosp"
export AOSP_KERNEL_BRANCH="common-android14-6.1"
launch_cvd --gpu_mode=gfxstream --resume=false -kernel_path=${AOSP_HOME}/${AOSP_KERNEL_BRANCH}/out/virtual_device_x86_64/dist/bzImage -initramfs_path=${AOSP_HOME}/${AOSP_KERNEL_BRANCH}/out/virtual_device_x86_64/dist/initramfs.img

Check that vendor module is loaded automatically:
```
adb root
adb shell lsmode | grep hello_world
```
```
hello_world            16384  0
```
Note that module is loaded in InitRamFS, there is no init message printed in dmesg.

Check that vendor module is included in filesystem:

adb shell find vendor* -name hello*

vendor_dlkm/lib/modules/extra/vendor/trongvq/hello_world.ko

Some notes here:

the kerner target //common:kernel_x86_64 is a base build,
the target kernel //common-modules/virtual-device:virtual_device_x86_64 is defined with addition modules
if the vendor kernel is build for //common:kernel_x86_64, the need changes in the build for /common/BUILD.bazel in the section which defines kernel_x86_64, but it needs more work.
for older Android version, please refer to this guide: https://vuquangtrong.github.io/blog/android/kernel-module.

Example: `virtio_rtc` Module

This is an example of a character device which mimics a virtual Real Time Clock (RTC) with only get, set, and tick functions.

Directorycommon-android14-6.1
- Directorycommon-modules/virtual-device # externa modules
  - BUILD.bazek # cuttlefish kernel target
- Directoryvendor/trongvq # vendor name
  - BUILD.bazel # vendor build file
  - Directoryvirtio_rtc # module name
    virtio_rtc.c # module implementation

Main functions implementation:

#define DEVICE_NAME "virtio_rtc"
#define CLASS_NAME "virtio"
#define BUFFER_SIZE 16
...
static long seconds = 0; // unprotected shared data!
static char str_seconds[BUFFER_SIZE]; // unprotected shared data!

static int thread_func(void *arg) {
    while (!kthread_should_stop()) {
        ssleep(1);
        seconds++;
    }
    return 0;
}
...
static ssize_t dev_read(struct file *filep, char *buffer, size_t len, loff_t *offset) {
    snprintf(str_seconds, BUFFER_SIZE, "%ld\n", seconds);
    return simple_read_from_buffer(buffer, len, offset, str_seconds, BUFFER_SIZE);
}

static ssize_t dev_write(struct file *filep, const char *buffer, size_t len, loff_t *offset) {
    ssize_t s = simple_write_to_buffer(str_seconds, BUFFER_SIZE, offset, buffer, len);
    if (kstrtol(str_seconds, 10, &seconds)) {
        return -EINVAL;
    }
    return s;
}
...
static int __init virtio_rtc_init(void) {
    ...
    task = kthread_run(thread_func, NULL, "virtio_rtc_thread");
    if (IS_ERR(task)) {
        printk(KERN_ALERT DEVICE_NAME "Failed to create the thread\n");
        return PTR_ERR(task);
    }
}
...

Rebuild kernel with vendor modules:

tools/bazel run //common-modules/virtual-device:virtual_device_x86_64_dist

Start Cuttlefish device:

export AOSP_HOME="$HOME/aosp"
export AOSP_KERNEL_BRANCH="common-android14-6.1"
source build/envsetup.sh
lunch aosp_cf_x86_64_auto-trunk_staging-userdebug
launch_cvd --gpu_mode=gfxstream --resume=false -kernel_path=${AOSP_HOME}/${AOSP_KERNEL_BRANCH}/out/virtual_device_x86_64/dist/bzImage -initramfs_path=${AOSP_HOME}/${AOSP_KERNEL_BRANCH}/out/virtual_device_x86_64/dist/initramfs.img

Access to the device’s shell as root:

adb shell
su

Read from and Write to the /dev/virtio_rtc device:

cat /dev/virtio_rtc
echo 100 > /dev/virtio_rtc