(adds tip for how to list all image recipes)
 
(30 intermediate revisions by the same user not shown)
Line 1: Line 1:
== Prerequisites ==
= Yocto and OpenEmbedded =
The package management tools used by the [https://yoctoproject.org Yocto project] have been developed by the [https://www.openembedded.org OpenEmbedded] project. The Yocto project basically uses these generic build tools to provide a basic Linux system called ''Poky''. Stable releases of Poky serve as standard platforms for third parties that provide additional features (e.g., hardware support layers).
<br />
<br />
Unless stated otherwise, the information on this page applies to '''version 4.0.5 (Kirkstone)''' of Yocto.
<br />
<br />
 
= Prerequisites =
* A Linux system supported by the Yocto project.<br />Assuming that [[Ubuntu]] is used. 
* Following package installed:
<pre class=terminal>
<pre class=terminal>
sudo apt install gcc g++ \
sudo apt install gcc g++ \
Line 6: Line 16:
<br />
<br />


== Images ==
= Bitbake =
Bitbake is the build tool from the OpenEmbedded project, which parses the project's metadata, resolves dependencies and the ordering of the build steps, runs the builds to create packages, to finally create an image from the packages.
<br />
=== Metadata Types ===
Yocto metadata, which controls how a system image is built, can be categorized at a high level into these four types:
* ''Machine''<br />the targeted hardware platform (e.g. Raspberry Pi 4)
* ''Distro''<br />common characteristic of a family of related images
* ''Recipes''<br />instructions on how to process the source files for a component of the image
* ''Image''<br />recipes that define the contents of generated system images
<br />
=== Layers ===
Layers are purposeful collections of metadata. Yocto layers often Layers often add the  provide that for a  that can be combined with other layers in order to directories in which
<br />
 
<br /> 
 
= Images =


Images are defined in recipe files. In order to separate them from package recipes, there is a convention to put image recipes into a folder named ''images''. Making use of this, we can list all images defined by all layers via
Images are recipes that ultimately inherit from '''meta/classes/image.bbclass''' and set the global variable '''IMAGE_INSTALL'''. Most images, however, will inherit from ''meta/classes/core-image.bbclass'' which inherits directly from ''image.bbclass'', provides a mapping from ''IMAGE_FEATURES'' to package groups, and adds the two package groups ''packagegroup-core-boot'' and ''packagegroup-base-extended'' to ''IMAGE_INSTALL''.
 
Images are defined in recipe files with the file extension 'bb'. In order to separate them from package recipes, there is a convention to put image recipes into a folder named ''images''. Making use of this, we can list all images defined by all layers via
<pre class="terminal">
<pre class="terminal">
cd poky
ls meta*/recipes*/images/*.bb
ls meta*/recipes*/images/*.bb
</pre>
</pre>
<br />
<br />


== Using devtool ==
=== ''IMAGE_FEATURES'' ===
The ''IMAGE_FEATURES'' variable is defined in ''meta/classes/image.bbclass'' with an empty value. It accepts only a predefined set of features (e.g., ''x11'', ''debug-tweaks'', ''dev-pkgs'', ''dbg-pkgs''). This affects how recipes will execute their tasks and which packages will be selected to go into the image. The image recipe ''meta/recipes-core/images/core-image-base.bb'', for example, adds the ''splash'' feature.
<br />
 
=== ''IMAGE_INSTALL'' ===
Holds a list of package groups whose packages (from the package feed area) will go into the filesystem image.
<br />
 
== Reference images ==
The following reference images are defined in the ''meta'' layer:
{| style="padding: 10px;"
! style="width: 20px" | || ||
|-
| || ''core-image-minimal'' || small image that just boots the target device
|-
| || ''core-image-minimal-dev'' || ''core-image-minimal'' with headers and libraries allowing development work
|-
| || style="padding-right: 10px" | ''core-image-minimal-initramfs'' || ''core-image-minimal'' with kernel support for in-RAM filesystem
|-
| || '''core-image-base''' || '''console-only system fully supporting the target hardware'''
|-
| || ''core-image-full-cmdline'' || console-only image that includes many system tools
|-
| || ''core-image-x11'' ||  image with basic X11 and a terminal
|-
|}
 
These image definitions are provided as a starting point for creating project-specific, custom image definitions.
<br />
<br />
 
== Creating a Custom Image ==
* In your custom layer, create the ''images'' subdirectory in one of your ''recipes-...' directory.
* In the ''images'' directory, create two image recipe files that inherit from ''core-image'':
<pre class="code">
SUMMARY = "My deployment image"
LICENSE = "MIT"
 
inherit core-image
IMAGE_FEATURES += "splash"
</pre>
and
<pre class="code">
SUMMARY = "My development image"
 
inherit core-image
require my-image.bb
 
IMAGE_FEATURES += "ssh-server-dropbear tools-debug debug-tweaks"
 
CORE_IMAGE_EXTRA_INSTALL += "i2c-tools "
</pre>
where the first image recipe is for deployment and the second is for development. Note that the development version includes the deployment version with the ''require'' statement, and adds to IMAGE_FEATURES and CORE_IMAGE_EXTRA_INSTALL.
<br />
 
== Image Filesystem Packaging ==
The system image generated by Bitbake can be packaged in the form of an image file, whose contents can be transferred on a boot medium (SD card or eMMC) such that the primary boot loader of the target device can discover the system and boot it.
 
Unless you restrict the type of packaging used for the image file, Yocto will generate multiple rootfs packages, each with a large footprint. To restrict the generated image files to a desired type, edit ''conf/local.conf'' of your project build directory and set the type.
<pre class="code">
#IMAGE_FSTYPES = "tar.gz"
#IMAGE_FSTYPES = "tar.bz2"
IMAGE_FSTYPES = "wic.bz2 wic.bmap"
</pre>
The wic ([https://pykickstart.readthedocs.io/en/latest/kickstart-docs.html kickstart]) format with accompanying [https://github.com/intel/bmap-tools bmap] is recommended.
<br />
 
== Installing the Rootfs Image ==
Using [https://github.com/intel/bmap-tools bmaptool] we can copy the generated ''wic'' image (with matching bmap file) to a mounted SD card or eMMC storage.
<pre class="terminal">
sudo bmaptool copy tmp/deploy/images/raspberrypi4-64/<wic.bz2 file> <drive>
</pre>
where ''<wic.bz2 file>'' is the root filesystem image wic.bz2 file, and ''<drive>'' is the drive at which the storage is mounted (for example, ''/dev/sdb'').
<br />
<br />
 
= Package Groups =
Package groups are recipes that inherit from '''meta/classes/packagegroup.bbclass''' and set the content of the global variable '''PACKAGES'''. Package groups are defined in recipe files with the file extension 'bb'.
<br />
<br />
Note that the ''packagegroup'' must be inherited before ''PACKAGES is defined.
<pre class="code">
inherit packagegroup
 
PACKAGES = '...'
</pre>
<br />
 
= Useful Image Features =
<br />
 
== Distro Features ==
Distro features are system features that affect multiple recipes. '''Large parts of the Poky image may need to be rebaked after making changes to the distro features!'''
<br />
 
=== systemd ===
After the kernel is done with its basic booting tasks, it passes control to the initialization manager. To use the ''systemd'' initialization manager instead of ''SysVinit'', make the following entries either in ''conf/distro/distro.conf'' or in your layer's ''conf/layer.conf'', or in ''conf/local.conf'':
<pre class="code">
DISTRO_FEATURES:append = " systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = ""
</pre>
<br />
 
=== sshd ===
The OpenSSH server is installed via
<pre class="code">
EXTRA_IMAGE_FEATURES:append = " ssh-server-openssh"
</pre>
This installs a socket-based SSH server, which runs only when a connection request comes in, thereby reducing memory consumption when no SSH client is connected.
<br />
<br />
To check the status of the SSH server, on the target device, type
<pre class="terminal">
systemctl status sshd.socket
</pre>
<br />
 
=== NFS ===
In Embedded Linux projects, NFS is often included in development images to enable faster code-deploy turnaround times.
<br />
In the ''conf/local.conf'' file of your Yocto build directory (or ''conf/layer.conf'' file of your layer) add
<pre class="code">
DISTRO_FEATURES:append = " nfs"
</pre>
<br />
On your Debian-based Yocto machine, [[Ubuntu#Setting_up_an_NFS_Server|set up an NFS server]] where the served directory in ''/etc/exports'' must be
<br />
: ${YOCTO_BUILD_DIR}'''/tmp/work/'''${TARGET}'''-poky-linux/'''${IMAGE}'''/1.0-r0/rootfs'''
For example,
: ''build-robofish-lt0/tmp/work/raspberrypi4_64-poky-linux/robofish-lt0-image-dev/1.0-r0/rootfs''
<br />
<br />
 
=== [https://vulkan.org Vulkan] and [https://opengl.org OpenGL] ===
<pre class="code">
DISTRO_FEATURES:append = " vulkan opengl x11 wayland"
</pre>
<br />
 
=== Bluetooth and NFC ===
<pre class="code">
DISTRO_FEATURES:append = " bluetooth nfc"
</pre>
<br />
 
= Recipes =
Recipes are Yocto metadata files that are interpreted by Bitbake and describe how a build task is performed.
<br />
== Patching A Device Tree ==
Follow these steps to patch an existing device tree by creating an ''append recipe'' in a custom layer.
<br />
<br />
In the custom layer, create the directory for a linux kernel append recipe.
<pre class="terminal">
mkdir -p ${LAYER_DIR}/recipes-kernel/linux/files
</pre>
where it is assumed that ''${LAYER_DIR}'' contains the path to the root directory of the custom Yocto layer.
<br />
<br />
Copy the kernel device tree file from your Yocto project build directory to the ''files'' directory of your new recipe. For a Raspberry Pi Compute Module 4, for example, type
<pre class="terminal">
cp ${YOCTO_BUILD_DIR}/tmp/work-shared/${MACHINE}/kernel-source/arch/arm/boot/dts/some.dts ${LAYER_DIR}/recipes-kernel/linux/files/
</pre>
where ''${YOCTO_BUILD_DIR}'' is the path to the Yocto project build directory, and ''${MACHINE}'' is the chosen target platform (e.g. "raspberrypi4-64").
<br />
<div class="sidenote">
If the ''tmp/work-shared'' directory does not exists anymore, you can recreate it by typing
<pre class="terminal">
bitbake -c menuconfig virtual/kernel
</pre>
and exiting the kernel configuration menu without making changes.
</div>
<br />
Make the necessary changes to the copied device tree file. Then create a git patch file using ''git diff'' as shown below
<pre class="terminal">
git diff --no-index {DTS_DIR}/some.dts some.dts > 0001-my_modifications-dts.patch
</pre>
<br />
Edit the generated patch file to fix the file paths at the top. The paths must be relative to the kernel sources directory:
: ''${YOCTO_BUILD_DIR}'''''/tmp/work-shared/'''''${MACHINE}'''''/kernel-source'''
The header should look like this:
<pre class="code">
diff --git a/arch/arm/boot/dts/some.dts b/some.dts
index 6e99da1..4567735 100644
--- a/arch/arm/boot/dts/some.dts
+++ b/arch/arm/boot/dts/some.dts
</pre>
where "--- a" and "+++ b" are followed by the same relative file path.
 
<br />
Create and edit the append recipe
<pre class="terminal">
vim ${LAYER_DIR}/recipes-kernel/linux/${KERNEL_NAME}_%.bbappend
</pre>
where ''${KERNEL_NAME}'' is the kernel name, which usually starts with "linux-" (for example, ''linux-raspberrypi''), and ''_%'' denotes all versions of the kernel package.
<br />
<div class="sidenote">
If in doubt, use the ''oe-pkgdata-util'' tool from within your build directory to find out the kernel name.
<pre class="terminal">
oe-pkgdata-util lookup-recipe kernel
</pre>
</div>
The contents of the recipe file should be similar to this:
<pre class="code">
FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
SRC_URI += "file://0001-my_modifications-dts.patch"
SRC_URI[md5sum] = "e2eba462f773f46fe6930a08323f5e1f"
SRC_URI[sha256sum] = "2ef2b2e9b33b5d2af0425940387bc6c1c2c305e6a685defbdc7cf3f584bc1edc"
</pre>
where ''0001-my_modifications-dts.patch'' needs to be replaced with the actual patch file name.
 
== Creating a New Device Tree ==
Creating a new device tree requires an input dts file which completely defines the hardware configuration. The ''KERNEL_DEVICETREE'' variable is used to make the Linux kernel select the new device tree during booting.
<pre class="code">
inherit kernel-devicetree
 
FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
 
# Make custom kernel with PRU enabled
SRC_URI += " \
    file://some.dts \
    file://0001-some.patch \
"
 
KERNEL_DEVICETREE = "some.dtb"
</pre>
<br />
 
= Accessing the OpenEmbedded package collection =
The OpenEmbedded project provides recipes for many useful programs that are not included with Yocto, like tmux, VLC, Gnome, various multimedia codecs, more filesystems, gRPC and much more.
<br />
To access the recipes for those software packages, clone the OpenEmbedded layers repository into your third-party layers directory: 
<pre class="terminal">
git clone git://git.openembedded.org/meta-openembedded
</pre>
<br />
 
= devtool =


Use '''devtool''' to make changes to an existing recipe and automatically generate an append recipe from those changes.
Use '''devtool''' to make changes to an existing recipe and automatically generate an append recipe from those changes.
<br />
<br />

Latest revision as of 2022-12-30T01:14:35

Yocto and OpenEmbedded

The package management tools used by the Yocto project have been developed by the OpenEmbedded project. The Yocto project basically uses these generic build tools to provide a basic Linux system called Poky. Stable releases of Poky serve as standard platforms for third parties that provide additional features (e.g., hardware support layers).

Unless stated otherwise, the information on this page applies to version 4.0.5 (Kirkstone) of Yocto.

Prerequisites

  • A Linux system supported by the Yocto project.
    Assuming that Ubuntu is used.
  • Following package installed:
sudo apt install gcc g++ \
  python3-distutils python-is-python3


Bitbake

Bitbake is the build tool from the OpenEmbedded project, which parses the project's metadata, resolves dependencies and the ordering of the build steps, runs the builds to create packages, to finally create an image from the packages.

Metadata Types

Yocto metadata, which controls how a system image is built, can be categorized at a high level into these four types:

  • Machine
    the targeted hardware platform (e.g. Raspberry Pi 4)
  • Distro
    common characteristic of a family of related images
  • Recipes
    instructions on how to process the source files for a component of the image
  • Image
    recipes that define the contents of generated system images


Layers

Layers are purposeful collections of metadata. Yocto layers often Layers often add the provide that for a that can be combined with other layers in order to directories in which


Images

Images are recipes that ultimately inherit from meta/classes/image.bbclass and set the global variable IMAGE_INSTALL. Most images, however, will inherit from meta/classes/core-image.bbclass which inherits directly from image.bbclass, provides a mapping from IMAGE_FEATURES to package groups, and adds the two package groups packagegroup-core-boot and packagegroup-base-extended to IMAGE_INSTALL.

Images are defined in recipe files with the file extension 'bb'. In order to separate them from package recipes, there is a convention to put image recipes into a folder named images. Making use of this, we can list all images defined by all layers via

cd poky
ls meta*/recipes*/images/*.bb


IMAGE_FEATURES

The IMAGE_FEATURES variable is defined in meta/classes/image.bbclass with an empty value. It accepts only a predefined set of features (e.g., x11, debug-tweaks, dev-pkgs, dbg-pkgs). This affects how recipes will execute their tasks and which packages will be selected to go into the image. The image recipe meta/recipes-core/images/core-image-base.bb, for example, adds the splash feature.

IMAGE_INSTALL

Holds a list of package groups whose packages (from the package feed area) will go into the filesystem image.

Reference images

The following reference images are defined in the meta layer:

core-image-minimal small image that just boots the target device
core-image-minimal-dev core-image-minimal with headers and libraries allowing development work
core-image-minimal-initramfs core-image-minimal with kernel support for in-RAM filesystem
core-image-base console-only system fully supporting the target hardware
core-image-full-cmdline console-only image that includes many system tools
core-image-x11 image with basic X11 and a terminal

These image definitions are provided as a starting point for creating project-specific, custom image definitions.

Creating a Custom Image

  • In your custom layer, create the images subdirectory in one of your recipes-...' directory.
  • In the images directory, create two image recipe files that inherit from core-image:
SUMMARY = "My deployment image"
LICENSE = "MIT"

inherit core-image
IMAGE_FEATURES += "splash"

and

SUMMARY = "My development image"

inherit core-image
require my-image.bb

IMAGE_FEATURES += "ssh-server-dropbear tools-debug debug-tweaks"

CORE_IMAGE_EXTRA_INSTALL += "i2c-tools "

where the first image recipe is for deployment and the second is for development. Note that the development version includes the deployment version with the require statement, and adds to IMAGE_FEATURES and CORE_IMAGE_EXTRA_INSTALL.

Image Filesystem Packaging

The system image generated by Bitbake can be packaged in the form of an image file, whose contents can be transferred on a boot medium (SD card or eMMC) such that the primary boot loader of the target device can discover the system and boot it.

Unless you restrict the type of packaging used for the image file, Yocto will generate multiple rootfs packages, each with a large footprint. To restrict the generated image files to a desired type, edit conf/local.conf of your project build directory and set the type.

#IMAGE_FSTYPES = "tar.gz"
#IMAGE_FSTYPES = "tar.bz2"
IMAGE_FSTYPES = "wic.bz2 wic.bmap"

The wic (kickstart) format with accompanying bmap is recommended.

Installing the Rootfs Image

Using bmaptool we can copy the generated wic image (with matching bmap file) to a mounted SD card or eMMC storage.

sudo bmaptool copy tmp/deploy/images/raspberrypi4-64/<wic.bz2 file> <drive>

where <wic.bz2 file> is the root filesystem image wic.bz2 file, and <drive> is the drive at which the storage is mounted (for example, /dev/sdb).

Package Groups

Package groups are recipes that inherit from meta/classes/packagegroup.bbclass and set the content of the global variable PACKAGES. Package groups are defined in recipe files with the file extension 'bb'.

Note that the packagegroup must be inherited before PACKAGES is defined.

inherit packagegroup

PACKAGES = '...'


Useful Image Features


Distro Features

Distro features are system features that affect multiple recipes. Large parts of the Poky image may need to be rebaked after making changes to the distro features!

systemd

After the kernel is done with its basic booting tasks, it passes control to the initialization manager. To use the systemd initialization manager instead of SysVinit, make the following entries either in conf/distro/distro.conf or in your layer's conf/layer.conf, or in conf/local.conf:

DISTRO_FEATURES:append = " systemd"
DISTRO_FEATURES_BACKFILL_CONSIDERED = "sysvinit"
VIRTUAL-RUNTIME_init_manager = "systemd"
VIRTUAL-RUNTIME_initscripts = ""


sshd

The OpenSSH server is installed via

EXTRA_IMAGE_FEATURES:append = " ssh-server-openssh"

This installs a socket-based SSH server, which runs only when a connection request comes in, thereby reducing memory consumption when no SSH client is connected.

To check the status of the SSH server, on the target device, type

systemctl status sshd.socket


NFS

In Embedded Linux projects, NFS is often included in development images to enable faster code-deploy turnaround times.
In the conf/local.conf file of your Yocto build directory (or conf/layer.conf file of your layer) add

DISTRO_FEATURES:append = " nfs"


On your Debian-based Yocto machine, set up an NFS server where the served directory in /etc/exports must be

${YOCTO_BUILD_DIR}/tmp/work/${TARGET}-poky-linux/${IMAGE}/1.0-r0/rootfs

For example,

build-robofish-lt0/tmp/work/raspberrypi4_64-poky-linux/robofish-lt0-image-dev/1.0-r0/rootfs



Vulkan and OpenGL

DISTRO_FEATURES:append = " vulkan opengl x11 wayland"


Bluetooth and NFC

DISTRO_FEATURES:append = " bluetooth nfc"


Recipes

Recipes are Yocto metadata files that are interpreted by Bitbake and describe how a build task is performed.

Patching A Device Tree

Follow these steps to patch an existing device tree by creating an append recipe in a custom layer.

In the custom layer, create the directory for a linux kernel append recipe.

mkdir -p ${LAYER_DIR}/recipes-kernel/linux/files

where it is assumed that ${LAYER_DIR} contains the path to the root directory of the custom Yocto layer.

Copy the kernel device tree file from your Yocto project build directory to the files directory of your new recipe. For a Raspberry Pi Compute Module 4, for example, type

cp ${YOCTO_BUILD_DIR}/tmp/work-shared/${MACHINE}/kernel-source/arch/arm/boot/dts/some.dts ${LAYER_DIR}/recipes-kernel/linux/files/

where ${YOCTO_BUILD_DIR} is the path to the Yocto project build directory, and ${MACHINE} is the chosen target platform (e.g. "raspberrypi4-64").

If the tmp/work-shared directory does not exists anymore, you can recreate it by typing

bitbake -c menuconfig virtual/kernel

and exiting the kernel configuration menu without making changes.


Make the necessary changes to the copied device tree file. Then create a git patch file using git diff as shown below

git diff --no-index {DTS_DIR}/some.dts some.dts > 0001-my_modifications-dts.patch


Edit the generated patch file to fix the file paths at the top. The paths must be relative to the kernel sources directory:

${YOCTO_BUILD_DIR}/tmp/work-shared/${MACHINE}/kernel-source

The header should look like this:

diff --git a/arch/arm/boot/dts/some.dts b/some.dts
index 6e99da1..4567735 100644
--- a/arch/arm/boot/dts/some.dts
+++ b/arch/arm/boot/dts/some.dts

where "--- a" and "+++ b" are followed by the same relative file path.


Create and edit the append recipe

vim ${LAYER_DIR}/recipes-kernel/linux/${KERNEL_NAME}_%.bbappend

where ${KERNEL_NAME} is the kernel name, which usually starts with "linux-" (for example, linux-raspberrypi), and _% denotes all versions of the kernel package.

If in doubt, use the oe-pkgdata-util tool from within your build directory to find out the kernel name.

oe-pkgdata-util lookup-recipe kernel

The contents of the recipe file should be similar to this:

FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
SRC_URI += "file://0001-my_modifications-dts.patch"
SRC_URI[md5sum] = "e2eba462f773f46fe6930a08323f5e1f"
SRC_URI[sha256sum] = "2ef2b2e9b33b5d2af0425940387bc6c1c2c305e6a685defbdc7cf3f584bc1edc"

where 0001-my_modifications-dts.patch needs to be replaced with the actual patch file name.

Creating a New Device Tree

Creating a new device tree requires an input dts file which completely defines the hardware configuration. The KERNEL_DEVICETREE variable is used to make the Linux kernel select the new device tree during booting.

inherit kernel-devicetree

FILESEXTRAPATHS:prepend := "${THISDIR}/files:"

# Make custom kernel with PRU enabled
SRC_URI += " \
    file://some.dts \
    file://0001-some.patch \
"

KERNEL_DEVICETREE = "some.dtb"


Accessing the OpenEmbedded package collection

The OpenEmbedded project provides recipes for many useful programs that are not included with Yocto, like tmux, VLC, Gnome, various multimedia codecs, more filesystems, gRPC and much more.
To access the recipes for those software packages, clone the OpenEmbedded layers repository into your third-party layers directory:

git clone git://git.openembedded.org/meta-openembedded


devtool

Use devtool to make changes to an existing recipe and automatically generate an append recipe from those changes.


Debug data: