LinuxBoot Introduction

This is the official “LinuxBoot Book” for the LinuxBoot project. The book:

• Describes the LinuxBoot project
• Explains why you would want to use LinuxBoot
• Describes the components that comprise LinuxBoot
• Highlights the differences between other boot processes and LinuxBoot
• Guides you through the steps needed to implement LinuxBoot

What is LinuxBoot?

LinuxBoot is a project that aims to replace specific firmware functionality with a Linux kernel and runtime. Over the years this project has grown to include various initiatives with the overarching goal of moving from obscure, complex firmware to simpler, open source firmware.

The goal of LinuxBoot is to reduce the role of firmware to a small, fixed-function core whose only purpose is to get a flash-based Linux kernel started. This “bare essentials” firmware prepares the hardware and starts a Linux kernel and a userland environment will run on the machine. Go is the recommended userland environment, but is not required.

Why LinuxBoot is needed

Sometimes firmware contains drivers and utilities. They can have bugs, or be unmaintained, which can be a source of problems and security issues. LinuxBoot replaces proprietary, closed-source, vendor-supplied firmware drivers with Linux drivers. This enables engineers writing Linux drivers and engineers writing firmware drivers to focus on one set of drivers. Those drivers will, as a result, have a larger set of contributors and reviewers, and because the drivers are part of Linux, standard industry coding infrastructure can be used to improve them. Finally, because these Linux drivers are currently being run 24x7 at scale, they will have fewer bugs.

What LinuxBoot does

LinuxBoot replaces many Driver Execution Environment (DXE) modules used by Unified Extensible Firmware Interface (UEFI) and other firmware, particularly the network stack and file system modules, with Linux applications.

LinuxBoot brings up the Linux kernel as a DXE in flash ROM instead of the UEFI shell. The Linux kernel, with a provided Go based userland, can then bring up the kernel that you want to run on the machine. The LinuxBoot firmware paradigm enables writing traditional firmware applications such as bootloader, debugging, diagnosis, and error detection applications as cross-architecture and cross-platform portable Linux applications.

When Linux boots it needs a root file system with utilities. One such root filesystem used for Linuxboot is based on u-root standard utilities written in Go. The following diagram shows the current state of the UEFI boot process and what is planned for the transition to LinuxBoot.

Benefits of using the Go userland environment and compiler

Go is a systems programming language created by Google. Go has strong typing, language level support for concurrency, inter-process communication via channels, runtime type safety and other protective measures, dynamic allocation and garbage collection, and closures. Go has a package syntax similar to Java that makes it easy to determine what packages a given program needs.

The modern language constructs make Go a much safer language than C. This safety is critical for network-attached embedded systems, which usually have network utilities written in C, including web servers, network servers including sshd, and programs that provide access to a command interpreter, itself written in C. All are proving to be vulnerable to the attack-rich environment that the Internet has become.

Even the most skilled programmers make simple mistakes that in C can be fatal, especially on network connected systems. Currently, even the lowest-level firmware in our PCs, printers, and thermostats is network-connected. These programming mistakes are either impossible to make in Go or, if made, are detected at runtime and result in the program exiting.

The case for using a high-level, safe language like Go in very low level embedded firmware might be stronger than for user programs, because exploits at the firmware level are nearly impossible to detect and mitigate.

The challenge to using Go in a storage-constrained environment such as firmware is that advanced language features lead to big binaries. Even a date program is about 2 MiB. One Go binary, implementing one function, is twice as large as a BusyBox binary implementing many functions. Currently, a typical BIOS FLASH part is 16 MiB. Fitting many Go binaries into a single BIOS flash part is not practical. The Go compiler is very fast and its sheer speed suggests a solution of having programs compiled only when they are used. With this approach, you can build a root file system that has almost no binaries except the Go compiler itself. The compiled programs and packages can be saved to a RAM-based file system. Another solution is to compile everything together into one BusyBox-style program. There are also other solutions that involve fetching things over the network, but compiling dynamically with Go or creating a BusyBox program are the recommended solutions.

Benefits of LinuxBoot with UEFI servers

Most server firmware is based on Intel’s Universal Extensible Firmware Interface (UEFI). LinuxBoot provides the following benefits over UEFI:

Reliability

• Improves boot reliability by replacing lightly-tested firmware drivers with hardened Linux drivers
• Proven approach for almost 20 years in military, consumer electronics, and supercomputing systems – wherever reliability and performance are paramount
• Fault Tolerance - Linux isolates processes** **(for example, when Pxeboot fails catastrophically, diskboot still works afterwards)

Security

• Move “Ring 0” bootloaders to “Ring 3”
• Pxeboot and diskboot do parsing and other logic in userspace
• Go provides memory safety and type safety
  • A buggy parser cannot easily affect other programs
• Kernel security patches can apply to firmware

Flexibility

• Can be used with coreboot, u-boot, OpenPOWER Abstraction Layer (OPAL), SlimBootLoader, ARM Trusted Firmware (ATF)
• Can boot multiple operating systems (Linux, Berkeley UNIX (BSD), XEN, Windows)
• Supports the following server mainboards:
  • QEMU emulated Q35 systems
  • Intel S2600WF
  • Dell R630
  • Winterfell Open Compute node
  • Leopard Open Compute node
  • Tioga Pass Open Compute node
  • Monolake Open Compute node (not tested)

Boot speed

• Improves boot time by removing unnecessary code; typically makes boot 20 times faster

Customization

• Allows customization of the initrd runtime to support site-specific needs (both device drivers as well as custom executables)

Engineering Productivity

• Write a driver once, not twice

  • Linux is open, measurable, reproducible, updatable
  • Linux already has drivers for almost everything

• Kernel Engineers = Firmware Engineers

  • Many more Engineers know Linux than know UEFI

• Reduced build time

  • 30s for initramfs
  • 15s for kernel (incremental)
  • ~15s to repack the bios image (using fiano/utk)
  • Total: ~1m for a new full bios image, ready to be tested

• Testing and debugging

  • Diskboot, Pxeboot already have unit tests
  • Easier to write tests using resources (like network) with Linux
  • Open-source projects such as u-root follow excellent software practices such as running automated test on each submitted change
  • Much easier to debug Go userspace applications
  • Test with a kernel in QEMU

Getting Started

You can try out LinuxBoot without needing to build anything! You can try out LinuxBoot needing only 3 commands.

We have made Initial Ram File System (initramfs) images available for four architectures: arm, aarch64, amd64 (a.k.a. x86_64), and riscv64.

For now, we only have a kernel ready for x86_64, so the instructions below apply to that.

First, you can get the initramfs image, which mainly contains Go programs from the u-root project.

curl -L -o u-root.cpio.xz https://github.com/linuxboot/u-root-builder/releases/download/v0.0.1/u-root_amd64_all.cpio.xz

Next, you will need to get a kernel. We use a pre-built kernel from Arch Linux.

curl -L -o linux.tar.zst https://archlinux.org/packages/core/x86_64/linux/download/
tar -xf linux.tar.zst

Now you are ready to test LinuxBoot out.

qemu-system-x86_64 -enable-kvm -machine q35 -nographic -append "console=ttyS0" \
  -kernel usr/lib/modules/*/vmlinuz -initrd u-root.cpio.xz

Or, for example, on Darwin:

qemu-system-x86_64 -machine q35 -nographic -append "console=ttyS0" \
  -kernel usr/lib/modules/*/vmlinuz -initrd u-root.cpio.xz

You will see the following:

[... varying message or two depending on qemu version and OS]
2023/12/12 22:37:52 Welcome to u-root!
                              _
   _   _      _ __ ___   ___ | |_
  | | | |____| '__/ _ \ / _ \| __|
  | |_| |____| | | (_) | (_) | |_
   \__,_|    |_|  \___/ \___/ \__|

/#

You can type uname:

/# uname
Linux
/#

To exit qemu, just run the poweroff command:

/# poweroff
[   14.442914] reboot: Power down

You have just run your first LinuxBoot kernel.

LinuxBoot Components

LinuxBoot consists of the following components:

1. BIOS
2. Linux kernel
3. u-root -> initramfs

BIOS

This does not have to be a specific BIOS; currently LinuxBoot supports UEFI and coreboot.

Linux kernel

LinuxBoot is not intended to be a runtime production kernel; rather, it is meant to replace specific UEFI functionality using Linux kernel capabilities and then boot the actual production kernel on the machine. Kernel configuration files specific to LinuxBoot provide the needed Linux kernel capabilities without bloating the size of the BIOS with unnecessary drivers.

These config files disable options that are not needed in the LinuxBoot kernel and add some patches that are needed.

Initial RAM filesystem (initramfs)

When Linux boots it needs a root file system that provides boot and startup utilities. LinuxBoot uses u-root to create an initramfs for this purpose.

What is an initramfs?

The initramfs is a root file system that is embedded within the firmware image itself. It is intended to be placed in a flash device along with the Linux kernel as part of the firmware image for LinuxBoot. The initramfs is essentially a set of directories bundled into a single cpio archive.

At boot time, the boot loader or firmware (for example, coreboot) loads the bzImage and initramfs into memory and starts the kernel. The kernel checks for the presence of the initramfs and, if found, unpacks it, mounts it as / and runs /init.

There are many types of initramfs, in this topic we focus on u-root. u-root is a Go userland (a set of programs and libraries written in Go that are used to interact with the kernel). It contains a toolset of standard Linux applications and commands.

u-root can create an initramfs in two different modes:

• source mode, which contains:
  • Go toolchain binaries
  • A simple shell
  • Go source for tools to be compiled on the fly by the shell
• Busybox (bb) mode: This is one busybox-like binary comprising all the requested utilities.

The initramfs provided by u-root implements the toolchain needed to securely boot the machine from the network, perform identity verification, communicate with different internal boot-related components, and kexec the next kernel.

u-root is an open source project hosted on GitHub. Within the u-root repository, we have executable commands in cmds and the packages containing libraries and implementations in pkg.

All about u-root

U-root is an embeddable root file system intended to be placed in a flash device as part of the firmware image, along with a Linux kernel. The program source code is installed in the root file system contained in the firmware flash part and compiled on demand. All the u-root utilities, roughly corresponding to standard Unix utilities, are written in Go, a modern, type-safe language with garbage collection and language-level support for concurrency and inter-process communication.

Unlike most embedded root file systems, which consist largely of binaries, u-root has only 5: an init program and 4 Go compiler binaries. When a program is first run, it, and any not-yet-built packages it uses are compiled to a RAM-based file system. The first invocation of a program takes a fraction of a second, as it is compiled. Packages are only compiled once, so the slowest build is always the first one, on boot, which takes about 3 seconds. Subsequent invocations are very fast, usually a millisecond or so.

U-root blurs the line between script-based distros such as Perl Linux[24] and binary-based distros such as BusyBox[26]. It has the flexibility of Perl Linux and the performance of BusyBox. Scripts and builtins are written in Go, not a shell scripting language. U-root is a new way to package and distribute file systems for embedded systems, and the use of Go promises a dramatic improvement in their security.

U-root and embedded systems

Embedding kernels and root file systems in BIOS flash is a common technique for gaining boot time performance and platform customization[25][14][23]. Almost all new firmware includes a multiprocess operating system with a full complement of file systems, network drivers, and protocol stacks, all contained in an embedded file system. In some cases, the kernel is only booted long enough to boot another kernel. In others, the kernel that is booted and the file system it contains constitute the operational environment of the device[15]. These so-called “embedded root file systems” also contain a set of standard Unix-style programs used for both normal operation and maintenance. Space on the device is at a premium, so these programs are usually written in C using the BusyBox toolkit[26], or in an interpretive language such as Perl[24] or Forth. BusyBox in particular has found wide usage in embedded appliance environments, as the entire root file system can be contained in under one MiB.

Embedded systems, which were once standalone, are now almost always network connected. Network connected systems face a far more challenging security environment than even a few years ago. In response to the many successful attacks against shell interpreters[11] and C programs[8], we have started to look at using a more secure, modern language in embedded root file systems, namely, Go[21][16].

Go is a new systems programming language created by Google. Go has strong typing; language level support for concurrency; inter-process communication via channels, a la Occam[13], Limbo[17], and Alef[27]; runtime type safety and other protective measures; dynamic allocation and garbage collection; closures; and a package syntax, similar to Java, that makes it easy to determine what packages a given program needs. The modern language constructs make Go a much safer language than C. This safety is critical for network-attached embedded systems, which usually have network utilities written in C, including web servers, network servers including sshd, and programs that provide access to a command interpreter, itself written in C. All are proving to be vulnerable to the attack-rich environment that the Internet has become. Buffer overflow attacks affecting C-based firmware code (among other things) in 2015 include GHOST and the so-called FSVariable.c bug in Intel’s UEFI firmware. Buffer overflows in Intel’s UEFI and Active Management Technology (AMT) have also been discovered in several versions in recent years.

Both UEFI[12] and AMT[4] are embedded operating systems, loaded from flash that run network-facing software. Attacks against UEFI have been extensively studied[9]. Most printers are network-attached and are a very popular exploitation target[6]. Firmware is not visible to most users and is updated much less frequently (if at all) than programs. It is the first software to run, at power on reset. Exploits in firmware are extremely difficult to detect, because firmware is designed to be as invisible as possible. Firmware is extremely complex; UEFI is roughly equivalent in size and capability to a Unix kernel. Firmware is usually closed and proprietary, with nowhere near the level of testing of kernels. These properties make firmware an ideal place for so-called advanced persistent threats[10][18][5]. Once an exploit is installed, it is almost impossible to remove, since the exploit can inhibit its removal by corrupting the firmware update process. The only sure way to mitigate a firmware exploit is to destroy the hardware.

U-root is an excellent option for embedded systems. U-root contains only 5 binaries, 4 of them from the Go toolchain, and the 5th is an init binary. The rest of the programs are contained in BIOS flash in source form, including packages. The search path is arranged so that when a command is invoked, if it is not in /bin, an installer is invoked instead which compiles the program into /bin. If the build succeeds, the command is executed. This first invocation takes a fraction of a second, depending on program complexity. After that, the RAM-based, statically linked binaries run in about a millisecond. Scripts are written in Go, not a shell scripting language, with two benefits: the shell can be simple, with fewer corner cases, and the scripting environment is substantially improved since Go is more powerful than most shell scripting languages, but also less fragile and less prone to parsing bugs.

U-root design

The u-root boot image is a build toolchain and a set of programs in source form. When first used, a program and any needed but not-yet-built packages are built and installed, typically in a fraction of a second. With later uses, the binary is executed. The root file system is almost entirely unformed on boot; /init sets up the key directories and mounts, including common ones such as /etc and /proc.

Since the init program itself is only 132 lines of code and is easy to change, the structure is very flexible and allows for many use cases, for example:

• Additional binaries: if the 3 seconds it takes to get to a shell is too long (some applications such as automotive computing require 800 ms startup time), and there is room in flash, some programs can be precompiled into /bin.
• Build it all on boot: if on-demand compilation is not desired, a background thread in the init process can build all the programs on boot.
• Selectively remove binaries after use: if RAM space is at a premium, once booted, a script can remove everything in /bin. Utilities or commands that are used will be rebuilt on demand.
• Always build on demand: run in a mode in which programs are never written to /bin and always rebuilt on demand. This is a very practical option given that program compilation is so fast.
• Lockdown: if desired, the system can be locked down once booted in one of several ways: the entire /src tree can be removed, for example, or just the compiler toolchain can be deleted.

U-root functionality

U-root is packaged as an LZMA-compressed initial RAM file system (initramfs) in cpio format. It is contained in a Linux compressed kernel image, also know as bzImage. The bootloader (for example, syslinux) or firmware (for example, coreboot) loads the bzImage into memory and starts it. The Linux kernel sets up a RAM-based root file system and unpacks the u-root file system into it. This initial root file system contains the Go toolchain (4 binaries), an init binary, the u-root program source, and the entire Go source tree, which provides packages needed for u-root programs.

All Unix systems start an init process on boot and u-root is no exception. The init for u-root sets up some basic directories, symlinks, and files. It builds a command installer and invokes the shell. This process is described in more detail below. The boot file system layout is shown in Table 1.

The src directory is where programs and u-root packages reside. The go/bin directory is for any Go tools built after boot; the go/pkg/tool directory contains binaries for various architecture/kernel combinations. The directory in which a compiler toolchain is placed provides information about the target OS and architecture, for example, the Go build places binaries for Linux on x86 64 in /go/pkg/tool/linux amd64/. Note that there is no /bin or many of the other directories expected in a root file system. The init binary builds them. It creates an empty /bin which is filled with binaries on demand as shown in Table 2.The u-root root file system has very little state.

For most programs to work, the file system must be more complete. Image space is saved by having init create additional file system structure at boot time: it fills in the missing parts of the root filesystem. It creates /dev and /proc and mounts them. It creates an empty /bin which is filled with binaries on demand.

In addition to /bin, there is a directory called /buildbin. Buildbin and the correct setup of $PATH are the keys to making on-demand compilation work. The init process sets $PATH to /go/bin:/bin:/buildbin:/usr/local/bin. Init also builds installcommand using the Go bootstrap builder and creates a complete set of symlinks. As a final step, init execs sh.

There is no /bin/sh at this point; the first sh found in $PATH is /buildbin/sh. This is a symlink to installcommand. Installcommand, once started, examines argv[0], which is sh, and takes this as instruction to build /src/cmds/sh/.go into /bin and then exec /bin/sh. There is no difference between starting the first shell and any other program. Hence, part of the boot process involves the construction of an installation tool to build a binary for a shell which is then run.

If a user wants to examine the source to the shell, they can cat /src/cmds/sh/.go. The cat command will be built and then show those files. U-root is intended for network-based devices and hence good network initialization code is essential. U-root includes a Go version of the IP and DHCP programs, along with the docker netlink package and a DHCP package.

Table 1 below shows the initial layout of a u-root file system.

All Go compiler and runtime source is included under /go/src. All u-root source is under /src and the compiler toolchain binaries are under /go/pkg.

Table 1: Initial layout of a u-root filesystem

Table 2 below shows the layout after /init has run.

Table 2: Layout after /init has run.

/buildbin contains symlinks to enable the on-demand compilation, and other standard directories and mount points are ready.

The u-root shell

U-root provides a shell that is stripped down to the fundamentals: it can read commands in using the Go scanner package; it can expand (that is, glob) the command elements, using the Go filepath package, and it can run the resulting commands, either programs or shell builtins. It supports pipelines and IO redirection. At the same time, the shell defines no language of its own for scripting and builtins. Instead, the u-root shell uses the Go compiler. In that sense, the u-root shell reflects a break in important ways with the last few decades of shell development, which has seen shells and their language grow ever more complex and, partially as a result, ever more insecure[19] and fragile[11].

The shell has several builtin commands, and you can extend it with builtin commands of your own. First, you need to understand the basic source structure of u-root shell builtins. Then, you will learn about user-defined builtins.

All shell builtins, including the ones that come with the shell by default, are written with a standard Go init pattern which installs one or more builtins.

Builtins in the shell are defined by a name and a function. One or more builtins can be described in a source file. The name is kept in a map and the map is searched for a command name before looking in the file system. The function must accept a string as a name and a (possibly zero-length) array of string arguments, and return an error. In order to connect the builtin to the map, a programmer must provide an init function which adds the name and function to the map. The init function is special in that it is run by Go when the program starts up. In this case, the init function just installs a builtin for the time command.

Figure 1 and Figure 2 below show the shell builtin for time.

// Package main is the 'root' of the package hierarchy for a program.
// This code is part of the main program, not another package,
// and is declared as package main.

package main

// A Go source file list all the packages on which it has a direct
// dependency.

import (
  "fmt"
  "os"
  "time"
  )

// init() is an optional function. If init () is present in a file,
// the Go compiler and runtime arrange for it to be called at
// program startup. It is therefore like a constructor.

func init () {
    // addBuiltIn is provided by the u−root shell for the addition of
    // builtin commands. Builtins must have a standard type:
    // - The first parameter is a string
    // - The second is a string array which may be 0 length
    // - The return is the Go error type
    // In this case, we are creating a builtincalled time that calls
    // the timecmd function.

  addBuiltIn ( "time " , timecmd )
  }

Figure 1: The code for the time builtin, Part I: setup

// The timecmd function is passed the name of a command to run,
// optional arguments, and returns an error. It:
// - gets the starttime using Now from the time package
// - runs the command using the u−root shell runit function
// - computes a duration using Since from the time package
// - if there is an error, prints the error to os.Stderr
// - uses fmt. Printf to print the duration to os.Stderr
// Note that since runtime always handles the error, by printing
// it, it always returns nil. Most builtins return the error.
// Here you can see the usage of the imported packages
// from the imports statement above.

func timecmd (name string, args [] string )
error {
      start: = time. Now ()
      err := run it (name, args)
        if err != nil {

      fmt. Fprintf (os. Stderr, ”%v\n”, err)

  }
  cost := time . Since (start)
  fmt.Printf(os.Stderr, ”%v ”,cost)

    // This function is special in that it handles the error, and hence
    // does not return an error.
    // Most other builtins return the error.

  return nil
  }

Figure 2: The code for the shell time builtin, Part II

Scripting and builtins

To support scripting and builtins, u-root provides two programs: script and builtin. The script program allows users to specify a Go fragment on the command line, and runs that fragment as a program. The builtin program allows a Go fragment to be built into the shell as a new command. Builtins are persistent; the builtin command instantiates a new shell with the new command built in. Scripts run via the script command are not persistent.

A basic hello builtin can be defined on the command line:

builtin hello '{ fmt.Printf("Hello\n") }'

The fragment is defined by the {} pair. Given a fragment that starts with a {, the builtin command generates all the wrapper boiler plate needed. The builtin command is slightly different from the script command in that the Go fragment is bundled into one argument. The command accepts multiple pairs of command name and Go code fragments, allowing multiple new builtin commands to be installed in the shell.

Builtin creates a new shell at /bin/sh with the source at /src/cmds/sh/. Invocations of /bin/sh by this shell and its children will use the new shell.

Processes spawned by this new shell can access the new shell source and can run the builtin command again and create a shell that further extends the new shell. Processes outside the new shell’s process hierarchy can not use this new shell or the builtin source. When the new shell exits, the builtins are no longer visible in any part of the file system. We use Linux mount name spaces to create this effect[22]. Once the builtin command has verified that the Go fragment is valid, it builds a new, private namespace with the shell source, including the new builtin source. From that point on, the new shell and its children will only use the new shell. The parent process and other processes outside the private namespace continue to use the old shell.

Figure 3 below shows an example usage of the script command.

This script implements printenv. Note that it is not a complete Go program in that it lacks a package statement, imports, a main function declaration, and a return at the end. All the boilerplate is added by the script command, which uses the Go imports package to scan the code and create the import statements required for compilation (in this case, both fmt and os packages are imported). Because the u-root shell is so simple, there is no need to escape many of these special characters. The complex parsing tasks have been offloaded to Go. Builtins are implemented in almost the same way. The builtin command takes the Go fragment and creates a standard shell builtin Go source file which conforms to the builtin pattern. This structure is easy to generate programmatically, building on the techniques used for the script command.

script{ fmt.Printf("%v\n", os.Environ()) }

Figure 3: Go fragment for a printenv script. Code structure is inserted and packages are determined automatically.

Environment variables

The u-root shell supports environment variables, but manages them differently than most Unix environments. The variables are maintained in a directory called /env; the file name corresponds to the environment variable name, and the files contents are the value. When it is starting a new process, the shell populates child process environment variables from the /env directory. The syntax is the same; $ followed by a name directs the shell to substitute the value of the variable in the argument by prepending /env to the path and reading the file.

The shell variables described above are relative paths; /env is prepended to them. In the u-root shell, the name can also be an absolute path. For example, the command script $/home/$USER/scripts/hello will substitute the value of the hello script into the command line and then run the script command. The ability to place arbitrary text from a file into an argument is proving to be extremely convenient, especially for script and builtin commands.

Using external packages and programs

No root file system can provide all the packages all users want, and u-root is no exception. You need to have the ability to load external packages from popular Linux distros. The tcz command can be used to load external packages from the TinyCore Linux distribution, also known as tinycore. A tinycore package is a mountable file system image, containing all the package files, including a file listing any additional package dependencies. To load these packages, u-root provides the tcz command which fetches the package and needed dependencies. Hence, if a user wants emacs, they need merely type tcz emacs, and emacs will become available in /usr/local/bin. The tinycore packages directory can be a persistent directory or it can be empty on each boot.

The tcz command is quite flexible as to what packages it loads and where they are loaded from. Users can specify the host name which provides the packages, the TCP port on which to connect, the version of tinycore to use, and the architecture. The tcz command must loopback mount each package as it is fetched, and hence must cache them locally. It will not refetch already cached packages. This cache can be volatile or maintained on more permanent storage. Performance varies depending on the network being used and the number of packages being loaded, but averages about 1 second per package on a WIFI-attached laptop. U-root also provides a small web server, called srvfiles, that can be used to serve locally cached tinycore packages for testing. The entire server is 18 lines of Go.

On-Demand Compilation

On-Demand compilation is one of the oldest ideas in computer science. Slimline Open Firmware (SLOF)[7] is a FORTHbased implementation of Open Firmware developed by IBM for some of its Power and Cell processors. SLOF is capable of storing all of Open Firmware as source in the flash memory and compiling components to indirect threading on demand[2].

In the last few decades, as our compiler infrastructure has gotten slower and more complex, true on-demand compilation has split into two different forms. First is the on-demand compilation of source into executable byte codes, as in Python. The byte codes are not native but are more efficient than source. If the python interpreter finds the byte code it will interpret that instead of source to provide improved performance. Java takes the process one step further with the Just In Time compilation of byte code to machine code[20] to boost performance.

Embedding kernel and root file systems in flash

The LinuxBIOS project[14][1], together with clustermatic[25], used an embedded kernel and simple root file system to manage supercomputing clusters. Due to space constraints of 1 MiB or less of flash, clusters embedded only a single-processor Linux kernel with a daemon. The daemon was a network bootloader that downloaded a more complex SMP kernel and root file system and started them. Clusters built this way were able to boot 1024 nodes in the time it took the standard PXE network boot firmware to find a working network interface.

Early versions of One Laptop Per Child used LinuxBIOS, with Linux in flash as a boot loader, to boot the eventual target. This system was very handy, as they were able to embed a full WIFI stack in flash with Linux, and could boot test OLPC images over WIFI. The continuing growth of the Linux kernel, coupled with the small flash size on OLPC, eventually led OLPC to move to Open Firmware.

AlphaPower shipped their Alpha nodes with a so-called Direct Boot Linux, or DBLX. This work was never published, but the code was partially released on sourceforge.net just as AlphaPower went out of business. Compaq also worked with a Linux-As-Bootloader for the iPaq.

Car computers and other embedded ARM systems frequently contain a kernel and an ext2 formatted file system in NOR flash, that is, flash that can be treated as memory instead of a block device. Many of these kernels use the so-called eXecute In Place[3] (XIP) patch, which allows the kernel to page binaries directly from the memory-addressable flash rather than copying it to RAM, providing a significant savings in system startup time. A downside of this approach is that the executables can not be compressed, which puts further pressure on the need to optimize binary size. NOR flash is very slow, and paging from it comes at a significant performance cost. Finally, an uncompressed binary image stored in NOR flash has a much higher monetary cost than the same image stored in RAM since the cost per bit is so much higher.

UEFI[12] contains a non-Linux kernel (the UEFI firmware binary) and a full set of drivers, file systems, network protocol stacks, and command binaries in the firmware image. It is a full operating system environment realized as firmware.

The ONIE project[23] is a more recent realization of the Kernel-in-flash idea, based on Linux. ONIE packs a Linux kernel and Busybox binaries into a very small package. Since the Linux build process allows an initial RAM file system (initramfs) to be built directly into the kernel binary, some companies are now embedding ONIE images into flash with coreboot. Sage Engineering has shown a bzImage with a small Busybox packed into a 4M image. ONIE has brought new life to an old idea: packaging a kernel and small set of binaries in flash to create a fast, capable boot system.

References

[1] AGNEW, A., SULMICKI, A., MINNICH, R., AND ARBAUGH, W. A. Flexibility in rom: A stackable open source bios. In USENIX Annual Technical Conference, FREENIX Track (2003), pp. 115–124.

[2] (AUTHOR OF SLOF), S. B. Personal conversation.

[3] BENAVIDES, T., TREON, J., HULBERT, J., AND CHANG, W. The enabling of an execute-in-place architecture to reduce the embedded system memory footprint and boot time. Journal of computers 3, 1 (2008), 79–89.

[4] BOGOWITZ, B., AND SWINFORD, T. Intel⃝R active management technology reduces it costs with improved pc manageability. Technology@ Intel Magazine (2004).

[5] CELEDA, P., KREJCI, R., VYKOPAL, J., AND DRASAR, M. Embedded malware-an analysis of the chuck norris botnet. In Computer Network Defense (EC2ND), 2010 European Conference on (2010), IEEE, pp. 3–10.

[6] CUI, A., COSTELLO, M., AND STOLFO, S. J. When firmware modifications attack: A case study of embedded exploitation. In NDSS (2013).

[7] DALY, D., CHOI, J. H., MOREIRA, J. E., AND WATERLAND, A. Base operating system provisioning and bringup for a commercial supercomputer. In Parallel and Distributed Processing Symposium, 2007. IPDPS 2007. IEEE International (2007), IEEE, pp. 1–7.

[8] DURUMERIC, Z., KASTEN, J., ADRIAN, D., HALDERMAN, J. A., BAILEY, M., LI, F., WEAVER, N., AMANN, J., BEEKMAN, J., PAYER, M., ET AL. The matter of heartbleed. In Proceedings of the 2014 Conference on Internet Measurement Conference (2014), ACM, pp. 475–488.

[9] KALLENBERG, C., AND BULYGIN, Y. All your boot are belong to us intel, mitre. cansecwest 2014.

[10] KALLENBERG, C., KOVAH, X., BUTTERWORTH, J., AND CORNWELL, S. Extreme privilege escalation on windows 8/uefi systems.

[11] KOZIOL, J., LITCHFIELD, D., AITEL, D., ANLEY, C., EREN, S., MEHTA, N., AND HASSELL, R. The Shellcoder’s Handbook. Wiley Indianapolis, 2004.

[12] LEWIS, T. Uefi overview, 2007.

[13] MAY,D.Occam.ACMSigplanNotices18,4(1983),69–79.

[14] MINNICH, R. G. Linuxbios at four. Linux J. 2004, 118 (Feb. 2004), 8–.

[15] MOON, S.-P., KIM, J.-W., BAE, K.-H., LEE, J.-C., AND SEO, D.-W. Embedded linux implementation on a commercial digital tv system. Consumer Electronics, IEEE Transactions on 49, 4 (Nov 2003), 1402–1407.

[16] PIKE, R. Another go at language design. Stanford University Computer Systems Laboratory Colloquium.

[17] RITCHIE, D. M. The limbo programming language. Inferno Programmer’s Manual 2 (1997).

[18] SACCO, A. L., AND ORTEGA, A. A. Persistent bios infection. In CanSecWest Applied Security Conference (2009).

[19] SAMPATHKUMAR, R. Vulnerability Management for Cloud Computing-2014: A Cloud Computing Security Essential. Rajakumar Sampathkumar, 2014.

[20] SUGANUMA, T., OGASAWARA, T., TAKEUCHI, M., YASUE, T., KAWAHITO, M., ISHIZAKI, K., KOMATSU, H., AND NAKATANI, T. Overview of the ibm java just-in-time compiler. IBM systems Journal 39, 1 (2000), 175–193.

[21] TEAM, G. The go programming language specification. Tech. rep., Technical Report http://golang. org/doc/doc/go spec. html, Google Inc, 2009.

[22] VAN HENSBERGEN, E., AND MINNICH, R. Grave robbers from outer space: Using 9p2000 under linux. In USENIX Annual Technical Conference, FREENIX Track (2005), pp. 83–94.

[23] VARIOUS. No papers have been published on onie; see onie.org.

[24] VARIOUS. No papers were published; see perllinux.sourceforge.net.

[25] WATSON, G. R., SOTTILE, M. J., MINNICH, R. G., CHOI, S.-E., AND HERTDRIKS, E. Pink: A 1024-node single-system image linux cluster. In High Performance Computing and Grid in Asia Pacific Region, 2004. Proceedings. Seventh International Conference on (2004), IEEE, pp. 454–461.

[26] WELLS, N. Busybox: A swiss army knife for linux. Linux J. 2000, 78es (Oct. 2000).

[27] WINTERBOTTOM, P. Alef language reference manual. Plan 9 Programmer’s Man (1995).

LinuxBoot Utilities

In order to bootstrap, build and maintain LinuxBoot projects, we provide a handful of utilities for extracting, reducing, reworking, and stitching firmware images.

UEFI Tool Kit

Authors: Ryan O'Leary, Gan Shun Lim and Andrea Barberio

In previous chapters, you learned how to read a raw ROM image from a flash part. If you've been following along, you know the next step is to insert a Linux kernel.

Inspecting and modifying ROM images is tricky and can involve a fair amount of tinkering. These images typically contain a number of file systems, drivers, tables, data structures and opaque blobs. They also differ significantly from the UNIX model of a file systems, thus cannot be reasonably mounted in Linux.

UEFI Tool Kit (UTK) is intended to be a one-stop-shop for reading, writing and modifying UEFI images -- the most common type of firmware image for x86 systems. UTK can parse a number of data structures including UEFI firmware volumes, Intel firmware descriptors and FIT.

In this chapter, we'll go over how to:

1. Install UTK
2. Inspect ROMs
3. Modify ROMs
4. Common pitfalls
5. Extend UTK with additional commands

Synopsis

$ make bzImage
$ sudo flashrom -r /tmp/ROM.bin
$ utk /tmp/ROM.bin replace_pe32 Shell arch/86/boot/bzImage save /tmp/NEWROM.bin
$ sudo flashrom -w /tmp/NEWROM.bin

Quick start

We assume you have a way to read and write the FLASH into a file.

Let's assume you have read FLASH into an image called ROM.bin and you have a kernel, called bzImage, which you want to insert into ROM.bin. Be sure the kernel is buildable as an EFI driver (DXE); see the pitfalls section. The easiest option is to replace the UEFI shell. This is a quick and easy way to get started. In the long term, you want to remove as much of UEFI as possible, but replacing the shell is always our first step on a new board.

Get the tool:

$ go get -u github.com/linuxboot/fiano/cmds/utk

Replace the shell:

$ utk ROM.bin replace_pe32 Shell bzImage save NEWROM.bin

After that, you can flash NEWROM.bin and test. If anything goes wrong, such as not enough space, you will need to refer to the more detailed instructions below.

Installation

At the time of writing, you must clone and build UTK from source -- binary distributions are not officially available. The source code resides in the Fiano project:

https://github.com/linuxboot/fiano/

Aside: what is the difference between Fiano and UTK? The Fiano project contains a few more tools besides UTK, but UTK is a big element.

We'll assume you already have Go installed. Check your installation with:

$ go version
go version go1.11 linux/amd64

Linux and the latest stable version of Go are recommended. Either download the official binary distributions of Go or install from source. See https://golang.org/ for details.

With Go, download and install UTK:

$ go get -u github.com/linuxboot/fiano/cmds/utk

Running the above line installs utk to your $GOPATH/bin directory (or $HOME/go/bin if the GOPATH environment variable is not set). Adding this directory to your $PATH is recommended.

Make sure it works with:

$ utk -h
Usage: utk [flags] <file name> [0 or more operations]

Operations:
  cat                   : cat a file with a regexp that matches a GUID
  comment               : Print one arg
  count                 : count the number of each firmware type
  dump                  : dump a firmware file
  dxecleaner            : automates removal of UEFI drivers
  dxecleaner_blacklist  : automates removal of UEFI drivers with a blacklist file
  extract               : extract the files to a directory
  find                  : find a file by GUID or Name
  flatten               : prints a JSON list of nodes
  insert_after          : insert a file after another file
  insert_before         : insert a file before another file
  insert_end            : insert a file at the end of a firmware volume
  insert_front          : insert a file at the beginning of a firmware volume
  json                  : produce JSON for the full firmware volume
  remove                : remove a file from the volume
  remove_pad            : remove a file from the volume and replace it with a pad file of the same size
  repack                : repack a per file compressed fv to a nested compressed fv
  replace_pe32          : replace a pe32 given a GUID and new file
  save                  : assemble a firmware volume from a directory tree
  table                 : print out important information in a pretty table
  validate              : perform extra validation checks

Don't fret if your list of operations differs. UTK is an evolving project!

Inspecting ROMs

Throughout this section, we'll demonstrate commands for inspecting a UEFI image. When confronted with a new image, run these commands to get a "lay of the land".

Start by downloading the UEFI image used in these examples:

$ wget https://github.com/linuxboot/fiano/raw/master/integration/roms/OVMF.rom

Aside: alternatively, all UTK operations should work with your own UEFI images. Simply substitute "OVMF.rom" with your own UEFI image in all the examples below. If you encounter any problems, please file an issue at https://github.com/linuxboot/fiano/issues.

First, it is advisable to print a count of each firmware element:

$ utk OVMF.rom count
{
        "FirmwareTypeCount": {
                "BIOSRegion": 1,
                "File": 118,
                "FirmwareVolume": 5,
                "Section": 365
        },
        "FileTypeCount": {
                "EFI_FV_FILETYPE_APPLICATION": 2,
                "EFI_FV_FILETYPE_DRIVER": 94,
                "EFI_FV_FILETYPE_DXE_CORE": 1,
                "EFI_FV_FILETYPE_FFS_PAD": 7,
                "EFI_FV_FILETYPE_FIRMWARE_VOLUME_IMAGE": 1,
                "EFI_FV_FILETYPE_FREEFORM": 3,
                "EFI_FV_FILETYPE_PEIM": 7,
                "EFI_FV_FILETYPE_PEI_CORE": 1,
                "EFI_FV_FILETYPE_RAW": 1,
                "EFI_FV_FILETYPE_SECURITY_CORE": 1
        },
        "SectionTypeCount": {
                "EFI_SECTION_DXE_DEPEX": 44,
                "EFI_SECTION_FIRMWARE_VOLUME_IMAGE": 2,
                "EFI_SECTION_GUID_DEFINED": 1,
                "EFI_SECTION_PE32": 99,
                "EFI_SECTION_RAW": 21,
                "EFI_SECTION_USER_INTERFACE": 99,
                "EFI_SECTION_VERSION": 99
        }
}

The definition of a "Firmware Element" is in order. Firmware images are hierarchical and can be represented as a tree. Each node in the tree is a "Firmware Element". Each element has a type such as "BIOSRegion", "FirmwareVolume", "File" and "Section" as seen above. Files (and sections) themselves have an additional type dictated by the UEFI spec. There are three major file types you should be aware of:

• EFI_FV_FILETYPE_DRIVER: This is the most numerous file type and is often called a "DXE". They persist in memory even after their main function exits.
• EFI_FV_FILETYPE_APPLICATION: Applications do not persist in memory after exiting. For example, the EFI Shell is an EFI Application.
• EFI_FV_FILETYPE_FIRMWARE_VOLUME_IMAGE: These file types allow nesting firmware volumes. You will see this when an entire firmware volume is compressed.

TODO: Diagram showing a tree of these firmware elements.

To view a human-readable tree of all the firmware elements, types and sizes, run:

$ utk OVMF.rom table | less
Node        GUID/Name                             Type                                   Size
BIOS                                                                                     0x400000
 FV         FFF12B8D-7696-4C8B-A985-2747075B4F50                                          0x84000
  Free                                                                                        0x0
 FV         8C8CE578-8A3D-4F1C-9935-896185C32DD3                                         0x348000
  File      9E21FD93-9C72-4C15-8C4B-E77F1DB2D792  EFI_FV_FILETYPE_FIRMWARE_VOLUME_IMAGE  0x1256a7
   Sec                                            EFI_SECTION_GUID_DEFINED               0x12568f
    Sec                                           EFI_SECTION_RAW                            0x7c
    Sec                                           EFI_SECTION_FIRMWARE_VOLUME_IMAGE       0xe0004
     FV     8C8CE578-8A3D-4F1C-9935-896185C32DD3                                          0xe0000
      File  1B45CC0A-156A-428A-AF62-49864DA0E6E6  EFI_FV_FILETYPE_FREEFORM                   0x2c
       Sec                                        EFI_SECTION_RAW                            0x14
      File  FFFFFFFF-FFFF-FFFF-FFFF-FFFFFFFFFFFF  EFI_FV_FILETYPE_FFS_PAD                    0x40
      File  52C05B14-0B98-496C-BC3B-04B50211D680  EFI_FV_FILETYPE_PEI_CORE                 0xc4fa
       Sec                                        EFI_SECTION_RAW                            0x3c
       Sec                                        EFI_SECTION_PE32                         0xc484
       Sec  PeiCore                               EFI_SECTION_USER_INTERFACE                 0x14
       Sec                                        EFI_SECTION_VERSION                         0xe
...

This format is compact and easy for humans reading, but not ideal for machine consumption. Use the json command to print everything (including much more metadata) as JSON:

$ utk OVMF.rom json | less

Combine utk with the JSON query command, jq (sudo apt-get install jq), and other UNIX commands to quickly write powerful queries. For example, the following lists all the GUIDs, sorted and without duplicates:

$ utk OVMF.rom json | jq -r '..|.GUID?|select(type=="string")' | sort -u
00000000-0000-0000-0000-000000000000
0167CCC4-D0F7-4F21-A3EF-9E64B7CDCE8B
0170F60C-1D40-4651-956D-F0BD9879D527
021722D8-522B-4079-852A-FE44C2C13F49
025BBFC7-E6A9-4B8B-82AD-6815A1AEAF4A
...

To only print the JSON for specific files, use the find command:

# The find command uses a regex to match on the name or GUID.
# These three examples find and print the JSON for the same file:
$ utk OVMF.rom find 'Sh.*'
$ utk OVMF.rom find 'Shell'
$ utk OVMF.rom find 7C04A583-9E3E-4F1C-AD65-E05268D0B4D1
{
        "Header": {
                "UUID": {
                        "UUID": "7C04A583-9E3E-4F1C-AD65-E05268D0B4D1"
                },
                "Type": 9,
                "Attributes": 0
        },
        "Type": "EFI_FV_FILETYPE_APPLICATION",
        "Sections": [
                {
                        "Header": {
                                "Type": 21
                        },
                        "Type": "EFI_SECTION_USER_INTERFACE",
                        "ExtractPath": "",
                        "Name": "Shell"
                },
                ...
        ],
        "ExtractPath": "",
        "DataOffset": 24
}

Note that UEFI uses GUIDs to identify files. Some files also have a name which is stored within the file's UI section. Like find, most of UTKs commands let you match a file by its name or GUID.

The examples up until now have only dealt with file metadata and not the file's contents. The extract <DIR> command extracts all the files from the image and saves them to <DIR>. <DIR>/summary.json lists all the paths to the extracted files along with their metadata.

$ utk OVMF.rom extract OVMF/

After modifying the files, they can be reassembled with:

$ utk OVMF/ save OVMF2.rom

Modifying ROMs

First, let's verify the image works by running it inside QEMU. This step is not absolutely necessary, but gives us confidence the image works before and after each change we make.

$ qemu-system-x86_64 -bios OVMF.rom -nographic -net none

For the provided OVMF.rom image, this should boot to the EDK2 shell.

TODO: include screenshot of the EDK2 shell

Multiple commands can be used together to form a pipeline. The first argument always loads the image into memory and the last argument typically writes the output. The commands in between operate on the image in memory and are reminiscent of a UNIX pipeline. The general syntax is:

utk <IMAGE or DIR>                 \
  <COMMAND0> <ARG0_0> <ARG0_1> ... \
  <COMMAND1> <ARG1_0> <ARG1_1> ... \
  ...

To see the pipeline in action, we introduce two new commands:

• remove <file GUID or NAME regex>: Remove a file from a firmware volume. The search has the same semantics as find.
• replace_pe32 <file GUID or NAME regex> <FILE>: Replace the pe32 section of a file with the given file. The search has the same semantics as find. The file must be a valid pe32 binary.
• save <FILE>: Save the firmware image to the given file. Usually, this is the last command in a pipeline.

The following pipeline removes some unnecessary drivers (anything that starts with Usb and the Legacy8259 driver which has the GUID 79ca4208-bba1-4a9a-8456-e1e66a81484e) and replaces the Shell with Linux. Often you need to remove drivers to make room for Linux which makes the pipeline convenient. This is the essence of LinuxBoot:

$ stat linux.efi
linux.efi: Linux kernel x86 boot executable bzImage, version 4.17.0
$ utk OVMF.rom \
  remove 'Usb.*' \
  remove 79ca4208-bba1-4a9a-8456-e1e66a81484e \
  replace_pe32 Shell linux.efi \
  save OVMF2.rom

That's all there to it! Try experimenting with the other commands such as insert.

Common Pitfalls

Kernel is not built as a DXE or has not enabled UEFI stub mode

In order to be properly bootable as a DXE, kernels must have the following enabled:

CONFIG_EFI=y
CONFIG_EFI_STUB=y

Files are missing from the Firmware Volume

When UTK does not recognize the compression format used by the particular image, the files within it are not listed.

In the wild, three compression schemes are common:

To determine which compression scheme you are using, search for the respective GUID in the json summary.

File size too big!

File size too big! File with GUID: XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX has length 543210, but is only 123450 bytes big

When saving a UEFI image, files are added successively to each firmware volume. The first file which overflows the volume's size causes this error.

If you were inserting files, you will need to delete existing files to make room.

There is a special cases where this error is generated without any operations:

utk OVMF.rom save OVMF2.rom

How can this be? No changes should be made to the image!

Not quite (and the complete list of differences can be found in the "binary equality section") -- compressed volumes are recompressed.

By default, UTK uses the Go compressor, which is generally worse than the compression found in most UEFI images. Pass --systemXZ=xz as the first argument to UTK to use a better compressor.

(TODO for everything after this point) Arbitrary data before or after the image.

Find a general solution which works for all images is a topic of research: #200.

Hard-coded addresses

Binary equality

TODO

Extending UTK

Visitor pattern means decoupling the structure from the operations.

• pkg/uefi: structure
• pkg/visitors: operations

Good resources:

1. https://sourcemaking.com/design_patterns/visitor
2. https://en.wikipedia.org/wiki/Visitor_pattern

A good visitor still works when new Firmware are introduced. A good Firmware still works when a new visitor is introduced.

AST

Abstract Syntax Tree -- this is a concept borrowed from compilers. When you're extracting the DXE to create a tree of structs containing a simplified model, you're essentially creating an AST. Then think about how patterns used in compiler architecture might apply to UTK.

Visitor Interface

Each visitor implements the following:

type Visitor interface {
    VisitFV(*FV) error
    VisitFile(*File) error
    VisitSection(*FileSection) error
    // ...
}

Think of a visitor as an "action" or a "transformation" being applied on the AST.

Visitor

A struct implementing Visitor performs a transformation on the AST, for example:

type RenameDXE struct {
    before, after string
}
func (v *RenameDXE) VisitFV(fv *FV) error {
    // Recursively apply on files in the FV.
    for i := range fv.Files {
        fv.Files[i].Apply(v)
    }
    return nil
}
func (v *RenameDXE) VisitFile(f *File) error {
    if f.Type == FILETYPE_DXE && f.Name == v.before {
        f.Name = after
    }
    return nil
}
func (v *RenameDXE) VisitSection(s *FileSection) error {
    return nil
}

You can imagine visitors being implemented for other actions, such as:

• Remove a DXE with the given GUID from the AST
• Replace a GUID with a file
• Validate that all the nodes in the tree are valid
• Find compressed files in the tree and decompress them
• Assembe the AST back into an image.
• Recursively write the AST to the filesystem (what you currently do with extract)
• Print an overview of the files to the terminal for debugging
• ...

It is easy to add more visitors without modifying existing code. Each action can be in a separate file.

Applying

Visitors are applied to the AST. Each node in the AST has an "Apply" method, for example:

func (f *File) Apply(v *visitor) error {
    return v.VisitFile(f)
}

This is so the visitors can be applied recursively over the AST.

To apply the above RenameDXE visitor, you'd run:

v := &RenameDXE{"Shell", "NotShell"}
fv.Apply(v)

Chaining Visitors Together

It would be exciting/useful to be able to chain these small actions together through the command line. For example:

utk extract bios.rom \
    remove a2dad2a-adadad-a2d2-ad23a3 \
    remove 9d8cd98-d9c8d9-d9c8-9d8c8c \
    replaceDXEWithFile bab8a98-a9ba89a-9aba-a98a9 linux.efi \
    validate \
    save new_bios.rom

Again, it is easy to write new actions in Go which modify nodes in the AST. Create a new file, new struct, and implement the visitFV/visitFile/visitSection methods to modify the AST.

TODO: reference the UEFI spec.

TODO: mention alternatives

• binwalk
• fresh0r/romdump
• UEFITool
• uefi-firmware-parser

The u-root cpu command

Do you want to have all the tools on your linuxboot system that you have on your desktop, but you can't get them to fit in your tiny flash part? Do you want all your desktop files visible on your linuxboot system, but just remembered there's no disk on your linuxboot system? Are you tired of using scp or wget to move files around? Do you want to run emacs or vim on the linuxboot machine, but know they can't ever fit? What about zsh? How about being able to run commands on your linuxboot machine and have the output appear on your home file system? You say you'd like to make this all work without having to fill out web forms in triplicate to get your organization to Do Magic to your desktop?

Your search is over: cpu is here to answer all your usability needs.

The problem: running your program on some other system

People often need to run a command on a remote system. That is easy when the remote system is the same as the system you are on, e.g., both systems are Ubuntu 16.04; and all the libraries, packages, and files are roughly the same. But what if the systems are different, say, Ubuntu 16.04 and Ubuntu 18.10? What if one is Centos, the other Debian? What if a required package is missing on the remote system, even though in all other ways they are the same?

While these systems are both Linux, and hence can provide Application Binary Interface (ABI) stability at the system call boundary, above that boundary stability vanishes. Even small variations between Ubuntu versions matter: symbol versions in C libraries differ, files are moved, and so on.

What is a user to do if they want to build a binary on one system, and run it on another system?

The simplest approach is to copy the source to that other system and compile it. That works sometimes. But there are limits: copying the source might not be allowed; the code might not even compile on the remote system; some support code might not be available, as for a library; and for embedded systems, there might not be a compiler on the remote system. Copy and compile is not always an option. In fact it rarely works nowadays, when even different Linux distributions are incompatible.

The next option is to use static linking. Static linking is the oldest form of binary on Linux systems. While it has the downside of creating larger binaries, in an age of efficient compilers that remove dead code, 100 gigabit networks, and giant disks and memory, that penalty is not the problem it once was. The growth in size of static binaries is nothing like the growth in efficiency and scale of our resources. Nevertheless, static linking is frowned upon nowadays and many libraries are only made available for dynamic linking.

Our user might use one of the many tools that package a binary and all its libraries into a single file, to be executed elsewhere. The u-root project even offers one such tool, called pox, for portable executables. Pox uses the dynamic loader to figure out all the shared libraries a program uses, and place them into the archive as well. Further, the user can specify additional files to carry along in case they are needed.

The problem here is that, if our user cares about binary size, this option is even worse. Deadcode removal won’t work; the whole shared library has to be carried along. Nevertheless, this can work, in some cases.

So our user packages up their executable using pox or a similar tool, uses scp to get it to the remote machine, logs in via ssh, and all seems to be well, until at some point there is another message about a missing shared library! How can this be? The program that packaged it up checked for all possible shared libraries.

Unfortunately, shared libraries are now in the habit of loading other shared libraries, as determined by reading text files. It’s no longer possible to know what shared libraries are used; they can even change from one run of the program to the next. One can not find them all just by reading the shared library itself. A good example is the name service switch library, which uses /etc/nsswitch.conf to find other shared libraries. If nsswitch.conf is missing, or a library is missing, some versions of the name service switch library will core dump.

Not only must our user remember to bring along /etc/nsswitch.conf, they must also remember to bring along all the libraries it might use. This is also true of other services such as Pluggable Authentication Modules (PAM). And, further, the program they bring along might run other programs, with their own dependencies. At some point, as the set of files grows, frustrated users might decide to gather up all of /etc/, /bin, and other directories, in the hope that a wide enough net might bring along all that’s needed. The remote system will need lots of spare disk or memory! We’re right back where we started, with too many files for too little space.

In the worst case, to properly run a binary from one system, on another system, one must copy everything in the local file system to the remote system. That is obviously difficult, and might be impossible if the remote system has no disk, only memory.

One might propose having the remote system mount the local system via NFS or Samba. While this was a common approach years ago, it comes with its own set of problems: all the remote systems are now hostage to the reliability of the NFS or Samba server. But there’s a bigger problem: there is still no guarantee that the remote system is using the same library versions and files that the user’s desktop is using. The NFS server might provide, e.g., Suse, to the remote system; the user’s desktop might be running Ubuntu. If the user compiles on their desktop, the binary might still not run on the remote system, as the Suse libraries might be different. This is a common problem.

Still worse, with an NFS root, everyone can see everyone’s files. It’s like living in an apartment building with glass walls. Glass houses only look good in architecture magazines. People want privacy.

We know what ssh provides; but what else do we need?

Ssh solves the problem of safely getting logged in to a remote machine. While this is no small accomplishment, it is a lot like being parachuted into a foreign land, where the rules are changed. It’s a lot nicer, when going to a new place, to be able to bring along some survival gear, if not your whole house!

Users need a way to log in to a machine, in a way similar to ssh, but they need to bring their environment with them. They need their login directory; their standard commands; their configuration files; and they need some privacy. Other users on the machine should not be able to see any of the things they bring with them. After all, everyone who goes camping wants to believe they are the only people at that campground!

How cpu provides what we need

cpu is a Go-based implementation of Plan 9's cpu command. It uses the go ssh package, so all your communications are as secure as ssh. It can be started from /sbin/init or even replace /sbin/init, so you have a tiny flash footprint. You can see the code at github.com:u-root/cpu. It's also small: less than 20 files, including tests.

cpu runs as both a client (on your desktop) and an ssh server (on your linuxboot machine). On your desktop, it needs no special privilege. On the linuxboot system, there is only one binary needed: the cpu daemon (cpud). As part of setting up a session, in addition to normal ssh operations, cpu sets up private name space at important places like /home/$USER, /bin, /usr,and so on. Nobody gets to see what other people’s files are.

Ssh provides remote access. Cpu goes one step further, providing what is called resource sharing -- resources, i.e., files from the client machine can be used directly on the remote machine, without needing to manually copy them. Cpud implements resource sharing by setting up a file systemmount on the remote machine and relaying file I/O requests back to the desktop cpu process. The desktop command services those requests; you don't need to run a special external server. One thing that is a bit confusing with cpu: the desktop client is a file server; the remote server’s Linux kernel is a file client. Cpu has to do a bit more work to accomplish its task.

Cpu will change your life. You can forget about moving files via scp: once you 'cpu in', the /home directory on your linuxboot node is your home directory. You can cd ~and see all your files. You can pick any shell you want, since the shell binary comes from your desktop, not flash. You don't have to worry about fitting zsh into flash ever again!

At Google we can now run chipsec, which imports 20M of Python libraries, because we have cpu and we can redirect chipsec output to files in our home directory.

Here is an example session:

In this command, we cpu to a PC Engines APU2. We have built a kernel and u-root initramfs containing just one daemon -- the cpu daemon -- into the flash image. The APU2 does not even need a disk; it starts running as a “cpu appliance.”

The bash is not on the cpu node; it will come from our desktop via the 9p mount.

    rminnich@xcpu:~/gopath/src/github.com/u-root/u-root$ cpu apu2
    root@(none):/#
    root@(none):/# ls ~
    IDAPROPASSWORD  go      ida-7.2  projects
    bin             gopath  papers   salishan2019random  snap
    root@(none):/# exit

The bash and ls command, and the shared libraries they need, do not exist on the apu2; cpu makes sure that the client provides them to the cpu server. The home directory is, similarly, made available to the remote machine from the local machine.

A big benefit of cpu is that, as long as the network works, users can create very minimal flash images, containing just the cpu daemon, just enough to get the network going. Once the network is up, users can 'cpu in', and everything they need is there. It actually looks like they are still logged in to their desktop, except, of course, truly local file systems such as /proc and /sys will come from the machine they are on, not their desktop.

An easy overview of how cpu works

Cpu, as mentioned, consists of a client and a server. The client is on your desktop (or laptop), and the server is on the remote system. Both client and server use an ssh transport, meaning that the “wire” protocol is ssh. In this way, cpu is just like ssh.

As mentioned above, the situation for cpu is a bit more complicated than for ssh. Cpu provides resource sharing, but not from the server to the client, but rather from the client to the server. The cpu client is a file server; the cpu server connects the kernel on the server machine to the file server in the client, as shown below. Things to note:

1. Cpud, on the remote or server machine, sets up a “private name space mount” of /tmp for the program. “Private name space mount” just means that only that program, and its children, can see what is in its private /tmp. Other, external programs continue to use /tmp, but they are different instantiations of /tmp.
2. The private name space mount of /tmp is on a filesystem in RAM. The data stored in /tmp is not visible to other processes, and not persistent.
3. cpud creates a directory, cpu, in the private /tmp; and mounts the server on it. This mount point is also invisible outside the process and its children.
4. To make sure that names like /bin/bash, and /usr/lib/libc.so work, cpud sets up bind mounts from, e.g., /tmp/cpu/bin to /bin. These are also private mounts, and do not affect any program outside the one cpud starts. Anytime the program and its children access files in /bin, /lib, /usr, /home/$USER, and so on, they are accessing files from the client machine via the built-in client file server.
5. The client cpu program passes the full environment from the client machine to cpud. When the client program requests that, e.g., bash be run, the cpud uses the PATH environment variable to locate bash. Because of the private name space mounts and binds, bash will be found in /bin/bash, and its libraries will be found in their usual place. This is an essential property of cpu, that the names used on the user’s machine work the same way on the remote machine. An overview of the process is shown below.

Cpu startup

The startup proceeds in several steps. Every session begins with an initial contact from the cpu client to the cpu server.

The first step the cpud does is set up the mounts back to the client. It then sets up the bind mounts such as /bin to /tmp/cpu/bin. In the following figure, we compress the Linux kernel mount and bind mounts shown above into a smaller box called “name space.”

Next, cpu and the cpud set up the terminal management.

Finally, cpud sets up the program to run. Because the PATH variable has been transferred to cpud, and the name space includes /bin and /lib, the cpud can do a standard Linux exec system call without having to locate where everything is. Native kernel mechanisms create requests as files are referenced, and the cpu file server support does the rest.

Why do we only show one program instead of many? From the point of view of cpud, it only starts one program. From the point of view of users, there can be many. But if there is more than one program to start, that is not the responsibility of cpud. If more than one program is run, they will be started by the program that cpud started, i.e., a command interpreter like the shell. Or it could be as simple as a one-off command like date. From the point of view of cpud, it’s all the same. Cpud will wait until the process it started, and all its children, have exited. But cpud’s responsibilities to start a program ends with that first program.

But what happens when cpud runs that first program? Here is where it gets interesting, and, depending on your point of view, either magical, confounding, or counterintuitive. We’ll go with magical.

Starting that first program

As mentioned above, cpud sets up mounts for a name space, and calls the Linux exec() call to start the program.

We can actually watch all the cpu file server operations. The file server protocol is called 9P2000. We are going to present a filtered version of the file I/O from running a remote date; in practice, you can watch all the opens, reads, writes, and closes the remote process performs.

The trace for running date starts right when the remote program has called exec, and the kernel is starting to find the program to run¹. The file opens look like this, on a user’s system:

Open /bin/date
Open /lib/x86_64-linux-gnu/ld-2.27.so
Open /etc/ld.so.cache
Open /lib/x86_64-linux-gnu/libc-2.27.so
Open /usr/lib/locale/locale-archive
Open /usr/share/zoneinfo/America/Los_Angeles

The kernel opened /bin/date, determined what libraries (files ending in .so) it needed, and opened them as well.

We can compare this with a local execution:

execve "/bin/date"
access "/etc/ld.so.nohwcap"
access "/etc/ld.so.preload"
openat "/etc/ld.so.cache"
access "/etc/ld.so.nohwcap"
openat "/lib/x86_64-linux-gnu/libc.so.6"
openat "/usr/lib/locale/locale-archive"
openat "/etc/localtime"

Note that several files do not show up in our trace; they are in /etc, and the cpud does not set up a bind mount over /etc. But the other files look very similar. You might wonder why the local version opens /etc/localtime, and the remote version opens /usr/share/zoneinfo/America/Los_Angeles.

The reason is that etc/localtime is a symlink:

lrwxrwxrwx 1 root root 39 May 29 12:47 /etc/localtime -> /usr/share/zoneinfo/America/Los_Angeles

The access to /etc/localtime does not get handled by the server; but the access to /usr/share/zoneinfo/America/Los_Angelesdoes.

What about different architectures? What if we are using an x86 but want to cpu to an ARM processor?

We can set the local cpu up to talk to a remote cpu that needs different binaries. We might have an entire ARM file system tree in ~/arm, for example. We would then invoke cpu as follows:

cpu -root ~/arm date

And the remote cpud, running on an ARM, would be provided with ARM binaries.

Learning how to use cpu

Cpu can be a hard thing to learn, not because it is difficult, but because it is different. To paraphrase Yoda, you have to unlearn what you have learned. Forget about copying files from here to there; when you cpu there, it looks like your files are waiting for you.

You can start experimenting and learning about cpu by just running it locally.

A set of binaries for you to try

In order for you to try it out, start by working with the set of cpu binaries at https://github.com/u-root/cpubinaries. With them, you can create a bootable, mountable USB image that you can download. The image contains a cpu client that runs on Linux, a private key, and, when booted, it starts a cpu daemon and waits to serve cpu clients. The cpu client is statically linked and hence should run on any Linux from the last 10 years or so.

The binaries include:

• A kernel (cpukernel) with a built-in initramfs containing cpud, as well as a public key. Also included, should you want to build your own, is the config file (cpu.config).
• A binary client program, cpu, as well as the private key to use. You can place this key in ~/.ssh or specify it via the -key option to cpu.
• A script to run the USB stick via qemu (TESTQEMU); and a script to run a cpu command (EXAMPLE).
• The extlinux.conf used for the USB stick.

usbstick.xz is a compressed USB stick image that is bootable. It will uncompress to about 7GB. You can use the TESTQEMU script to try it out, or use dd to write it to a USB stick and boot that stick on an x86 system.

Be careful how you use the keys; they're public. You should really only use them as part of the demo.

The cpukernel was built using the github.com:linuxboot/mainboards repo. If you clone this repo, the following commands will rebuild the kernel:

• cd mainboards/intel/generic
• make fetch
• make cpukernel

How to use the cpu binaries

You’ll first need to start the server, and we show the entire sequence below, including unpacking the image:

xz -d usbstick.xz

How you run qemu depends on whether you want graphics or not: if you are not in a windowing environment, add -nographic to the command below. In any event, at the boot: prompt, you can hit return or wait:

bash QEMU -hda usbstick

SeaBIOS (version 1.13.0-1)
iPXE (http://ipxe.org) 00:03.0 CA00 PCI2.10 PnP PMM+3FF90750+3FED0750 CA00
                                                                           Booting from Hard Disk...
SYSLINUX 6.03 EDD 20171017 Copyright (C) 1994-2014 H. Peter Anvin et al
boot:
.
.
.
Freeing unused kernel image (rodata/data gap) memory: 568K
rodata_test: all tests were successful
Run /init as init process

At this point, the cpu daemon is running, and you can try the cpu command:

rminnich@minnich:/home/cpubinaries$ ./cpu -key cpu_rsa localhost date
Fri 16 Oct 2020 04:21:04 PM PDT

You can log in and notice that things are the same:

rminnich@minnich:/home/cpubinaries$ ./cpu -key cpu_rsa localhost

root@192:/home/cpubinaries# ls

cpu cpukernel cpu_rsa.pub extlinux.conf QEMU usbstick

cpu.config cpu_rsa EXAMPLE LICENSE README.md

root@192:/home/cpubinaries#

Note that you end up in the same directory on the remote node that you are in on the host; all the files are there. We can run any program on the remote node that we have on the host:

root@192:/home/cpubinaries# which date
/usr/bin/date
root@192:/home/cpubinaries# date
Fri 16 Oct 2020 04:25:01 PM PDT
root@192:/home/cpubinaries# ldd /usr/bin/date
mount
	linux-vdso.so.1 (0x00007ffd83784000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007efdb93db000)
	/lib64/ld-linux-x86-64.so.2 (0x00007efdb95e4000)
root@192:/home/cpubinaries# mount
...
cpu on /tmp type tmpfs (rw,relatime)
127.0.0.1 on /tmp/cpu type 9p (rw,nosuid,nodev,relatime,sync,dirsync,uname=rminnich,access=client,msize=65536,trans=fd,rfd=9,wfd=9)
rootfs on /tmp/local type rootfs (rw,size=506712k,nr_inodes=126678)
127.0.0.1 on /lib type 9p (rw,nosuid,nodev,relatime,sync,dirsync,uname=rminnich,access=client,msize=65536,trans=fd,rfd=9,wfd=9)
127.0.0.1 on /lib64 type 9p (rw,nosuid,nodev,relatime,sync,dirsync,uname=rminnich,access=client,msize=65536,trans=fd,rfd=9,wfd=9)
127.0.0.1 on /usr type 9p (rw,nosuid,nodev,relatime,sync,dirsync,uname=rminnich,access=client,msize=65536,trans=fd,rfd=9,wfd=9)
127.0.0.1 on /bin type 9p (rw,nosuid,nodev,relatime,sync,dirsync,uname=rminnich,access=client,msize=65536,trans=fd,rfd=9,wfd=9)
127.0.0.1 on /etc type 9p (rw,nosuid,nodev,relatime,sync,dirsync,uname=rminnich,access=client,msize=65536,trans=fd,rfd=9,wfd=9)
127.0.0.1 on /home type 9p (rw,nosuid,nodev,relatime,sync,dirsync,uname=rminnich,access=client,msize=65536,trans=fd,rfd=9,wfd=9)
root@192:/home/cpubinaries#

As you can see, /tmp/cpu is mounted via 9p back to the cpu client (recall that the cpu client is a 9p server, so your files are visible on the remote node). Further, you can see mounts on /usr, /bin, /etc, and so on. For this reason, we can run date and it will find its needed libraries in /usr, as the ldd command demonstrates.

Making cpu easier to use

If you get tired of typing -keys, do the following: put your own cpu_rsa in ~/.ssh; and copy the cpu binary to bin (or build a new one).

Warning! The cpu keys we provide in the repo are only to be used for this demo. You should not use them for any other purpose, as they are in a github repo and hence open to the world.

What if you don’t want all the name space?

Sometimes, you don’t want all the /usr and /bin directories to be replaced with those from your machine. You might, for example, cpu into an ARM system, and hence only need a /home, but nothing else.

The -namespace switch lets you control the namespace. It is structured somewhat like a path variable, with :-seperated components. The default value is /lib:/lib64:/usr:/bin:/etc:/home. You can modify it or even force it to be empty: -namespace="", for example. If it is empty, cpud will only mount the 9p server on /tmp/cpu.

This following example will cpu to an ARM64 host, sharing /home, but nothing else.

cpu arm -namespace=/home /bin/date

For an different architecture system, we might want to specify that the /bin, /lib, and other directories have a different path on the remote than they have locally. The -namespace switch allows this via an = sign:

cpu -namespace /lib:/lib64:/usr:/bin:/etc:/home arm /bin/date

In this case, /bin, /usr, and /lib on the remote system are supplied by /arm/bin, /arm/lib, and /arm/usr locally.

If we need to test cpu without doing bind mounts, we can specify a PWD that requires no mounts and an empty namespace:

PWD=/ cpu -namespace="" -9p=false h /bin/ls
bbin
bin
buildbin
dev
env
etc
go
home
init
...

There is a bit of a subtlety about the interaction of the namespace and 9p switches, which we are still discussing: the -namespace value can override the -9p switch.

If you set -9p=false but have a non-empty namespace variable, then 9p will be set to true. So in this example, the -9p switch has no effect:

cpu -9p=false h ls

Why is this? Because the default value of -namespace is non-empty. The open question: should -9p=false force the namespace to be empty; or should a none-empty namespace for -9p to be true? For now, we have chosen the latter approach.

Another possible approach is to log conflicting settings of these two switches and exit:

cpu -9p=false h ls
error: 9p is false but the namespace is non-empty; to force an empty namespace use -namespace=""

We welcome comments on this issue.

cpu and Docker

Maintaining file system images is inconvenient. We can use Docker containers on remote hosts instead. We can take a standard Docker container and, with suitable options, use docker to start the container with cpu as the first program it runs.

That means we can use any Docker image, on any architecture, at any time; and we can even run more than one at a time, since the namespaces are private.

In this example, we are starting a standard Ubuntu image:

docker run -v /home/rminnich:/home/rminnich -v /home/rminnich/.ssh:/root/.ssh -v /etc/hosts:/etc/hosts --entrypoint /home/rminnich/go/bin/cpu -it ubuntu@sha256:073e060cec31fed4a86fcd45ad6f80b1f135109ac2c0b57272f01909c9626486 h
Unable to find image 'ubuntu@sha256:073e060cec31fed4a86fcd45ad6f80b1f135109ac2c0b57272f01909c9626486' locally
docker.io/library/ubuntu@sha256:073e060cec31fed4a86fcd45ad6f80b1f135109ac2c0b57272f01909c9626486: Pulling from library/ubuntu
a9ca93140713: Pull complete
Digest: sha256:073e060cec31fed4a86fcd45ad6f80b1f135109ac2c0b57272f01909c9626486
Status: Downloaded newer image for ubuntu@sha256:073e060cec31fed4a86fcd45ad6f80b1f135109ac2c0b57272f01909c9626486
WARNING: The requested image's platform (linux/arm64/v8) does not match the detected host platform (linux/amd64) and no specific platform was requested
1970/01/01 21:37:32 CPUD:Warning: mounting /tmp/cpu/lib64 on /lib64 failed: no such file or directory
# ls
bbin  buildbin	env  go    init     lib    proc  tcz  ubin  var
bin   dev	etc  home  key.pub  lib64  sys	 tmp  usr
#

Note that the image was update and then started. The /lib64 mount fails, because there is no /lib64 directory in the image, but that is harmless.

On the local host, on which we ran docker, this image will show up in docker ps:

rminnich@a300:~$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS     NAMES
b92a3576229b   ubuntu    "/home/rminnich/go/b…"   9 seconds ago   Up 9 seconds             inspiring_mcnulty

Even though the binaries themselves are running on the remote ARM system.

cpu and virtiofs

While 9p is very general, because it is transport-independent, there are cases where we can get much better performance by using a less general file system. One such case is with virtofs.

Because virtiofs is purely from guest kernel vfs to host kernel vfs, via virtio transport, it has been measured to run at up to 100 times faster.

We can use virtiofs by specifying virtiofs mounts. The cpud will look for an environemnt variable, CPU_FSTAB, which is in fstab(5) format. The client can specify an fstab in one of two ways: o via the -fstab switch, in which case the client will populate the CPU_FSTAB variable with the contents of the file o by passing the CPU_FSTAB environment variable, which happens by default

On the client side, the file specified via the -fstab takes precedence over any value of the CPU_FSTAB environment variable. On the server side, cpud does not use the -fstab switch, only using the environment variable.

Here is an example of using the CPU_FSTAB variable with one entry:

CPU_FSTAB="myfs /mnt virtiofs rw 0 0" cpu v

In this case, the virtiofs server had the name myfs, and on the remote side, virtiofs was mounted on /mnt.

For the fstab case, the command looks like this:

cpu -fstab fstab v

The fstab in this case would be

myfs /mnt virtiofs rw 0 0

Note that both the environment variable and the fstab can have more than one entry, but they entries must be separate by newlines. Hence, this will not work:

CPU_FSTAB=`cat fstab` cpu v

as shells insist on converting newlines to spaces.

The fstab can specify any file system. If there is a mount path to, e.g., Google drive, and it can be specified in fstab format, then cpu clients can use Google Drive files. Note, again, that these alternative mounts do not use the 9p server built in to the cpu client; they use the file systems provided on the cpu server machine.

There are thus several choices for setting up the mounts

• 9p support by the cpu client
• 9p supported by the cpu client, with additional mounts via -fstab or -namespace
• 9p without any bind mounts, i.e. -9p=false -namespace "", in which case, on the remote machine, files from the client are visible in /tmp/cpu, but no bind mounts are done; with additional mounts provided by fstab mounts are provided
• no 9p mounts at all, when -namespace="" -9p=false; with optional additional mounts via fstab
• if there are no 9p mounts, and no fstab mounts, cpu is equivalent to ssh.

Notes

DUT, a simple Device Under Test utility.

Points of contact: Ron Minnich

DUT is a simple Device Under Test program that gives you control of a node. It is intended to make very fast startup and control easy.

DUT is one program implementing three operations. The first, tester, is run on a test control system, such as your desktop; the second, called device, is run on the device; the third, called ssh and also run on the device, starts an ssh server assuming one is present.

DUT is intended to be very limited, with more sophisticated operations, should they be needed, being done over SSH.

DUT is found at github.com:linuxboot/dut.

This chapter describes how we build and use DUT.

Components

DUT is intended to be built into a u-root image. First one must fetch it:

go get github.com/linuxboot/dut
# ignore the warning message.

DUT source tree is structured such that a program called uinit is produced. This is convenient for u-root usage.

Building it into a u-root image is easy:

go run $(GOPATH)/src/github.com/u-root/u-root -build=bb minimal github.com/linuxboot/dut/uinit

I almost always add an sshd to u-root; it's just too handy. U-root sshd does not support passwords, so you have to supply the public key:

go run $(GOPATH)/src/github.com/u-root/u-root -build=bb -files key.pub minimal  github.com/linuxboot/dut/uinit   github.com/u-root/u-root/xcmds/sshd

DUT on the device

On boot, the standard init program will find dut, and run it. The standard mode on a device is device mode, and dut will bring up the ethernet, currently using 192.168.0.2, and assuming the tester is 192.168.0.1 (this should be fixed ...). It will then attempt to connect to a uinit running in 'tester' mode on 192.168.0.1. Once connected, it functions as a server and waits for requests.

DUT on the controller

Running on the controller is easy:

uinit -m tester

On the controller, the program waits for a connection and then starts issuing commands to the device. The controller has the option of calling the following RPC functions:

RPCWelcome - return a welcome message
RPCExit - exit the testing mode
RPCReboot - reboot the system
RPCKexec - kexec a kernel
RPCSsh - start the sshd

Each of these RPCs takes arguments and returns a result, with Welcome being the most fun:

______________
< welcome to DUT >
  --------------
         \   ^__^
          \  (oo)\_______
             (__)\       )\/\
                 ||----w |
                 ||     ||

The current tester mode performs an RPC sequence I use for DXE cleaning, namely, a Welcome, followed by a Reboot, followed by a Welcome. This sequence verifies that I can get network going from power on, do a reboot, and reconnect after a reboot. It's been good for finding out if a particular DXE can be removed.

Once the second Welcome has happened, if an sshd is installed, it will have been started, and you can do additional commands.

Future work

Obviously, much more can be done. But this is a useful foundation on which to build DUT environments.

Implementing LinuxBoot

The aim of LinuxBoot is to reduce complexity and obscure firmware by moving that functionality into kernel and userspace.

This chapter describes the procedures from a LinuxBoot workshop where an Atomic Pi board with UEFI firmware was converted to run LinuxBoot. The build materials associated with this are found at digitalloggers/atomicpi.

Read the below and consult the Makefile for the details of how it was implemented.

A quick refresher on UEFI

UEFI has three sections:

• SEC ("Boot")
• PEI ("Very early chip setup and DRAM programming")
• DXE ("DRAM code")

DXE process is very complex; some systems have 750 DXEs.

LinuxBoot replaces most of the UEFI software with Linux. LinuxBoot has an initramfs provided by u-root.

The above are stored inside a flash filesystem (FFS) inside a region of flash on your motherboard (the BIOS region). Another important region of flash is the ME region.

The Management Engine (ME) is an x86 CPU embedded in the Intel Platform Controller Hub (PCH). It runs the Minix operating system which boots first and enables hardware such as clocks and GPIOs. ME checks the contents of flash memory and is used to implement "BootGuard". If you reflash and the ME is in "BootGuard" mode, your machine will be unusable. You need to run a tool called me_cleaner on the image to disable BootGuard.

How do you get LinuxBoot on your hardware

Start with a board running standard UEFI and proceed from "zero changes to FLASH" to "max changes" in 4 steps:

• Boot from USB stick via UEFI shell command or netboot (zero changes)
• Find a way to read flash and write flash
• Understand the flash layout
• Prepare linux kernel and initrd/initramfs payload.
• Replace UEFI Shell code section with Linux kernel and associated initrd (change part of one thing)
• Remove as many DXEs as possible (change by removal). This change:
  • Speeds boot
  • Reduces panic possibilities
  • Removes exploits
  • In production, it has solved problems
• Clear ME region for initrd storage
• Replace some DXEs with open source components (change by replacement)

One of the challenges in the above is in finding (or reclaiming) enough space in flash to shoehorn your kernel and initrd into.

Tools of the trade

There are two tools you use when you modify the UEFI flash image: utk and me_cleaner.

The ME Cleaner tool:

/usr/bin/python2 me_cleaner.py -s imagefile.bin

me_cleaner sets the high assurance platform (HAP) bit. HAP provides a way to disable a feature on Intel chips that does not allow us to modify the UEFI image and install LinuxBoot. Setting the bit with me_cleaner disables the "feature". Note that this does not always work; check with the LinuxBoot community.

When you run me_cleaner:

~/projects/linuxboot/me_cleaner/me_cleaner.py -s /tmp/rom.bin

you should see output similar to the following:

By applying me_cleaner, it has been observed that almost 4M of flash ram can be reclaimed for use. That 4M is enough to store a reasonably full featured compressed initrd image.

The utk tool can:

• Remove DXEs
• Insert new DXEs
• Replace the binary code of a DXE with a kernel
• Reallocate space from the ME region to the BIOS region ("tighten")

LinuxBoot Implementation steps

Step 1: boot Linux via netboot / UEFI shell

• netboot: standard BIOS-based PXE boot
  • Netboot is probably the most common working boot method on UEFI
  • We have never seen a system that did not have a net boot
• UEFI Shell (mentioned only for completeness)
  • Install Linux on FAT-32 media with a name of your choice (e.g. "kernel")
    • FAT-32, also known as MS-DOS file system
  • Boot kernel at UEFI Shell prompt
  • We've run into a few systems that don't have a UEFI shell

Working with a system that only has a net interface

If the system only has a net interface, you use Dynamic Host Configuration Protocol (DHCP), using broadcast DISCOVER, and Trivial File Transfer Protocol (TFTP) to get the boot information you need.

Configuration information is provided by REPLY to a DHCP request. The REPLY returns an IP, server, and a configuration file name that provides:

• Identity
• What to boot
• Where to get it

Data is provided by TFTP. HTTP downloading takes a fraction of a second even for 16M kernels. With TFTP it's very slow and TFTP won't work with initramfs much large than 32MiB. Most LinuxBoot shops use or are transitioning to HTTP.

Note: Boot images require a kernel(bzImage) + an initramfs + a command line. They can be loaded as three pieces or compiled and loaded as one piece, as described in this section.

Step 2: read & write the flash

There are two main ways to read and write the flash - hardware and software.

Hardware: It is worth buying a Pomona 5250 SOIC Clip adapter to read directly by hardware to have something to roll back to if anything goes wrong. Avoid cheap SOIC clip adapters that don't allow you to use standard jumper leads. For a good example of using a Raspberry Pi 3/4 to read/write, see Sakaki's EFI Install Guide/Disabling the Intel Management Engine

Software: With a working boot image, use flashrom to read an image of your flash. To write you may need to disable flash protections (look for "ME Manufacturing mode" jumpers on your motherboard). Figure on generally using software methods for reading & writing flash, but with hardware to drop back to.

Step 3: Familiarise yourself with the flash layout and identify free space

Open your flash image with UEFITool, and locate the filesystem containing the DXE's (it will have the Shell or Shell_Full in it ). Check how much volume free space is in that filesystem - this will be an initial limit when you come to place your kernel and initramfs in it in step 5.

Step 4: Prepare linux/u-root payload

Start small and work your way up.

• Use the tiny.config to configure your first kernel, and embed a small initramfs in-kernel (the u-root cpu payload is an excellent starting point).
• One can have a full kernel/initramfs in around 2M of flash.
• A more full featured kernel might consume 2M and a u-root bb distribution 4M, which may well exceed the volume free space.
• When there isn't enough space in this filesystem, one can either start removing unused DXE's (step 6), or use space formerly used by the ME Region (step 7).

Step 5: replace Shell binary section

• UEFI Shell is a DXE
  • DXEs are Portable Executable 32-bit binaries (PE32)
  • They have multiple sections, one of them being binary code
  • You need a flash image (in this case called firmware.bin). You can get it via vendor website, flashrom, or other mechanism.
• The following utk command replaces the Shell code section with a Linux kernel:
  • utk firmware.bin replace_pe32 Shell bzImage save new.bin
  • Note: It's always a PE32, even for 64-bit kernels. new.bin is a filename of your choosing.
• After running utk, you can reflash

Step 6a: remove as many DXEs as possible

• You can do an initial mass removal based on your current knowledge
• utk automates removing DXEs: this is the DXE cleaner
  • utk removes a DXE, reflashes, checks if it boots, repeat This part should be easy: DXE can have a dependency section. In practice, it's hard: because dependency sections are full of errors and omissions. A lot of UEFI code does not check for failed DXE loads.

Step 6b: place your initramfs in me_cleaned region

• Run me_cleaner and then utk tighten on the source image, then inspect the image using UEFITool. If successful, there will now be padding at the beginning of the BIOS region of a substantial size.
• This padding space can be used, without the filesystem's knowledge, to stash an initramfs. The kernel is informed of the location this initramfs as an initrd kernel parameter.
  • Use the base address of this padding region to calculate the offset in the flash image where the initrd is stashed using dd.
  • Use the address (not base address) as the initramfs location in memory to pass as a kernel parameter.

Step 7: replace closed-source with open source

• If you can build a DXE from source, you can use utk to remove the proprietary one and replace it with one built from source. You can get DXE source from the tianocore/EDK2 source repo at github.com. The GitHub repo has a limited number of DXEs in source form; i.e., you can't build a full working image using it.
• There are scripts that let you compile individual DXEs, including the UEFI Shell and Boot Device Selection (BDS). These two DXEs have been compiled and are used in the Atomic Pi. Source-based BDS was needed to ensure the UEFI Shell was called.
• You only need the UEFI Shell built long enough to replace it with Linux.

Final step: reflash the image

• "Native" reflash: Boot the system whatever way is easiest: netboot, usb, local disk, and run flashrom -p internal -w _filename.bin_ where filename.bin is a filename of your choosing.
• Run flashrom with an external device such as an sf100. There may be a header on the board, or you might have to use a clip. flashrom -p dediprog:voltage=1.8 -w _filename.bin_

The voltage option is required for the Atomic Pi.

LinuxBoot using coreboot, u-root and systemboot

Points of contact: Andrea Barberio, David Hendricks

This chapter describes how to build a LinuxBoot firmware based on coreboot, u-root and systemboot. The examples will focus on x86_64, and the coreboot builds will cover virtual and physical OCP hardware.

Quick Start with coreboot

Run these commands in a directory you create or in /tmp; do so because it creates some files and directories:

$ go get github.com/linuxboot/corebootnerf
$ go run github.com/linuxboot/corebootnerf --fetch
... lots and lots of output!

This produces a coreboot image in coreboot-4.9/build/coreboot.rom You can now run this rom image:

$  qemu-system-x86_64 -serial stdio -bios coreboot-4.9/build/coreboot.rom

And see how it looks when you put this in a coreboot ROM image.

Components

The final image is built on top of multiple open-source components:

• coreboot, used for the platform initialization. Silicon and DRAM initialization are done here.
• Linux, used to initialize peripherals and various device drivers like file systems, storage and network devices; network stack; a multiuser and multitasking environment.
• u-root, an user-space environment that provides basic libraries and utilities to work in a Linux environment.
• systemboot, an additional set of libraries and tools on top of u-root, that provide a bootloader behaviour for various booting scenarios. systemboot was merged into u-root.

These components are built in reverse order. u-root and systemboot are built together in a single step.

Building u-root

The first step is building the initramfs. This is done using the u-root ramfs builder, with additional tools and libraries from systemboot.

u-root is written in Go. We recommend using a relatively recent version of the Go toolchain. At the time of writing the latest is 1.11, and we recommend using at least version 1.10. Previous versions may not be fully supported.

Adjust your PATH to include ${GOPATH}/bin, in order to find the u-root command that we will use in the next steps.

Then, fetch u-root and its dependencies:

go get -u github.com/u-root/u-root

Then build the ramfs in busybox mode, and add fbnetboot, localboot, and a custom uinit to wrap everything together:

u-root -build=bb core github.com/u-root/u-root/cmds/boot/{uinit,localboot,fbnetboot}

This command will generate a ramfs named /tmp/initramfs_${os}_${arch}.cpio, e.g. /tmp/initramfs.linux_amd64.cpio. You can specify an alternative output path with -o. Run u-root -h for additional command line parameters.

Note: the above command will include only pure-Go commands from u-root. If you need to include other files or non-Go binaries, use the -file option in u-root. For example, you may want to include static builds of kexec or flashrom, that we build on https://github.com/systemboot/binaries .

Then, the initramfs has to be compressed. This step is necessary to embed the initramfs in the kernel as explained below, in order to maintain the image size smaller. Linux has a limited XZ compressor, so the compression requires specific options:

xz --check=crc32 --lzma2=dict=512KiB /tmp/initramfs.linux_amd64.cpio

which will produce the file /tmp/initramfs.linux_amd64.cpio.xz.

The kernel compression requirements are documented under Documentation/xz.txt (last checked 2018-12-03) in the kernel docs.

Building a suitable Linux kernel

A sample config to use with Qemu can be downloaded here: linux-4.19.6-linuxboot.config.

You need a relatively recent kernel. Ideally a kernel 4.16, to have support for VPD variables, but a 4.11 can do the job too, if you don't care about boot entries and want "brute-force" booting only.

We will build a kernel with the following properties:

• small enough to fit most flash chips, and with some fundamental kernel features
• that can run Go programs (mainly futex and epoll support)
• with the relevant storage and network stack and drivers
• with kexec support, so it can boot a new kernel
• with kexec signature verification disabled (optional)
• with devtmpfs enabled, since we don't use udev
• XZ support to decompress the embedded initramfs
• VPD (Vital Product Data) (optional)
• TPM support (optional)
• embed the u-root initramfs
• and last but not least, "linuxboot" as default host name :)

Download kernel sources

You can either download a tarball from kernel.org, or get it via git and use a version tag. We recommend at least a kernel 4.16, in order to have VPD variables support.

# download the kernel tarball. Replace 4.19.6` with whatever kernel version you want
wget https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.19.6.tar.xz
tar xvJf linux-4.19.6.tar.xz
cd linux-4.19.6
make tinyconfig

You can also check out the linux-stable branch, that will point to the latest stable commit. You need to download it via git as follows:

git clone --depth 1 -b linux-stable
git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git
cd linux-stable
make tinyconfig

Some more information about tiny configs can be found at https://tiny.wiki.kernel.org (last checked 2018-12-01).

A few fundamental features

Assuming we are running on x86_64, some basic features to enable are:

• 64-bit kernel
• General setup → Configure standard kernel features → Enable support for printk
• General setup → Configure standard kernel features → Multiple users, groups and capabilities support (this is not strictly required on LinuxBoot)
• Processor type and features → Built-in kernel command line (customize your command line here if needed, e.g. earlyprintk=serial,ttyS0,57600 console=ttyS0,57600)
• Executable file formats / Emulations → Kernel support for ELF binaries (you may want to enable more formats)
• Networking support → Networking options → TCP/IP networking
• Networking support → Networking options → The IPv6 protocol
• Device Drivers → Character devices → Enable TTY
• Device Drivers → Character devices → Serial drivers → 8250/16550 and compatible serial support
• Device Drivers → Character devices → Serial drivers → Console on 8250/16550 and compatible serial port
• File systems → Pseudo filesystems → /proc file system support
• File systems → Pseudo filesystems → sysfs file system support

Requirements for Go 1.11

Go requires a few kernel features to work properly. At the time of writing, you need to enable CONFIG_FUTEX in your kernel config. Older versions of Go may require CONFIG_EPOLL.

In menuconfig:

• General setup → Configure standard kernel features (expert users) → Enable futex support
• General setup → Configure standard kernel features (expert users) → Enable eventpoll support

Additional information about Go's minimum requirements can be found at https://github.com/golang/go/wiki/MinimumRequirements (last checked 2018-12-01).

Enable devtmpfs

Our system firmware uses u-root, which does not have (intentionally) an udev equivalent. Therefore, to have /dev/ automatically populated at boot time you should enable devtmps.

Simply enable CONFIG_DEVTMPFS and CONFIG_DEVTMPFS_MOUNT in your kernel config

In menuconfig:

• Device drivers → Generic Driver Options → Maintain a devtmpfs filesystem to mount at /dev
• Device drivers → Generic Driver Options → Automount devtmpfs at /dev, after the kernel mounted the rootfs

Additional drivers

This really depends on your hardware. You may want to add all the relevant drivers for the platforms you plan to run LinuxBoot on. For example you may need to include NIC drivers, file system drivers, and any other device that you need at boot time.

For example, enable SCSI disk, SATA drivers, EXT4, and e1000 NIC driver. In menuconfig:

• Bus options → PCI support
• Enable the block layer
• Device drivers → Block devices (required for SCSI and SATA)
• Device drivers → SCSI device support → SCSI disk support
• Device drivers → Serial ATA and Parallel ATA drivers
• File systems → The Extended 4 (ext4) filesystem
• Networking support (required for e1000)
• Device drivers → Network device support → Ethernet driver support → Intel(R) PRO/1000 Gigabit Ethernet support

Enable XZ kernel and initramfs compression support

The u-root-based RAMFS will be compressed with XZ and embedded in the kernel. Hence you need to enable XZ compression support. Make sure to have at least CONFIG_HAVE_KERNEL_XZ, CONFIG_KERNEL_XZ, CONFIG_DECOMPRESS_XZ.

In menuconfig:

• General setup → Kernel compression mode → XZ
• General setup → Initial RAM filesystem and RAM disk (initramfs/initrd) support → Support initial ramdisk/ramfs compressed using XZ

Enable VPD

VPD stands for Vital Product Data. We use VPD to store boot configuration for localboot and fbnetboot, similarly to UEFI's boot variables. Linux supports VPD out of the box, but you need at least a kernel 4.16.

Make sure to have CONFIG_GOOGLE_VPD enabled in your kernel config.

In menuconfig:

• Firmware drivers → Google Firmware Drivers → Coreboot Table Access - ACPI → Vital Product Data

TPM support

This also depends on your needs. If you plan to use TPM, and this is supported by your platform, make sure to enable CONFIG_TCG_TPM.

In menuconfig:

• Device drivers → Character devices → TPM Hardware Support → (enable the relevant drivers)

Include the initramfs

As mentioned above, the kernel will embed the compressed initramfs image. Your kernel configuration should point to the appropriate file using the CONFIG_INITRAMFS_SOURCE directive. E.g.

CONFIG_INITRAMFS_SOURCE="/path/to/initramfs_linux.x86_64.cpio.xz"`

In menuconfig:

• General setup → Initial RAM filesystem and RAM disk (initramfs/initrd) support → Initramfs source file(s)

Default hostname

We use "linuxboot" as default hostname. You may want to adjust it to a different value, You need to set CONFIG_DEFAULT_HOSTNAME for the purpose, e.g.

CONFIG_DEFAULT_HOSTNAME="linuxboot"

In menuconfig:

• General setup → Default hostname

Build the kernel

Once your configuration is ready, build the kernel as usual:

make -j$(nproc --ignore=1)

The image will be located under arch/${ARCH}/boot/bzImage if your architecture supports bzImage (e.g. x86).

For more details on how to build a kernel, see https://kernelnewbies.org/KernelBuild (last checked 2018-12-01).

Building coreboot

In this step we will build coreboot using the Linux kernel image that we built at the previous step as payload. This build is for a Qemu x86 target, the process may be somehow different for other platforms.

Steps overview:

• download coreboot from the git repo
• build the compiler toolchain
• configure coreboot for Qemu, and to use our bzImage as payload
• build coreboot.rom

Download coreboot

Our preferred method is to download coreboot from the git repository:

git clone https://review.coreboot.org/coreboot.git
cd coreboot

Build the compiler toolchain

This step is required to have, among other things, reproducible builds, and a compiler toolchain that is known to work with coreboot.

make crossgcc-i386 CPUS=$(nproc) BUILD_LANGUAGES=c

The step above may ask you to install a few additional libraries or headers, do so as requested, with the exception of gcc-gnat, that we won't need.

Configure coreboot for Qemu and our payload

Run make menuconfig to enter the coreboot configuration menus. Then:

Specify the platform we will run on:

• Mainboard → Mainboard vendor → Emulation
• Mainboard → Mainboard Model → QEMU x86 q35/ich9 (aka qemu -M q35, since v1.4)

Specify a large enough flash chip and CBFS size:

• Mainboard → ROM chip size → 16 MB
• Mainboard → Size of CBFS filesystem in ROM → 0x1000000

Specify our payload:

• Payload → Add a payload → A Linux payload
• Payload → Linux path and filename → path to your bzImage

Then save your configuration and exit menuconfig.

Build coreboot

This is done with a simple

make -j$(nproc)

The coreboot build system will clone the relevant submodules, if it was not done already, and will build a coreboot ROM file that will contain the initialization code, and our bzImage payload. The output file is at build/coreboot.rom.

If everything works correctly you will get an output similar to the following:

This image contains the following sections that can be manipulated with this tool:

'COREBOOT' (CBFS, size 16776704, offset 512)

It is possible to perform either the write action or the CBFS add/remove actions on every section listed above.
To see the image's read-only sections as well, rerun with the -w option.
    CBFSPRINT  coreboot.rom

FMAP REGION: COREBOOT
Name                           Offset     Type           Size   Comp
cbfs master header             0x0        cbfs header        32 none
fallback/romstage              0x80       stage           15300 none
fallback/ramstage              0x3cc0     stage           51805 none
config                         0x10780    raw               155 none
revision                       0x10880    raw               576 none
cmos_layout.bin                0x10b00    cmos_layout       548 none
fallback/dsdt.aml              0x10d80    raw              6952 none
fallback/payload               0x12900    simple elf    5883908 none
(empty)                        0x5af140   null         10800216 none
bootblock                      0xffbdc0   bootblock       16384 none

Built emulation/qemu-q35 (QEMU x86 q35/ich9)

Putting everything together

TODO

Defining boot entries

TODO

Running on a virtual machine

The image built with the above steps can run on a QEMU virtual machine, using the machine type q35, as specified in the coreboot mainboard section. Assuming that your coreboot image is located at build/coreboot.rom, you can run the following command:

sudo qemu-system-x86_64\        # sudo is required to enable KVM below
    -M q35 \                    # the machine type specified in the coreboot mainboard configuration
    -enable-kvm \               # use KVM to avail of hardware virtualization extensions
    -bios build/coreboot.rom \  # the coreboot ROM to run as system firmware
    -m 1024 \                   # the amount of RAM in MB
    -object rng-random,filename=/dev/urandom,id=rng0 \
                                # RNG to avoid DHCP lockups when waiting for entropy
    -nographic                  # redirect all the output to the console

If everything has been done correctly you should see, in order, the output from coreboot, linux, u-root, and systemboot. You can press ctrl-c when Systemboot instructs you to do so, to enter the u-root shell.

Running on real OCP hardware

TODO

Glossary

• BIOS: Originally, BIOS was the software built into computers to send simple instructions to the hardware, allowing input and output before the operating system was loaded. It was a binary blob with no standardized structure that was responsible for initializing CPU and memory, and jumping to a hard-coded position on the master block of the first disk drive. BIOS has been largely replaced by UEFI. Many UEFI implementations still offer a "BIOS compatibility mode" which makes it behave like an old BIOS, with its features.
• busybox: Busybox is a single userspace binary which includes versions of a large number of system commands, including a shell. This package can be very useful for recovering from certain types of system failures, particularly those involving broken shared libraries. There are multiple implementations of busybox, such as git.busybox.net/busybox and github.com/u-root/u-root.
• coreboot: A project to develop open source boot firmware for various architectures. Its design philosophy is to do the bare minimum necessary to ensure that hardware is usable and then pass control to a different program called the payload. The payload can then provide user interfaces, file system drivers, various policies etc. to load the OS.
• DHCP: A networking protocol that runs on a DHCP server and that automatically assigns an IP address from a pre-configured pool to any machine that queries it on boot up.
• EDK II: An open source reference implementation of an UEFI-compliant firmware, originally developed by Intel
• firmware: A specific class of computer software that provides low-level control for a device's specific hardware. It is installed at the time of manufacturing and is the first program that runs when a computer is turned on. It checks to see what hardware components the computing device has, wakes the components up, and hands them over to the operating system that is to be installed on the machine. The current x86 firmware is based on Intel’s Universal Extensible Firmware Interface (UEFI).
• Heads_: An open source firmware for laptops and servers, aimed at strong platform security. Developed by Trammell Hudson, based on stripped UEFI plus Linux, and BusyBox instead of u-root.
• iSCSI_: A protocol that provides a way to make network-attached storage appear to be a local device to the hosts using it, allowing it to be (among other things) mounted as a regular local file system.
• kexec: A system call that enables you to load and boot into another kernel from the currently running kernel. kexec performs the function of the boot loader from within the kernel.
• LinuxBIOS: A project originated in 1999 from Ron Minnich, Stefan Reinauer and others. It was an experiment in the idea of running Linux as firmware. At that time Linux was not mature enough for a hardware initialization project, and while LinuxBIOS was successful in several performance-and-reliability critical environments, it didn't see mass adoption. It later became coreboot.
• LinuxBoot: LinuxBoot is not a product, but rather a concept. It's the idea of booting Linux (OS) with Linux (system firmware). In a way, the same concept pioneered by LinuxBIOS. It is like a Linux distribution, but for firmware. It is a collection of various open source components, combined to work as a consistent firmware OS.
• NERF: The original name for the LinuxBoot project composed of stripped UEFI plus Linux plus u-root. The name stands for Non-Extensible Reduced Firmware, as opposed to UEFI's Unified Extensible Firmware Interface. NERF is an UEFI replacement that is more compact and less extensible. While extensibility is nice and often desirable, too much extensibility can make a complex project very hard to maintain and keep secure.
• Open Source Firmware: OSF can be used to refer to Open Source Firmware or Open System Firmware depending on the context.
• Open System Firmware (OSF): An official subproject of the Open Compute Project (OCP). OSF has been developed in the open, by various members of OCP that were interested in having open source system firmware. OSF defines a set of guidelines with contributions from Microsoft, Google, Facebook, Intel, 9elements, TwoSigma, and several other companies.
• OVMF: Open Virtual Machine Firmware. Open Virtual Machine Firmware is a build of EDK II for virtual machines. It includes full support for UEFI, including Secure Boot, allowing use of UEFI in place of a traditional BIOS in your EFI Initialization (PEI)|UEFI stage which runs before RAM is initialized, from cache and ROM. PEI is mostly C-code running in 32-bit protected flat mode. The main goal of the PEI stage is to detect RAM. As soon as RAM is detected and configured, PEI stage give control to the DXE through DXE Initial Program Load (IPL) driver
• production kernel: LinuxBoot is not intended to be a runtime production kernel; rather, it is meant to replace specific UEFI functionality using Linux kernel capabilities and then boot the actual production kernel (prodkernel) on the machine. Kernel configuration files specific to LinuxBoot provide the needed Linux kernel capabilities without bloating the size of the BIOS with unnecessary drivers.
• QEMU: An emulator that performs hardware virtualization. QEMU is a hosted virtual machine monitor.
• Secure Boot Preverifier (SEC): In UEFI, the SEC stage initializes the CPU cache-as-RAM (CAR) and gives control to the PEI dispatcher. It is 99.9% assembly code (32-bit protected mode).
• u-boot: A very popular open source firmware and bootloader. Not to be confused with u-root.
• u-root: A modern, embedded userspace environment for Linux, with bootloader tools. See the section on u-root.
• UEFI: Unified Extensible Firmware Interface. It is Intel’s specification of a standard for system firmware. UEFI defines everything from the layout on the flash chip, to how to interface to peripherals, enables boot from disk or from a network, defines how UEFI applications work, etc). It is not an implementation, it's a standard. EDK II and OpenEDK II are UEFI implementations. UEFI is not closed source per-se, but in practice most implementations are.
• userland/userspace: A set of programs and libraries that are used to interact with the kernel.

coreboot, LinuxBoot, NERF... What?

Naming is hard. You may have heard of coreboot, NERF, u-root, systemboot, etc. If you are confused, well, you're not alone.

LinuxBIOS

LinuxBIOS is a project originated in 1999 from Ron Minnich, Stefan Reinauer and others. It is not much younger than UEFI, but they were already experimenting the idea of running Linux as firmware! Like many great ideas, it was way ahead of its time. At that time Linux was not mature enough for hardware initialization project, and while LinuxBIOS was successful in several performance-and-reliability critical environments, it didn't see mass adoption.

coreboot

LinuxBIOS became coreboot in 2008. It is effectively the same project that evolved over time. coreboot (spelled lowercase) is a complete open source system firmware package, aimed at replacing proprietary implementations. It's also one of the most mature and well-maintained open source firmware projects.

coreboot supports a wide varieties of platforms, and has a modular architecture. It provides platform initialization (CPU, DRAM, PCI, ACPI, SMBIOS, etc), a filesystem (CBFS) suitable for on-firmware storage, integration with vendor extensions and blobs, a friendly license, and a wide variety of payloads.

This modular design also enables various bootloader scenarios, through coreboot payloads. Among the various options, there are:

• SeaBIOS, a very popular open source BIOS implementation
• depthcharge, the blazing-fast boot payload for ChromeOS/Chromebooks
• LinuxBoot, of course! More details below
• UEFI, via the open-source EDK II.

LinuxBoot

LinuxBoot is not a product, but rather a concept. It's the idea of booting Linux (OS) with Linux (system firmware). In a way, the same concept pioneered by LinuxBIOS.

"LinuxBoot" is also often used as an umbrella name at Facebook to indicate how we do open source firmware, i.e. coreboot + Linux + u-root + systemboot. Imagine it like a Linux distribution, but for firmware. It is a collection of various open source components, glued together to work as a consistent firmware OS.

Depending on who you are talking to, you may hear "LinuxBoot" used as a reference to "stripped UEFI, plus Linux". This is because LinuxBoot, when originally created, was meant to run a Linux kernel on top of a stripped UEFI firmware, not on coreboot. See also "NERF" below.

NERF

This is the original name for the stripped UEFI, plus Linux, plus u-root. The name stands for Non-Extensible Reduced Firmware, as opposed to UEFI's Unified Extensible Firmware Interface. Basically, saying that NERF is an UEFI replacement that prefers to be more compact, less extensible, and a bit more opinionated. While extensibility is nice and often desirable, too much extensibility and too many "yes" can make a complex project very hard to maintain and keep secure.

NERF started from Ron Minnich (one of the coreboot founders) at Google, and is now developed by a few other folks that are now part of Google's "NERF team". This name, in my understanding, is mostly used within Google, while "LinuxBoot" is becoming a more common name for this effort.

Heads

Heads is an open source firmware for laptops and servers, aimed at strong platform security. Developed by Trammell Hudson, this is based on stripped UEFI plus Linux, and BusyBox instead of u-root. More info at https://trmm.net/Heads .

Open System Firmware

Open System Firmware, or in short OSF, is an official subproject of the Open Compute Project (OCP). OSF has been developed in the open, by various members of OCP that were interested in having open source system firmware. OSF defines a set of guidelines with contributions from Microsoft, Google, Facebook, Intel, 9elements, TwoSigma, and several other companies.

The important thing to keep in mind is that Open System Firmware is a project name, not an implementation, nor an idea. An implementation (like LinuxBoot or OpenEDK2) can be OSF-compliant if it follows the aforementioned guidelines.

Currently, Open System Firmware has two work streams:

• LinuxBoot, led by Google, Facebook, 9elements, ITRenew, TwoSigma, and others
• OpenEDK II, led by Microsoft and Intel.

Open Source Firmware

While this may sound obvious, it's worth noting that OSF can be used to refer to "Open Source Firmware" or "Open System Firmware" depending on the context. Confusing? I couldn't agree more.

BIOS

BIOS is the good old, imperscrutable, unstructured, non-standard way of initializing a hardware platform in the pre-UEFI days. In other words it's a binary blob with no standardized structure, that is responsible for initializing CPU and memory, and jumping to a hard-coded position on the MBR of the first disk drive.

BIOS has been largely replaced by the (much better) UEFI over the past 20 years. Many UEFI implementations still offer a "BIOS compatibility mode" which make it behave like an old BIOS, with its (lack of) features.

BIOS is also a misused term for system firmware nowadays. You may still hear "BIOS" in reference to system firmware, either it's UEFI or even LinuxBoot. However, "BIOS" refers to a specific type of firmware, and UEFI is definitely not BIOS, just like LinuxBoot is not BIOS.

UEFI

It's a complex specification of a standard for system firmware. It defines everything from the layout on the flash chip, to how to interface to peripherals, boot from disk or from network, how UEFI applications work, etc). It is not an implementation, it's a standard. EDK II and OpenEDK II are UEFI implementations.

UEFI is not closed source per-se, but in practice most implementations are. Typically IBVs and ODMs would take a snapshot of the reference implementation EDK II, and base their work on that, with their patches and additional components.

EDK II

It is the open source reference implementation of an UEFI-compliant firmware, originally developed by Intel. See https://github.com/tianocore/edk2 .

u-boot

u-boot is another very popular open source firmware and bootloader.

u-root

u-root is not u-boot! They are two completely different projects.

u-root is modern, embedded userspace environment for Linux, with bootloader tools. u-root has several advantages:

• it is written in a memory-safe language, Go, and it's compiled to avoid native code (CGo is disabled). No segfaults at firmware space!
• it has a bb (busybox) mode: it's more space-efficient than compiling the programs individually, because it automatically merges the source code of the programs and libraries you need, into one.
• it has a source mode: you have the source code of your tools in the firmware image, so you can modify any tool in place and re-run it, instead of rebuilding the whole image and reflashing, rebooting, and re-running the tool
• it can run any unmodified Go program, it doesn't have to be written specifically for u-root (while with BusyBox it needs to be part of that source tree, using different libraries). This allows using any Go library or program written for non-embedded environments
• is blazing fast to build: seconds instead of minutes

Case Studies

This chapter contains case studies for various solutions.

Table of Contents

1. Google study
2. OCP TiogaPass

The LinuxBoot project at Google

Google runs workloads across a number of clusters each with up to tens of thousands of machines. Firmware runs on these machines when they first start up. Google is pushing the state-of-the-art in many places including firmware. The discussion here about Google's implementation of LinuxBoot is limited to replacing specific UEFI firmware functionality with a Linux kernel and runtime. Over the years this project has grown to include various initiatives with the overarching goal of moving from obscure, complex firmware to simpler, open source firmware.

Team

There have been a number of contributors to the Google LinuxBoot project including:

• Ron Minnich (technical lead)
• Gan-shun Lim
• Ryan O'Leary
• Prachi Laud
• Chris Koch
• Xuan Chen
• Andrew Sun

Ryan O'Leary is one of the Open Compute Platform Foundation Open System Firmware project volunteer leads and Ron Minnich is the Open Compute Platform Foundation Incubation Committee Representative.

Goal

The primary goal of Google's LinuxBoot is to modernize the firmware by simplifying it to technologies engineers understand and trust. In UEFI systems, LinuxBoot consists of a "full stack" solution of stripped-down UEFI firmware, a Linux kernel, and an initramfs with tools written in Go. Although these components all make up one bundle stored in ROM, there are three parts: the closed-source EFI firmware, a Linux kernel, and u-root. The Linux kernel is an unmodified kernel. The user-space initramfs image with Go tools for system booting is available as u-root. Due to this modularity, LinuxBoot can be used with a variety of systems. In many cases, for example, the same kernel and initramfs have been used, without recompilation, on both AMD and Intel x86 boards. The UEFI on these boards is always specific to the board, however.

Converting a UEFI firmware image to use LinuxBoot

The conversion to LinuxBoot starts with generic UEFI. A UEFI computer boots in four main phases. The security phase (SEC) and the Pre-EFI Initialization Stage (PEI) are responsible for low-level operations to prepare the hardware and are usually specific to the hardware they are implemented for. After these two stages, the Driver Execution Environment (DXE) loads various drivers, and then the Boot Device Select (BDS) phase begins.

It is not possible to modify the SEC and PEI stages, as their components are tightly coupled to the chips on the board; even small changes to the chips require new SEC and PEI stages. LinuxBoot starts during the DXE stage, resulting in most of the drivers (and their associated attack surface) not being loaded. Instead, a Linux kernel is loaded as if it were a driver. By loading during the DXE, LinuxBoot runs after the first two stages of the UEFI, but takes over after that point, replacing the UEFI drivers. It therefore completely replaces a large portion of the boot process.

Phases of the project

Google's LinuxBoot project is focused on moving UEFI boot functionality into the kernel and userspace. That is, converting UEFI firmware to run LinuxBoot. The project has taken the standard UEFI boot process and converted it to LinuxBoot for production environments. The steps to reach this goal are described below.

Step 1. Reduce or replace UEFI components

UEFI contains proprietary, closed-source, vendor-supplied firmware drivers and firmware. LinuxBoot replaces many Driver Execution Environment (DXE) modules used by UEFI and other firmware, particularly the network stack and file system modules, with Linux applications.

The following diagram shows the phases of the UEFI boot process. The items in red are components that are either reduced or eliminated with LinuxBoot. The dark blue items on the left cannot be changed.

In the real FLASH part, the SEC and PEI are actually only 10% of total, so we reduce the size of those boxes in this and following diagrams.

Another part of the conversion process was to modify the UEFI boot process to boot a LinuxBoot image as shown below.

Step 2. Delete or replace as many proprietary DXEs as required to make step 3 work. In most cases, none need to be removed.

Step 3. Replace the UEFI shell with a Linux kernel + u-root

When Linux boots it needs a root file system with utilities. LinuxBoot provides a file system based on u-root standard utilities written in Go.

Step 4. Through trial and error, continue to remove DXEs until you can't remove anymore.

The DXEs are delivered as binary blobs. There are three ways to handle them:

1. The most desirable is to remove them and let Linux drivers take over what they did. This works well for USB, network, disk, and other drivers, as well as network protocols and file systems. In fact we have resolved many system reliability and performance issues just by removing DXEs!

2. The second way is to replace the DXE with an open source driver. This is less desirable, as the DXE environment is not as hardened as the Linux kernel environment.

3. The final, least desired option, is to continue to use the DXE. This is required if the DXE contains proprietary code that "tweaks" chipset settings, for example, memory timing or other controls, and there is no chance of ever bringing them to open source.

Step 5. Replace closed source DXEs with open source

If we can build a DXE from source, we can use utk to:

• Remove the proprietary one
• Replace it with one built from source

Step 6. Next steps: complete LinuxBoot

LinuxBoot is currently in production, but the LinuxBoot project development continues to provide an open-source solution that does the following:

1. Brings up the Linux kernel as a DXE in flash ROM instead of the UEFI shell.
2. Provides a Go based userland that can then bring up the kernel that you want to run on the machine.
3. Enables writing traditional firmware applications such as bootloader, debugging, diagnosis, and error detection applications as cross-architecture and cross-platform portable Linux applications.

The complete LinuxBoot solution is shown in the following diagram.

OCP TiogaPass Case Study

Points of contact: Jonathan Zhang, Andrea Barberio, David Hendricks, Adi, Morgan Jang, Johnny Lin

This case study describes information for firmware development community to use OCP platform TiogaPass, made by Wiwynn Corporation.

It contains following sections:

• Quick Start
• Details
  • How to build
  • How to operate
  • Platform info
• Support
  • Hardware support
  • Community support
  • Professional support

Quick Start

• Order the hardware if you have not done so.
• Download or build the firmware binary. The current solution is to boot embedded Linux kernel and initramfs as UEFI payload. Please contact Wiwynn to get a UEFI binary after ordering.
• Flash the firmware.
  • Copy the downloaded firmware to OpenBMC.
  • From OpenBMC

  fw-util mb --update bios --force ./<firmware image name>

• Boot and enjoy.
  • From OpenBMC

  power-util mb reset
  sol-util mb

Details

How to build

Follow Build Details for details on how to get the source code, and how to build.

Boot flow of the current firmware solution is: Power on --> minimized UEFI --> Linuxboot --> target OS.

In near feature, the boot flow will be: power on --> Coreboot --> Linuxboot --> target OS.

Build Details

• Download the code from linuxboot github

  git clone https://github.com/linuxboot/linuxboot.git

• You need to apply Wiwiynn's linuxboot patch for now

cd linuxboot
wget -O TiogaPass.patch https://github.com/johnnylinwiwynn/linuxboot/commit/28ae8450b3b05c6e6b8c74e29d0974ccf711d5e6.patch
git am TiogaPass.patch

• Build the kernel bzImage (has embeded initramfs) for linuxboot, please reference Building u-root and Building a suitable Linux kernel for how to build the bzImage. You can always customize your Linux kernel configuration to suit your needs, please reference Wiwynn's kernel configuration file as a sample linux_config.
• Place the tioga.rom into linuxboot/boards/tioga which is provided from Wiwynn after ordering, and also put your bzImage to the root folder of linuxboot, and then make

  cp path/to/tioga.rom linuxboot/boards/tioga
  cp path/to/bzImage linuxboot
  cd linuxboot
  BOARD=tioga make

• You should see the built image at build/tioga/linuxboot.rom.

How to operate

Follow TBD section for details on:

• How to flash. The image can be flashed either out-of-band, or from Linuxboot u-root shell, or from targetOS shell.
• How to run Linuxboot u-root shell commands.

Platform info

The SKU contains TiogaPass board, a debug card, a VGA card, a power adapter. The details can be obtained from Here.

Platform design details (including the design spec and schematics) can be found from Here.

Support

Hardware support

Hardware support can be obtained from Wiwynn Corporation(http://www.wiwynn.com/english)

Community support

OCP Open System Firmware is where industry collaborates on how to move forward with OSF. The OCP OSF project has regular recorded meetings and a mailing list.

Linuxboot open source community is the community you can ask any technical questions. Linuxboot community has a slack channel, a IRC channel, a mailing list and regular meetings.

Professional support

Following companies provides professional support services:

** TBD **

F.A.Q.

This chapter contains the Frequently Asked Questions

Troubleshooting

Q: u-root dhcp client take ages, what is the problem ?

A: the problem is a lack of early entropy:

1. If your platform has a hwrng enable it with CONFIG_ARCH_RANDOM and trust it with CONFIG_RANDOM_TRUST_CPU.
2. If your platform don't have hwrng, add uroot.nohwrng to your kernel command line so u-root use a non-blocking random implementation.