Bering-uClibc 5.x - Developer Guide - Adding a Hardware Architecture Variant

From bering-uClibc
Jump to: navigation, search
Adding a Hardware Architecture Variant
Prev Bering-uClibc 5.x - Developer Guide Next


Introduction

A major enhancement added in Bering-uClibc 5.x is the ability to target non-x86 runtime platforms. In principle it is now possible to build Bering-uClibc 5.x for SPARC, MIPS or other CPU architectures. These notes provide guidance on what changes are required to add support for a brand new target architecture variant.

The addition of support for an ARM926 processor on an ARM® Versatile™ Platform Baseboard is used as an example. This particular example has been chosen because ARM CPUs are common on low-cost embedded hardware and because the Versatile board has excellent support from the QEMU qemu-system-arm emulator.

The first step is to understand exactly what hardware the target platform consists of. In particular:

  • What is the model number of the CPU?
    • The default CPU for the Versatile Platform Baseboard is the ARM926EJ-S
  • What is the architecture of the CPU?
    • The ARM926EJ-S implements the ARMv5TEJ instruction set.
    • The ARM9 CPU family is "bi-endian" but defaults to little-endian, so code should be compiled little-endian.
  • What are the characteristics of the supporting platform?
    • The ARM Versatile Platform Baseboard is recognized as a standard "machine" target for the Linux kernel.


Note: These notes are intended to provide guidelines rather than a fully prescriptive recipe to follow.

Warning: The cross-compilation build system is under active development and these notes reflect the situation at the time of writing. If recent changes have been made they may be out of step with the Bering-uClibc 5.x code in Git.


Linux Kernel CPU Architecture Selection

The standard Linux kernel source tree includes CPU architecture specific code for quite a number of CPU types. This code is in the "arch" directory within the kernel source tree and it is sensible to review the contents of this directory. If you have not already extracted the Kernel source run: 

./buildtool.pl source kernel
cd source/*/linux/linux-3.2/arch/

Each of the directory names under "arch" represents a fundamental "architecture" variant. The Bering-uClibc 5.x toolchain references this via the ARCH variable.

Note: There are a few "special cases", which include i386 and x86_64! Refer to the comments and code in source/linux/linux-3.2/Makefile (starting around line 174) for further details.

Since there is a sub-directory of "arch" called "arm" that is what we need to set the "ARCH" variable to when building a toolchain to target the Versatile Platform Board. Details of how and where to do that are provided below.

In addition to the fundamental CPU architecture setting the kernel recognizes a further level of "machine" specification. For example, under the umbrella architecture of i386 we have the "true" i386 and also i486, Pentium 4, Geode LX etc. and it is possible to select between those when compiling a kernel.

The exact details of what "machines" can be selected vary depending on the value of ARCH:

  • For i386 there are entries in the kernel .config file like the following: 
    CONFIG_M686=y
  • For arm the permissible options are governed by the names of files with names like arch/arm/mach-machinename (for example arch/arm/mach-versatile) and then there are entries in the kernel .config file like the following: 
    CONFIG_ARCH_VERSATILE=y

Since different users run different machines which demand incompatible settings of the kernel .config variables the option to build for multiple machine variants has been part of the Bering-uClibc toolchain since Bering-uClibc 4.x.

The Bering-uClibc 5.x toolchain uses the variable KARCHS to specify a space-separated list of "machines" to build for using a single toolchain.

Note: As will be seen by the later description of how these settings are processed there is nothing "magic" about the values in KARCHS. They are just unique string labels used to identify patch files for the kernel .config and these patch files can contain system-specific settings in addition to more generic CPU architecture settings. It is often more appropriate to choose a "system" name like alix rather than a "CPU" name like geode.

For the Versatile Platform Board the relevant machinename setting is versatile.

GCC and Binutils CPU Architecture Selection

The toolchain is responsible for building code for the target environment and it relies on the GCC (cross-)compiler to do most of the work.

The GNU toolset (most notably "configure") has a well-established way of identifying different target platforms by a hyphen-separated list of the key characteristics known as the "configuration name". This was initially the triplet cpu-manufacturer-kernel but is now more commonly the quadruplet cpu-manufacturer-kernel-os (though this is still often referred to as a "triplet"). For example, i486-unknown-linux-uclibc refers to:

  • an i486 CPU, installed in
  • an unknown hardware platform ("unknown" as in "we don't care whether a PC is made by HP, IBM, Dell etc."), running
  • the linux kernel, and
  • a uclibc C library-based operating system

The first field ("cpu") is of particular interested here. Having identified the Kernel CPU Architecture (ARCH) refer to the appropriate sub-page of the GCC "Hardware Models and Configurations" page in order to understand what options are available.

For example, on the ARM Options sub-page there is a definition of the permissible values for the -march command-line option to GCC and related tools. One of the permissible values is "armv5te" which is a close match for the ARMv5TEJ architecture which we know the ARM926EJ-S CPU uses.

This (the setting for -march) also forms the first entry in the hyphen-separated "configuration name" string. For Bering-uClibc 5.x the second and third entries in this string are always "unknown" and "linux" respectively. The fourth entry is somewhat more variable; often is it simply "uclibc" but for some platforms it is necessary to include extra information in this part if the "configuration name". In particular, for ARM platforms, it seems to be necessary to include the string "gnueabi" in order to specify use of the GNU EABI (in place of the default OABI). Since the Versatile Platform Board uses an ARM CPU and the EABI is desirable the fourth entry should be "uclibcgnueabi" making the full string "armv5te-unknown-linux-uclibcgnueabi".

The Bering-uClibc 5.x toolchain references this "configuration name" via the GNU_TARGET_NAME variable.

Since this "configuration name" captures all the characteristics of the target system which need to be hard-coded into the toolchain it is a good string to use to identify and differentiate multiple toolchains. The buildtool.pl, buildpacket.pl and buildimage.pl scripts therefore use this "configuration name" as their "toolchainname" and they set the environment variable $GNU_TARGET_NAME based on the specified (or default) toolchainname.

The setting for -march ensures that the generated code will run on all CPUs which are compatible with that CPU architecture. For example, code compiled for i486 will also run on all later x86-compatible processors. GCC and related tools make it possible to optimise code for a particular CPU while retaining compatibility with other CPUs by specifying the -mtune command-line option. The permissible values for this are specified on the same page as for -march above. For the Versatile Platform Board there is an exact match for the actual CPU: arm926ej-s so this needs to be specified as the value for -mtune.


High-Level Toolchain Configuration

Once the required values for the ARCH, KARCHS and GNU_TARGET_NAME variables and the settings for the -march and -mtune command-line options have been identified it is time to start configuring a toolchain to target those settings.

The default toolchain target for Bering-uClibc 5.x is i486-unknown-linux-uclibc and this is specified as the default by the following lines in conf/buildtool.conf: 

# default toolchain - override with "-t toolchain" argument to buildtool.pl
Toolchain=i486-unknown-linux-uclibc

As the comment says this can be overridden by specifying "-t toolchainname" to buildtool.pl (and "--toolchain ToolchainName" to buildpacket.pl). Alternatively the default value can be changed by editing conf/buildtool.conf.

At the time of writing (2012-04-01) the tools/buildall.sh and buildlwp.sh scripts only look at the default setting in conf/buildtool.conf. They do not accept command-line arguments to specify the toolchain.

Script buildimage.pl gets its setting of toolchainname from the relevant buildimage.cfg file since each image has to be generated with the corresponding toolchain and it doesn't make sense to override this with a command-line argument.

All of the build .pl scripts set environment variable $GNU_TARGET_NAME based on the specified (or default) toolchainname and $GNU_TARGET_NAME is used internally in other scripts and configuration files where toolchain-specific processing is required.

Most of the configuration is performed by creating a new toolchain makefile to be included by make/MasterInclude.mk via the following lines in that file: 

# Include per-toolchain Makefiles
include $(BT_BUILDROOT)/make/toolchain/*.mk

In other words, every .mk file in the $(BT_BUILDROOT)/make/toolchain/ directory is automatically included, and IF-THEN logic within those included files determines which settings are made active.

The default toolchain configuration is specified in file make/toolchain/i486-unknown-linux-uclibc.mk which looks something like this: 

#
# Included Makefilefile for i486-unknown-linux-uclibc toolchain
# Intended for generic x86 target
#
ifeq ($(GNU_TARGET_NAME),i486-unknown-linux-uclibc)
# Primary kernel architecture
export ARCH:=i386
# Space-separated list of kernel sub-archs to generate
export KARCHS:=i686 i486 geode
# Available kernel archs with pci-express support
export KARCHS_PCIE:=i686
# Arch-specific CFLAGS
export ARCH_CFLAGS:=-march=i486 -mtune=pentiumpro
# Name of kernel image
export KERN_IMAGE:=bzImage
# Name of OpenSSL target
export OPENSSL_TARGET:=linux-elf
...
<lines omitted>
...
endif

The "<lines omitted>" look like (for example): 

export ac_cv_sizeof_int=4

and are covered later in this document.

For the Versatile Platform Board we need to copy the default toolchain file to make/toolchain/armv5te-unknown-linux-uclibcgnueabi.mk and adjust the contents to read as follows: 

#
# Included Makefilefile for armv5te-unknown-linux-uclibcgnueabi toolchain
# Intended for ARM Versatile Platform Board target
#
ifeq ($(GNU_TARGET_NAME),armv5te-unknown-linux-uclibcgnueabi)
# Primary kernel architecture
export ARCH:=arm
# Space-separated list of kernel sub-archs to generate
export KARCHS:=versatile
# Arch-specific CFLAGS
export ARCH_CFLAGS:=-march=armv5te -mtune=arm926ej-s
# Name of kernel image
export KERN_IMAGE:=zImage
# Name of OpenSSL target
export OPENSSL_TARGET:=linux-armv4

endif

If other settings in addition to -march and -mtune are required they should be appended to ARCH_CFLAGS. For example, some ARM processors need the FPU type to be specified with e.g. -mfpu=vfp.

The variables KERN_IMAGE and OPENSSL_TARGET accommodate some Package-specific platform differences. Refer to repo/kernel/buildtool.mk and repo/openssl/buildtool.mk respectively for usage of these variables.

Kernel Configuration File

If you were to try to build the new toolchain at this point it would fail with an error message because the build scripts will not be able to locate a kernel .config patch file with the right name. (The kernel source must be processed before building the toolchain executables in order to extract the header files.)

There needs to be a file called repo/linux/Bering-KVER.config-KARCH.patch for each KARCH in KARCHS, and this file must contain "diff" output which converts the base repo/linux/Bering-KVER.config into a specific kernel .config file suitable for KARCH.

For the Versatile Platform Board KARCH = versatile so the full file name is repo/linux/Bering-KVER.config-versatile.patch. This name needs to be added to repo/linux/buildtool.cfg and the file must be created in the repo/linux/ directory.

Constructing a suitable and fully correct patch file is non-trivial and requires a good understanding of the kernel configuration options. One possible procedure is as follows:

  • Create (e.g. "touch") an empty patch file with the right name in the repo/ directory and specify this in repo/buildtool.cfg. With an empty patch file the starting configuration will be the same as the "base" configuration for Bering-uClibc 5.x
  • Run: 
    buildtool.pl -t armv5te-unknown-linux-uclibcgnueabi source linux
  • This will recognize that the .config file is not compatible with the specified ARCH and prompt for new values for the kernel configuration variables which must be changed while preserving those which are valid. In a separate shell run: 
    tail -f log/buildtoollog
    to see the prompts from make oldconfig but answer the prompts in the shell where buildtool.pl is running.
  • At this stage it is OK to accept the default values for all of the settings.

This procedure carries across the majority of the standard Bering-uClibc 5.x kernel configuration but does not take account of the requirements of the target hardware and it is generally necessary to adjust the configuration.

A good way to make minor adjustments to configuration settings is to go to directory source/$GNU_TARGET_NAME/linux/linux-$KARCH/ and to run: 

make ARCH=arm menuconfig

Consult e.g. source/armv5te-unknown-linux-uclibcgnueabi/linux/linux-3.2.13/arch/arm/configs/versatile_defconfig for the default settings recommended for the versatile target.

Once a good .config file has been generated the "patch" file must be (re-)created. Locate the generated .config file (should be source/$GNU_TARGET_NAME/linux/linux-$KARCH/.config) and generate the patch file with commands like the following: 

cp .config ../Bering-$KVER.config-$KARCH
cd ..
diff -c ../Bering-$KVER.config Bering-$KVER.config-$KARCH > Bering-$KVER.config-$KARCH.patch


uClibc Configuration File

Just like the kernel, uClibc has a .config file which needs to be tailored for the new toolchain. For uClibc the file needs to be called repo/toolchain/config.$GNU_TARGET_NAME and this is a "full" file rather than a "patch".

For the Versatile Platform Board the full file name is repo/toolchain/config.armv5te-unknown-linux-uclibcgnueabi. This name needs to be added to repo/toolchain/buildtool.cfg and the file must be created in the repo/toolchain/ directory.

As with the kernel .config it is non-trivial to create a file with the right contents. One possible procedure is as follows:

  • Copy the file for the default toolchain and edit it to reflect the correct ARCH and the correct value for CROSS_COMPILER_PREFIX.
  • Run: 
    buildtool.pl -t toolchainname build toolchain
  • This will recognize that some different options need to be selected and prompt for new values for the uClibc configuration variables which must be changed.
    • For some reason the uClibc "make oldconfig" doesn't behave the same way as the kernel "make oldconfig" and refuses to accept entries when the console input is redirected.
      • That was because "make oldconfig" specified "$(MAKEOPTS)" which runs a multi-threaded build. Now removed (no performance benefit from a multi-threaded build to this step).
    • Instead, go to the directory containing the "live" uClibc .config file and run: 
      make menuconfig
      on the build host.
  • Locate the generated .config file (should be source/toolchainname/toolchain/uClibc-0.9.3*/.config) and use that as the "real" file with a command like the following:
    cp .config ../config.GNU_TARGET_NAME

Like the kernel, uClibc has both "generic" (architecture) and "specifc" (CPU) configuration entries. The "specific" entry is something like: 

CONFIG_ARM926T=y

There are references to these sort of configuration variables in source/toolchainname/toolchain/uClibc-0.9.3*/Rules.mak - for example: 

CPU_CFLAGS-$(CONFIG_ARM926T)+=-mtune=arm9e -march=armv5te

Recognize those? The implication is that the generated uClibc library will run on any armv5te processor but is optimized for the arm9e, like the kernel.

Note: The above is correct for uClibc 0.9.32 but the "specific" architecture configuration variables have been removed in uClibc 0.9.33.


Toolchain Build

That should be it. Running: 

buildtool.pl -t toolchainname build toolchain

should create a toolchain based on the specified configuration settings.

In reality you will probably get build errors and will need to refine the contents of the kernel and uClibc .config files in order to get a successful toolchain build.

For the Versatile Platform Board, try: 

buildtool.pl -t armv5te-unknown-linux-uclibcgnueabi build toolchain

(This works for me as of today Davidmbrooke 21:00, 31 March 2012 (UTC))


The steps performed as part of the toolchain build are described below.

Source Processing

Within conf/sources.cfg the "toolchain" source Package is declared to be dependent on the "linux" source Package so the kernel source gets processed first. In order to build the linux "source" target:

  • The kernel source .tar.bz2 file is unpacked
  • The kernel source patches are applied
  • For each entry in KARCHS
    • The generic kernel .config file is patched with the specific KARCH patch to create a specific .config file
    • The "make oldconfig" command is run (with appropriate command-line arguments)
    • The "make headers_install" command is run (with appropriate command-line arguments)
    • The generated header files are copied to the toolchain/$GNU_TARGET_NAME/usr/include/ directory

Once the "linux" source Package has been processed the "toolchain" source Package processing can start.

  • The uClibc source is extracted and the Bering-uClibc 5.x uClibc source patches are applied.
  • The binutils source is extracted.
  • The GCC source is extracted.
  • The mod-utils source is extracted (required for depmod).

Build Processing

Once the "source" processing has completed the "build" processing can start. The sequence is as follows:

  • The uClibc .config file is processed as part of "make install_headers" for uClibc.
    • This adds uClibc header files to the ones installed for the kernel above.
  • The "stage 1" binutils files are compiled.
  • The "stage 1" GCC compiler is compiled.
  • The "stage 2" GCC compiler is compiled.
  • The uClibc library is compiled.
  • The "stage 2" binutils files are compiled.
  • The mod-utils files are compiled.
  • The results of the toolchain build are copied to the staging/$GNU_TARGET_NAME/ directory.

Hints and tips for debugging toolchain build failures

  • You can get more verbose diagnostics from the uClibc build by setting environment variable "V" to either 1 or 2. See source/toolchainname/uClibc-0.9.3*/Makefile.help for more details.


Bering-uClibc Build

Once the new toolchain has been built the procedure to build Bering-uClibc 5.x itself should be the same as for the default platform. Note that the most of the code is much better tested on x86 platforms than on other CPUs and there are likely to be more bugs and build failures.

Most of the Bering-uClibc 5.x code is platform-independent but there are some exceptions. The key ones are noted below.

Platform-Specific Components

Syslinux, Isolinux, Pxelinux

Syslinux, Isolinux and Pxelinux are so common that it is easy to forget they are x86 / PC specific. A possible alternative is U-Boot. However many embedded platforms have their own specific boot loader solutions.

Cross Compilation Challenges and Workarounds

Autoconf

One particular challenge with cross-compiling applications which use "configure" is that they try to compile and execute applications on the build host in order to infer things about the target host. Sometimes this works; sometimes it does not.

In cases where it does not work it is possible to "prime" configure's cache with the correct selections by setting environment variables to specify these. That is what all those "export ac_cv_*" lines are for. If you get errors when building applications you can try to establish which variable configure is looking for (check the configure script) and set it appropriately. Use the following guidelines when deciding where to set the variable:

  • IF the variable is specific to a single application (e.g. samba_cv_CC_NEGATIVE_ENUM_VALUES=yes which is only for the SAMBA application) AND the setting is the same for all toolchains THEN set it in the application's buildtool.mk
    • Review the usage of CONFDEFS in repo/samba/buildtool.mk as an example
  • IF the variable is not specific to a single application (e.g. if it relates to one library which is used by multiple applications) AND the setting is the same for all toolchains THEN set it in make/MasterInclude.mk
  • IF the variable is specific to a single toolchain (whether or not it relates to more than one application) THEN set it in the per-toolchain makefile (e.g. make/toolchain/armv5te-unknown-linux-uclibcgnueabi.mk)

Build Host strip

Some applications' Makefiles call the strip utility explicitly, and the build host's strip utility will not typically understand the target host's executable file format.

In some cases the Makefile permits an override by specifying the STRIP environment variable, so adding this to the "make install" line in buildtool.mk as "$(GNU_TARGET_NAME)-strip" fixes the problem.

Other applications' Makefiles use the install utility to copy files to the destination directory and sometimes these use the '-s' command-line option which specifies that binaries should be stripped as part of the install step. By default this uses the build host's strip executable, which assumes the build host's CPU architecture.

It is possible to specify a different strip executable by adding --strip-program=PROGRAM to the install call within the application's Makefile.


Testing with QEMU

(Not sure this belongs here - Maybe move to a different page? Davidmbrooke 20:21, 6 April 2012 (UTC))

Once you have successfully built Bering-uClibc 5.x with your new toolchain it is best to try to test it with QEMU before testing on physical hardware.

For the ARM Versatile Platform Board example try the following: 

qemu-system-arm -m 256 -machine versatilepb -cpu arm926 -kernel linux-versatile -initrd initrd-versatile.lrp -hda sda.raw \
-append "rw root=/dev/ram0 LEAFCFG=/dev/sda1:vfat" -serial stdio

Where:

  • linux-versatile is the kernel image file, copied from the staging/$GNU_TARGET_NAME/boot/ directory
  • initrd-versatile.lrp is the matching initd file, copied from the package/$GNU_TARGET_NAME/ directory
  • sda.raw is a hard disk image file created as described here and populated with usual Bering-uClibc 5.x files (leaf.cfg, *.lrp etc.)

Logging kernel boot in qemu

To get kernel output dumped to a file outside the virtual system, add e.g. "-serial file:/tmp/qemu-output.log" to the qemu command line. When booting the virtual system, add "console=ttyS0" to the kernel boot parameters.

This output is particularly helpful if you are having trouble booting the system, in which case you may also wish to remove "rhgb" and "quiet" from the kernel boot parameters.

Prev Up Next
Retrieved from "http://bering-uclibc.zetam.org/index.php?title=Bering-uClibc_5.x_-_Developer_Guide_-_Adding_a_Hardware_Architecture_Variant&oldid=1998"