Difference between revisions of "Bering-uClibc 5.x - Developer Guide - Adding a Hardware Architecture Variant"

From bering-uClibc
Jump to: navigation, search
(Added navigation header and footer)
m (fix navigation)
 
(26 intermediate revisions by 3 users not shown)
Line 2: Line 2:
 
! colspan="3" align="center" | [[Bering-uClibc 5.x - Developer Guide - Adding a Hardware Architecture Variant|Adding a Hardware Architecture Variant]]
 
! colspan="3" align="center" | [[Bering-uClibc 5.x - Developer Guide - Adding a Hardware Architecture Variant|Adding a Hardware Architecture Variant]]
 
|-
 
|-
| width="20%" align="left"  | [[Bering-uClibc 5.x - Developer Guide - Preparing the Build Environment|Prev]]
+
| width="20%" align="left"  | [[Bering-uClibc 5.x - Developer Guide - Adding a Kernel Architecture Variant|Prev]]
 
! width="60%" align="center" | [[Bering-uClibc 5.x - Developer Guide]]
 
! width="60%" align="center" | [[Bering-uClibc 5.x - Developer Guide]]
| width="20%" align="right"  |
+
| width="20%" align="right"  | [[Bering-uClibc 5.x - Developer Guide - Policies and Guidelines|Next]]
 
|}
 
|}
 
----
 
----
Line 11: Line 11:
 
==Introduction==
 
==Introduction==
 
A major enhancement added in [[Bering-uClibc 5.x]] is the ability to target non-x86 runtime platforms.
 
A major enhancement added in [[Bering-uClibc 5.x]] is the ability to target non-x86 runtime platforms.
In principle it would now be possible to build [[Bering-uClibc 5.x]] for SPARC, MIPS or other CPU architectures.
+
In principle it is now possible to build [[Bering-uClibc 5.x]] for SPARC, MIPS or other CPU architectures.
 
These notes provide guidance on what changes are required to add support for a brand new target architecture variant.
 
These notes provide guidance on what changes are required to add support for a brand new target architecture variant.
The addition of support for the ARM11 processor on the [http://en.wikipedia.org/wiki/Raspberry_Pi Raspberry Pi] single board computer is used as an example.
+
 
 +
The addition of support for an ARM926 processor on an [http://www.arm.com/products/tools/development-boards/versatile/platform-baseboards.php ARM<sup>®</sup> Versatile™ Platform Baseboard] is used as an example.
 +
This particular example has been chosen because ARM CPUs are common on low-cost embedded hardware and because the Versatile board has excellent support from the QEMU <tt>qemu-system-arm</tt> emulator.
  
 
The first step is to understand exactly what hardware the target platform consists of.
 
The first step is to understand exactly what hardware the target platform consists of.
 
In particular:
 
In particular:
 
* What is the model number of the CPU?
 
* What is the model number of the CPU?
** The Raspberry Pi has a [http://www.broadcom.com/products/BCM2835 Broadcom BCM2835] "system on a chip" integrated circuit containing an [http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.ddi0301h/index.html ARM1176JZF-S] CPU
+
** The default CPU for the Versatile Platform Baseboard is the [http://www.arm.com/products/processors/classic/arm9/arm926.php ARM926EJ-S]
* What is the "architecture" of the CPU?
+
* What is the architecture of the CPU?
** The ARM1176JZF-S CPU implements the [http://en.wikipedia.org/wiki/ARM11 ARMv6] architecture standard
+
** The ARM926EJ-S implements the [http://en.wikipedia.org/wiki/ARM_architecture ARMv5TEJ] instruction set.
** This appears to fit within the Broadcom "BCMRING" "system on a chip" family.
+
** The ARM9 CPU family is "bi-endian" but defaults to little-endian, so code should be compiled little-endian.
*** Not ''completely'' sure about this yet, but it seems the most likely candidate [[User:Davidmbrooke|Davidmbrooke]] 14:30, 24 March 2012 (UTC)
+
* What are the characteristics of the supporting platform?
 +
** The ARM Versatile Platform Baseboard is recognized as a standard "machine" target for the Linux kernel.
  
  
Line 36: Line 39:
 
If you have not already extracted the Kernel source run:
 
If you have not already extracted the Kernel source run:
 
  ./buildtool.pl source kernel
 
  ./buildtool.pl source kernel
  cd source/linux/linux-3.2.*/arch
+
  cd source/*/linux/linux-3.2/arch/
  
 
Each of the directory names under "<code class="filename">arch</code>" represents a fundamental "architecture" variant.
 
Each of the directory names under "<code class="filename">arch</code>" represents a fundamental "architecture" variant.
The [[Bering-uClibc]] toolchain references this via the <code>'''ARCH'''</code> variable.
+
The [[Bering-uClibc 5.x]] toolchain references this via the <code>'''ARCH'''</code> variable.
  
 
'''Note:''' There are a few "special cases", which include <code class="filename">i386</code> and <code class="filename">x86_64</code>!
 
'''Note:''' There are a few "special cases", which include <code class="filename">i386</code> and <code class="filename">x86_64</code>!
Refer to the comments and code in <code class="filename">source/linux/linux-*.*.*/Makefile</code> (starting around line 174) for further details.
+
Refer to the comments and code in <code class="filename">source/linux/linux-3.2/Makefile</code> (starting around line 174) for further details.
  
 
Since there is a sub-directory of "<code class="filename">arch</code>" called  
 
Since there is a sub-directory of "<code class="filename">arch</code>" called  
"<code class="filename">arm</code>" that is what we need to set the "<code>ARCH</code>" variable to when building a toolchain to target the Raspberry Pi.
+
"<code class="filename">arm</code>" that is what we need to set the "<code>ARCH</code>" variable to when building a toolchain to target the Versatile Platform Board.
 
Details of how and where to do that are provided below.
 
Details of how and where to do that are provided below.
  
Line 53: Line 56:
 
The exact details of what "machines" can be selected vary depending on the value of <code>ARCH</code>:
 
The exact details of what "machines" can be selected vary depending on the value of <code>ARCH</code>:
 
* For <code>i386</code> there are entries in the kernel <code class="filename">.config</code> file like the following: <pre>CONFIG_M686=y</pre>
 
* For <code>i386</code> there are entries in the kernel <code class="filename">.config</code> file like the following: <pre>CONFIG_M686=y</pre>
* For <code>arm</code> the permissible options are governed by the names of files with names like <code class="filename">arch/arm/mach-''machinename''</code> (for example <code class="filename">arch/arm/mach-bcmring</code>) and then there are entries in the kernel <code class="filename">.config</code> file like the following: <pre>CONFIG_ARCH_BCMRING=y</pre>
+
* For <code>arm</code> the permissible options are governed by the names of files with names like <code class="filename">arch/arm/mach-''machinename''</code> (for example <code class="filename">arch/arm/mach-versatile</code>) and then there are entries in the kernel <code class="filename">.config</code> file like the following: <pre>CONFIG_ARCH_VERSATILE=y</pre>
  
 
Since different users run different machines which demand incompatible settings of the kernel <code class="filename">.config</code> variables the option to build for multiple machine variants has been part of the [[Bering-uClibc]] toolchain since [[Bering-uClibc 4.x]].
 
Since different users run different machines which demand incompatible settings of the kernel <code class="filename">.config</code> variables the option to build for multiple machine variants has been part of the [[Bering-uClibc]] toolchain since [[Bering-uClibc 4.x]].
  
 
The [[Bering-uClibc 5.x]] toolchain uses the variable <code>'''KARCHS'''</code> to specify a space-separated list of "machines" to build for ''using a single toolchain''.
 
The [[Bering-uClibc 5.x]] toolchain uses the variable <code>'''KARCHS'''</code> to specify a space-separated list of "machines" to build for ''using a single toolchain''.
 
For the Raspberry Pi the relevant setting is (probably) <code>bcmring</code>.
 
  
 
'''Note:''' As will be seen by the later description of how these settings are processed there is nothing "magic" about the values in <code>KARCHS</code>. They are just unique string labels used to identify patch files for the kernel <code class="filename">.config</code> and these patch files can contain system-specific settings in addition to more generic CPU architecture settings.
 
'''Note:''' As will be seen by the later description of how these settings are processed there is nothing "magic" about the values in <code>KARCHS</code>. They are just unique string labels used to identify patch files for the kernel <code class="filename">.config</code> and these patch files can contain system-specific settings in addition to more generic CPU architecture settings.
It may be more appropriate to choose a "system" name like <tt>alix</tt> rather than a "CPU" name like <tt>geode</tt>.
+
It is often more appropriate to choose a "system" name like <tt>alix</tt> rather than a "CPU" name like <tt>geode</tt>.
  
 +
For the Versatile Platform Board the relevant ''machinename'' setting is <code>versatile</code>.
  
 
==GCC and Binutils CPU Architecture Selection==
 
==GCC and Binutils CPU Architecture Selection==
Line 79: Line 81:
 
Having identified the Kernel CPU Architecture (<code>ARCH</code>) refer to the appropriate sub-page of [http://gcc.gnu.org/onlinedocs/gcc/Submodel-Options.html the GCC "Hardware Models and Configurations" page] in order to understand what options are available.
 
Having identified the Kernel CPU Architecture (<code>ARCH</code>) refer to the appropriate sub-page of [http://gcc.gnu.org/onlinedocs/gcc/Submodel-Options.html the GCC "Hardware Models and Configurations" page] in order to understand what options are available.
  
For example, on the [http://gcc.gnu.org/onlinedocs/gcc/ARM-Options.html ARM Options] sub-page there is a definition of the permissible values for the <code>-march</code> command-line option to GCC and related tools. One of the permissible values is "<code>armv6</code>" which is an obvious match for the ARMv6 architecture which we know the ARM11 CPU family uses.
+
For example, on the [http://gcc.gnu.org/onlinedocs/gcc/ARM-Options.html ARM Options] sub-page there is a definition of the permissible values for the <code>-march</code> command-line option to GCC and related tools. One of the permissible values is "<code>armv5te</code>" which is a close match for the ARMv5TEJ architecture which we know the ARM926EJ-S CPU uses.
  
The [[Bering-uClibc 5.x]] toolchain references this (the setting for <code>-march</code>) via the <code>'''GNU_ARCH'''</code> variable.
+
This (the setting for <code>-march</code>) also forms the first entry in the hyphen-separated "configuration name" string.
 
+
For [[Bering-uClibc 5.x]] the second and third entries in this string are always "<tt>unknown</tt>" and "<tt>linux</tt>" respectively.
This also forms the first entry in the hyphen-separated "configuration name" string. Since the other three entries in this string are always the same for [[Bering-uClibc 5.x]] we therefore know what this full string is.
+
The fourth entry is somewhat more variable; often is it simply "<tt>uclibc</tt>" but for some platforms it is necessary to include extra information in this part if the "configuration name". In particular, for ARM platforms, it seems to be necessary to include the string "<tt>gnueabi</tt>" in order to specify use of the GNU EABI (in place of the default OABI).
For the Raspberry Pi the full string is "<code>armv6-unknown-linux-uclibc</code>".
+
Since the Versatile Platform Board uses an ARM CPU and the EABI is desirable the fourth entry should be "<tt>uclibcgnueabi</tt>" making the full string "<code>armv5te-unknown-linux-uclibcgnueabi</code>".
  
 
The [[Bering-uClibc 5.x]] toolchain references this "configuration name" via the <code>'''GNU_TARGET_NAME'''</code> variable.
 
The [[Bering-uClibc 5.x]] toolchain references this "configuration name" via the <code>'''GNU_TARGET_NAME'''</code> variable.
Line 91: Line 93:
 
The <code class="filename">buildtool.pl</code>, <code class="filename">buildpacket.pl</code> and <code class="filename">buildimage.pl</code> scripts therefore use this "configuration name" as their "toolchainname" and they set the environment variable <code>$GNU_TARGET_NAME</code> based on the specified (or default) toolchainname.
 
The <code class="filename">buildtool.pl</code>, <code class="filename">buildpacket.pl</code> and <code class="filename">buildimage.pl</code> scripts therefore use this "configuration name" as their "toolchainname" and they set the environment variable <code>$GNU_TARGET_NAME</code> based on the specified (or default) toolchainname.
  
The <code>GNU_ARCH</code> variable ensures that the generated code will run on all CPUs which are compatible with that CPU architecture.
+
The setting for <code>-march</code> ensures that the generated code will run on all CPUs which are compatible with that CPU architecture.
 
For example, code compiled for <tt>i486</tt> will also run on all later x86-compatible processors.
 
For example, code compiled for <tt>i486</tt> will also run on all later x86-compatible processors.
 
GCC and related tools make it possible to optimise code for a particular CPU while retaining compatibility with other CPUs by specifying the <code>-mtune</code> command-line option. The permissible values for this are specified on the same page as for <code>-march</code> above.
 
GCC and related tools make it possible to optimise code for a particular CPU while retaining compatibility with other CPUs by specifying the <code>-mtune</code> command-line option. The permissible values for this are specified on the same page as for <code>-march</code> above.
For the Raspberry Pi there is an exact match for the actual CPU: <code>arm1176jzf-s</code>.
+
For the Versatile Platform Board there is an exact match for the actual CPU: <code>arm926ej-s</code> so this needs to be specified as the value for <code>-mtune</code>.
 
+
The [[Bering-uClibc 5.x]] toolchain references this (the setting for <code>-mtune</code>) via the <code>'''GNU_TUNE'''</code> variable.
+
  
  
 
==High-Level Toolchain Configuration==
 
==High-Level Toolchain Configuration==
Once the required values for <code>ARCH</code>, <code>KARCHS</code>, <code>GNU_ARCH</code>, <code>GNU_TUNE</code> and <code>GNU_TARGET_NAME</code> have been identified it is time to start configuring a toolchain to target those.
+
Once the required values for the <code>ARCH</code>, <code>KARCHS</code> and <code>GNU_TARGET_NAME</code> variables and the settings for the <code>-march</code> and <code>-mtune</code> command-line options have been identified it is time to start configuring a toolchain to target those settings.
Most of the configuration is performed by editing file <code class="filename">make/MasterInclude.mk</code>.
+
  
 
The default toolchain target for [[Bering-uClibc 5.x]] is <code>i486-unknown-linux-uclibc</code> and this is specified as the default by the following lines in <code class="filename">conf/buildtool.conf</code>:
 
The default toolchain target for [[Bering-uClibc 5.x]] is <code>i486-unknown-linux-uclibc</code> and this is specified as the default by the following lines in <code class="filename">conf/buildtool.conf</code>:
Line 107: Line 106:
 
  Toolchain=i486-unknown-linux-uclibc
 
  Toolchain=i486-unknown-linux-uclibc
  
As the comment says this can be overridden by specifying "<code>-t ''toolchainname''</code>" to <code class="filename">buildtool.pl</code> (and "<code>--toolchain ''ToolchainName''</code>" to <code class="filename">buildpacket.pl</code> and <code class="filename">buildimage.pl</code>).
+
As the comment says this can be overridden by specifying "<code>-t ''toolchainname''</code>" to <code class="filename">buildtool.pl</code> (and "<code>--toolchain ''ToolchainName''</code>" to <code class="filename">buildpacket.pl</code>).
 
Alternatively the default value can be changed by editing <code class="filename">conf/buildtool.conf</code>.
 
Alternatively the default value can be changed by editing <code class="filename">conf/buildtool.conf</code>.
At the time of writing (2012-03-24) the <code class="filename">tools/buildall.sh</code> script only looks at the default setting in <code class="filename">conf/buildtool.conf</code>.
 
  
All of the build <code class="filename">.pl</code> scripts set environment variable <code>GNU_TARGET_NAME</code> based on the specified (or default) ''toolchainname'' and <code>GNU_TARGET_NAME</code> is used internally in other scripts and configuration files where toolchain-specific processing is required.
+
At the time of writing (2012-04-01) the <code class="filename">tools/buildall.sh</code> and <code class="filename">buildlwp.sh</code> scripts only look at the default setting in <code class="filename">conf/buildtool.conf</code>. They do not accept command-line arguments to specify the toolchain.
 +
 
 +
Script <code class="filename">buildimage.pl</code> gets its setting of <tt>''toolchainname''</tt> from the relevant <code class="filename">buildimage.cfg</code> file since each image has to be generated with the corresponding toolchain and it doesn't make sense to override this with a command-line argument.
 +
 
 +
All of the build <code class="filename">.pl</code> scripts set environment variable <code>$GNU_TARGET_NAME</code> based on the specified (or default) ''toolchainname'' and <code>$GNU_TARGET_NAME</code> is used internally in other scripts and configuration files where toolchain-specific processing is required.
 +
 
 +
Most of the configuration is performed by creating a new toolchain makefile to be included by <code class="filename">make/MasterInclude.mk</code> via the following lines in that file:
 +
# Include per-toolchain Makefiles
 +
include $(BT_BUILDROOT)/make/toolchain/*.mk
 +
In other words, every <code class="filename">.mk</code> file in the <code class="filename">$(BT_BUILDROOT)/make/toolchain/</code> directory is automatically included, and IF-THEN logic within those included files determines which settings are made active.
  
The main configuration file which reacts to the setting for <code>GNU_TARGET_NAME</code> is <code class="filename">make/MasterInclude.mk</code> and this is where corresponding values for <code>ARCH</code>, <code>GNU_ARCH</code> etc. must be specified.
+
The default toolchain configuration is specified in file <code class="filename">make/toolchain/i486-unknown-linux-uclibc.mk</code> which looks something like this:
The standard <code class="filename">make/MasterInclude.mk</code> has a skeleton IF - THEN - ELSE structure which needs to be extended for each new toolchain target. The lines for the default toolchain look like this:
+
#
 +
# Included Makefilefile for i486-unknown-linux-uclibc toolchain
 +
# Intended for generic x86 target
 +
#
 
  ifeq ($(GNU_TARGET_NAME),i486-unknown-linux-uclibc)
 
  ifeq ($(GNU_TARGET_NAME),i486-unknown-linux-uclibc)
  # primary kernel arch
+
  # Primary kernel architecture
 
  export ARCH:=i386
 
  export ARCH:=i386
  # available kernel archs
+
  # Space-separated list of kernel sub-archs to generate
 
  export KARCHS:=i686 i486 geode
 
  export KARCHS:=i686 i486 geode
  # available kernel archs with pci-express support
+
  # Available kernel archs with pci-express support
 
  export KARCHS_PCIE:=i686
 
  export KARCHS_PCIE:=i686
  # set target subarch here
+
  # Arch-specific CFLAGS
  export GNU_ARCH:=i486
+
  export ARCH_CFLAGS:=-march=i486 -mtune=pentiumpro
  # set target optimization here
+
  # Name of kernel image
  export GNU_TUNE:=pentiumpro
+
export KERN_IMAGE:=bzImage
 +
# Name of OpenSSL target
 +
  export OPENSSL_TARGET:=linux-elf
 +
...
 +
<lines omitted>
 +
...
 +
endif
  
For the Raspberry Pi we need to add a new block of lines below that:
+
The "<tt><lines omitted></tt>" look like (for example):
else ifeq ($(GNU_TARGET_NAME),armv6-unknown-linux-uclibc)
+
# primary kernel arch
+
export ARCH:=arm
+
# available kernel archs
+
export KARCHS:=bcmring
+
# set target subarch here
+
export GNU_ARCH:=armv6
+
# set target optimization here
+
export GNU_TUNE:=arm1176jzf-s
+
 
+
You will notice other lines within the IF - THEN - ELSE structure for the default toolchain - for example:
+
 
  export ac_cv_sizeof_int=4
 
  export ac_cv_sizeof_int=4
Don't worry about copying those for the new toolchain yet; they are covered later in this document.
+
and are covered later in this document.
 +
 
 +
For the Versatile Platform Board we need to copy the default toolchain file to <code class="filename">make/toolchain/armv5te-unknown-linux-uclibcgnueabi.mk</code> and adjust the contents to read as follows:
 +
#
 +
# Included Makefilefile for armv5te-unknown-linux-uclibcgnueabi toolchain
 +
# Intended for ARM Versatile Platform Board target
 +
#
 +
ifeq ($(GNU_TARGET_NAME),armv5te-unknown-linux-uclibcgnueabi)
 +
# Primary kernel architecture
 +
export ARCH:=arm
 +
# Space-separated list of kernel sub-archs to generate
 +
export KARCHS:=versatile
 +
# Arch-specific CFLAGS
 +
export ARCH_CFLAGS:=-march=armv5te -mtune=arm926ej-s
 +
# Name of kernel image
 +
export KERN_IMAGE:=zImage
 +
# Name of OpenSSL target
 +
export OPENSSL_TARGET:=linux-armv4
 +
 +
endif
 +
If other settings in addition to <code>-march</code> and <code>-mtune</code> are required they should be appended to <code>ARCH_CFLAGS</code>. For example, some ARM processors need the FPU type to be specified with e.g. <tt>-mfpu=vfp</tt>.
  
 +
The variables <tt>KERN_IMAGE</tt> and <tt>OPENSSL_TARGET</tt> accommodate some Package-specific platform differences.
 +
Refer to <code class="filename">repo/kernel/buildtool.mk</code> and <code class="filename">repo/openssl/buildtool.mk</code> respectively for usage of these variables.
  
 
==Kernel Configuration File==
 
==Kernel Configuration File==
Line 149: Line 176:
 
There needs to be a file called <code class="filename">repo/linux/Bering-''KVER''.config-''KARCH''.patch</code> for each <code>KARCH</code> in <code>KARCHS</code>, and this file must contain "diff" output which converts the base <code class="filename">repo/linux/Bering-''KVER''.config</code> into a specific kernel <code class="filename">.config</code> file suitable for <code>KARCH</code>.
 
There needs to be a file called <code class="filename">repo/linux/Bering-''KVER''.config-''KARCH''.patch</code> for each <code>KARCH</code> in <code>KARCHS</code>, and this file must contain "diff" output which converts the base <code class="filename">repo/linux/Bering-''KVER''.config</code> into a specific kernel <code class="filename">.config</code> file suitable for <code>KARCH</code>.
  
For the Raspberry Pi <code>KARCH</code> = <tt>bcmring</tt> so the full file name is <code class="filename">repo/linux/Bering-''KVER''.config-bcmring.patch</code>.
+
For the Versatile Platform Board <code>KARCH</code> = <tt>versatile</tt> so the full file name is <code class="filename">repo/linux/Bering-''KVER''.config-versatile.patch</code>.
 
This name needs to be added to <code class="filename">repo/linux/buildtool.cfg</code> and the file must be created in the <code class="filename">repo/linux/</code> directory.
 
This name needs to be added to <code class="filename">repo/linux/buildtool.cfg</code> and the file must be created in the <code class="filename">repo/linux/</code> directory.
  
Constructing a suitable and fully correct patch file is non-trivial.
+
Constructing a suitable and fully correct patch file is non-trivial and requires a good understanding of the kernel configuration options.
 
One possible procedure is as follows:
 
One possible procedure is as follows:
* Create (e.g. "touch") an empty patch file with the right name.
+
* Create (e.g. "touch") an empty patch file with the right name in the <code class="filename">repo/</code> directory and specify this in <code class="filename">repo/buildtool.cfg</code>. With an empty patch file the starting configuration will be the same as the "base" configuration for [[Bering-uClibc 5.x]]
* Run: <pre>buildtool.pl -t toolchainname source linux</pre>
+
* Run: <pre>buildtool.pl -t armv5te-unknown-linux-uclibcgnueabi source linux</pre>
* This will recognize that the <code class="filename">.config</code> file is not compatible with the specified <code>ARCH</code> and prompt for new values for the kernel configuration variables which must be changed.
+
* This will recognize that the <code class="filename">.config</code> file is not compatible with the specified <code>ARCH</code> and prompt for new values for the kernel configuration variables which must be changed while preserving those which are valid. In a separate shell run: <pre>tail -f log/buildtoollog</pre> to see the prompts from <tt>make oldconfig</tt> but answer the prompts in the shell where <code class="filename">buildtool.pl</code> is running.
** Run: <pre>tail -f log/buildtoollog</pre> to see the prompts from <tt>make oldconfig</tt> and answer the prompts in the shell where <code class="filename">buildtool.pl</code> is running.
+
* At this stage it is OK to accept the default values for all of the settings.
* Locate the generated <code class="filename">.config</code> file (should be <code class="filename">source/''toolchainname''/linux/linux-''KARCH''/.config</code>) and use that to generate the "real" patch file with a command like the following:<pre>diff -c ../Bering-KVER.config .config > ../Bering-KVER.config-KARCH.patch</pre>
+
 
** This populates the actual differences into the patch file so that no prompts are displayed the next time the build is run.
+
This procedure carries across the majority of the standard [[Bering-uClibc 5.x]] kernel configuration but does not take account of the requirements of the target hardware and it is generally necessary to adjust the configuration.
 +
 
 +
A good way to make minor adjustments to configuration settings is to go to directory <code class="filename">source/$GNU_TARGET_NAME/linux/linux-$KARCH/</code> and to run:
 +
make ARCH=arm menuconfig
 +
Consult e.g. <code class="filename">source/armv5te-unknown-linux-uclibcgnueabi/linux/linux-3.2.13/arch/arm/configs/versatile_defconfig</code> for the default settings recommended for the <tt>versatile</tt> target.
 +
 
 +
Once a good <code class="filename">.config</code> file has been generated the "patch" file must be (re-)created.
 +
Locate the generated <code class="filename">.config</code> file (should be <code class="filename">source/$GNU_TARGET_NAME/linux/linux-$KARCH/.config</code>) and generate the patch file with commands like the following:
 +
cp .config ../Bering-$KVER.config-$KARCH
 +
cd ..
 +
diff -c ../Bering-$KVER.config Bering-$KVER.config-$KARCH > Bering-$KVER.config-$KARCH.patch
  
  
Line 166: Line 203:
 
For uClibc the file needs to be called <code class="filename">repo/toolchain/config.''$GNU_TARGET_NAME''</code> and this is a "full" file rather than a "patch".  
 
For uClibc the file needs to be called <code class="filename">repo/toolchain/config.''$GNU_TARGET_NAME''</code> and this is a "full" file rather than a "patch".  
  
For the Raspberry Pi the full file name is <code class="filename">repo/toolchain/config.armv6-unknown-linux-uclibc</code>.
+
For the Versatile Platform Board the full file name is <code class="filename">repo/toolchain/config.armv5te-unknown-linux-uclibcgnueabi</code>.
 
This name needs to be added to <code class="filename">repo/toolchain/buildtool.cfg</code> and the file must be created in the <code class="filename">repo/toolchain/</code> directory.
 
This name needs to be added to <code class="filename">repo/toolchain/buildtool.cfg</code> and the file must be created in the <code class="filename">repo/toolchain/</code> directory.
  
Line 174: Line 211:
 
* Run: <pre>buildtool.pl -t toolchainname build toolchain</pre>
 
* Run: <pre>buildtool.pl -t toolchainname build toolchain</pre>
 
* This will recognize that some different options need to be selected and prompt for new values for the uClibc configuration variables which must be changed.
 
* This will recognize that some different options need to be selected and prompt for new values for the uClibc configuration variables which must be changed.
* Locate the generated <code class="filename">.config</code> file (should be <code class="filename">source/''toolchainname''/toolchain/uClibc-0.9.32*/.config</code>) and use that as the "real" file with a command like the following:<pre>cp .config ../config.GNU_TARGET_NAME</pre>
+
** For some reason the uClibc "<tt>make oldconfig</tt>" doesn't behave the same way as the kernel "<tt>make oldconfig</tt>" and refuses to accept entries when the console input is redirected.
 +
*** That was because "<tt>make oldconfig</tt>" specified "<tt>$(MAKEOPTS)</tt>" which runs a multi-threaded build. Now removed (no performance benefit from a multi-threaded build to this step).
 +
** Instead, go to the directory containing the "live" uClibc <code class="filename">.config</code> file and run: <pre>make menuconfig</pre> on the build host.
 +
* Locate the generated <code class="filename">.config</code> file (should be <code class="filename">source/''toolchainname''/toolchain/uClibc-0.9.3*/.config</code>) and use that as the "real" file with a command like the following:<pre>cp .config ../config.GNU_TARGET_NAME</pre>
 +
 
 +
Like the kernel, uClibc has both "generic" (architecture) and "specifc" (CPU) configuration entries.
 +
The "specific" entry is something like:
 +
CONFIG_ARM926T=y
 +
There are references to these sort of configuration variables in <code class="filename">source/''toolchainname''/toolchain/uClibc-0.9.3*/Rules.mak</code> - for example:
 +
CPU_CFLAGS-$(CONFIG_ARM926T)+=-mtune=arm9e -march=armv5te
 +
Recognize those? The implication is that the generated uClibc library will run on any <tt>armv5te</tt> processor but is optimized for the <tt>arm9e</tt>, like the kernel.
 +
 
 +
'''Note:''' The above is correct for uClibc 0.9.32 but the "specific" architecture configuration variables have been removed in uClibc 0.9.33.
  
  
Line 180: Line 229:
 
That should be it. Running:
 
That should be it. Running:
 
  buildtool.pl -t ''toolchainname'' build toolchain
 
  buildtool.pl -t ''toolchainname'' build toolchain
should create a toolchain based on the specified configuration settings.
+
''should'' create a toolchain based on the specified configuration settings.
  
 
In reality you will probably get build errors and will need to refine the contents of the kernel and uClibc <code class="filename">.config</code> files in order to get a successful toolchain build.
 
In reality you will probably get build errors and will need to refine the contents of the kernel and uClibc <code class="filename">.config</code> files in order to get a successful toolchain build.
  
(No, it doesn't currently work for me either :-) [[User:Davidmbrooke|Davidmbrooke]] 20:30, 24 March 2012 (UTC))
+
For the Versatile Platform Board, try:
 +
buildtool.pl -t armv5te-unknown-linux-uclibcgnueabi build toolchain
  
 +
(This works for me as of today [[User:Davidmbrooke|Davidmbrooke]] 21:00, 31 March 2012 (UTC))
  
==Cross Compilation Challenges==
+
 
One particular challenge with cross-compiling applications which use "configure" is that they try to compile and execute applications on the build host in order to infer things about the target host. Sometimes this works; sometimes it does not.
+
The steps performed as part of the toolchain build are described below.
 +
 
 +
===Source Processing===
 +
Within <code class="filename">conf/sources.cfg</code> the "<tt>toolchain</tt>" source Package is declared to be dependent on the "<tt>linux</tt>" source Package so the kernel source gets processed first.
 +
In order to build the linux "<tt>source</tt>" target:
 +
* The kernel source <code class="filename">.tar.bz2</code> file is unpacked
 +
* The kernel source patches are applied
 +
* For each entry in <tt>KARCHS</tt>
 +
** The ''generic'' kernel <code class="filename">.config</code> file is patched with the ''specific'' <tt>KARCH</tt> patch to create a ''specific'' <code class="filename">.config</code> file
 +
** The "<tt>make oldconfig</tt>" command is run (with appropriate command-line arguments)
 +
** The "<tt>make headers_install</tt>" command is run (with appropriate command-line arguments)
 +
** The generated header files are copied to the <code class="filename">toolchain/$GNU_TARGET_NAME/usr/include/</code> directory
 +
 
 +
Once the "<tt>linux</tt>" source Package has been processed the "<tt>toolchain</tt>" source Package processing can start.
 +
* The uClibc source is extracted and the [[Bering-uClibc 5.x]] uClibc source patches are applied.
 +
* The binutils source is extracted.
 +
* The GCC source is extracted.
 +
* The mod-utils source is extracted (required for depmod).
 +
 
 +
===Build Processing===
 +
Once the "<tt>source</tt>" processing has completed the "<tt>build</tt>" processing can start. The sequence is as follows:
 +
* The uClibc <code class="filename">.config</code> file is processed as part of "<tt>make install_headers</tt>" for uClibc.
 +
** This adds uClibc header files to the ones installed for the kernel above.
 +
* The "stage 1" binutils files are compiled.
 +
* The "stage 1" GCC compiler is compiled.
 +
* The "stage 2" GCC compiler is compiled.
 +
* The uClibc library is compiled.
 +
* The "stage 2" binutils files are compiled.
 +
* The mod-utils files are compiled.
 +
* The results of the toolchain build are copied to the <code class="filename">staging/$GNU_TARGET_NAME/</code> directory.
 +
 
 +
===Hints and tips for debugging toolchain build failures===
 +
* You can get more verbose diagnostics from the uClibc build by setting environment variable "V" to either 1 or 2. See <code class="filename">source/''toolchainname''/uClibc-0.9.3*/Makefile.help</code> for more details.
 +
 
 +
 
 +
==Bering-uClibc Build==
 +
Once the new toolchain has been built the procedure to build [[Bering-uClibc 5.x]] itself should be the same as for the default platform.
 +
Note that the most of the code is much better tested on x86 platforms than on other CPUs and there are likely to be more bugs and build failures.
 +
 
 +
Most of the [[Bering-uClibc 5.x]] code is platform-independent but there are some exceptions. The key ones are noted below.
 +
 
 +
===Platform-Specific Components===
 +
====Syslinux, Isolinux, Pxelinux====
 +
Syslinux, Isolinux and Pxelinux are so common that it is easy to forget they are x86 / PC specific.
 +
A possible alternative is [http://www.denx.de/wiki/U-Boot U-Boot].
 +
However many embedded platforms have their own specific boot loader solutions.
 +
 
 +
===Cross Compilation Challenges and Workarounds===
 +
====Autoconf====
 +
One particular challenge with cross-compiling applications which use "configure" is that they try to compile and execute applications on the ''build'' host in order to infer things about the ''target'' host. Sometimes this works; sometimes it does not.
  
 
In cases where it does not work it is possible to "prime" configure's cache with the correct selections by setting environment variables to specify these.
 
In cases where it does not work it is possible to "prime" configure's cache with the correct selections by setting environment variables to specify these.
 
That is what all those "<tt>export ac_cv_*</tt>" lines are for.
 
That is what all those "<tt>export ac_cv_*</tt>" lines are for.
If you get errors when building applications you can try to establish which variable configure is looking for and set it appropriately in the block of lines that relate to the toolchain.
+
If you get errors when building applications you can try to establish which variable configure is looking for (check the <code class="filename">configure</code> script) and set it appropriately.
 +
Use the following guidelines when deciding where to set the variable:
 +
* '''IF''' the variable is specific to a single ''application'' (e.g. <tt>samba_cv_CC_NEGATIVE_ENUM_VALUES=yes</tt> which is only for the SAMBA application) '''AND''' the setting is the same for all toolchains '''THEN''' set it in the application's <code class="filename">buildtool.mk</code>
 +
** Review the usage of <tt>CONFDEFS</tt> in <code class="filename">repo/samba/buildtool.mk</code> as an example
 +
* '''IF''' the variable is ''not'' specific to a single ''application'' (e.g. if it relates to one library which is used by multiple applications) '''AND''' the setting is the same for all toolchains '''THEN''' set it in <code class="filename">make/MasterInclude.mk</code>
 +
* '''IF''' the variable ''is'' specific to a single ''toolchain'' (whether or not it relates to more than one ''application'') '''THEN''' set it in the per-toolchain makefile (e.g. <code class="filename">make/toolchain/armv5te-unknown-linux-uclibcgnueabi.mk</code>)
 +
 
 +
====Build Host <tt>strip</tt>====
 +
Some applications' Makefiles call the <code class="filename">strip</code> utility explicitly, and the build host's <code class="filename">strip</code> utility will not typically understand the target host's executable file format.
 +
 
 +
In some cases the Makefile permits an override by specifying the <tt>STRIP</tt> environment variable, so adding this to the "<tt>make install</tt>" line in <code class="filename">buildtool.mk</code> as "<tt>$(GNU_TARGET_NAME)-strip</tt>" fixes the problem.
 +
 
 +
Other applications' Makefiles use the <code class="filename">install</code> utility to copy files to the destination directory and sometimes these use the '<tt>-s</tt>' command-line option which specifies that binaries should be stripped as part of the install step.
 +
By default this uses the build host's <code class="filename">strip</code> executable, which assumes the build host's CPU architecture.
 +
 
 +
It is possible to specify a different <code class="filename">strip</code> executable by adding <tt>--strip-program=''PROGRAM''</tt>  to the <code class="filename">install</code> call within the application's Makefile.
 +
 
 +
 
 +
==Testing with QEMU==
 +
(Not sure this belongs here - Maybe move to a different page? [[User:Davidmbrooke|Davidmbrooke]] 20:21, 6 April 2012 (UTC))
 +
 
 +
Once you have successfully built [[Bering-uClibc 5.x]] with your new toolchain it is best to try to test it with QEMU before testing on physical hardware.
 +
 
 +
For the ARM Versatile Platform Board example try the following:
 +
qemu-system-arm -m 256 -machine versatilepb -cpu arm926 -kernel '''linux-versatile''' -initrd '''initrd-versatile.lrp''' -hda '''sda.raw''' \
 +
-append "rw root=/dev/ram0 LEAFCFG=/dev/sda1:vfat" -serial stdio
 +
Where:
 +
* <code class="filename">linux-versatile</code> is the kernel image file, copied from the <code class="filename">staging/$GNU_TARGET_NAME/boot/</code> directory
 +
* <code class="filename">initrd-versatile.lrp</code> is the matching initd file, copied from the <code class="filename">package/$GNU_TARGET_NAME/</code> directory
 +
* <code class="filename">sda.raw</code> is a hard disk image file created as described [[Bering-uClibc 5.x - User Guide - Appendices - Working with Disk Image Files|here]] and populated with usual [[Bering-uClibc 5.x]] files (<code class="filename">leaf.cfg</code>, <code class="filename">*.lrp</code> etc.)
 +
 
 +
===Logging kernel boot in qemu===
 +
To get kernel output dumped to a file outside the virtual system, add e.g. "-serial file:/tmp/qemu-output.log" to the qemu command line. When booting the virtual system, add "console=ttyS0" to the kernel boot parameters.
  
 +
This output is particularly helpful if you are having trouble booting the system, in which case you may also wish to remove "rhgb" and "quiet" from the kernel boot parameters. 
  
 
----
 
----
 
{| summary="Navigation footer" width="100%"
 
{| summary="Navigation footer" width="100%"
| width="40%" align="left"  | [[Bering-uClibc 5.x - Developer Guide - Preparing the Build Environment|Prev]]
+
| width="40%" align="left"  | [[Bering-uClibc 5.x - Developer Guide - Adding a Kernel Architecture Variant|Prev]]
 
| width="20%" align="center" | [[Bering-uClibc 5.x - Developer Guide|Up]]
 
| width="20%" align="center" | [[Bering-uClibc 5.x - Developer Guide|Up]]
| width="40%" align="right"  |  
+
| width="40%" align="right"  | [[Bering-uClibc 5.x - Developer Guide - Policies and Guidelines|Next]]
 
|}
 
|}
  
 
[[Category:Bering-uClibc 5.x]]
 
[[Category:Bering-uClibc 5.x]]
 
[[Category:Developer Guide]]
 
[[Category:Developer Guide]]

Latest revision as of 14:02, 10 August 2014

Adding a Hardware Architecture Variant
Prev Bering-uClibc 5.x - Developer Guide Next


Introduction

A major enhancement added in Bering-uClibc 5.x is the ability to target non-x86 runtime platforms. In principle it is now possible to build Bering-uClibc 5.x for SPARC, MIPS or other CPU architectures. These notes provide guidance on what changes are required to add support for a brand new target architecture variant.

The addition of support for an ARM926 processor on an ARM® Versatile™ Platform Baseboard is used as an example. This particular example has been chosen because ARM CPUs are common on low-cost embedded hardware and because the Versatile board has excellent support from the QEMU qemu-system-arm emulator.

The first step is to understand exactly what hardware the target platform consists of. In particular:

  • What is the model number of the CPU?
    • The default CPU for the Versatile Platform Baseboard is the ARM926EJ-S
  • What is the architecture of the CPU?
    • The ARM926EJ-S implements the ARMv5TEJ instruction set.
    • The ARM9 CPU family is "bi-endian" but defaults to little-endian, so code should be compiled little-endian.
  • What are the characteristics of the supporting platform?
    • The ARM Versatile Platform Baseboard is recognized as a standard "machine" target for the Linux kernel.


Note: These notes are intended to provide guidelines rather than a fully prescriptive recipe to follow.

Warning: The cross-compilation build system is under active development and these notes reflect the situation at the time of writing. If recent changes have been made they may be out of step with the Bering-uClibc 5.x code in Git.


Linux Kernel CPU Architecture Selection

The standard Linux kernel source tree includes CPU architecture specific code for quite a number of CPU types. This code is in the "arch" directory within the kernel source tree and it is sensible to review the contents of this directory. If you have not already extracted the Kernel source run:

./buildtool.pl source kernel
cd source/*/linux/linux-3.2/arch/

Each of the directory names under "arch" represents a fundamental "architecture" variant. The Bering-uClibc 5.x toolchain references this via the ARCH variable.

Note: There are a few "special cases", which include i386 and x86_64! Refer to the comments and code in source/linux/linux-3.2/Makefile (starting around line 174) for further details.

Since there is a sub-directory of "arch" called "arm" that is what we need to set the "ARCH" variable to when building a toolchain to target the Versatile Platform Board. Details of how and where to do that are provided below.

In addition to the fundamental CPU architecture setting the kernel recognizes a further level of "machine" specification. For example, under the umbrella architecture of i386 we have the "true" i386 and also i486, Pentium 4, Geode LX etc. and it is possible to select between those when compiling a kernel.

The exact details of what "machines" can be selected vary depending on the value of ARCH:

  • For i386 there are entries in the kernel .config file like the following:
    CONFIG_M686=y
  • For arm the permissible options are governed by the names of files with names like arch/arm/mach-machinename (for example arch/arm/mach-versatile) and then there are entries in the kernel .config file like the following:
    CONFIG_ARCH_VERSATILE=y

Since different users run different machines which demand incompatible settings of the kernel .config variables the option to build for multiple machine variants has been part of the Bering-uClibc toolchain since Bering-uClibc 4.x.

The Bering-uClibc 5.x toolchain uses the variable KARCHS to specify a space-separated list of "machines" to build for using a single toolchain.

Note: As will be seen by the later description of how these settings are processed there is nothing "magic" about the values in KARCHS. They are just unique string labels used to identify patch files for the kernel .config and these patch files can contain system-specific settings in addition to more generic CPU architecture settings. It is often more appropriate to choose a "system" name like alix rather than a "CPU" name like geode.

For the Versatile Platform Board the relevant machinename setting is versatile.

GCC and Binutils CPU Architecture Selection

The toolchain is responsible for building code for the target environment and it relies on the GCC (cross-)compiler to do most of the work.

The GNU toolset (most notably "configure") has a well-established way of identifying different target platforms by a hyphen-separated list of the key characteristics known as the "configuration name". This was initially the triplet cpu-manufacturer-kernel but is now more commonly the quadruplet cpu-manufacturer-kernel-os (though this is still often referred to as a "triplet"). For example, i486-unknown-linux-uclibc refers to:

  • an i486 CPU, installed in
  • an unknown hardware platform ("unknown" as in "we don't care whether a PC is made by HP, IBM, Dell etc."), running
  • the linux kernel, and
  • a uclibc C library-based operating system

The first field ("cpu") is of particular interested here. Having identified the Kernel CPU Architecture (ARCH) refer to the appropriate sub-page of the GCC "Hardware Models and Configurations" page in order to understand what options are available.

For example, on the ARM Options sub-page there is a definition of the permissible values for the -march command-line option to GCC and related tools. One of the permissible values is "armv5te" which is a close match for the ARMv5TEJ architecture which we know the ARM926EJ-S CPU uses.

This (the setting for -march) also forms the first entry in the hyphen-separated "configuration name" string. For Bering-uClibc 5.x the second and third entries in this string are always "unknown" and "linux" respectively. The fourth entry is somewhat more variable; often is it simply "uclibc" but for some platforms it is necessary to include extra information in this part if the "configuration name". In particular, for ARM platforms, it seems to be necessary to include the string "gnueabi" in order to specify use of the GNU EABI (in place of the default OABI). Since the Versatile Platform Board uses an ARM CPU and the EABI is desirable the fourth entry should be "uclibcgnueabi" making the full string "armv5te-unknown-linux-uclibcgnueabi".

The Bering-uClibc 5.x toolchain references this "configuration name" via the GNU_TARGET_NAME variable.

Since this "configuration name" captures all the characteristics of the target system which need to be hard-coded into the toolchain it is a good string to use to identify and differentiate multiple toolchains. The buildtool.pl, buildpacket.pl and buildimage.pl scripts therefore use this "configuration name" as their "toolchainname" and they set the environment variable $GNU_TARGET_NAME based on the specified (or default) toolchainname.

The setting for -march ensures that the generated code will run on all CPUs which are compatible with that CPU architecture. For example, code compiled for i486 will also run on all later x86-compatible processors. GCC and related tools make it possible to optimise code for a particular CPU while retaining compatibility with other CPUs by specifying the -mtune command-line option. The permissible values for this are specified on the same page as for -march above. For the Versatile Platform Board there is an exact match for the actual CPU: arm926ej-s so this needs to be specified as the value for -mtune.


High-Level Toolchain Configuration

Once the required values for the ARCH, KARCHS and GNU_TARGET_NAME variables and the settings for the -march and -mtune command-line options have been identified it is time to start configuring a toolchain to target those settings.

The default toolchain target for Bering-uClibc 5.x is i486-unknown-linux-uclibc and this is specified as the default by the following lines in conf/buildtool.conf:

# default toolchain - override with "-t toolchain" argument to buildtool.pl
Toolchain=i486-unknown-linux-uclibc

As the comment says this can be overridden by specifying "-t toolchainname" to buildtool.pl (and "--toolchain ToolchainName" to buildpacket.pl). Alternatively the default value can be changed by editing conf/buildtool.conf.

At the time of writing (2012-04-01) the tools/buildall.sh and buildlwp.sh scripts only look at the default setting in conf/buildtool.conf. They do not accept command-line arguments to specify the toolchain.

Script buildimage.pl gets its setting of toolchainname from the relevant buildimage.cfg file since each image has to be generated with the corresponding toolchain and it doesn't make sense to override this with a command-line argument.

All of the build .pl scripts set environment variable $GNU_TARGET_NAME based on the specified (or default) toolchainname and $GNU_TARGET_NAME is used internally in other scripts and configuration files where toolchain-specific processing is required.

Most of the configuration is performed by creating a new toolchain makefile to be included by make/MasterInclude.mk via the following lines in that file:

# Include per-toolchain Makefiles
include $(BT_BUILDROOT)/make/toolchain/*.mk

In other words, every .mk file in the $(BT_BUILDROOT)/make/toolchain/ directory is automatically included, and IF-THEN logic within those included files determines which settings are made active.

The default toolchain configuration is specified in file make/toolchain/i486-unknown-linux-uclibc.mk which looks something like this:

#
# Included Makefilefile for i486-unknown-linux-uclibc toolchain
# Intended for generic x86 target
#
ifeq ($(GNU_TARGET_NAME),i486-unknown-linux-uclibc)
# Primary kernel architecture
export ARCH:=i386
# Space-separated list of kernel sub-archs to generate
export KARCHS:=i686 i486 geode
# Available kernel archs with pci-express support
export KARCHS_PCIE:=i686
# Arch-specific CFLAGS
export ARCH_CFLAGS:=-march=i486 -mtune=pentiumpro
# Name of kernel image
export KERN_IMAGE:=bzImage
# Name of OpenSSL target
export OPENSSL_TARGET:=linux-elf
...
<lines omitted>
...
endif

The "<lines omitted>" look like (for example):

export ac_cv_sizeof_int=4

and are covered later in this document.

For the Versatile Platform Board we need to copy the default toolchain file to make/toolchain/armv5te-unknown-linux-uclibcgnueabi.mk and adjust the contents to read as follows:

#
# Included Makefilefile for armv5te-unknown-linux-uclibcgnueabi toolchain
# Intended for ARM Versatile Platform Board target
#
ifeq ($(GNU_TARGET_NAME),armv5te-unknown-linux-uclibcgnueabi)
# Primary kernel architecture
export ARCH:=arm
# Space-separated list of kernel sub-archs to generate
export KARCHS:=versatile
# Arch-specific CFLAGS
export ARCH_CFLAGS:=-march=armv5te -mtune=arm926ej-s
# Name of kernel image
export KERN_IMAGE:=zImage
# Name of OpenSSL target
export OPENSSL_TARGET:=linux-armv4

endif

If other settings in addition to -march and -mtune are required they should be appended to ARCH_CFLAGS. For example, some ARM processors need the FPU type to be specified with e.g. -mfpu=vfp.

The variables KERN_IMAGE and OPENSSL_TARGET accommodate some Package-specific platform differences. Refer to repo/kernel/buildtool.mk and repo/openssl/buildtool.mk respectively for usage of these variables.

Kernel Configuration File

If you were to try to build the new toolchain at this point it would fail with an error message because the build scripts will not be able to locate a kernel .config patch file with the right name. (The kernel source must be processed before building the toolchain executables in order to extract the header files.)

There needs to be a file called repo/linux/Bering-KVER.config-KARCH.patch for each KARCH in KARCHS, and this file must contain "diff" output which converts the base repo/linux/Bering-KVER.config into a specific kernel .config file suitable for KARCH.

For the Versatile Platform Board KARCH = versatile so the full file name is repo/linux/Bering-KVER.config-versatile.patch. This name needs to be added to repo/linux/buildtool.cfg and the file must be created in the repo/linux/ directory.

Constructing a suitable and fully correct patch file is non-trivial and requires a good understanding of the kernel configuration options. One possible procedure is as follows:

  • Create (e.g. "touch") an empty patch file with the right name in the repo/ directory and specify this in repo/buildtool.cfg. With an empty patch file the starting configuration will be the same as the "base" configuration for Bering-uClibc 5.x
  • Run:
    buildtool.pl -t armv5te-unknown-linux-uclibcgnueabi source linux
  • This will recognize that the .config file is not compatible with the specified ARCH and prompt for new values for the kernel configuration variables which must be changed while preserving those which are valid. In a separate shell run:
    tail -f log/buildtoollog
    to see the prompts from make oldconfig but answer the prompts in the shell where buildtool.pl is running.
  • At this stage it is OK to accept the default values for all of the settings.

This procedure carries across the majority of the standard Bering-uClibc 5.x kernel configuration but does not take account of the requirements of the target hardware and it is generally necessary to adjust the configuration.

A good way to make minor adjustments to configuration settings is to go to directory source/$GNU_TARGET_NAME/linux/linux-$KARCH/ and to run:

make ARCH=arm menuconfig

Consult e.g. source/armv5te-unknown-linux-uclibcgnueabi/linux/linux-3.2.13/arch/arm/configs/versatile_defconfig for the default settings recommended for the versatile target.

Once a good .config file has been generated the "patch" file must be (re-)created. Locate the generated .config file (should be source/$GNU_TARGET_NAME/linux/linux-$KARCH/.config) and generate the patch file with commands like the following:

cp .config ../Bering-$KVER.config-$KARCH
cd ..
diff -c ../Bering-$KVER.config Bering-$KVER.config-$KARCH > Bering-$KVER.config-$KARCH.patch


uClibc Configuration File

Just like the kernel, uClibc has a .config file which needs to be tailored for the new toolchain. For uClibc the file needs to be called repo/toolchain/config.$GNU_TARGET_NAME and this is a "full" file rather than a "patch".

For the Versatile Platform Board the full file name is repo/toolchain/config.armv5te-unknown-linux-uclibcgnueabi. This name needs to be added to repo/toolchain/buildtool.cfg and the file must be created in the repo/toolchain/ directory.

As with the kernel .config it is non-trivial to create a file with the right contents. One possible procedure is as follows:

  • Copy the file for the default toolchain and edit it to reflect the correct ARCH and the correct value for CROSS_COMPILER_PREFIX.
  • Run:
    buildtool.pl -t toolchainname build toolchain
  • This will recognize that some different options need to be selected and prompt for new values for the uClibc configuration variables which must be changed.
    • For some reason the uClibc "make oldconfig" doesn't behave the same way as the kernel "make oldconfig" and refuses to accept entries when the console input is redirected.
      • That was because "make oldconfig" specified "$(MAKEOPTS)" which runs a multi-threaded build. Now removed (no performance benefit from a multi-threaded build to this step).
    • Instead, go to the directory containing the "live" uClibc .config file and run:
      make menuconfig
      on the build host.
  • Locate the generated .config file (should be source/toolchainname/toolchain/uClibc-0.9.3*/.config) and use that as the "real" file with a command like the following:
    cp .config ../config.GNU_TARGET_NAME

Like the kernel, uClibc has both "generic" (architecture) and "specifc" (CPU) configuration entries. The "specific" entry is something like:

CONFIG_ARM926T=y

There are references to these sort of configuration variables in source/toolchainname/toolchain/uClibc-0.9.3*/Rules.mak - for example:

CPU_CFLAGS-$(CONFIG_ARM926T)+=-mtune=arm9e -march=armv5te

Recognize those? The implication is that the generated uClibc library will run on any armv5te processor but is optimized for the arm9e, like the kernel.

Note: The above is correct for uClibc 0.9.32 but the "specific" architecture configuration variables have been removed in uClibc 0.9.33.


Toolchain Build

That should be it. Running:

buildtool.pl -t toolchainname build toolchain

should create a toolchain based on the specified configuration settings.

In reality you will probably get build errors and will need to refine the contents of the kernel and uClibc .config files in order to get a successful toolchain build.

For the Versatile Platform Board, try:

buildtool.pl -t armv5te-unknown-linux-uclibcgnueabi build toolchain

(This works for me as of today Davidmbrooke 21:00, 31 March 2012 (UTC))


The steps performed as part of the toolchain build are described below.

Source Processing

Within conf/sources.cfg the "toolchain" source Package is declared to be dependent on the "linux" source Package so the kernel source gets processed first. In order to build the linux "source" target:

  • The kernel source .tar.bz2 file is unpacked
  • The kernel source patches are applied
  • For each entry in KARCHS
    • The generic kernel .config file is patched with the specific KARCH patch to create a specific .config file
    • The "make oldconfig" command is run (with appropriate command-line arguments)
    • The "make headers_install" command is run (with appropriate command-line arguments)
    • The generated header files are copied to the toolchain/$GNU_TARGET_NAME/usr/include/ directory

Once the "linux" source Package has been processed the "toolchain" source Package processing can start.

  • The uClibc source is extracted and the Bering-uClibc 5.x uClibc source patches are applied.
  • The binutils source is extracted.
  • The GCC source is extracted.
  • The mod-utils source is extracted (required for depmod).

Build Processing

Once the "source" processing has completed the "build" processing can start. The sequence is as follows:

  • The uClibc .config file is processed as part of "make install_headers" for uClibc.
    • This adds uClibc header files to the ones installed for the kernel above.
  • The "stage 1" binutils files are compiled.
  • The "stage 1" GCC compiler is compiled.
  • The "stage 2" GCC compiler is compiled.
  • The uClibc library is compiled.
  • The "stage 2" binutils files are compiled.
  • The mod-utils files are compiled.
  • The results of the toolchain build are copied to the staging/$GNU_TARGET_NAME/ directory.

Hints and tips for debugging toolchain build failures

  • You can get more verbose diagnostics from the uClibc build by setting environment variable "V" to either 1 or 2. See source/toolchainname/uClibc-0.9.3*/Makefile.help for more details.


Bering-uClibc Build

Once the new toolchain has been built the procedure to build Bering-uClibc 5.x itself should be the same as for the default platform. Note that the most of the code is much better tested on x86 platforms than on other CPUs and there are likely to be more bugs and build failures.

Most of the Bering-uClibc 5.x code is platform-independent but there are some exceptions. The key ones are noted below.

Platform-Specific Components

Syslinux, Isolinux, Pxelinux

Syslinux, Isolinux and Pxelinux are so common that it is easy to forget they are x86 / PC specific. A possible alternative is U-Boot. However many embedded platforms have their own specific boot loader solutions.

Cross Compilation Challenges and Workarounds

Autoconf

One particular challenge with cross-compiling applications which use "configure" is that they try to compile and execute applications on the build host in order to infer things about the target host. Sometimes this works; sometimes it does not.

In cases where it does not work it is possible to "prime" configure's cache with the correct selections by setting environment variables to specify these. That is what all those "export ac_cv_*" lines are for. If you get errors when building applications you can try to establish which variable configure is looking for (check the configure script) and set it appropriately. Use the following guidelines when deciding where to set the variable:

  • IF the variable is specific to a single application (e.g. samba_cv_CC_NEGATIVE_ENUM_VALUES=yes which is only for the SAMBA application) AND the setting is the same for all toolchains THEN set it in the application's buildtool.mk
    • Review the usage of CONFDEFS in repo/samba/buildtool.mk as an example
  • IF the variable is not specific to a single application (e.g. if it relates to one library which is used by multiple applications) AND the setting is the same for all toolchains THEN set it in make/MasterInclude.mk
  • IF the variable is specific to a single toolchain (whether or not it relates to more than one application) THEN set it in the per-toolchain makefile (e.g. make/toolchain/armv5te-unknown-linux-uclibcgnueabi.mk)

Build Host strip

Some applications' Makefiles call the strip utility explicitly, and the build host's strip utility will not typically understand the target host's executable file format.

In some cases the Makefile permits an override by specifying the STRIP environment variable, so adding this to the "make install" line in buildtool.mk as "$(GNU_TARGET_NAME)-strip" fixes the problem.

Other applications' Makefiles use the install utility to copy files to the destination directory and sometimes these use the '-s' command-line option which specifies that binaries should be stripped as part of the install step. By default this uses the build host's strip executable, which assumes the build host's CPU architecture.

It is possible to specify a different strip executable by adding --strip-program=PROGRAM to the install call within the application's Makefile.


Testing with QEMU

(Not sure this belongs here - Maybe move to a different page? Davidmbrooke 20:21, 6 April 2012 (UTC))

Once you have successfully built Bering-uClibc 5.x with your new toolchain it is best to try to test it with QEMU before testing on physical hardware.

For the ARM Versatile Platform Board example try the following:

qemu-system-arm -m 256 -machine versatilepb -cpu arm926 -kernel linux-versatile -initrd initrd-versatile.lrp -hda sda.raw \
-append "rw root=/dev/ram0 LEAFCFG=/dev/sda1:vfat" -serial stdio

Where:

  • linux-versatile is the kernel image file, copied from the staging/$GNU_TARGET_NAME/boot/ directory
  • initrd-versatile.lrp is the matching initd file, copied from the package/$GNU_TARGET_NAME/ directory
  • sda.raw is a hard disk image file created as described here and populated with usual Bering-uClibc 5.x files (leaf.cfg, *.lrp etc.)

Logging kernel boot in qemu

To get kernel output dumped to a file outside the virtual system, add e.g. "-serial file:/tmp/qemu-output.log" to the qemu command line. When booting the virtual system, add "console=ttyS0" to the kernel boot parameters.

This output is particularly helpful if you are having trouble booting the system, in which case you may also wish to remove "rhgb" and "quiet" from the kernel boot parameters.


Prev Up Next