Bering-uClibc 5.x - Developer Guide - Compiling Source Code

From bering-uClibc
Revision as of 23:25, 25 December 2012 by Kapeka (Talk | contribs) (Creating the makefile for syslinux)

Jump to: navigation, search
Compiling Source Code
Prev Bering-uClibc 5.x - Developer Guide Next


What is buildtool?

Buildtool is a set of Perl scripts and Perl modules to build LRP files (LEAF Packages) from source. Everything in the build process is automated, from extracting the sources and applying patches to configuring and building the source and creating the LRP Package.

Why bother?

In very early LEAF releases it was usual to compile the sources and build the Packages manually. This could be error-prone, and where upstream source upgrades happened regularly it was tedious to have to repeat builds for new versions manually.

The current practice is for every Package to be generated automatically according to simple configuration definitions. This means that no special knowledge is required to re-build a particular Package, because all of the steps are captured in the configuration files. Some effort is required to prepare the initial buildtool configuration but many Packages are fairly "standard" and it is often possible to adapt the configuration from a similar package quite quickly.

Buildtool usage


usage: ./ [option] command [pkgname|srcname] [...]

describe [pkgname|srcname]       shows descriptionlines of package
list [sourced|built]             shows a list of built/sourced packages and sources
dumpenv [pkgname|srcname]        dump the environment of buildtool and package
source [pkgname|srcname]         downloads, unpacks and patches
                                the wanted package/source
build [pkgname|srcname]          the same as source, but builds
                                and installs sources/packages also
pkglist [pkgname|srcname]        create a list with all dependencies
                                for the package given or all if no name given
buildclean [pkgname|srcname]     removes everything that is outside
                                the source dir
srcclean [pkgname|srcname]       same as buildclean + call make srcclean
remove [pkgname|srcname]         same as buildclean + remove everything from dldir
distclean                        remove everything
maketar                          make a tar for distribution
-v                               just print version and exit
-f                               allows you to force the command even if the internal
                                state of buildtool states it has nothing to do
-O                               Do not override default Server entries with the ones
                                found in package/source buildtool config
-D                               Download nothing, use files in Source dir (useful for devel)
-d                               Only to be used in conjunction with the "source" target. 
                                Only download files, don't invoke the source action on
-t toolchain                     Build using the specified toolchain
                                (or actually build that toolchain if srcname = "toolchain")
                                Value of "toolchain" is e.g. i486-unknown-linux-uclibc

Getting information about the sources that can be built

Buildtool can only build sources/packages that it has a configuration for (see the next section on how to create such a configuration). To get a listing of the sources that are configured for buildtool type:

 ./ describe

This will give you a list (and a short description) of all sources/packages that can be built. An example output looks like this:

 ./ describe

 The following packages and sources are available:

 openssh          openssh
 tcp_wrappers     Wietse Venema's TCP wrappers library
 helloworld       classical example
 openssl          Secure Sockets Layer and cryptography
                  libraries and tools
 zlib             zLib shared libraries
 linux            kernel source only package

 toolchain         this holds everything to build cross build env
 kernel           kernel pseudo package

The difference between sources and packages

You might be a bit confused why in the above output there are Sources and Packages. The main reason for having sources and packages separately is that there are some sources out there that produce programs used in several packages. Or in other words, it sometimes makes sense to just build the sources (like util-linux for example) only once and create several packages out of the resulting binaries. (Although the developer team did not strictly obey this idea in the past.)

Getting information about sources and packages that already has been built

The command

./ list

shows a list of already build packages/sources.

Downloading the sources for a package


./ source somepackageName

will cause the sources of somepackageName to be "downloaded" (and before that, all its dependencies, if there are any that haven't been downloaded yet). Upstream sources are now copied into the Bering-uClibc 5.x Git repository, since this avoids problems when upstream download sites go off-line and makes it easier to distribute copies of all the source code as required for compliance with some Open Source licenses. As a result, these sources are "downloaded" (from the Internet) when the Sourceforge-hosted Git repository is "cloned", but an important further step is to create symbolic links to the "local repository" (directory repo/) from the source/ directory, so this is what now happens when sources are "downloaded".

After that, the makefile for this source is executed for target source. This has the effect that the sources will be extracted, and patches (if there are any) are applied.

If all this is completed without error, the sources of somepackageName are ready to be compiled.

Compiling the sources for a package


./ build somepackageName

will cause buildtool to actually compile the source and install it to the build/ directory (and all its dependencies). After this is completed successfully, all binaries required for the lrp package (linked against uClibc) reside in the build/[toolchain] directory.

Note that [toolchain] in the example above will be replaced with the toolchain you choosed in conf/buildtool.conf as toolchain. By default it points to i486-unknown-linux-uclibc.

Creating buildtool sources/packages


For each package, five to six files control how it's built:

This file defines which files to include - usually the servers configuration file conf/servers.cfg, local definitions in conf/sources.local and all configuration files in the directory conf/sources.d.
Defines the servers where the files for the packages are downloaded from. The most used "server" location is localrepo, which points to your local copy of LEAF's git repository. That way you can build the toolchain and the packages even if an upstream site is not available. Therefor, and because it's a requirement of Sourceforge, when hosting LEAF, that all sources have to be available, you usually don't have to touch this file. Instead add the sources along with the other files needed to build a package into the git repository.
The conf/sources.d directory holds the configuration files for sources and packages. For each package/source a new file with the package/source name in conf/sources.d with the extension cfg has to be created. The configuration file defines the Server (referring to a server defined in conf/servers.cfg) where all files belonging to a package (or source) can be downloaded from (usually localrepo), the Directory on that server, the Revision (usually HEAD) a short Description of the package (source) and the dependencies for that package (source). All other files that are part of a package and should be downloaded from somewhere have to be explicitly named in the buildtool.cfg which is specific to the package.
New since Bering-uClibc 4.x there is a fourth file, conf/sources.local, which is "included" at the end of conf/sources.cfg and is used to define local sources/packages, typically those being developed and debugged and not yet ready to be loaded into the master repository.

In addition there are two more package specific files:

(package specific - will be downloaded with the information from conf/sources.d/packagename.cfg). Defines where to get the sources/patches and also has the definitions for the package itself (used by buildpacket).
(package specific - will be downloaded with the information from buildtool.cfg). Makefile for actually building the sources.

Step by step guide to creating a simple config

This section should give you a quick glance at how the configuration for a specific source is created (but it won't show the hours of fighting with some source because it refuses to build - you will experience that soon enough when you create your own package. I don't want to discourage you, but cross-compiling often is more difficult than building on the target system). For a more complete overview, please see the following sections of this chapter.

For this exercise, we will create a package called hdsupp.lrp, which contains utilities for preparing a harddisk or Compact Flash disk for running under Bering uClibc. This package will contain fdisk, syslinux and mkfs.msdos.

Because there is no combined tarball for syslinux, fdisk and mkfs.msdos, we will have to create 3 different source definitions for buildtool. We will start with syslinux, since that is the easiest one.

By the time you read this, hdsupp will already be part of the buildtool checkout. This means that the settings for syslinux, util-linux and dosfstools in conf/sources.d will already be there. You can decide to either just skip that section, or (for the sake of learning something) delete that part of conf/sources.d and enter the information again manually. The following sections will assume that buildtool knows nothing about syslinux, util-linux and dosfstools.

Creating the source definition for syslinux, util-linux, dosfstools

First of all we need to create the files conf/sources.d/syslinux.cfg, conf/sources.d/util-linux.cfg and conf/sources.d/dosfstools.cfg containing a source definition for syslinux, util-linux, dosfstools.

We start with conf/sources.d/syslinux.cfg

<Source syslinux>
  Server = localrepo
  Directory = syslinux
  Description = Syslinux boot-loader plus an MBR
    Name  = toolchain

followed by conf/sources.d/util-linux.cfg

<Source util-linux>
  Server = localrepo
  Directory = util-linux
  Description = provides fdisk and mkfs.minix
    Name  = toolchain

and finally conf/sources.d/dosfstools.cfg

<Source dosfstools>
  Server = localrepo
  Directory = dosfstools
  Description = provides mkfs.dos
    Name  = toolchain

For now, we'll just pretend that the buildtool.cfg file will be in the leaf Git repository. In this section, we will create everything needed for syslinux - the other sources will be taken care of later.

Next, we need to create a directory called syslinux in repo. Doing

mkdir repo/syslinux

will do the job. This directory will contain buildtool.cfg, and the syslinux sources.

Create a new file repo/syslinux/buildtool.cfg and enter the following information.

  Server = localrepo
  Directory = syslinux

<File syslinux-4.03.tar.bz2>
  Server = localrepo
  envname = SOURCE_TGZ

The two server definitions tell buildtool which server it should download the files from. In almost every case it is "localrepo" which refers to your local copy of the Bering-uClibc 5.x Git repository. Next we define the file definitions themselves.

Note the envname parameter - with this definition in the cfg-file, buildtool will set an environment variable (named SOURCE_TGZ in this case) containing the filename of the syslinux sources (syslinux-4.03.tar.bz2 in our example). This saves us from having to hardcode any version-specific information into the makefile. Specifying envname is not mandatory, but it keeps things simple for upgrading to a new version of a source.

Creating the makefile for syslinux

Obviously, this guide will not be able to teach you how to write makefiles, or what to do when there are errors during build. We'll simply focus on creating a makefile that produces the binaries and conforms to what buildtool expects.

Create a new file repo/syslinux/ and enter the following contents:

 # buildtool makefile for syslinux

 SOURCE_DIR:=$(shell $(BT_TGZ_GETDIRNAME) $(SOURCE_TGZ) 2>/dev/null )
 ifeq ($(SOURCE_DIR),)
 SOURCE_DIR:=$(shell cat DIRNAME)
   -mkdir $(TARGET_DIR)

   export LANG=en_US  # Hack for Redhat 9
   export CC=$(TARGET_CC)
   $(MAKE) -C $(SOURCE_DIR) syslinux
   cp -a $(SOURCE_DIR)/mbr.bin $(TARGET_DIR)/usr/bin/mbr.bin
   mkdir -p $(BT_STAGING_DIR)/usr/bin
   cp -a $(TARGET_DIR)/usr/bin/syslinux $(BT_STAGING_DIR)/usr/sbin/
   cp -a $(TARGET_DIR)/usr/bin/mbr.bin $(BT_STAGING_DIR)/usr/sbin/

   rm -rf $(TARGET_DIR)
   $(MAKE) -C $(SOURCE_DIR) clean

As with all makefiles, make sure that the delimiter in front of the commands (like $(MAKE) -C $(SOURCE_DIR) clean above) is a tab and not spaces.

Creating the source definition for fdisk

On Debian, fdisk (and also mkfs.minix, which we will use for the hdsupp package as well) is part of the util-linux package. So, we use the sources that are luckily already provided in the Bering-uClibc local repository.

Create a directory repo/util-linux/ and a new file repo/util-linux/buildtool.cfg and enter the following information.


Prev Up Next