This is the third in a series of blog posts on how to use VersionVault to help generate accurate and comprehensive Software Bills of Materials (SBOM) for your applications.

The first two blog posts covered authoritative audited builds of your application and 3^rd-party components that are application dependencies.  They demonstrated the wisdom and benefits behind, “Put everything in a VOB!”  These benefits extend to placing your build tools in a VOB, too.

Applying this practice to build tools not only makes it easy to answer the what/where/who/when/how questions to assemble an SBOM, it provides significant additional benefits.  As noted in the previous post, building with 3^rd-party components that are managed in VOBs isolates builds from the environment of the machines performing the build.  Extending this practice to build tools improves isolation yet further.  It no longer matters if the correct version and patch level of the build tools are installed on the build machines (or installed at all).  Baselines of product builds/releases can also include the versions of build tools used.  If VersionVault audited builds are utilized, the build tools will be included in the audit records (configuration records) yielding authoritative traceability to the versions of the tools used (see the “Example Build Audit Report” section in the first blog post of this series for an example).  This practice significantly eases the burden of creating accurate, comprehensive Software Bills of Materials when you release your application.

Structuring Build Tool Imports

There are different ways of structuring the imports of build tools.  Which approach is appropriate depends on the organization of the build tool(s) and the needs of the runtime environment to correctly execute the tool(s).

Forms of Build Tools

There are several common forms of build tools:       

1. Part of location-independent package – Often an individual build tool is part of a package of related build and development tools (e.g. some IDEs [Integrated Development Environments], which might include a C++ compiler bundled with other tools). These tools detect their location at runtime and automatically find other dependencies within the package without any customization.
2. Part of location-dependent package – This form is the same as the previous “location-independent” package but has some configuration information needed at runtime to locate other dependencies within the package.
3. Standalone tool with flexible OS dependencies – Many tools are not bundled with other build tools (e.g. ‘flex’ lexical analyzer generator, ‘bison’ parser generator).
4. Standalone tool with specific OS dependencies – Some tools depend on particular versions of operating system dependencies (e.g. shared libraries) which may not be present on all build machines.

Selecting Build Tool Version

Just like with 3^rd-party components, there are two ways of structuring how the version of a build tool is chosen:

1. SCM configuration-based version selection – With this approach, new releases of the build tool(s) will be overlaid on the predecessor release, changed files checked in, and everything labeled (or baselined) with a descriptive label (e.g. V2.7.5). The config spec for your application builds will specify the version of the component to use (e.g. “element /vobs/tools/…/mytool/… V2.7.5”).
2. Build environment-based version selection – With this approach, new releases of the build tool(s) are placed in an entirely new directory with new elements created for each file. The new directory would be named with the release of the build tool (e.g. “/vobs/tools/…/mytool/v2.7.5”).  The build environment/scripts will specify the directory name to use for the appropriate version of the build tool.

An application may build with mixture of build tools using different approaches, as appropriate.

Build Tool Root Directories

Build Tool Root Directories may be multi-level with each level representing important distinctions.  The top-level directory will typically be the name of the component (e.g. “/vobs/tools/that_build_tool”).  If the component is operating system (OS) and architecture-specific (e.g. binaries for Red Hat Linux 8.x on Intel x86 64-bit), subdirectories may represent the target architecture.  This can be structured based on your preferences.  Examples include:

• …/LINUX/linux_x86_64_rhat8
• …/LINUX/rhat8/x86_64
• …/rhat8_x86_64

Give thought to how many combinations and permutations you will need to support and that will likely influence your choice of structure.

The entire architecture-specific SDK would then be imported in its normal structure in that Build Tool Root Directory.  For an SDK including a compiler for the C++ language, you might expect to see a ‘bin’ directory, an ‘include’ directory, a ‘lib’ directory, and perhaps a ‘man’ directory in the Build Tool Root Directory.

Importing the Build Tool

Most build tools should be imported in their entirety and in their normal structure.  The last-modified-timestamp should be preserved to aid with traceability.  Supply a descriptive comment for checked in files.  Apply a meaningful label to new versions (e.g. “V2.7.5”).

In general, the easiest way to import a build tool is to use ‘clearfsimport’ ( https://help.hcltechsw.com/versionvault/2.0.1/oxy_ex-1/com.ibm.rational.clearcase.cc_ref.doc/topics/clearfsimport.html ).  Run ‘clearfsimport’ in a dynamic view as the VOB owner or root, if possible.  To optimize ‘clearfsimport’ performance, consider running it on the server hosting the VOB using a local view.

Importing Symbolic Links

VersionVault supports symbolic links in VOBs.  However, symbolic links (symlinks) to absolute path names are unlikely to work as intended.  Thus, any absolute symlinks in the “import from” location should be recreated as relative symlinks before the import.  Alternatively, imported absolute symlinks should recreated as relative symlinks after the import.

For example, if you are importing a Java Development Kit (JDK), it might have an absolute symlink:

rwxrwxrwx. 1 root root 11 Apr 16  2020 /opt/jdk/bin/java -> /opt/jdk/jre/bin/java

The symlink in the VOB after the import should look something like this:

rwxrwxrwx. 1 root root 11 Apr 16  2020 /vobs/tools/jdk8/rhat8_x86_64/bin/java -> ../jre/bin/java

Build Tool Version/Provenance/License Information

As part of each import, you should capture pertinent build tool version detail and provenance information.  Refer to my previous post for guidelines on collecting and recording this information.  Also ensure that your use of the imported tool(s) aligns with any licensing requirements.

Organizing Build Tools in VOBs

Giving thought to your unique requirements and reflecting that in the organization of the imported build tools will minimize or eliminate the need to reorganize them in the future.  Refer to my previous post for guidelines.

More Detail about Location-Dependent Build Tools

Location-dependent build tools require some configuration information which is needed at runtime to locate other dependencies within the package.  Determining how to configure the tools to operate in a non-default location involves a bit of research, primarily reading the tools’ documentation.

Some tools provide command line options to supply the configuration information.  For example, the GNU C compiler, ‘gcc’, has a ‘—sysroot’ flag to control where it will search for system include files and libraries (supply the location in the VOB where you imported the appropriate version of those files) and a ‘-Wl,option’ flag to pass other options to the linker:

      –sysroot=dir : Use dir as the logical root directory for headers and libraries.  For example, if the compiler normally searches for headers in /usr/include and libraries in /usr/lib, it instead searches dir/usr/include and dir/usr/lib.

     -Wl,option : Pass option as an option to the linker.  If option contains commas, it is split into multiple options at the commas.  You can use this syntax to pass an argument to the option.

Other tools may require setting one or more environment variables to indicate the location of the dependency(s).

There are two common approaches to supplying the configuration information.

1. Provide configuration information in the build environment.  With this approach, the configuration information is typically provided in variables that are defined globally and referenced by the appropriate build steps/scripts. For instance, in a Makefile, you might define a Make variable something like “CC_DEP_FLAGS = -sysroot=/vobs/sys/LINUX/rhat8.2/x86_64”.  Scripts that invoke ‘gcc’ would include that variable in the execution line (e.g. “gcc $CC_DEP_FLAGS …”).
2. Provide configuration information in a wrapper script. With this approach, the configuration information is supplied in a script executed by the build step instead of invoking the build tool directly.  The wrapper script will define any necessary environment variables and/or command line flags and then invoke the actual build tool.

More Detail about Standalone tools with specific OS dependencies

If a build tool depends on shared libraries that are not normally installed on all build and developer machines, you can either install the missing libraries everywhere or you can import them into a VOB and create a wrapper script that sets environment variables such as LD_LIBRARY_PATH and/or LD_RUN_PATH to point the tool to the appropriate location in the VOB(s) with the imported libraries.

Dealing with old, incompatible build tools

Creating a highly successful software system can be both a blessing and a curse.  The blessings are obvious.  One potential curse is the need to support the software system for decades including on platforms and/or OS releases that may no longer be officially supported by the OS vendor.  Utilizing cross-compilers will sometimes help with this challenge.  Another approach is to continue using the unsupported platform / OS which may work as long as your overall build system continues to function correctly on the old environment.  Sometimes there is another option.

Kernel ABI compatibility

Operating system vendors typically provide binary compatibility across some number of major releases allowing software systems built on older OS releases to run on the next major version or two.  A build tool originally released on MyOS V2.0 might function correctly on MyOS V3.x but not MyOS V4.x or later, depending on the vendor.  Even though the later ABIs (Application Binary Interfaces) may have changed, it’s sometime possible to take advantage of the fact that kernel interfaces are often still compatible.

Although typically not officially supported by the OS vendor, it is sometimes possible to execute an old incompatible build tool on a later OS version by bringing the shared libraries the tool depends on from the old OS over to the later OS.  Tools like ‘ldd -v’ and ‘strace’ can help identify the shared library dependencies.  Import not just the build tool but all of the identified shared libraries into a VOB.  Create a wrapper script that causes the build tool to load the imported shared libraries instead of those installed on the system.  This can typically be done by defining environment variables such as LD_LIBRARY_PATH or LD_RUN_PATH.

While this technique may not always work, sometimes it can greatly ease the challenge of continuing to maintain your software system on older OS versions no longer supported by the vendor.

Conclusion

Building your applications using build tools maintained in VOBs requires a bit of up-front work.  That work usually pays off well by ensuring correct and consistent builds regardless of what is installed on the build or developer machines.  It also allows complete authoritative traceability to the exact versions of all build tools used if you utilize VersionVault’s audited build mechanisms.  This all makes it much easier to generate accurate, comprehensive SBOMs.