diff options
Diffstat (limited to 'docs/DriverInternals.rst')
| -rw-r--r-- | docs/DriverInternals.rst | 400 |
1 files changed, 400 insertions, 0 deletions
diff --git a/docs/DriverInternals.rst b/docs/DriverInternals.rst new file mode 100644 index 0000000..c779555 --- /dev/null +++ b/docs/DriverInternals.rst @@ -0,0 +1,400 @@ +========================= +Driver Design & Internals +========================= + +.. contents:: + :local: + +Introduction +============ + +This document describes the Clang driver. The purpose of this document +is to describe both the motivation and design goals for the driver, as +well as details of the internal implementation. + +Features and Goals +================== + +The Clang driver is intended to be a production quality compiler driver +providing access to the Clang compiler and tools, with a command line +interface which is compatible with the gcc driver. + +Although the driver is part of and driven by the Clang project, it is +logically a separate tool which shares many of the same goals as Clang: + +.. contents:: Features + :local: + +GCC Compatibility +----------------- + +The number one goal of the driver is to ease the adoption of Clang by +allowing users to drop Clang into a build system which was designed to +call GCC. Although this makes the driver much more complicated than +might otherwise be necessary, we decided that being very compatible with +the gcc command line interface was worth it in order to allow users to +quickly test clang on their projects. + +Flexible +-------- + +The driver was designed to be flexible and easily accommodate new uses +as we grow the clang and LLVM infrastructure. As one example, the driver +can easily support the introduction of tools which have an integrated +assembler; something we hope to add to LLVM in the future. + +Similarly, most of the driver functionality is kept in a library which +can be used to build other tools which want to implement or accept a gcc +like interface. + +Low Overhead +------------ + +The driver should have as little overhead as possible. In practice, we +found that the gcc driver by itself incurred a small but meaningful +overhead when compiling many small files. The driver doesn't do much +work compared to a compilation, but we have tried to keep it as +efficient as possible by following a few simple principles: + +- Avoid memory allocation and string copying when possible. +- Don't parse arguments more than once. +- Provide a few simple interfaces for efficiently searching arguments. + +Simple +------ + +Finally, the driver was designed to be "as simple as possible", given +the other goals. Notably, trying to be completely compatible with the +gcc driver adds a significant amount of complexity. However, the design +of the driver attempts to mitigate this complexity by dividing the +process into a number of independent stages instead of a single +monolithic task. + +Internal Design and Implementation +================================== + +.. contents:: + :local: + :depth: 1 + +Internals Introduction +---------------------- + +In order to satisfy the stated goals, the driver was designed to +completely subsume the functionality of the gcc executable; that is, the +driver should not need to delegate to gcc to perform subtasks. On +Darwin, this implies that the Clang driver also subsumes the gcc +driver-driver, which is used to implement support for building universal +images (binaries and object files). This also implies that the driver +should be able to call the language specific compilers (e.g. cc1) +directly, which means that it must have enough information to forward +command line arguments to child processes correctly. + +Design Overview +--------------- + +The diagram below shows the significant components of the driver +architecture and how they relate to one another. The orange components +represent concrete data structures built by the driver, the green +components indicate conceptually distinct stages which manipulate these +data structures, and the blue components are important helper classes. + +.. image:: DriverArchitecture.png + :align: center + :alt: Driver Architecture Diagram + +Driver Stages +------------- + +The driver functionality is conceptually divided into five stages: + +#. **Parse: Option Parsing** + + The command line argument strings are decomposed into arguments + (``Arg`` instances). The driver expects to understand all available + options, although there is some facility for just passing certain + classes of options through (like ``-Wl,``). + + Each argument corresponds to exactly one abstract ``Option`` + definition, which describes how the option is parsed along with some + additional metadata. The Arg instances themselves are lightweight and + merely contain enough information for clients to determine which + option they correspond to and their values (if they have additional + parameters). + + For example, a command line like "-Ifoo -I foo" would parse to two + Arg instances (a JoinedArg and a SeparateArg instance), but each + would refer to the same Option. + + Options are lazily created in order to avoid populating all Option + classes when the driver is loaded. Most of the driver code only needs + to deal with options by their unique ID (e.g., ``options::OPT_I``), + + Arg instances themselves do not generally store the values of + parameters. In many cases, this would simply result in creating + unnecessary string copies. Instead, Arg instances are always embedded + inside an ArgList structure, which contains the original vector of + argument strings. Each Arg itself only needs to contain an index into + this vector instead of storing its values directly. + + The clang driver can dump the results of this stage using the + ``-ccc-print-options`` flag (which must precede any actual command + line arguments). For example: + + .. code-block:: console + + $ clang -ccc-print-options -Xarch_i386 -fomit-frame-pointer -Wa,-fast -Ifoo -I foo t.c + Option 0 - Name: "-Xarch_", Values: {"i386", "-fomit-frame-pointer"} + Option 1 - Name: "-Wa,", Values: {"-fast"} + Option 2 - Name: "-I", Values: {"foo"} + Option 3 - Name: "-I", Values: {"foo"} + Option 4 - Name: "<input>", Values: {"t.c"} + + After this stage is complete the command line should be broken down + into well defined option objects with their appropriate parameters. + Subsequent stages should rarely, if ever, need to do any string + processing. + +#. **Pipeline: Compilation Job Construction** + + Once the arguments are parsed, the tree of subprocess jobs needed for + the desired compilation sequence are constructed. This involves + determining the input files and their types, what work is to be done + on them (preprocess, compile, assemble, link, etc.), and constructing + a list of Action instances for each task. The result is a list of one + or more top-level actions, each of which generally corresponds to a + single output (for example, an object or linked executable). + + The majority of Actions correspond to actual tasks, however there are + two special Actions. The first is InputAction, which simply serves to + adapt an input argument for use as an input to other Actions. The + second is BindArchAction, which conceptually alters the architecture + to be used for all of its input Actions. + + The clang driver can dump the results of this stage using the + ``-ccc-print-phases`` flag. For example: + + .. code-block:: console + + $ clang -ccc-print-phases -x c t.c -x assembler t.s + 0: input, "t.c", c + 1: preprocessor, {0}, cpp-output + 2: compiler, {1}, assembler + 3: assembler, {2}, object + 4: input, "t.s", assembler + 5: assembler, {4}, object + 6: linker, {3, 5}, image + + Here the driver is constructing seven distinct actions, four to + compile the "t.c" input into an object file, two to assemble the + "t.s" input, and one to link them together. + + A rather different compilation pipeline is shown here; in this + example there are two top level actions to compile the input files + into two separate object files, where each object file is built using + ``lipo`` to merge results built for two separate architectures. + + .. code-block:: console + + $ clang -ccc-print-phases -c -arch i386 -arch x86_64 t0.c t1.c + 0: input, "t0.c", c + 1: preprocessor, {0}, cpp-output + 2: compiler, {1}, assembler + 3: assembler, {2}, object + 4: bind-arch, "i386", {3}, object + 5: bind-arch, "x86_64", {3}, object + 6: lipo, {4, 5}, object + 7: input, "t1.c", c + 8: preprocessor, {7}, cpp-output + 9: compiler, {8}, assembler + 10: assembler, {9}, object + 11: bind-arch, "i386", {10}, object + 12: bind-arch, "x86_64", {10}, object + 13: lipo, {11, 12}, object + + After this stage is complete the compilation process is divided into + a simple set of actions which need to be performed to produce + intermediate or final outputs (in some cases, like ``-fsyntax-only``, + there is no "real" final output). Phases are well known compilation + steps, such as "preprocess", "compile", "assemble", "link", etc. + +#. **Bind: Tool & Filename Selection** + + This stage (in conjunction with the Translate stage) turns the tree + of Actions into a list of actual subprocess to run. Conceptually, the + driver performs a top down matching to assign Action(s) to Tools. The + ToolChain is responsible for selecting the tool to perform a + particular action; once selected the driver interacts with the tool + to see if it can match additional actions (for example, by having an + integrated preprocessor). + + Once Tools have been selected for all actions, the driver determines + how the tools should be connected (for example, using an inprocess + module, pipes, temporary files, or user provided filenames). If an + output file is required, the driver also computes the appropriate + file name (the suffix and file location depend on the input types and + options such as ``-save-temps``). + + The driver interacts with a ToolChain to perform the Tool bindings. + Each ToolChain contains information about all the tools needed for + compilation for a particular architecture, platform, and operating + system. A single driver invocation may query multiple ToolChains + during one compilation in order to interact with tools for separate + architectures. + + The results of this stage are not computed directly, but the driver + can print the results via the ``-ccc-print-bindings`` option. For + example: + + .. code-block:: console + + $ clang -ccc-print-bindings -arch i386 -arch ppc t0.c + # "i386-apple-darwin9" - "clang", inputs: ["t0.c"], output: "/tmp/cc-Sn4RKF.s" + # "i386-apple-darwin9" - "darwin::Assemble", inputs: ["/tmp/cc-Sn4RKF.s"], output: "/tmp/cc-gvSnbS.o" + # "i386-apple-darwin9" - "darwin::Link", inputs: ["/tmp/cc-gvSnbS.o"], output: "/tmp/cc-jgHQxi.out" + # "ppc-apple-darwin9" - "gcc::Compile", inputs: ["t0.c"], output: "/tmp/cc-Q0bTox.s" + # "ppc-apple-darwin9" - "gcc::Assemble", inputs: ["/tmp/cc-Q0bTox.s"], output: "/tmp/cc-WCdicw.o" + # "ppc-apple-darwin9" - "gcc::Link", inputs: ["/tmp/cc-WCdicw.o"], output: "/tmp/cc-HHBEBh.out" + # "i386-apple-darwin9" - "darwin::Lipo", inputs: ["/tmp/cc-jgHQxi.out", "/tmp/cc-HHBEBh.out"], output: "a.out" + + This shows the tool chain, tool, inputs and outputs which have been + bound for this compilation sequence. Here clang is being used to + compile t0.c on the i386 architecture and darwin specific versions of + the tools are being used to assemble and link the result, but generic + gcc versions of the tools are being used on PowerPC. + +#. **Translate: Tool Specific Argument Translation** + + Once a Tool has been selected to perform a particular Action, the + Tool must construct concrete Jobs which will be executed during + compilation. The main work is in translating from the gcc style + command line options to whatever options the subprocess expects. + + Some tools, such as the assembler, only interact with a handful of + arguments and just determine the path of the executable to call and + pass on their input and output arguments. Others, like the compiler + or the linker, may translate a large number of arguments in addition. + + The ArgList class provides a number of simple helper methods to + assist with translating arguments; for example, to pass on only the + last of arguments corresponding to some option, or all arguments for + an option. + + The result of this stage is a list of Jobs (executable paths and + argument strings) to execute. + +#. **Execute** + + Finally, the compilation pipeline is executed. This is mostly + straightforward, although there is some interaction with options like + ``-pipe``, ``-pass-exit-codes`` and ``-time``. + +Additional Notes +---------------- + +The Compilation Object +^^^^^^^^^^^^^^^^^^^^^^ + +The driver constructs a Compilation object for each set of command line +arguments. The Driver itself is intended to be invariant during +construction of a Compilation; an IDE should be able to construct a +single long lived driver instance to use for an entire build, for +example. + +The Compilation object holds information that is particular to each +compilation sequence. For example, the list of used temporary files +(which must be removed once compilation is finished) and result files +(which should be removed if compilation fails). + +Unified Parsing & Pipelining +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Parsing and pipelining both occur without reference to a Compilation +instance. This is by design; the driver expects that both of these +phases are platform neutral, with a few very well defined exceptions +such as whether the platform uses a driver driver. + +ToolChain Argument Translation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +In order to match gcc very closely, the clang driver currently allows +tool chains to perform their own translation of the argument list (into +a new ArgList data structure). Although this allows the clang driver to +match gcc easily, it also makes the driver operation much harder to +understand (since the Tools stop seeing some arguments the user +provided, and see new ones instead). + +For example, on Darwin ``-gfull`` gets translated into two separate +arguments, ``-g`` and ``-fno-eliminate-unused-debug-symbols``. Trying to +write Tool logic to do something with ``-gfull`` will not work, because +Tool argument translation is done after the arguments have been +translated. + +A long term goal is to remove this tool chain specific translation, and +instead force each tool to change its own logic to do the right thing on +the untranslated original arguments. + +Unused Argument Warnings +^^^^^^^^^^^^^^^^^^^^^^^^ + +The driver operates by parsing all arguments but giving Tools the +opportunity to choose which arguments to pass on. One downside of this +infrastructure is that if the user misspells some option, or is confused +about which options to use, some command line arguments the user really +cared about may go unused. This problem is particularly important when +using clang as a compiler, since the clang compiler does not support +anywhere near all the options that gcc does, and we want to make sure +users know which ones are being used. + +To support this, the driver maintains a bit associated with each +argument of whether it has been used (at all) during the compilation. +This bit usually doesn't need to be set by hand, as the key ArgList +accessors will set it automatically. + +When a compilation is successful (there are no errors), the driver +checks the bit and emits an "unused argument" warning for any arguments +which were never accessed. This is conservative (the argument may not +have been used to do what the user wanted) but still catches the most +obvious cases. + +Relation to GCC Driver Concepts +------------------------------- + +For those familiar with the gcc driver, this section provides a brief +overview of how things from the gcc driver map to the clang driver. + +- **Driver Driver** + + The driver driver is fully integrated into the clang driver. The + driver simply constructs additional Actions to bind the architecture + during the *Pipeline* phase. The tool chain specific argument + translation is responsible for handling ``-Xarch_``. + + The one caveat is that this approach requires ``-Xarch_`` not be used + to alter the compilation itself (for example, one cannot provide + ``-S`` as an ``-Xarch_`` argument). The driver attempts to reject + such invocations, and overall there isn't a good reason to abuse + ``-Xarch_`` to that end in practice. + + The upside is that the clang driver is more efficient and does little + extra work to support universal builds. It also provides better error + reporting and UI consistency. + +- **Specs** + + The clang driver has no direct correspondent for "specs". The + majority of the functionality that is embedded in specs is in the + Tool specific argument translation routines. The parts of specs which + control the compilation pipeline are generally part of the *Pipeline* + stage. + +- **Toolchains** + + The gcc driver has no direct understanding of tool chains. Each gcc + binary roughly corresponds to the information which is embedded + inside a single ToolChain. + + The clang driver is intended to be portable and support complex + compilation environments. All platform and tool chain specific code + should be protected behind either abstract or well defined interfaces + (such as whether the platform supports use as a driver driver). |
