## Buildapp - Create executables with SBCL or CCL

Buildapp is an application for SBCL or CCL that configures and saves an executable Common Lisp image. It is similar to cl-launch and hu.dwim.build. Buildapp is available under a BSD-style license. The latest version is 1.5.6, released on November 7th, 2015.

Download shortcut: http://www.xach.com/lisp/buildapp.tgz

## Contents

- Installation
- Example Use
- Overview
- Limitations
- Implementation
- Feedback
- License

## Installation

Buildapp does not require any libraries. To compile it with SBCL you simply run `make install`. To compile with a different lisp, just specify it after the make command: `make install LISP=<my_favorite_lisp>`. For example, you wanted to compile it under CCL you'd say `make install LISP=ccl` By default, it is installed in `/usr/local/bin`; to use another location, use `make DESTDIR=/path install`.

You can also create the buildapp binary by loading the buildapp system with asdf and running `(buildapp:build-buildapp)`.

## Example Use

Here's a small application:

    $ buildapp \
        --eval '(defun main (argv) (declare (ignore argv)) (write-line "Hello, world"))' \
        --entry main \
        --output hello-world
    [undoing binding stack and other enclosing state... done]
    [saving current Lisp image into hello-world:
    writing 6352 bytes from the read-only space at 0x20000000
    writing 4064 bytes from the static space at 0x20100000
    writing 44834816 bytes from the dynamic space at 0x1000000000
    done]

    $ ./hello-world
    Hello, world

The following creates a toy curl-like application. (It's not quite practical, because any errors will land you in the interactive debugger.)

    $ buildapp --output lisp-curl --asdf-path ~/src/clbuild/systems/ \
        --load-system drakma \
        --eval '(defun main (args) (write-string (drakma:http-request (second args))))' \
        --entry main
    ;; loading system sb-grovel (needed by drakma)
    ;;  from /usr/local/lib/sbcl/sb-grovel/
    ;; loading system sb-posix (needed by cl+ssl)
    ;;  from /usr/local/lib/sbcl/sb-posix/
    ;; loading system trivial-gray-streams (needed by chunga, cl+ssl, flexi-streams)
    ;;  from /home/xach/src/clbuild/source/trivial-gray-streams/
    ;; loading system flexi-streams (needed by drakma, cl+ssl)
    ;;  from /home/xach/src/clbuild/source/flexi-streams/
    ;; loading system alexandria (needed by cffi, babel)
    ;;  from /home/xach/src/clbuild/source/alexandria/
    ;; loading system trivial-features (needed by cffi, babel)
    ;;  from /home/xach/src/clbuild/source/trivial-features/
    ;; loading system babel (needed by cffi)
    ;;  from /home/xach/src/clbuild/source/babel/
    ;; loading system cffi (needed by cl+ssl)
    ;;  from /home/xach/src/clbuild/source/cffi/
    ;; loading system cl+ssl (needed by drakma)
    ;;  from /home/xach/src/clbuild/source/cl+ssl/
    ;; loading system sb-bsd-sockets (needed by usocket)
    ;;  from /usr/local/lib/sbcl/sb-bsd-sockets/
    ;; loading system usocket (needed by drakma)
    ;;  from /home/xach/src/clbuild/source/usocket/
    ;; loading system chunga (needed by drakma)
    ;;  from /home/xach/src/clbuild/source/chunga/
    ;; loading system cl-base64 (needed by drakma)
    ;;  from /home/xach/src/clbuild/source/cl-base64/
    ;; loading system puri (needed by drakma)
    ;;  from /home/xach/src/clbuild/source/puri/
    ;; loading system drakma
    ;;  from /home/xach/src/clbuild/source/drakma/
    [undoing binding stack and other enclosing state... done]
    [saving current Lisp image into lisp-curl:
    writing 6352 bytes from the read-only space at 0x20000000
    writing 5472 bytes from the static space at 0x20100000
    writing 61722624 bytes from the dynamic space at 0x1000000000
    done]

    $ ./lisp-curl http://xach.com/
    <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
           "http://www.w3.org/TR/REC-html40/loose.dtd">
    <HTML>
    <HEAD>
    <TITLE>www.xach.com</TITLE>
    </HEAD>
    ...

Here's how the l1sp.org redirection service application is built:

    $ make
    buildapp --output l1sp --entry redirector:main \
                     --asdf-path /opt/l1sp/systems \
                     --require sb-aclrepl \
                     --eval '(pushnew :hunchentoot-no-ssl *features*)' \
                     --load-system swank \
                     --eval '(setf swank:*log-output* nil)' \
                     --load-system redirector
    ;; loading system sb-grovel
    ;;  from /usr/local/lib/sbcl/sb-grovel/
    ;; loading system sb-bsd-sockets
    ;;  from /usr/local/lib/sbcl/sb-bsd-sockets/
    ;; loading system sb-introspect
    ;;  from /usr/local/lib/sbcl/sb-introspect/
    ;; loading system sb-posix
    ;;  from /usr/local/lib/sbcl/sb-posix/
    ;; loading system sb-cltl2
    ;;  from /usr/local/lib/sbcl/sb-cltl2/
    ;; loading system swank
    ;;  from /opt/l1sp/src/slime/
    ;; loading system html-template (needed by redirector)
    ;;  from /opt/l1sp/src/html-template-0.9.1/
    ;; loading system sb-rotate-byte (needed by sb-md5)
    ;;  from /usr/local/lib/sbcl/sb-rotate-byte/
    ;; loading system sb-md5 (needed by redirector)
    ;;  from /usr/local/lib/sbcl/sb-md5/
    ;; loading system cl-who (needed by redirector)
    ;;  from /opt/l1sp/src/cl-who-0.11.1/
    ;; loading system cl-ppcre (needed by hunchentoot, redirector)
    ;;  from /opt/l1sp/src/cl-ppcre-2.0.0/
    ;; loading system url-rewrite (needed by hunchentoot)
    ;;  from /opt/l1sp/src/url-rewrite-0.1.1/
    ;; loading system rfc2388 (needed by hunchentoot)
    ;;  from /opt/l1sp/src/rfc2388/
    ;; loading system md5 (needed by hunchentoot)
    ;;  from /opt/l1sp/src/md5-1.8.5/
    ;; loading system cl-fad (needed by hunchentoot)
    ;;  from /opt/l1sp/src/cl-fad-0.6.2/
    ;; loading system cl-base64 (needed by hunchentoot)
    ;;  from /opt/l1sp/src/cl-base64-3.3.2/
    ;; loading system trivial-gray-streams (needed by flexi-streams)
    ;;  from /opt/l1sp/src/trivial-gray-streams-2006-09-16/
    ;; loading system flexi-streams (needed by chunga)
    ;;  from /opt/l1sp/src/flexi-streams-1.0.7/
    ;; loading system chunga (needed by hunchentoot)
    ;;  from /opt/l1sp/src/chunga-0.4.3/
    ;; loading system hunchentoot (needed by redirector)
    ;;  from /opt/l1sp/src/hunchentoot-0.15.7/
    ;; loading system redirector
    ;;  from /opt/l1sp/src/redirector/
    [undoing binding stack and other enclosing state... done]
    [saving current Lisp image into l1sp:
    writing 6176 bytes from the read-only space at 0x20000000
    writing 4064 bytes from the static space at 0x20100000
    writing 61042688 bytes from the dynamic space at 0x1000000000
    done]

    $ ./l1sp
    ;; Swank started at port: 7717.
    CL-USER(1):

The `(setf swank:*log-output* nil)` eval is needed to avoid problems when the image restarts.

`redirector::main` looks like this:

    (defun main (argv)
      (declare (ignore argv))
      (load "/opt/l1sp/etc/init.lisp")
      (sb-impl::toplevel-repl nil))

Here's an example of the `--dispatched-entry` option, which was inspired by the desire to have a dozen different small utilities embedded in one big executable and called based on the binary name. First, the support files:

    ;;;; utils.lisp

    (defpackage #:utils
      (:use #:cl))

    (in-package #:utils)

    (defun main (argv)
      (let ((name (pathname-name (first argv))))
        (format *error-output*
                "Unknown binary name ~S, try using cl-echo, cl-ls, or cl-true~%"
                name)
        (sb-ext:quit :unix-status 1)))

    ;;;; ls.lisp

    (defpackage #:ls
      (:use #:cl))

    (in-package #:ls)

    (defun main (argv)
      (declare (ignore argv))
      (dolist (file (directory "*.*"))
        (write-line (namestring file))))

    ;;;; echo.lisp

    (defpackage #:echo
      (:use #:cl))

    (in-package #:echo)

    (defun main (argv)
      (format t "~{~A~^ ~}~%" (rest argv)))

    ;;;; true.lisp

    (defpackage #:true
      (:use #:cl))

    (in-package #:true)

    (defun main (argv)
      (declare (ignore argv))
      (sb-ext:quit :unix-status 0))

Buliding it all together looks like this:

    $ buildapp --output utils \
       --load utils.lisp --dispatched-entry /utils:main \
       --load ls.lisp --dispatched-entry cl-ls/ls:main \
       --load echo.lisp --dispatched-entry cl-echo/echo:main \
       --load true.lisp --dispatched-entry cl-true/true:main
    ;; loading file #P"/tmp/demo/utils.lisp"
    ;; loading file #P"/tmp/demo/ls.lisp"
    ;; loading file #P"/tmp/demo/echo.lisp"
    ;; loading file #P"/tmp/demo/true.lisp"
    [undoing binding stack and other enclosing state... done]
    [saving current Lisp image into utils:
    writing 6352 bytes from the read-only space at 0x20000000
    writing 4064 bytes from the static space at 0x20100000
    writing 45223936 bytes from the dynamic space at 0x1000000000
    done]

    $ ln -sf utils cl-ls

    $ ln -sf utils cl-echo

    $ ln -sf utils cl-true

    $ ./cl-ls
    /tmp/demo/cl-true
    /tmp/demo/echo.lisp
    /tmp/demo/ls.lisp
    /tmp/demo/true.lisp
    /tmp/demo/utils
    /tmp/demo/utils.lisp

    $ ./cl-echo Hello world
    Hello world

    $ ./cl-true && echo $?
    0

## Overview

Here is the usage output of buildapp:

    Usage: buildapp --output OUTPUT-FILE [--flag1 value1 ...]

    Required flags:
      --output OUTPUT-FILE      Use OUTPUT-FILE as the name of the executable
                                  to create

    Entry-point flags:
      --entry NAME              Use the function identified by NAME as the
                                  executable's toplevel function. Called
                                  with SB-EXT:*POSIX-ARGV* as its only
                                  argument. If NAME has a colon, it is
                                  treated as a package separator,
                                  otherwise CL-USER is the implied
                                  package.
      --dispatched-entry DNAME  Specify one possible entry function, depending
                                  on the name of the file that is used to
                                  start the application. The syntax of
                                  DNAME is APPLICATION-NAME/ENTRY-NAME. If the
                                  name used to start the executable matches
                                  APPLICATION-NAME, use ENTRY-NAME as the
                                  entry point. This can be used to choose
                                  one of many possible entry points by
                                  e.g. symlinking names to the application
                                  executable. If APPLICATION-NAME is empty, the
                                  specified ENTRY-NAME is used as a default
                                  if no other application names match. There
                                  may be any number of dispatched entry points,
                                  but only one default.

    Action flags:
      --load FILE               Load FILE. CL:*PACKAGE* is bound to the CL-USER
                                  package before loading
      --load-system NAME        Load an ASDF system identified by NAME
      --require NAME            Use CL:REQUIRE to load NAME
      --eval CODE               Use CL:EVAL to evaulate CODE. The code is read
                                  with CL:READ-FROM-STRING in the CL-USER package

    There may be any number of load/load-system/require/eval flags. Each
    is executed in command-line order before creating an executable.

    Load path flags:
      --load-path DIRECTORY     When handling a --load, search DIRECTORY for
                                  files to load
      --asdf-path DIRECTORY     When handling a --load-system, search DIRECTORY
                                  for ASDF system files to load
      --asdf-tree DIRECTORY     When handling a --load-system, search DIRECTORY
                                  and all its subdirectories for ASDF system
                                  files to load
      --manifest-file FILE      When handling a --load-system, read a list of
                                  ASDF system file pathnames from FILE as
                                  possible matching systems.

    There may be any number of load-path/asdf-path/asdf-tree/manifest-file
    flags. They take priority in command-line order.

    Other flags:
      --compress-core           Compress the core or executable; requires
                                  configuration support in SBCL
      --core-only               Make a core file only, not an executable
      --dynamic-space-size MB   Pass a --dynamic-space-size option to SBCL
                                  when building; value is megabytes
      --help                    Show this usage message
      --logfile FILE            Log compilation and load output to FILE
      --sbcl PATH-TO-SBCL       Use PATH-TO-SBCL instead of the sbcl program
                                  found in your PATH environment variable

    For the latest documentation, see http://www.xach.com/lisp/buildapp/

## Limitations

Buildapp is limited in scope. It aims to make the following steps easy:

- Define an application environment by loading files, loading ASDF systems, evaluating code, and using CL:REQUIRE
- Dump an executable image with an arbitrary startup function

By design, it does not handle the following tasks:

- Initialization tasks at app startup (use SB-EXT:*INIT-HOOKS* for that)
- Debugger management (use `*DEBUGGER-HOOK*` for that)
- Command-line processing (use a library such as COMMAND-LINE-ARGUMENTS for that)
- Obtaining libraries (use clbuild, LibCL, cl-librarian, asdf-install, etc. for that)

## Implementation

Buildapp works like this:

1. It processes the command-line and creates an object that captures the command-line requirements: the output file, any eval/load/load-system/require actions, the entry function, etc.
2. It creates a new Lisp file (the dumpfile) with all the commands needed to implement the command-line options.
3. It runs either "sbcl" with `sb-ext:run-program` or "ccl" with `ccl:run-program`. In either case lisp is invoked with no init files (it doesn't read user or system rc files) and loads the dumpfile:
   1. The first few commands of the dumpfile establish a specialized loading environment:
      - The debugger is changed with `*invoke-debugger-hook*` to simply quit with a special exit code instead of entering the normal debugger (in CCL the same behavior is accomplished using the `:application` and `:error-handler` arguments to `ccl:save-application`)
      - Most output (ASDF system compilation output, low-level error messages, etc) is redirected to a log stream; that stream can be directed to a file with the `--logfile` argument
      - Stale compiled files are automatically recompiled with an `:around` method on `asdf:perform`
   2. There are some sanity checks: Is the output file writable? Does this version of sbcl support the required `:save-runtime-options` argument?
   3. The dumpfile performs the eval/load/load-system/require actions. Each operation is evaluated in the `cl-user` package and with a fresh binding of `sb-ext:*invoke-debugger-hook*` in sbcl or `ccl::*debugger-hook*` in ccl to the buildapp debugger function. If the binding is modified, the new value is saved.
   4. The dumpfile clears itself out of the environment:
      - Remove extra ASDF methods with `remove-method`
      - Reset the debugger hook to the saved value or NIL if no value was saved.
      - Delete the dumpfile package with `delete-package`
   5. The dumpfile then creates an executable with `save-lisp-and-die` in sbcl or `ccl:save-application` in ccl. This ends the lisp subprocess.
4. It deletes the dumpfile.

The ASDF central registry is temporarily extended with the `--asdf-path` and `--asdf-tree` arguments at load time, and reverts back to the default central registry value after that. To avoid conflicts with this behavior, changes to the central registry should be done at startup time instead of application load time.

## Feedback

If you have any questions or comments about buildapp, please email me, Zach Beane.

## License

Copyright © 2010 Zachary Beane, All Rights Reserved

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE AUTHOR 'AS IS' AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.