# parachute

1.4.0

An extensible and cross-compatible testing framework.

## About parachute

Parachute is a simple-to-use and extensible testing framework.

## How To

In Parachute, things are organised as a bunch of named tests within a package. Each test can contain a bunch of test forms that make up its body. By default `true`, `false`, `fail`, `is`, `isnt`, `is-values`, `isnt-values`, `of-type`, and `finish` are provided as test forms.

    (define-test numbers
      (of-type integer 5)
      (true (numberp 2/3))
      (false (numberp :keyword))
      (is-values (values 0 "1")
        (= 0)
        (equal "1")))

The name of a test can be either a symbol or a string, though both get coerced to a string internally.

    (define-test "A more descriptive name, perhaps")

Tests can have dependencies in order to ensure that you don't get cascade failures. If a test's dependency fails, it is automatically skipped.

    (define-test equality
      :depends-on (numbers)
      (is = 0 1))             ; Bogus

    (define-test arithmetic
      :depends-on (equality)
      (is = 5 (+ 2 3)))

Dependencies can also be logically combined if you require more complicated dependency logic.

    (define-test unexpected-failure-backup-test
      :depends-on (:and equality (:not arithmetic))
      (of-type number (+ 2 3))) ; Maybe everything broke?

Often times it also makes sense to organise tests according to a hierarchy. For example you could have tests that reflect your type hierarchy, or other kind of structure within your system. You can also use this to create simple test suites.

    (define-test suite)

    (define-test test-a
      :parent suite
      #|...|#)

    (define-test (suite test-b)
      #|...|#)

    (define-test other-suite
      #|...|#
      (define-test other-test
         #|...|#))

Sometimes it is then useful to skip children if you know that they are either faulty or incomplete and shouldn't yet be tested as part of the greater scheme.

    (define-test suite
      :skip (test-a))

If you need to skip individual test forms rather than a whole test, you can use the `skip` form.

    (define-test stuff
      (true :pass)
      (skip "Not ready yet"
        (is = 5 (some-unimplemented-function 10))))

If you need to skip tests depending on the implementation, or the presence of other feature combinations, you can use `skip-on`.

    (define-test stuff
      (skip-on (clisp) "Not supported on clisp."
        (is equal #p"a/b/" (merge-pathnames "b/" "a/"))))

In order to ensure that there is no accidental sequential dependency between test forms or children, you can use the `:serial NIL` option, which will shuffle the test forms and children each time before evaluating them.

    (define-test random
      :serial NIL
      (true 1)
      (true 2)
      (true 3)
      (true 4)
      (true 5))

If you need to wrap your test forms in some kind of environment, then the shuffling won't work automatically. However, you can fix this by wrapping the forms in a `with-shuffling` form.

    (define-test random-2
      (let ((a 0))
        (with-shuffling
          (is = 1 (incf a))
          (is = 2 (incf a))
          (is = 3 (incf a)))))

In case your code will cause changes to the global environment, you probably will want to fix it in place to make sure they are restored to their former values after the test completes. Parachute allows you to automatically fix variables, functions, and macros in place.

    (define-test float-format
      :fix (*read-default-float-format*)
      (of-type single-float (read-from-string "0.5"))
      (setf *read-default-float-format* 'double-float)
      (of-type double-float (read-from-string "0.5")))

You can also tell it to hold all the symbols accessible to a certain package in place by specifying a package designator as a keyword, gensym, or string. In any case, note that these fixtures only apply to the test itself, not to any child tests that might be associated.

Using `with-fixtures`, fixing can also be done locally. It expects an evaluated list of bindings to fix in place.

    (define-test float-format2
      (with-fixtures '(*read-default-float-format*)
        (setf *read-default-float-format* 'double-float)
        (of-type double-float (read-from-string "0.5")))
      (of-type single-float (read-from-string "0.5")))

If you have a user-defined binding to a symbol you can also make the fixture system aware of it so that it'll capture the bindings automatically.

    (define-fixture-capture my-binding (symbol)
      (when (my-binding-bound-p symbol)
        (values (my-binding-value symbol) T)))

    (define-fixture-restore my-binding (symbol value)
      (setf (my-binding-value symbol) value))

Sometimes the compiler will already complain for some tests that you expect to fail, for instance when the type inference is too good. In that case you can force Parachute to only compile the test forms when the test is evaluated. This is also useful when you're continuously working on macros and the like and don't want to recompile the test all the time. Parachute does not do this by default in order to give you useful compiler feedback in case you mess up writing your tests, and in order to avoid having to pay the time to invoke the compiler when it usually isn't necessary. Note that the compilation of the forms will be factored into your timing results.

    (define-test compare-numbers
      :compile-at :execute
      (fail (= :5 5)))

Finally, Parachute also allows you to enforce timing constraints on tests to make sure that they complete within a specified limit.

    (define-test too-slow
      :time-limit 0.5
      (sleep 1))

Tests are run under a `report` context, which gathers a bunch of `result` objects that are generated as part of the evaluation and is responsible for presenting them to the user in some hopefully useful manner. The standard report is `plain`, which reports things as text to the REPL.

    (test 'arithmetic)

There are two more report types included in the standard distribution, namely `quiet` and `interactive`. The former should not produce any output or user-interaction whatseover.

    (test 'arithmethic :report 'quiet)

The latter on the other hand will present you the debugger with a handful of useful restarts whenever a test fails. This allows you to iteratively and dynamically develop and refine tests.

    (test 'arithmetic :report 'interactive)

And that should cover most of it. Parachute does not have any fancy ASDF integration, however I don't believe any is needed anyway. Simply modifying your main system and test system as follows should be sufficient.

    (asdf:defsystem main-system
      #|...|#
      :in-order-to ((asdf:test-op (asdf:test-op :test-system))))

    (asdf:defsystem test-system
      #|...|#
      :perform (asdf:test-op (op c) (uiop:symbol-call :parachute :test :test-package)))

This should allow you to run the tests via ASDF like so: `(asdf:test-system :main-system)`.

## Compatibility Layers

Parachute offers "zero-cost switching" by implementing basic compatibility layers that simulate another test framework's syntax and behaviour. This allows you to switch over from another test framework hopefully without having to change a single test in your suite.

Usually the only thing you need to do to get this done is change the name of the framework in your test system dependencies to one prefixed with `parachute-`. However, since there's so many test frameworks, and each layer requires quite a bit of work and investigation, only some are supported.

The layers only mirror the public API of the test frameworks and leave out any extension mechanisms. They also do not attempt to mirror the output format or interaction method by which the other frameworks functioned, as there would be little point to mirroring them exactly.

Compatibility layers exist for:

- 5am
- lisp-unit
- prove
  Note: make sure to both change the `:depends-on` and `:defsystem-depends-on` to `parachute-prove` if you happen to use the ASDF extension.

## Extending Parachute

### Test and Result Evaluation

Parachute follows its own evaluation semantics in order to run tests. Primarily this means that most everything goes through one central function called `eval-in-context`. This functions allows you to customise evaluation based on both what the context is, and what the object being "evaluated" is.

Usually the context is a report object, but other situations might also be conceived. Either way, it is your responsibility to add methods to this function when you add a new `result` type, some kind of `test` subclass, or a new `report` type that you want to customise according to your desired behaviour.

The evaluation of results is decoupled from the context and reports in the sense that their behaviour does not, by default, depend on it. At the most basic, the `result` class defines a single `:around` method that takes care of recording the `duration` of the test evaluation, setting a default `status` after finishing without errors, and skipping evaluation if the status is already set to something other than `:unknown`.

Next we have a result object that is interesting for anything that actually produces direct test results-- `value-result`. Upon evaluation, if the `value` slot is not yet bound, it calls its `body` function and stores the return value thereof in the `value` slot.

However, the result type that is actually used for all standard test forms is the `comparison-result`. This also takes a comparator function and an expected result to compare against upon completion of the test. If the results match, then the test status is set to `:passed`, otherwise to `:failed`.

Since Parachute allows for a hierarchy in your tests, there have to be aggregate results as well, and indeed there are. Two of them, actually. First is the base case, namely `parent-result` which does two things on evaluation: one, it binds `*parent*` to itself to allow other results to register themselves upon construction, and two it sets its status to `:failed` if any of the `children` have failed.

Finally we have the `test-result` which takes care of properly evaluating an actual `test` object. What this means is to evaluate all `dependencies` before anything else happens, and to check the time limit after everything else has happened. If the time limit has exceeded, set the `description` accordingly and mark the result as `:failed`. For its main `eval-in-context` method however it checks whether any of the dependencies have failed, and if so, mark itself as `:skipped`. Otherwise it calls `eval-in-context` on the actual test object.

The default evaluation procedure for a test itself is to simply call all the functions in the `tests` list in a `with-fixtures` environment.

And that describes the semantics of default test procedures. Actual test forms like `is` are created through macros that emit an `(eval-in-context *context* (make-instance 'comparison-result #|...|#))` form. The `*context*` object is automatically bound to the context object on call of `eval-in-context` and thus always refers to the current context object. This allows results to be evaluated even from within opaque parts like user-defined functions.

### Report Generation

Finally we come to the question of how to generate a report and interact with the evaluation process. The most primitive idea for a report is one that doesn't do anything at all, except for perhaps catching stray errors. This is implemented by the `quiet` report object, which only has a single `eval-in-context` `:around` method that has a handler-case around the rest.

It should be possible to get any kind of reporting behaviour you want by adding methods that specialise on your report object to `eval-in-context`. For the simple case where you want something that prints to the REPL but has a different style than the preset `plain` report, you can simply subclass that and specialise on the `report-on` and `summarize` functions that then produce the output you want.

Since you can control pretty much every aspect of evaluation rather closely, very different behaviours and recovery mechanisms are also possible to achieve. One final aspect to note is `result-for-testable`, which should return an appropriate result object for the given testable. This should only return fresh result objects if no result is already known for the testable in the given context. The standard tests provide for this, however they only ever return a standard `test-result` instance. If you need to customise the behaviour of the evaluation for that part, it would be a wise idea to subclass `test-result` and make sure to return instances thereof from `result-for-testable` for your report.

Finally it should be noted that if you happen to create new `result` types that you might want to run using the default reports, you should add methods to `format-result` that specialise on the keywords `:oneline` and `:extensive` for the type. These should return a string containing an appropriate description of the test in one line or extensively, respectively. This will allow you to customise how things look to some degree without having to create a new report object entirely.

## System Information

Version: 1.4.0
Dependencies:

- documentation-utils
- form-fiddle

Author: Nicolas Hafner
License: zlib
Homepage: https://Shinmera.github.io/parachute/

## Definition Index

- ### PARACHUTE

  - ORG.SHIRAKUMO.PARACHUTE
  No documentation provided.
  - EXTERNAL SPECIAL-VARIABLE

    #### *CONTEXT*

    Source

        Variable storing the current context of the evaluation.

        While EVAL-IN-CONTEXT already carries the context as an argument,
        in certain situations like when opaque functions are evaluated, testers
        within the execution context must be able to access the context object
        in order to evaluate properly.

  - EXTERNAL SPECIAL-VARIABLE

    #### *FIXTURE-CAPTURES*

    Source

        A map of names to fixture capture functions.

        See DEFINE-FIXTURE-CAPTURE

  - EXTERNAL SPECIAL-VARIABLE

    #### *FIXTURE-RESTORES*

    Source

        A map of names to fixture restore functions.

        See DEFINE-FIXTURE-RESTORE

  - EXTERNAL SPECIAL-VARIABLE

    #### *PARENT*

    Source

        Variable storing the immediate parent in the current evaluation context.

        This is used in order to register a result within its parent.

  - EXTERNAL CLASS

    #### COMPARISON-RESULT

    Source

        A result that compares against an expected value by some kind of comparator.

        This result sets its status based on the boolean return value of a test
        of its value against an expected, preset value.

        See VALUE-FORM
        See EXPECTED
        See COMPARISON
        See COMPARISON-GEQ
        See VALUE-RESULT
        See VALUE-EXPECTED-P

  - EXTERNAL CLASS

    #### FINISHING-RESULT

    Source

        A result that only passes if its evaluation proceeds without escaping.

        See VALUE-RESULT

  - EXTERNAL CLASS

    #### INTERACTIVE

    Source

        An interactive test report that shows the debugger on every failure, with restarts that allow you to decide what to do next.

        See REPORT

  - EXTERNAL CLASS

    #### MULTIPLE-VALUE-COMPARISON-RESULT

    Source

        A result that compares against a list of expected values and comparators.

        This test sets its status to :passed if, and only if, for every value pair
        from the VALUE, EXPECTED, and COMPARISON lists, the GEQ test succeeds. In
        other words, if we have VALUES (1 "2"), EXPECTED (1.0 "2"),
        COMPARISON (= equal), then the result would be :passed.

        See VALUE-FORM
        See EXPECTED
        See COMPARISON
        See COMPARISON-GEQ
        See MULTIPLE-VALUE-RESULT
        See COMPARISON-RESULT

  - EXTERNAL CLASS

    #### MULTIPLE-VALUE-RESULT

    Source

        A result that carries a list of test result values.

        Typically the VALUE slot is only set once the result is evaluated. Note that
        if the VALUE slot is bound at the time that the result is evaluated, the
        BODY will not be run.

        See BODY
        See VALUE
        See RESULT

  - EXTERNAL CLASS

    #### PARENT-RESULT

    Source

        A result that does not directly perform a test, but rather serves as an aggregate for multiple tests.

        This result will set its status to :FAILED automatically if it notices that
        after evaluation one or more of its child results have the status :FAILED.

        See RESULTS
        See RESULT
        See FIND-CHILD-RESULT
        See RESULTS-WITH-STATUS
        See ADD-RESULT

  - EXTERNAL CLASS

    #### PLAIN

    Source

        A plain-text report that prints test results as they go by as well as a summary of the failures at the end.

        See REPORT
        See OUTPUT
        See REPORT-ON

  - EXTERNAL CLASS

    #### QUIET

    Source

        A quiet report that doesn't report nor do anything special.

        Simply returns the report object on SUMMARIZE.

        See REPORT

  - EXTERNAL CLASS

    #### REPORT

    Source

        Base class for all report objects.

        Reports are responsible for gathering and representing the
        results of a test evaluation. The manner in which the results are
        presented to the user is up to the test in question.

        See PARENT-RESULT
        See SUMMARIZE

  - EXTERNAL CLASS

    #### RESULT

    Source

        Base container object for test results of any kind.

        A result always has:
          EXPRESSION  -- An expression describing what this result is about.
          STATUS      -- One of :UNKNOWN :FAILED :PASSED :SKIPPED.
          DURATION    -- The duration (in seconds) the evaluation took, or NIL.
          DESCRIPTION -- A string describing the result, or NIL.

        See EXPRESSION
        See STATUS
        See DURATION
        See DESCRIPTION

  - EXTERNAL CLASS

    #### TEST

    Source

        Container for test collections.

        Manages all the data that ties different collections together
        and manages the various options you might want to have for tests.

        Note that merely initializing a test instance will not register it
        to be found by FIND-TEST. It will also not tie it in with the rest
        of the tests. An error will be signalled, if a parent is designated
        that does not exist, and warnings will be signalled if dependencies
        are designated that do not exist. The PARENT may be either a test
        designator name, which will be resolved within HOME, or a list of
        a package designator to use as the HOME and a name.

        In order for the test to be tied in and registered to be findable,
        use (SETF (FIND-TEST name home) test). In order to remove it, simply
        use REMOVE-TEST.

        See NAME
        See HOME
        See DESCRIPTION
        See PARENT
        See CHILDREN
        See DEPENDENCIES
        See REFERENCED-DEPENDENCIES
        See FIXTURES
        See TIME-LIMIT
        See SKIPPED-CHILDREN
        See REFERENCED-SKIPS
        See TESTS
        See SERIAL
        See FIND-TEST
        See REMOVE-TEST
        See DEFINE-TEST

  - EXTERNAL CLASS

    #### TEST-RESULT

    Source

        A result object for tests.

        This takes care of properly evaluating the test by following these steps:
        1. Run all tests in the test's DEPENDENCIES as per EVAL-DEPENDENCY-COMBINATION
        2. Check that all dependencies have a :PASSED status as per the dependency
           combination by running CHECK-DEPENDENCY-COMBINATION
        3. For each child in the test's CHILDREN
           create a result by RESULT-FOR-TESTABLE
           if the child is part of the test's SKIPPED-CHILDREN
           mark its result as :SKIPPED and EVAL-IN-CONTEXT it.
           Otherwise EVAL-IN-CONTEXT the child's result directly.

        The methods implemented by PARENT-RESULT and RESULT will in turn
        take care to implement proper status setting and avoiding duplicate
        evaluation.

        See PARENT-RESULT
        See EVAL-DEPENDENCY-COMBINATION
        See CHECK-DEPENDENCY-COMBINATION

  - EXTERNAL CLASS

    #### VALUE-RESULT

    Source

        A result that carries some kind of test result value.

        Typically the VALUE slot is only set once the result is evaluated. Note that
        if the VALUE slot is bound at the time that the result is evaluated, the
        BODY will not be run.

        See BODY
        See VALUE
        See RESULT

  - EXTERNAL FUNCTION

    #### CALL-WITH-FIXTURES

    - FUNCTION
    - FIXTURES
    Source

        Calls the function with the given list of fixtures.

        This ensures that the values are stored before the function is
        called and restored once the function is completed.

        See CAPTURE-FIXTURES
        See RESTORE-FIXTURES
        See WITH-FIXTURE

  - EXTERNAL FUNCTION

    #### CAPTURE-FIXTURES

    - FIXTURE
    Source

        Returns a list of fixtures for the given kind.

        If the given name is a symbol that is not a keyword and has a
        package associated with it, then a list of a single fixture that
        captures every appropriate binding for the symbol is returned.
        Otherwise the name is interpreted as a package for which a list
        of fixtures for every symbol accessible in the package is returned.

        See RESTORE-FIXTURES
        See PACKAGE-FIXTURES
        See DEFINE-FIXTURE-CAPTURE

  - EXTERNAL FUNCTION

    #### FEATUREP

    - EXPR
    Source

        Returns T if the given feature expression matches the set of *FEATURES*

        The expression follows the standard feature syntax. See 24.1.2.1 in the CLHS.

  - EXTERNAL FUNCTION

    #### FIND-TEST

    - NAME
    - &OPTIONAL
    - PACKAGE-ISH
    Source

        Finds the test by name if possible.

        If the name is a symbol, the package defaults to the symbol's package.
        If the name is a string, the package defaults to *PACKAGE*.
        If no explicit package has been given and no test has been found, the
        search is automatically retried with the package set to *PACKAGE*. This
        should avoid confusion in the case of imported symbols and tests defined
        under the local package.

        This function can also be used as a place in order to register a test.
        When a test is set, the preexisting test if any is first removed by
        REMOVE-TEST. This ensures that potentially removed options need to be
        properly updated within other affected tests. After that, the test is
        added to the list of children within its parent, if one is set. Finally
        the test is registered within the test index.

        See REMOVE-TEST
        See *TEST-INDEX*

  - EXTERNAL FUNCTION

    #### (SETF FIND-TEST)

    - TEST-INSTANCE
    - NAME
    - &OPTIONAL
    - PACKAGE-ISH
    Source
    No documentation provided.
  - EXTERNAL FUNCTION

    #### PACKAGE-FIXTURES

    - NAME
    Source

        Returns a list of fixtures for all symbols accessible in the given package.

  - EXTERNAL FUNCTION

    #### PACKAGE-TESTS

    - PACKAGE
    Source

        Returns a list of all the tests defined in the given package.

        Signals an error if the PACKAGE cannot be resolved to a package object.

        See *TEST-INDEXES*

  - EXTERNAL FUNCTION

    #### REMOVE-TEST

    - NAME
    - &OPTIONAL
    - PACKAGE-ISH
    Source

        Removes the test by name if possible.

        When a test is removed, it is also automatically removed from its
        parent's child list. However, weak references such as within a test's
        dependencies cannot be updated as there is no backreference.

        See FIND-TEST
        See *TEST-INDEX*

  - EXTERNAL FUNCTION

    #### RESTORE-FIXTURES

    - FIXTURES
    Source

        Restores the bindings stored in the fixtures to their saved values.

        See CAPTURE-FIXTURES
        See DEFINE-FIXTURE-RESTORE

  - EXTERNAL FUNCTION

    #### TEST

    - DESIGNATOR
    - &REST
    - ARGS
    - &KEY
    - REPORT
    - &ALLOW-OTHER-KEYS
    Source

        Run a test.

        DESIGNATOR is resolved to a list of tests to run under a single
        report. You can specify the report class by the REPORT keyword
        argument. All other arguments are used as initargs for the REPORT
        creation.

        After the report has been created, each test is run via
        EVAL-IN-CONTEXT. Finally SUMMARIZE is called on the report.

        See REPORT
        See RESOLVE-TESTS
        See EVAL-IN-CONTEXT
        See SUMMARIZE

  - EXTERNAL FUNCTION

    #### TEST-PACKAGES

    Source

        Returns a list of all packages that define tests.

        See *TEST-INDEXES*

  - EXTERNAL FUNCTION

    #### TEST-TOPLEVEL

    - DESIGNATOR/S
    - &REST
    - ARGS
    Source

        Run tests as the toplevel.

        This function calls TEST for each of the given test/s and the
        provided arguments. Once all tests have been completed, it will
        cause the implementation to quit.

        The exit code will be 100 if any of the tests failed, and 0
        if all of them passed. This is useful for cases like in automated
        continuous integration environments where the exit code determines
        success or failure of the build.

        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### CHECK-EVALUATABLE

    - CONTEXT
    - TEST

        Checks whether the thing is evaluatable in the context and signals an error if not.

        This is primarily used to error out early on tests that are specified wrong.

        See EVAL-IN-CONTEXT

  - EXTERNAL GENERIC-FUNCTION

    #### CHILDREN

    - OBJECT

        The list of child test instances.

        Note that the return value of this will be shuffled randomly if the test is
        marked as non-serial. Child tests will be evaluated after the test itself.

        See SERIAL
        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF CHILDREN)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### COMPARISON

    - OBJECT

        The comparison function designator that compares the expected and actual values of the test.

        See COMPARISON-RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF COMPARISON)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### COMPARISON-GEQ

    - OBJECT

        The value to which the comparison result must be GEQ to.

        Defaults to T.

        See COMPARISON-RESULT
        See GEQ

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF COMPARISON-GEQ)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### DEPENDENCIES

    - TEST

        The logical combination of dependant tests.

        DEPENDENCIES ::= (LOGOP DEPENDENCY*)
        LOGOP        ::= :OR | :AND | :NOT
        DEPENDENCY   ::= test-object | DEPENDENCIES

        If a dependency is referenced that does not exist, an error is signalled.
        Dependant tests must be evaluated before the test itself.

        See REFERENCED-DEPENDENCIES
        See CHECK-DEPENDENCY-COMBINATION
        See EVAL-DEPENDENCY-COMBINATION
        See RESOLVE-DEPENDENCY-COMBINATION
        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### DESCRIPTION

    - OBJECT

        The description of what the test/result is about as a string or NIL.

        See RESULT
        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF DESCRIPTION)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### DURATION

    - OBJECT

        The duration the evaluation of the result object took in seconds or NIL.

        This is automatically recorded in standard methods for EVAL-IN-CONTEXT.
        The value may be NIL if the test has not yet been evaluated at all.

        See RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF DURATION)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### EVAL-IN-CONTEXT

    - CONTEXT
    - TEST

        Evaluates the thing within the given context.

        This may seem like a rather generic description. In specific, this is mostly
        used in order to provide a customizable evaluation behaviour during the
        evaluation of tests under a given report. Which is to say that usually the
        context will be some report object and the thing being tested some kind of
        result or test instance.

        See REPORT
        See RESULT
        See CHECK-EVALUATABLE

  - EXTERNAL GENERIC-FUNCTION

    #### EXPECTED

    - OBJECT

        A value that should be equal (under some predicate) to what the test evaluates to.

        See COMPARISON-RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF EXPECTED)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### EXPRESSION

    - OBJECT

        The expression that the result object is about.

        See RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF EXPRESSION)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### FIND-CHILD-RESULT

    - TEST
    - RESULT

        Attempts to find the result object associated with the given test.

        This simply tests by EQ against the EXPRESSION of each child within the result.

        See PARENT-RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### FIXTURES

    - OBJECT

        The list of fixed symbols or packages.

        See WITH-FIXTURE
        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF FIXTURES)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### FORMAT-RESULT

    - RESULT
    - TYPE

        Formats the result into the requested representation as a string.

        The representation type can, by default, be either :ONELINE or :EXTENSIVE.

        See RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### HOME

    - OBJECT

        The home package of the test.

        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### NAME

    - OBJECT

        The name of the test, which is always a string.

        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### OUTPUT

    - OBJECT

        The stream that the PLAIN report outputs to during REPORT-ON and SUMMARIZE.

        See PLAIN
        See REPORT-ON
        See SUMMARIZE

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF OUTPUT)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### PARENT

    - OBJECT

        The parent test object, if any.

        If a test has a parent, then it must also itself be contained in the parent's
        children list.

        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF PARENT)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### REFERENCED-DEPENDENCIES

    - OBJECT

        The logical combination of test designators that reference a dependant test.

        DEPENDENCIES ::= (DEPENDENCY*) | (LOGOP DEPENDENCY*)
        LOGOP        ::= :OR | :AND | :NOT
        DEPENDENCY   ::= test-name | (home test-name) | DEPENDENCIES

        See DEPENDENCIES
        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF REFERENCED-DEPENDENCIES)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### REFERENCED-SKIPS

    - OBJECT

        The list of test designators that reference a skipped test.

        See SKIPPED-CHILDREN
        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF REFERENCED-SKIPS)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### REPORT-ON

    - RESULT
    - REPORT

        Causes the result to be printed to standard output under the formatting of the report.

        See REPORT
        See RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### RESULT-FOR-TESTABLE

    - TEST
    - CONTEXT

        Returns an appropriate RESULT instance for the given testable and context.

        You should specialize this if you need special kinds of result types for your
        report.

        See RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### RESULTS

    - OBJECT

        The vector of child results within the parent.

        See PARENT-RESULT
        See ADD-CHILD

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF RESULTS)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### RESULTS-WITH-STATUS

    - STATUS
    - RESULT

        Returns a list of results that are a child of the result and have the requested status.

        See PARENT-RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### SERIAL

    - OBJECT

        Whether the tests and children should be evaluated in order, or randomly.

        This affects the return values of TESTS and CHILDREN.

        See CHILDREN
        See TESTS
        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF SERIAL)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### SKIPPED-CHILDREN

    - TEST

        The list of the children that should be skipped.

        If a child is referenced that does not exist, an error is signalled.
        If a child is within this list, it will not be evaluated after the test
        and instead be marked as skipped automatically.

        See REFERENCED-SKIPS
        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### STATUS

    - OBJECT

        The status of the result object.

        Should be one of :UNKNOWN :FAILED :PASSED :SKIPPED. By default after
        initialization the status is :UNKNOWN. The status is automatically
        changed to :PASSED if the result is still :UNKNOWN after EVAL-IN-CONTEXT
         completes.

        See RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF STATUS)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### SUMMARIZE

    - REPORT

        Should cause the report to produce some kind of summary.

        How this summary is presented and what is contained in it is up to
        the report itself.

        See REPORT.

  - EXTERNAL GENERIC-FUNCTION

    #### TESTS

    - OBJECT

        The list of functions to execute in order to actually perform the tests.

        Note that the return value of this will be shuffled randomly if the test is
        marked as non-serial.

        See SERIAL
        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF TESTS)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### TESTS-WITH-STATUS

    - STATUS
    - REPORT

        Returns a list of TEST objects that have been evaluated and whose reports have the requested status.

        See REPORT
        See RESULTS-WITH-STATUS

  - EXTERNAL GENERIC-FUNCTION

    #### TIME-LIMIT

    - OBJECT

        The time limit of the test, if any.

        In order for the test to succeed, the evaluation must succeed before
        the time limit is reached. Otherwise the test is marked as having failed.

        See TEST

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF TIME-LIMIT)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### VALUE

    - OBJECT

        The value of the result object that is obtained by running its body.

        See BODY
        See VALUE-RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF VALUE)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### VALUE-EXPECTED-P

    - RESULT
    - VALUE
    - EXPECTED
    Source
    No documentation provided.
  - EXTERNAL GENERIC-FUNCTION

    #### VALUE-FORM

    - OBJECT

        A literal representation of the form that produces the value for the test.

        See COMPARISON-RESULT

  - EXTERNAL GENERIC-FUNCTION

    #### (SETF VALUE-FORM)

    - NEW-VALUE
    - OBJECT
    No documentation provided.
  - EXTERNAL MACRO

    #### DEFINE-FIXTURE-CAPTURE

    - NAME
    - ARGS
    - &BODY
    - BODY
    Source

        Defines a capture function to fix a symbol binding.

        A capture function should take a symbol and return a value to
        capture for it, and T as the secondary value if anything should
        be captured at all for the kind of binding the function handles.

        See *FIXTURE-CAPTURES*

  - EXTERNAL MACRO

    #### DEFINE-FIXTURE-RESTORE

    - NAME
    - ARGS
    - &BODY
    - BODY
    Source

        Defines a restore function to fix a symbol binding.

        A restoration function should take a symbol and a value and
        make sure to restore the binding it handles to the given value.

        See *FIXTURE-RESTORES*

  - EXTERNAL MACRO

    #### DEFINE-TEST

    - NAME
    - &BODY
    - ARGUMENTS-AND-BODY
    Source

        Defines a test.

        NAME can be either a string, a symbol, or a list of a parent name and a name.

        BODY ::= KARG* form*
        KARG ::= keyword expression

        Each karg in the body is used as an initialization argument to the test
        instance with the exception of :TEST-CLASS, which is used as the class-name
        to create a test instance with and defaults to TEST. The values of the
        keyword arguments are always passed quoted.

        If you set a parent in the NAME, you cannot use the :PARENT karg and vice-
        versa. The following initargs are automatically always passed by DEFINE-TEST
        and should thus not appear as a karg: :NAME :TESTS :PARENT

        You can define child tests within the body of the test if they appear as
        "toplevel" forms. They will be taken out of the body forms and emitted
        as definition forms after the parent form, with the parent automatically set
        appropriately. As such, subtests defined in this manner cannot cause any
        effects on the surrounding environment.

        Each form within the body will be turned into a separate test function.
        This is done so that the tests can be shuffled randomly if the test is not
        set as serial. This means that if you need to wrap multiple testers in any
        other form, the inner tests cannot be shuffled accordingly. However, you can
        use the WITH-SHUFFLING macro.

        The karg :COMPILE-AT directs whether each test form should be compiled at
        definition time (default, value :COMPILE-TIME) or at test evaluation time
        (value :EXECUTE). The latter is achieved by, instead of a straight-up LAMBDA
        for each test form, it wraps it in a LAMBDA that calls COMPILE on the source
        form.

        The karg :PARENT can be either a test designator, or a list of a package
        designator and a test name. The latter case allows you to set the test of
        another home package as the parent, useful if you split your tests across
        multiple packages.

        Note that the dependencies of the test can be a combined by logic operators
        of :AND :OR and :NOT. See REFERENCED-DEPENDENCIES for the necessary structure.

        If the test object to be defined already exists, the existing instance is
        updated as necessary instead of creating a new one. The update proceeds in two
        steps: If the requested :TEST-CLASS is different from the instance's class the
        disparity is attempted to be reconciled through CHANGE-CLASS. The instance's
        fields are then always updated via REINITIALIZE-INSTANCE. This should ensure
        that, in the normal case of redefining tests via DEFINE-TEST, test objects
        returned by FIND-TEST are EQ to each other under the same arguments.

        See TEST
        See FIND-TEST
        See REMOVE-TEST
        See REFERENCED-DEPENDENCIES

  - EXTERNAL MACRO

    #### FAIL

    - FORM
    - &OPTIONAL
    - TYPE
    - DESCRIPTION
    - &REST
    - FORMAT-ARGS
    Source

        A tester that succeeds if the form signals an error of the requested type, defaulting to ERROR.

  - EXTERNAL MACRO

    #### FALSE

    - FORM
    - &OPTIONAL
    - DESCRIPTION
    - &REST
    - FORMAT-ARGS
    Source

        A tester that succeeds if the form returns NIL as the value.

  - EXTERNAL MACRO

    #### FINISH

    - FORM
    - &OPTIONAL
    - DESCRIPTION
    - &REST
    - FORMAT-ARGS
    Source

        A tester that succeeds if the form returns without escaping.

  - EXTERNAL MACRO

    #### IS

    - COMP
    - EXPECTED
    - FORM
    - &OPTIONAL
    - DESCRIPTION
    - &REST
    - FORMAT-ARGS
    Source

        A tester that succeeds if the value the form returns is equal to the expected value under the given comparator.

  - EXTERNAL MACRO

    #### IS-VALUES

    - FORM
    - &BODY
    - BODY
    Source

        A tester that succeeds if each value the form returns is equal to the corresponding expected value under the corresponding comparator.

        The body consists of lists of two values, the first being the comparator,
        the second being the expected value. A third, ignored value can be
        supplied as well for aesthetic purposes.

  - EXTERNAL MACRO

    #### ISNT

    - COMP
    - EXPECTED
    - FORM
    - &OPTIONAL
    - DESCRIPTION
    - &REST
    - FORMAT-ARGS
    Source

        A tester that succeeds if the value the form returns is not equal to the expected value under the given comparator.

  - EXTERNAL MACRO

    #### ISNT-VALUES

    - FORM
    - &BODY
    - BODY
    Source

        A tester that succeeds if each value the form returns is not equal to the corresponding expected value under the corresponding comparator.

        The body consists of lists of two values, the first being the comparator,
        the second being the expected value. A third, ignored value can be
        supplied as well for aesthetic purposes.

  - EXTERNAL MACRO

    #### OF-TYPE

    - TYPE
    - FORM
    - &OPTIONAL
    - DESCRIPTION
    - &REST
    - FORMAT-ARGS
    Source

        A tester that succeeds if the form returns a value that is of the requested type.

  - EXTERNAL MACRO

    #### SKIP

    - DESC
    - &BODY
    - TESTS
    Source

        Skips the tests in the body by avoiding their evaluation and marking their status as :SKIPPED.

  - EXTERNAL MACRO

    #### SKIP-ON

    - FEATURES
    - DESC
    - &BODY
    - TESTS
    Source

        Skips the tests in the body if any of the given feature expressions match.

        If one of the feature expression matches, the evaluation of the tests in the body is skipped
        and the status of these tests is forced to :SKIPPED. Otherwise, the tests are evaluated as
        normal.

        Note that if you have tests that will not /compile/ on a given feature combination, you still
        need to exclude those tests with reader conditionals as well.

        See FEATUREP

  - EXTERNAL MACRO

    #### TRUE

    - FORM
    - &OPTIONAL
    - DESCRIPTION
    - &REST
    - FORMAT-ARGS
    Source

        A tester that succeeds if the form returns a non-NIL value.

  - EXTERNAL MACRO

    #### WITH-FIXTURES

    - FIXTURES
    - &BODY
    - BODY
    Source

        Shorthand macro to evaluate the body with the fixtures saved away.

        See CALL-WITH-FIXTURES

  - EXTERNAL MACRO

    #### WITH-FORCED-STATUS

    - STATUS
    - &OPTIONAL
    - DESCRIPTION
    - &REST
    - FORMAT-ARGS
    - &BODY
    - TESTS
    Source

        Forces the requested status upon the tests in the body without evaluating any value-results.

  - EXTERNAL MACRO

    #### WITH-SHUFFLING

    - &BODY
    - BODY
    Source

        Causes each form in the body to be evaluated in random order each time the with-shuffling form is evaluated.

        This is done by taking each form, wrapping it in a lambda, shuffling the list of
        lambdas, and then funcalling each in the shuffled order.

        See SHUFFLE