[Text of a post to comp.lang.lisp.]

So I was thinking about Ron Garret's lexicons, as I had once thought
about and played around with T locales back in the mid-1980s, and
having the same experience I had back then: that they were
interesting, but they didn't seem to do all the things I wanted them
to and thought they should, somehow, be able to do.  I had puzzled
over this, back then, but never managed to come up with a better
mechanism.

And then, working on another project, I was writing a piece of code of
the form

(defun foo (...)
  (let (... some bindings ...)
    (labels (... lots of clauses ...)
      ...))

and I had the thought -- wouldn't it be nice if there were some way to
write this so that I could call the various `labels' functions
interactively, for testing and debugging?  And then I had an idea.
What if, instead of writing

(defun foo (x)
  (let ((y (bar x)))            ; `bar' is a global function
    (labels ((baz (z) ... x ... y ...))
      (baz (quux x)))))         ; `quux' is a global function

we could instead write the two top-level forms

(defcontext foo-context (x)

  (deflex y (bar x))            ; a lexical `defvar'

  (defun baz (z) ... x ...  y ...))

(defun foo (q)
  (with-context (foo-context q)
    (baz (quux x)))

Here the `defcontext' declares, not a lexical environment, but a
lexical environment _generator_ -- a function that constructs lexical
environments with a known set of bound names (in this case, `x', `y',
and `baz').  `with-context' does two things: it invokes the generator
with the supplied argument(s) to construct a "context instance", and
it arranges for its body to be evaluated in the lexical environment
that binds the names declared in the context to the values in the
instance.

Notice that the body of `baz' is written in a lexical environment
containing the bindings of `x' and `y'.  The effect is similar to that
of dynamic binding; consider for comparison (ignoring `y' for the
moment):

(defvar *x*)

(defun baz1 (z) ... *x* ...)

(defun foo1 (q)
  (let ((*x* q))
    (baz (quux *x*))))

This will do the same thing in many cases.  But the context
implementation is absolutely lexical; you could write

(defun foo (q)
  (with-context (foo-context q)
    #'baz))

and get back a function which is closed over the supplied value of
`x' (the value which was passed to `foo' as `q').

The primary price to be paid for this is that where `baz1', the
dynamic version, can be compiled separately, the entire context `foo-
context' has to be compiled at one time.  If one were to make heavy
use of `defcontext', one could easily wind up with a large source file
consisting entirely of a single `defcontext' form.  This will
certainly slow down incremental compilation a bit.  On modern
hardware, though, I think it's unlikely to be a problem (though I have
yet to try it).

And there's an upside too.  Since all the `defun' forms in a single
`defcontext' body are gathered into a single `labels', you get what
amounts to automatic block compilation -- the calls between these
routines have less overhead than calls between ordinary top-level
functions.  (In most implementations, at least.)  Also, the references
to the context variables -- its parameters plus those additional
variables defined with `deflex' -- are quite fast; there's no need to
deal with the overhead of special binding (which is not as cheap as it
used to be, in this brave new SMP world).

Some may recall that I have previously expressed in this forum my
dislike of dynamic binding, my feeling that it's a hack, that obscures
the true structure of the program and is at least a little bit bug-
prone.  Yet the alternatives that have been available have not been
entirely satisfactory -- the best, usually, has been to gather up the
desired bits of context into an object and pass this object around
explicitly everywhere it's needed.  I am glad to finally have an
alternative to suggest that permits the same elegance of notation as
dynamic binding without (what I see as) the downside.

Another subtopic I'd like to raise before I turn to the implementation
details.  I have seen, on various occasions and in various places
around the Web, people criticizing CL for not having a "real" module
system.  The package system, these people say, doesn't cut it.  I have
some sympathy with their position.  So it's natural to wonder, should
I be calling these "modules" rather than "lexical contexts"?  It's
tempting.  But when I think about features what a "real" module system
should have, one of them is the ability for modules to define types,
and I haven't thought enough yet about how to do that.  I also need to
do some background reading on other module systems to see if there's
any other functionality worth including.

Unfortunately, while CL has notions of variable names, function names,
and even macro names with non-global scopes, it does not have any
notion of a type name with a non-global scope.  Types are normally
named by symbols, which means that the only scoping mechanism for them
is the package; it also means that type names live in their own
namespace.  It's possible this could be circumvented, at least
partially, by thinking of types as the values of variables rather than
global names in the type namespace.  CL makes some provision for this
by allowing classes (that is, class objects) to be used as type
specifiers; but the support is not complete -- it's not the case that
every type specifier has a corresponding class, and it's also not the
case that classes which are the values of variables (and may not be
named at all, or not with the same name as the variable) can be used
in every context (consider `defmethod' parameter specializers).  So
I'm not sure what can be done here.

So anyway, I don't yet think of this as a module system.  Just to put
it in context: CL already has two other modularity mechanisms,
packages and classes.  I certainly don't think that these lexical
contexts are going to supplant classes, despite a certain
resemblance.  And they can't supplant packages either.  My sense is
that lexical contexts complement these other mechanisms rather than
subsuming them.

Well, there's more I could say about that, but for now I want to move
on to my initial implementation.  Here's what I have.

The macro `defcontext' takes, as you see in the examples above, a
parameter list and a body.  The parameter list is just like the
ordinary parameter list of a function, with one exception (ha, you
guys are going to love this! [donning flame-retardant suit]): if you
want the parameter name to be bound in the function namespace rather
than the variable namespace, you can indicate that by marking the
parameter name with #'.  That is, such a parameter can be referenced
within the body of the `defcontext' as a function name rather than a
variable name; for instance

  (defcontext foo (x #'f)
    (defun bar () (f x)))

Why did I do this?  Obviously, it isn't strictly necessary.  I just
thought it would be common to want to supply functional parameters to
a context and that it would be nice to be able to invoke them without
`funcall', and this occurred to me.  The only little weirdness is that
you can't use `function' as the name of an optional or keyword
parameter; conversely, if you have an optional or keyword function
parameter, you must supply a default (`nil' wouldn't make much sense
anyway).  Since the two situations -- an optional parameter named
`function' with a default, or an optional function parameter with no
default -- are indistinguishable, the implementation disallows either.

The body of the `defcontext' must contain only these kinds of forms:

  (import-context (context-name arg0 arg1 ...))

The named context is instantiated with the supplied arguments; the
names it exports are made available to code in the invoking context.
(Currently there is no way to get any of the imported names re-
exported from the invoking context; there should be.)  If two `import-
context' forms import the same name, the later one shadows the earlier
one.

  (deflex name init-form ["doc-string"])

Defines a lexical variable.  The init-form is evaluated in an
environment that includes the parameters, all names imported via
`import-context', and all preceding `deflex' variables (but _not_
functions defined with `defun').  The variable name will be exported
from the context.

  (defmacro ...)

Defines a macro.  The macro will be visible to the init-forms of
`deflex' forms as well as to the bodies of `defun' forms.  It will
also be exported from the context.  The macro bodies can see only the
context parameters and imported names.

  (defun ...)

Defines a function.  Functions defined in a context may be mutually
recursive.  The name will be exported from the context.

I could allow these forms to be intermixed arbitrarily, but I thought
it might lead to confusion -- users might think that by putting a
`deflex' after a `defun', they could make the function available to
the variable's init-form -- so the implementation restricts the
ordering: `import-context' forms must come first, followed by `deflex'
and `defmacro' forms in any combination (I figure these don't tend to
interact much), followed by `defun'.

To use a context, all you have to do, as shown in the examples above,
is to use `with-context' to instantiate it and enter the lexical
environment containing its exported names.

Oh, one more thing.  `with-context' forms have a compile-time
dependency on the corresponding `defcontext' similar to that which a
struct accessor invocation has on the corresponding `defstruct'.  The
implementation goes to some effort to minimize this, however.  As long
as fasl continuity is maintained for the `defcontext' form, it can be
edited and recompiled arbitrarily without necessarily having to
recompile the `with-context' forms that reference it.  The only
strange thing that can happen is if one version of the `defcontext' is
compiled that exports a name `foo', a `with-context' is compiled that
references `foo', and then a new version of the `defcontext' is
compiled that no longer contains `foo'; in that case, when the `with-
context' is executed, the value provided for `foo' will be a symbol
whose name contains a message telling you to recompile the `with-
context'.  (This recompilation will fail, since `foo' is no longer
exported(*), but the compiler will give you a nice error to that
effect.)

(Obviously there are other ways to get your code out of sync, but
they're all familiar; e.g., changing the context parameter list
incompatibly.)

(* Unless there's an outer `foo', previously shadowed, that now
becomes visible, of course.)

So, here it is.  This code is EXPERIMENTAL.  Use At Your Own Risk,
Your Mileage May Vary, Post No Bills, etc.

The best way to see how it works is to macroexpand some `defcontext'
forms. 
