Macros in concert with modules
==============================

One reason that the standard Scheme language does not support a module system yet is the issue of macros and modularity.  There are several issues to deal with:

  * that compilation of code that uses macros requires presence of those macros' definitions, which prevents true separate compilation, because those macros may be from other modules;

  * that a macro's expansion must preserve referential transparency and hygiene, for example in cases where it refers to names from within the module in which it was defined, even if those names weren't exported; and

  * that a macro's code may be arbitrary Scheme code, which in turn can use other modules, so one module's compile-time, when macros are expanded, is another's run-time, when the code used in macros is executed by the expander: this makes a tower of phases of code evaluation over which some coherent control must be provided.

Scheme48's module system tries to address all of these issues coherently and comprehensively.  Although it cannot offer total separate compilation, it can offer incremental compilation, and compiled modules can be dumped to the file system & restored in the process of incremental compilation. (While such facilities are not built-in to Scheme48, there is a package to do this, which will probably be integrated at some point soon into Scheme48.)

Scheme48's module system is also very careful to preserve non-local module references from a macro's expansion.  Macros in Scheme48 are required to perform hygienic renaming in order for this preservation, however; see Explicit renaming macros.  For a brief example, consider the delay syntax for lazy evaluation.  It expands to a simple procedure call:

     (delay EXPRESSION)
       ==> (make-promise (lambda () EXPRESSION)) lisp

     However, make-promise is not exported from the scheme
     structure.  The expansion works correctly due to the hygienic renaming
     performed by the delay macro transformer: when it hygienically
     renames make-promise, the output contains not the symbol but a
     special token that refers exactly to the binding of make-promise
     from the environment in which the delay macro transformer was
     defined.  Special care is taken to preserve this information.  Had
     delay expanded to a simple S-expression with simple symbols, it
     would have generated a free reference to make-promise, which
     would cause run-time undefined variable errors, or, if the module in
     which delay was used had its own binding of or imported a
     binding of the name make-promise, delay's expansion
     would refer to the wrong binding, and there could potentially be
     drastic and entirely unintended impact upon its semantics.

     Finally, Scheme48's module system has a special design for the tower of
     phases, called a reflective tower. (This would be more
     accurately named `syntactic tower,' as it has nothing to do with
     reflection.)  Every storey represents the environment available at
     successive macro levels.  That is, when the right-hand side of a macro
     definition or binding is evaluated in an environment, the next storey
     in that environment's reflective tower is used to evaluate that macro
     binding.  For example, in this code, there are two storeys used in the
     tower:

          (define (foo ...bar...)
            (let-syntax ((baz ...quux...))
              ...zot...)) lisp

          In order to evaluate code in one storey of the reflective tower, it is
          necessary to expand all macros first.  Most of the code in this example
          will eventually be evaluated in the first storey of the reflective
          tower (assuming it is an ordinary top-level definition), but, in order
          to expand macros in that code, the let-syntax must be expanded.
          This causes ...quux... to be evaluated in the second
          storey of the tower, after which macro expansion can proceed, and long
          after which the enclosing program can be evaluated.

          The module system provides a simple way to manipulate the reflective
          tower.  There is a package clause, for-syntax, that simply
          contains package clauses for the next storey in the tower.  For
          example, a package with the following clauses:

               (open scheme foo bar)
               (for-syntax (open scheme baz quux)) lisp

               has all the bindings of scheme, foo, & bar, at the
               ground storey; and the environment in which macros' definitions are
               evaluated provides everything from scheme, baz, &
               quux.

               With no for-syntax clauses, the scheme structure is
               implicitly opened; however, if there are for-syntax clauses,
               scheme must be explicitly opened. (This is actually only
               in the default config package of the default development environment.
               The full mechanism is very general.)  Also, for-syntax clauses
               may be arbitrarily nested: reflective towers are theoretically infinite
               in height.  (They are internally implemented lazily, so they grow
               exactly as high as they need to be.)

               Here is a simple, though contrived, example of using for-syntax.
               The while-loops structure exports while, a macro similar
               to C's while loop.  While's transformer unhygienically
               binds the name exit to a procedure that exits from the loop.
               It necessarily, therefore, uses Explicit renaming macros,
               explicit renaming macros in order to break hygiene; it also, in the
               macro transformer, uses the destructure macro to destructure the
               input form (see Library utilities, in particular, the structure
               destructuring for destructuring S-expressions).

                    (define-structure while-loops (export while)
                      (open scheme)
                      (for-syntax (open scheme destructuring))
                      (begin
                        (define-syntax while
                          (lambda (form r compare)
                            (destructure (((WHILE test . body) form))
                              `(,(r 'CALL-WITH-CURRENT-CONTINUATION)
                                 (,(r 'LAMBDA) (EXIT)
                                   (,(r 'LET) (r 'LOOP) ()
                                     (,(r 'IF) ,test
                                       (,(r 'BEGIN)
                                         ,@body
                                         (,(r 'LOOP)))))))))
                          (CALL-WITH-CURRENT-CONTINUATION LAMBDA LET IF BEGIN)))) lisp

                    This next while-example structure defines an example procedure
                    foo that uses while.  Since while-example has no
                    macro definitions, there is no need for any for-syntax clauses;
                    it imports while from the while-loops structure only at
                    the ground storey, because it has no macro bindings to evaluate the
                    transformer expressions of:

                         (define-structure while-example (export foo)
                           (open scheme while-loops)
                           (begin
                             (define (foo x)
                               (while (> x 9)
                                 (if (integer? (sqrt x))
                                     (exit (expt x 2))
                                     (set! x (- x 1))))))) lisp