Chapter 41

sandbox.ss: Sandboxed Evaluation

To load: (require (lib "sandbox.ss"))

The main function that is provided by this module is make-evaluator. The rest of this module is mostly for customization and interaction with sandboxed evaluators. This module can be used in both MzScheme and MrEd (in the latter, a separate eventspace is used for the sandbox).

Note: this module does not provide a test-suite framework, but can be used as the evaluation engine for one. Evaluating expressions throws exceptions in the usual ways -- but exceptions are raised for additional problems like accessing the file system, running out of time or memory, etc.

(make-evaluator language requires input-program)      PROCEDURE

This function is used to create an evaluator from a given input-program, using a given language and requires specification. The result is a sandboxed evaluator, working in an environment that is completely protected against malicious or buggy code.

The input-program holds the input program in one of the following ways:

• an input port will be used to read the program;

• a string or a byte string holding the complete input;

• a path that names a file holding the input;

• an S-expression or a syntax value -- used as-is (s-expressions are converted to syntax first, see also get-uncovered-expressions below.). Note that it is a single expressions, not a list, but see below for additional ways of invoking make-evaluator.

In the first three cases, the contents is read using the sandbox-reader (see below), with line-counting enabled for sensible error messages, and with 'program as the source (used for testing coverage). In the last case, the input is expected to be the complete program, and is converted to a syntax (using 'program as the source), unless it already is a syntax. See below for providing multiple arguments.

The language specification can be:

• A symbol indicating a built-in language (currently, only 'mzscheme, 'r5rs, or a symbol indicating a teaching language: 'beginner, 'beginner-abbr, 'intermediate, 'intermediate-lambda, or 'advanced. The teaching languages and the 'r5rs imply additional customization of the environment (currently only read-related parameters¹⁰ are set).

• A list that begins with a 'lib, 'file, or 'planet, which stands for the language defined by this (quoted) module specification, or a string specifying a relative module file name directly.

• A list that begins with a 'begin means that the code will not be evaluated in a module context at all, instead, it will simply be evaluated in a new namespace, after evaluating the expressions in the tail of this list¹¹.

The requires list specifies additional code to load for the input program. It can be one of:

• a list of module specifications to load into the program;

• a list that begins with a 'begin is arbitrary code that is prefixed into the submitted program.

There are two additional ways to call make-evaluator. The first can be used when you want to provide the program as a sequence of S-expressions or syntax values, but you want more than a single form:

(make-evaluator language requires input-program ···)      PROCEDURE

You can also provide no expressions at all, which is a convenient way to get a clean sandbox. For example, to get an empty MzScheme read-eval-print loop, or a module-based evaluator¹²:

(define mz-repl-eval (make-evaluator '(begin) '()))
(define mz-module-eval (make-evaluator 'mzscheme '()))

The '(begin) language specification avoids a module-based read-eval-print loop, as described above.

The third form for calling make-evaluator, is for cases where you have code that is already in a complete module form:

(make-evaluator input-program)      PROCEDURE

The input-program argument must specify code that has a single module form. The form is inspected and determines the language that is to be used. This means that these two evaluators:

(define ev1 (make-evaluator '(module foo mzscheme ...)))
(define ev2 (make-evaluator '(begin) '()
                            '(module foo mzscheme ...)
                            '(require foo)))

are similar, except that the first is a module-based read-eval-print loop.

make-evaluator returns an evaluation function that works in the context of the given program. In most cases, this means that evaluation happens in the namespace of the program's module, unless the language argument is a 'begin expression. '(begin) is therefore a convenient way to get a plain read-eval-print loop.

The evaluation function expects as input a value similar to the input-program argument to make-evaluator: a string or byte string holding a sequence of expressions, a path for a file holding expressions, an S-expression, or a syntax object. If the evaluator receives an eof value, it will be terminated and raise errors from that point on (kill-evaluator terminates the evaluator without raising an error, see below).

The evaluator operates in an isolated and limited environment:

• it uses a new custodian and namespace, under MrEd it is also put in its own eventspace;

• since the evaluator is dynamic, both run-time and syntax-time exceptions can be caught in a uniform way;

• the evaluator works under the sandbox-security-guard that restricts file system and network access (see below);

• each evaluation is wrapped in a call-with-limits (see sandbox-eval-limits and set-eval-limits below).

Evaluation can also be instrumented to track evaluation information if sandbox-coverage-enabled is set.

41.1  Customizing Evaluators

The evaluators that make-evaluator creates can be customized via several parameters. Note that these parameters affect newly created evaluators, changing them does not have an effect on already running evaluators.

(sandbox-init-hook [thunk])      PROCEDURE

A parameter that holds a thunk to be called for initializing a new evaluator. The hook is called just before the program is evaluated, in a newly created evaluator context. It can be used to setup environment parameters related to reading, writing, evaluation, etc. Note that certain languages ('r5rs and the teaching languages) have initializations specific to the language -- the init-hook is used after that initialization, so it is possible to override some settings.

(sandbox-reader [proc])      PROCEDURE

A parameter that holds a one-argument function that reads all expressions from the current-input-port. The function will be used to read program source (unless an S-expression or a syntax object is provided). The reader function will receive a value to be used as input-source, and it should read its current-input and return a list of syntax objects (with the given value for the source). The default reader does a simple loop with (read-syntax src-arg).

(sandbox-input [input-spec])      PROCEDURE

This is a parameter that specifies the input for evaluations that happen in an evaluator. It defaults to #f, which means that evaluators work in a dynamic context where no input is available. It can also be set to:

• values that are accepted as an input-program argument to make-evaluator (except for S-expressions and syntax values): a string or a byte string specify the complete input, an input port is used as is, and a path indicates an input file;

• the symbol 'pipe, which will make it use a pipe for input, and put-input can return the input end of the pipe or write to it;

• a thunk, which will be invoked to get a port (e.g., using current-input-port means that the evaluator input is the same as the calling context's input).

(sandbox-output [output-spec])      PROCEDURE

A parameter that specifies the output for evaluations that happen in a make-evaluator function. It defaults to #f, which simply discards all such output. It can also be set to:

• an output port, which will be used as is;

• the symbol 'bytes, which will make get-output (see below) return the complete output as a byte string;

• the symbol 'string, similar to the above, but uses a string;

• the symbol 'pipe, which will make it use a pipe for output, and get-output returns the input end of the pipe;

• a thunk, which will be invoked to get a port (e.g., using current-output-port means that the evaluator output is not diverted).

(sandbox-error-output [error-spec])      PROCEDURE

This parameter is similar to sandbox-output above, but applies to the error output. (See also get-error-output below.) Note that the sandbox's error output is set after its output, so using current-output-port for this parameter means that it is linked to the output port specified by sandbox-output.

The default is current-error-port which means that the error output of the generated evaluator goes to the calling context's error port.

(sandbox-coverage-enabled [thunk])      PROCEDURE

This is a boolean parameter that controls whether syntactic coverage information is collected by sandbox evaluators. If it set to true, the mzlib/private/sandbox-coverage.ss module will be required at the new sandbox top-level so coverage information is collected. You can retrieve this information using the get-uncovered-expressions function that is described below.

(sandbox-namespace-specs [specs])      PROCEDURE

A parameter that holds a list of values that specify how to create a namespace for evaluation in make-evaluator. The first item in the list is a thunk that creates the namespace, and the rest are require specs for modules that are to be attached to the created namespace (using namespace-attach-module). The default is namespace creator function is make-namespace if running in MzScheme, or make-namespace-with-mred if in MrEd, and no module specs in both cases.

The module specs are needed for sharing module instantiations between the sandbox and the caller. For example, sandbox code that return posn values (from lang/posn.ss) will not be recognized as such by your own code by default, since the sandbox will have its own struct type for posns. To be able to use such values, you should have (lib "posn.ss" "lang") in the list of module specs (the rest of the sandbox-namespace-specs value).

If you're testing code that uses a teaching language, the following piece of code can be helpful:

(sandbox-namespace-specs
 (let ([specs (sandbox-namespace-specs)])
   `(,(car specs)
     ,@(cdr specs)
     (lib "posn.ss" "lang")
     ,@(if mred? '((lib "cache-image-snip.ss" "mrlib")) '()))))

(sandbox-override-collection-paths [paths])      PROCEDURE

A parameter that holds a list of collection directories. An evaluator that is created by make-evaluator will put these directories (ones that actually exist) in front of the collections in current-library-collection-paths -- so you can put collection there that will override normal ones. This is useful for cases when you want to test code using an alternate test-friendly version of a collection, for example, testing code that uses GUI (like the ``world'' teachpack) can be done using a fake library that provides the same interface but no actual interaction. The default is null.

(sandbox-security-guard [guard-proc])      PROCEDURE

A parameter that holds a security guard that is used by sandboxed evaluations. The default value is a security guard that forbids all I/O except for things in sandbox-path-permissions, and uses the sandbox-network-guard for network connections. These parameters are described next.

(sandbox-path-permissions [permissions])      PROCEDURE

This parameter configures the behavior of the default sandbox security guard by listing paths and access modes that are allowed for them. The contents of this parameter is a list of specifications, each one is a list of an access mode (a symbol) and a byte-regexp for paths that are granted this access.

The access mode symbol is one of: 'execute, 'write, 'delete, 'read, or 'exists. These symbols are in decreasing order: each implies access for the following modes too (e.g., 'read allows reading or checking for existence).

The path regexp is used to identify paths that are granted access. It can also be given as a path (or a string or a byte string) which will be (made into a complete path, expanded, and simplified, then) converted to a regexp that allows the path and sub-directories (e.g., /foo/bar apply to /foo/bar/baz).

The default value is null, but when an evaluator is created it is augmented by 'read permissions that make it possible to use collection libraries (including sandbox-override-collection-paths), as well as files that the program requires.

(sandbox-network-guard [proc])      PROCEDURE

This parameter holds a procedure that is used (as is) by the default sandbox-security-guard. The default forbids all network connection.

(sandbox-eval-limits [limits])      PROCEDURE

A parameter that determines the default limits on each use of a make-evaluator function -- including the initial evaluation of the input program. Its value should be a list of two numbers, the first is a timeout value in seconds, and the second is a memory limit in megabytes. Either one can be #f for disabling the corresponding limit (or the parameter can be set to #f to disable all limits, in case more are available in future versions). When limits are set, call-with-limits (see below) is wrapped around each use of the evaluator, so consuming too much time or memory results in an exception. You can change the limits of a running evaluator using set-eval-limits below.

41.2  Interacting with Evaluators

(kill-evaluator evaluator)      PROCEDURE

Releases the resources that are held by the evaluator (shuts down the evaluator's custodian). Attempts to use an evaluator after killing it will raise an error, attempts to kill a dead evaluator are ignored. Releases resources, from now on using it will raise an error. This is similar to sending an eof value to the evaluator (except that an eof value will raise an error immediately).

(set-eval-limits evaluator sec mb)      PROCEDURE

Changes the per-expression limits that the evaluator uses to sec seconds and mb megabytes (either one can be #f indicating no limit). This procedure should be used to modify an existing evaluator limits -- changing the sandbox-eval-limits parameter (see above) does not affect existing evaluators. See also call-with-limits below.

(put-input evaluator)      INP

procedure

If sandbox-input is 'pipe when an evaluator is created, then this procedure can be used to retrieve the output port end of the pipe (when used with no arguments), or to add a string or a byte string into the pipe. It can also be used with eof, which closes the pipe.

(get-output evaluator)      PROCEDURE

(get-error-output evaluator)      PROCEDURE

These functions return the output (error output) of the evaluator, in a way that depends on the setting of sandbox-output (sandbox-error-output):

• if it was 'pipe, then get-output returns the input port end of the created pipe;

• if it was 'bytes or 'string then get-output returns the accumulated output and resets the evaluator's output to a new output string or byte string (so each call returns a different piece of the evaluator's output);

• otherwise it returns #f.

(get-uncovered-expressions evaluator [prog? src])      PROCEDURE

Retrieves uncovered expressions (a list of syntax values) from an evaluator. This can only be done if the sandbox-coverage-enabled parameter is turned on, otherwise an error is raised indicating that no coverage information is available.

The prog? argument specifies whether these are expressions that were uncovered after only the original input program was evaluated (#t) or after all later uses of the evaluator (#f). (Using #t retrieves a list that is saved after the input program is evaluated, and before the evaluator is used, so the result is always the same.) The default value is #t, which is useful for testing student programs, where you want to find out if a submission has sufficient test coverage; using #f is useful for writing test suites for a program, where you want to ensure that your tests cover the whole code.

The second optional argument, src, specifies that the result should be filtered to hold only syntax values with that source field (using syntax-source). The default is a symbol, 'program, which is the source associated with the input program by the default sandbox-reader -- which means that you only get syntax values from the input program, not from required modules and not from expressions that were passed to the evaluator. You can either provide a different value for filtering, or use #f which will avoid any filtering.

The resulting list of syntax values has at most one expression for each position and span. This means that the contents may be unreliable, but the position information is (it always indicates source code that would be painted red in DrScheme when coverage information is used).

Note that if your input program is a sequence of syntax values, then you should wither make sure that they have 'program as the source field, or use the src argument. Note that if you use a sequence of S-expressions for an input program, then coverage information will be unreliable, since each expression is assigned a single source location (the first will appear to come from the first line and first character, the second from the second line etc, each with a span value of 1).

41.3  Miscellaneous

mred?      BOOLEAN

A boolean value, bound to #t if we're currently running in MrEd, #f if in plain MzScheme. The idea is that you can use this module from either MzScheme or MrEd. This can help writing code that adapts to the executable that was used.

(call-with-limits sec mb thunk)      PROCEDURE

This function executes the given thunk with memory and time restrictions: if execution consumes more than mb megabytes or more than sec seconds, then the computation is aborted and the exn:fail:resource exception is raised. Otherwise the result of the thunk is returned as usual (a value, multiple values, or an exception). Each of the two limits can be #f to specify no limit¹³.

This is used by sandboxed evaluators, according to the sandbox-eval-limits setting and uses of set-eval-limits: each expression evaluation is protected from timeouts and memory problems. This means that you usually have no need for call-with-limits -- but you may want to limit a whole testing session instead of each expression (e.g., when you want to run tests faster).

(with-limits sec mb body ···)      SYNTAX

A macro version of call-with-limits.

(exn:fail:resource? exn)      PROCEDURE

(exn:fail:resource-resource exn)      PROCEDURE

A predicate and accessor for exceptions that are raised by call-with-limits. The resource field holds a symbol, either 'time or 'memory.