From oleg@pobox.com Sun Jan 10 15:28:32 1999 From: oleg@pobox.com To: oleg@pobox.com Subject: Testing Scheme functions and special forms: catching expected errors Date: Sun, 10 Jan 1999 23:29:44 GMT Reply-To: oleg@pobox.com Keywords: validation, verification, test case, catching errors, Scheme Newsgroups: comp.lang.scheme Organization: Deja News - The Leader in Internet Discussion Summary: Tools to validate error reporting code of Scheme functions and special forms X-Article-Creation-Date: Sun Jan 10 23:29:44 1999 GMT X-Http-User-Agent: Mozilla/4.08 (Macintosh; I; PPC, Nav) Content-Length: 6588 Status: OR A while ago there has been a discussion about tools that help compose test cases. Given a function, a sample input and the expected value, the tool will check to see that the result of function application matches the expectation; the tool will then note that fact in a test report. This however leaves out the case of invalid inputs. Many functions are written to recognize invalid inputs and special conditions, and report an error. This error detection code ought to be tested as well. Alas, capturing an error and recovering from it is rather platform-specific. This article will show two validation forms that make sure evaluating an expression indeed ends up in an error as expected. One of the tools is more portable than the other; often you need both, depending on how a particular Scheme system and a function report a run-time error. Validating error reporting in special forms poses even a bigger challenge. This article will offer one solution as well. A basic error-capturing primitive to be discussed is (failed? . stmts) It evaluates one or several expressions and returns a boolean result indicating if the evaluation of 'stmts' resulted in an error. Obviously 'failed?' must be a special form. Otherwise, the 'stmts' will be evaluated at parameter-passing time: the expected error will then be generated before 'failed?' code gets a chance to run and set appropriate traps. There appear to be two basic mechanism of reporting errors -- throwing exceptions -- in Scheme. Most of the systems provide a function that spells like 'error'; this function prints its arguments and then does something attracting attention. Scheme libraries, SLIB in particular, use that function to report a situation that makes further evaluation impossible. This 'error' can easily be trapped as: ; Try to execute the thunk, and return #f if execution succeeded ; If an error occurred during the execution, it is caught, and ; the thunk-failed? procedure returns #t (define (thunk-failed? thunk) (let* ((orig-error error) ; save the original 'error' primitive (caught (call-with-current-continuation (lambda (cont) (set! error ; redefine the error primitive (lambda (msg . args) (display "\ncaught error: ") (display msg) (for-each display args) (newline) (cont #t))) (thunk) #f)))) (set! error orig-error) caught)) (define-macro (failed? . stmts) `(thunk-failed? (lambda () ,@stmts))) (failed? (+ 1 2)) ==> #f (failed? (+ 1 3) (error 3) (/ 3 0)) ==> caught error: 3 #t File vinput-parse.scm (and other validation code of http://pobox.com/~oleg/ftp/Scheme/util.html) give many examples of using the failed? form. For example, (assert (failed? (expect (test-assert-curr-char "bacd" '(#\a #\space)) #\a))) (assert (failed? (expect-parse-result "xxxd" (skip-until '(#\a #\space #\c)) '(#f . #f)))) (assert (failed? (expect-parse-result "cccx" (next-token '(#\a #\space #\x) '(#\d)) '(#f . #f)))) Another way of reporting run-time errors is by throwing an exception. Unfortunately, at present this is very system-specific. Here's how exception capturing is done in Gambit: ; Check out that the 'form' has failed indeed ; (as it was supposed to) (define-macro (must-have-failed form) `(assert (failed? (##catch-all (lambda (sig args) (error "catching " sig args)) (lambda () ,form))))) This code essentially "relays" the exception to the 'error' function. Note, Gambit-specific as it is, this mechanism might become rather wide-spread. A low-level exception handling proposal discussed at 1998 Scheme workshop appears to be rather close to the Gambit implementation. Marc Feeley is also an editor of the forthcoming Scheme exceptions standard. The must-have-failed form can be used as > (must-have-failed (/ 3 0)) caught error: catching ##signal.runtime-error(Division by zero / (3 0)) #t > (must-have-failed (/ 3 1)) *** ERROR IN (stdin)@27.1 -- failed assertion (failed? (##catch-all (lambda (sig args) (error "catching " sig args)) (lambda () (/ 3 1)))) 1> ,t Or, in more serious applications: [snipped from vreaddir.scm] (let ((non-existing-file-name (OS:tmpnam))) (cerr "testing taking of file status of a non-existing file " non-existing-file-name nl) (must-have-failed (OS:make-file-info non-existing-file-name))) (cerr "Trying to scan " file-name " which is *not* a directory") (must-have-failed (OS:for-each-file-in-directory file-name error)) [snipped from vtreap.scm] (treap 'clear!) (assert (treap 'empty?)) (assert (zero? (treap 'size))) (must-have-failed (treap 'get-min)) A form '(must-be-a-syntax-error form)' traps an expected syntax error in 'form'. As before, must-be-a-syntax-error should be able to delay evaluation of the 'form' until it sets up a trap. Note that evaluation delay afforded by wrapping an expression into lambda will no longer work here. As 'form' is assumed syntactically wrong, the error will be generated when parsing or (pre)compiling of the form. We should therefore put off the moment the interpreter scans the 'form' until the run-time. Once we spelled that out, the solution becomes obvious: ; Check to see that 'form' has indeed a wrong syntax (define-macro (must-be-a-syntax-error form) `(call-with-current-continuation (lambda (k) (##catch-signal '##signal.syntax-error (lambda x (display "catching a syntax error: ") (display x) (newline) (k #f)) (lambda () (eval ',form) (error "No syntax error detected, unexpectedly")))))) This must-be-a-syntax-error is a special form of the second order, so to speak. We can now test that standard Scheme forms indeed report bad syntax where expected: > (must-be-a-syntax-error (let 1)) catching a syntax error: (##signal.syntax-error (#(#(source2) (let 1) #f #f) Ill-formed special form: let)) #f > (must-be-a-syntax-error (let ((x 0) (x 1)) x)) catching a syntax error: (##signal.syntax-error (#(#(source2) x #f #f) Duplicate variable in bindings)) #f > must-be-a-syntax-error (or 1 ())) catching a syntax error: (##signal.syntax-error (#(#(source2) () #f #f) Ill-formed expression)) #f Note that the system caught expected syntax and semantic errors, reported them, and then went on with the next test case. Validation code for a special form LAND* (vland.scm) shows more examples.