Web and CGI Programming in Scheme

Tools and working examples of HTML, XML, FORM, HTTP and CGI programming in Scheme


 

Search a database of weather observation stations

A sample CGI script illustrating: This script is used in production, to search an Air Weather Service Master Station Library Catalog, a database of all weather observation stations. To see the script in action, follow a link to a metnet site below.

This script contains a great deal of generic code for web services and for common form processing chores. Many other CGI scripts of mine are written by cloning of this script and making appropriate changes.

Version
The current version is 3.3, Sep 28, 2002.
References
search-mslib.scm [16K]
Commented source code
Requires

My own standard prelude

CGI processing utilities

and other string parsing, POSIX interfacing, etc. procedures documented on this site.
 

get-url and web agents

Web agents are applications that surf the web on behalf of a user and, among other things, extract pieces of useful information from HTML pages filled with advertisements and other fluff. An article below presents and discusses one such web robot. The robot fetches last trading prices for specific stocks, in a format suitable for further processing or analysis. The agent's code implements a function get-url. This general-purpose function can be used independently, to retrieve a page or a file from a web server. The function handles direct and proxied connections.
Version
The current version is 1.1, Dec 26, 1999.
References

Tcl, Scheme, Web, robot [Re: Why Scheme?] [plain text file]
An article posted on a comp.lang.scheme newsgroup on Sun, 26 Dec 1999 22:50:47 GMT. It contains the source code for the stock quote robot in Scheme and Tcl. The Scheme code includes the source for a get-url function.

Uni-, Bi-, and TCP pipes as regular files
A detailed explanation of this trick and of its applications.

vext-io.scm [18K]
Code that illustrates and validates extended file names, uni-, bi-directional and tcp://-like pipes.

Requires
Input parsing primitives
 

Validating a URL

This simple Scheme code checks the existence of a resource given its URL. If the resource, e.g., a web page, really exists, the code determines and prints out the size of the resource.

This code is an example of submitting an HTTP request and of parsing an HTTP response -- done entirely in Scheme.

Version
The current version is 1.1, Feb 21, 1999.
References

Way extended handling of files and "files" [plain text file]
An article posted on a comp.lang.scheme newsgroup on Sun, 21 Feb 1999 22:57:26 GMT

Uni-, Bi-, and TCP pipes as regular files
A detailed explanation of this trick and of its applications.

vext-io.scm [18K]
Code that illustrates and validates extended file names, uni-, bi-directional and tcp://-like pipes.

Running an HTTP transaction

Requires
Input parsing primitives
 

CGI processing utilities

A collection of several procedures for common CGI processing. The most notable of these procedures is
     CGI:lookup NAME TYPE [DEFAULT-VALUE]
The procedure looks up a NAME (a symbol) among CGI variables. The result is converted to TYPE. If the NAME is not CGI-bound and the DEFAULT-VALUE is specified, the procedures returns the DEFAULT-VALUE. Otherwise, the procedure signals an error. A failure of the TYPE-conversion also triggers an error. The TYPE is a symbol, one of the following: token, tokens, string, io, int, number. The detailed source code comments describe how the procedure handles all four cases of CGI parameters: param=, param, param=val, and param=val1+val2.

The collection of utilities includes several other useful procedures: to obtain a query string from a GET or a POST request, to parse the query string into an associative list of parameter bindings, to determine the name of a remote user, to handle a signal raised in a CGI script.

Version
The current version is 3.4, Sep 30, 2002.
References

cgi-util.scm [11K]
The commented source code

Requires

Input parsing primitives

Exceptions handling SRFI-12 is required for the function CGI:exception-handler

OS:getenv procedure, which on many Scheme systems can be defined as getenv

 

A CGI script utility for parsing of a QUERY_STRING

-- procedure+: CGI:url-unquote QUERY-STRING
converts a CGI-quoted string like
     parm1=val1&parm2=val2+val%253+val4&%7Eparm3=&parm4
into an associative list
     ((parm4) (~parm3 "")
      (parm2 "val2" "val%3" "val4")
      (parm1 "val1"))

The parsing is done by a finite state machine, which takes into account (i) the current state: looking for a parm, looking for a value; (ii) an action-prefix: =, +, or &; (iii) the token that follows the action prefix.

This function is intended for parsing of a QUERY_STRING, POST action messages, or similar parameters passed by an HTTP server to a CGI script.

Version
The current version is 1.2, Nov 17, 1997.
References
CGI-unquote.scm [5K]
A commented source code with a set of verification tests.
CGI processing utilities
Requires
Input parsing primitives
 

Parsing of MIME Entities

Parse a content-type string:   MIME:parse-content-type CTYPE-STR
Given a content-type string CTYPE-STR of a form media-type [; attr=value]* the procedure returns the list of associations (attr . value) where attr is a symbol and value is a string. The media-type is returned as an association with a tag =mime-type. See Sections 2.2 and 3.6 of RFC2616 (HTTP/1.1) for the syntax of the Content-Type string.
For example, given a content-type string
     text/html; cid=10;name=" val "
the function returns a list
     ((=mime-type . "text/html") (cid . "10") (name . " val "))
Read MIME headers from a port:   MIME:read-headers PORT
This procedure reads MIME headers from the PORT. The port will be positioned after the empty line that terminates the headers.
The procedure can be used to read the headers of e-mail messages or the response headers of an HTTP server.
Platforms
The package has been tested on Bigloo 2.4b, GambitC 3.0, SCM 5d2 and MIT Scheme 7.5.2.
Version
The current version is 1.1, Jul 8, 2002.
References

mime.scm [6K]
Commented source code

vmime.scm [3K]
Commented validation code

Running an HTTP transaction

Requires
Input parsing primitives
 

Running an HTTP transaction

This code implements the basic flow of a HTTP transaction, as defined in a HTTP 1.1 document [RFC 2068]. That is, this code:

http-transaction REQ-METHOD REQ-URL REQ-PARMS RESPONSE-HANDLER

Here, REQ-METHOD is a string, typically "GET" or "POST", although many others may be allowed. REQ-URLis an absolute URL of the HTTP server. REQ-PARMS is an associative list, a list of (name . value) pairs. The list may be regarded as "keyword arguments" of the http-transaction procedure. All keyword arguments are optional: if omitted or specified with a value #f, a suitable default value will be used. The following "keyword parameters" are supported: http-proxy, user-agent, http-req, logger. See the title comments in the code for the full description.

RESPONSE-HANDLER is a procedure RESP-CODE RESP-HEADERS RESP-PORT => ANY. Here RESP-CODE is a number, which is one of the HTTP codes, e.g., 200, 304, 404, or 500. The argument RESP-HEADERS is a list of pairs (http-header-name . http-header-val) where http-header-name is an upper-case symbol. This list carries the HTTP server response headers. In addition to the standard header names defined by HTTP 1.1, a special pair (HTTP-RESPONSE . the-whole-response-line) tells the response handler the entire HTTP response line. The argument RESP-PORT is an input port from which to read the body of the reply, if any.

The validation code gives several examples of invoking http-transaction to retrieve a web page, to submit a web form, or to make other HTTP remote procedure call.

Platforms
The package has been tested on Bigloo 2.4b and GambitC 3.0.
Version
The current version is 2.0, Aug 23, 2002.
References

http.scm [11K]
Commented source code

vhttp.scm [4K]
Commented validation code

Speaking HTTP: A File-Uploader Tool
USENIX ;login: -- vol. 25, No. 2 -- April 2000, pp. 6-14.

Requires

Input parsing primitives

Parsing of MIME Entities

Exceptions handling SRFI-12

 

tail-f: file watching CGI script

This CGI code is activated by an HTTP server when serving a URL like
/cgi-bin/tail-f?period=600&file=/opt/apache/logs/access

This code watches the given file, sending its last nlines lines to the client. This script does not quit after the first peek into the file, but rather keeps watching the file, checking up on it every period seconds. The watching stops when a user agent (a browser, that is) closes the connection: e.g., in response to the user's pressing the 'stop' button. The watching also terminates when the file disappears. All in all, this is similar to a familiar UNIX tail -f command, working across the network.

Opening the tail-f URL starts off a server-web-browser finite automaton. The latter quickly goes through several states, without any user intervention. When the Finite State Machine reaches its final state, it leaves the result of the computation in a browser window: a frame set with a HTML form on the top, and the tail lines of the file. The form in the top frame presents the parameters of the script -- file name, watch period, nlines -- and their specified or implied values. You can enter or modify them right on the form, and then click on the watch button. The bottom frame shows the tail of the file, automatically updated via a server push.

When scanning a file, we deliberately avoid a direct positioning of the file. For one thing, fseek() etc. functions are not standard in Scheme. For another, treating a file as a purely sequential stream enables us to handle files which are not regular files per se: named (FIFO) pipes, raw devices, serial ports, etc.

Version
The current version is 2.0, Mar 21, 1997.
References

An extensively commented source code. [plain text file]
The code implements all CGI functionality, tail-f functionality, and the server push. It also declares and implements a circular buffer object, which can be used independently.

The Web IS a computer, HTML its language (and a pushy Scheme CGI setting it off) [plain text file]
A USENET article posted on comp.lang.scheme, comp.programming, comp.infosystems.www.authoring.cgi, and comp.infosystems.www.misc newsgroups on Tue Mar 25 13:17:13 CST 1997.

Requires
My own standard prelude


Last updated October 11, 2002

This site's top page is http://okmij.org/ftp/

oleg-at-pobox.com or oleg-at-okmij.org
Your comments, problem reports, questions are very welcome!

Converted from SXML by SXML->HTML