Metcast Taker


  1. Introduction
  2. Virtual Taker Interface
  3. Metcast Taker's logs
  4. References

Metcast Taker

A Metcast Taker is a publishing server, a web application server that takes data from a w-shover and publishes them into a Metcast Channel. To be more precise, the Metcast Taker receives incoming data and metadata and inserts them into tables of a Metcast Channels database. The data are stored as a BLOB. The taker also records the timestamp and the identity of the remote client. For a virtual channel, processing, translation and perhaps storage of published data are performed by a separate application -- a virtual taker. The main Metcast Taker still does common chores of credential verification, authorization, and record-keeping.

The Metcast taker may also function as a remote CGI script: it may run on a computer different from the one of an HTTP server. For example, the taker can be launched by an inetd of a remote computer, communicating with the HTTP daemon through a TCP or a similar pipe. The taker is smart enough to figure out the local or remote mode of its invocation. If the Metcast Taker operates in a remote mode, it must be represented by a local CGI proxy on a computer that runs the HTTP daemon. It is this proxy that gets called by the daemon when it receives a PUT request. The proxy queries its process environment, determines all necessary CGI information, opens a connection to the taker (if it had not been already open), sends the CGI information followed by request's content. The proxy then relays the remote taker's response to the HTTP server.

Virtual Taker Interface

A channel is virtual if a channel administrator specified a particular taker application when he created the channel. The application -- a sh command line to be precise -- will process that virtual channel's incoming data. This virtual taker application is invoked in a manner similar to that of a regular CGI script. That is, the incoming content is fed onto application's stdin. All other parameters -- MCHANNELS_ID, CONTENT_LENGTH, REMOTE_HOST, REMOTE_USER, etc, are passed via POSIX process environment.

A virtual channel taker must tell the outcome of its processing by sending a response on its stdout -- again, in a manner similar to that of a CGI application. If a virtual channel taker has successfully handled the content on its own, it should reply:

     Status: 201 Created ...etc information... CRLF
     ( other HTTP-headers if needed CRLF )*

This information is handed over to the HTTP daemon that accepted w-shove's connection.

If a virtual channel taker has processed the incoming content and wants the result inserted into a channel, the taker should write on its sdtout:

     Status: 200 OK
     Content-type: media-type
     ( Content-length: length )?
     ( other HTTP-headers if needed CRLF )*

The Content-itself is then stored into the channel as if it was received from the original publisher.

All other virtual taker's responses are considered an error, and sent to the HTTP server preceded by a "400 VC taker Error" HTTP response line.


Metcast Taker's logs

The Metcast Taker logs every publishing request and the results of its processing. To keep logs of potentially concurrently running takers separate, the name or an IP address of the publishing host is used as a suffix in the name of a log file. If a particular log file does not exist, it is created; if it did exist, the new information is appended to it. The logs may therefore grow very big and need periodical pruning/trimming. All log files are created with a prefix LOG:PREFIX, which may include a directory (path) prefix, as in logs/jmvlog-taker. This means that logs are contained in a local directory logs, which may be a symbolic link pointing elsewhere.

For example, a MORIAH report submitted as

    w-shove w-shove-rest.conf "text/plain; cid=17" d:\MORIAH\report.txt
leaves the following trace in a Metcast Taker log:
    ----> New remote request[192.XX.XX.XX] => /cgi-bin/oleg/rest/taker-cgi-proxy by WShove/1.2
    PUT  /d:\transport\XNPS_Ob.mor "text/plain; cid=17" 78
    Publishing into virtual channel: launching a taker ./MORIAH-vtaker
    4/2 0127 MORIAH report of 78 bytes
    virtual channel response: 201  Accepted MORIAH report
as well as a line in HTTP server's access log: - moriah [01/Apr/1999:17:16:28 -0800] "PUT /cgi-bin/oleg/rest/taker-cgi-proxy/d:\transport\XNPS_Ob.mor HTTP/1.0" 201 0 "-" "WShove/1.2" 1



For more details, see the comments to the Metcast Taker source code,
and to a sample virtual taker code,



$Id: Metcast-Taker.html,v 1.2 2001/02/01 23:37:53 oleg Exp oleg $