Development History of Metcast Channels

Notes on an early (prior to 4.0) version of the system

Subscription service
Metcast Channels vs. ...
1. ...Metoc Virtual Library (MVL)
2. ...WebJMV (aka Thumbnails)
Usage scenarios (from a Metcast Channel presentation).
Metcast Channels Table of Contents, in a Channel Definition Format (CDF)
Setting up Metcast Channels under IE 4.x

Metcast Channel Subscription service

A subscriber service is one front-end user interface to Metcast Channels. A MetcastClient application is another. The subscriber lets a user scan or search through available Metcast channels, subscribe to selected channels, view and update his current subscription, and obtain more information about a channel. The subscriber, a CGI script, presents a self-explanatory form. Clicking on highlighted strings (e.g., "subscription schedule", "update interval") brings a specific help on that particular topic. Or at least, is supposed to. At present, the script simply shows the topic id, for which no help is written yet. Unimplemented the help service may be, it is designed to be rather universal: all help requests are routed to a single CGI script, which then selects the appropriate text document from its database and sends to the user. This way, no matter how elaborate the help system may become, no external links are in danger of being broken and in need of updating.

The subscriber script lets the user view all the channels of a particular category or published by a particular originator. In addition, the user may enter one or several keywords to search through channel descriptions. For example, the user may enter "satellite" to view all channels that carry satellite-related data; or "satellite visual" to restrict the search only to vis sat pictures. In addition, the user may further limit the search only to data produced by NASA. See usage scenarios below for more detail. The results of the search are presented in a scrollable list. The user may select one or several channels on this list and add these products to his current subscription. The user can obtain more information about the channel, or unsubscribe from it, by pressing corresponding buttons. Note that the subscription service works off a "live" database: whenever a channel administrator adds, updates, or deletes a channel, the change becomes visible to the subscription service instantly.

When the user has settled down on his subscription, he can press a button to activate it. This sends a special "channel activation record"; the user's browser is supposed to handle the document of this type (if not, the user would be prompted for the handler. He should select chan_act.bat file, for a WinNT/Win95 platform).

Processing of a channel activation record creates a new folder (usually named MetcastCh, unless specified otherwise). This folder would initially contain three files:

   retriever.conf, things.mbl, sub.html

The first two files have all necessary information for the Metcast retriever to request and deliver the data from subscribed channels. Specifically, retriever.conf contains the URL of a Metcast server and the request schedule. The file things.mbl is a DiscreteThings request itself, in the Metcast request language. These files are created automatically as a part of the subscription activation. Thus the subscriber service acts as a user front-end to the request language.

The last part of processing of the request activation record is launching of a retriever using the configuration files above. The channel thus opens and starts delivering things as requested. The channel (retriever) stays in the background to query for updates if the user has chosen that option. You can stop the channel and reactivate it later merely by launching the retriever and passing it the names of the two config files in the MetcastCh directory; you do not need to contact the subscriber service again. It's possible to configure your system to open channels whenever the system starts up or a particular user logs in.

The third file in the MetcastCh directory, sub.html, is a bookmark that contains the subscription. Whenever a user clicks on this bookmark, a browser launches and shows the subscription page exactly as the user has left it last time, with all selected channels and subscription schedule. The user may add or delete a channel, change the schedule, and activate the subscription again.

The title comments in subscriber's source code explain its functionality in full detail. The channel activation record is a multi-part MIME document, which is handled by the retriever itself. The last part of the document instructs the retriever to launch another retriever, this time as a channel. Thus the retriever is called tail-recursively. The channel activation record is a text MIME document, and can be sent via e-mail and processed at a later time, asynchronously.

Metcast Channels and a Metoc Virtual Library (MVL)

It is no coincidence that the subscriber web page looks very similar to a search page of MVL

http://www.navo.navy.mil/noframe/select.products.htm

The subscriber service has the same goal as that of MVL: to maintain a repository of products, present a list of products to a user, let the user browse and search through the list, select products of some particular interest, and download/view them. The Metcast Channel service goes beyond MVL in that:

The subscriber service offers keyword search capability
In MVL, the user browses and downloads separate products; with the subscriber service, he browses and selects channels. A channel may contain one or several products updated periodically. The channel can be used to download its contents once; even here Metcast offers an advantage in allowing specification of time constraints: you can ask for a product only if it was modified within a certain time. In addition, you can subscribe to a channel and receive further updates of the selected products automatically, without any need to go through the search and selection process again.
Metcast is more efficient than MVL and easier to manage, as its repository is a single database rather than a tangle of files. The database schema is itself one of the Metcast channels, which makes it possible to distribute/clone repositories (see also below).

Thus Metcast subsumes MVL.

The Metcast subscription service presents a bare-bone form that lets a user search for a product based on a variety of criteria. It is certainly possible to make a better looking and more customized form for a particular group of users, as a MTIS project clearly demonstrates. The MTIS search form looks far nicer, yet it uses the same Metcast subscription mechanism -- and the same CGI script, eventually. This is another example how Metcast tools can be used with a number of policies and end-user interfaces.

Metcast Channels vs. WebJMV (aka Thumbnails)

It goes without saying that all the products currently offered on the FNMOC pages can be registered with Metcast and offered on a "subscription" basis. This gives a user an advantage of an efficient search, an ability to receive automatic updates; Metcast also eases the burden of maintaining repository of products as the data are stored and described in one location (a database) within a simple and flexible schema; Metcast Web pages are generated on the fly and do not need additional storage.

You can easily enter all of the satellite imagery, special products, etc. into Metcast and distribute them right away. You can register grid products, by describing the corresponding areas of interest as channels or categories.

Thus Metcast Channels can be used to serve the sets of products that are defined by a FNMOC web administrator for geographical areas of interest, categories, etc. These sets of products are typically represented as "thumbnails". This is exactly how www.fnmoc.navy.mil currently operates. In addition, Metcast Channels application can generate the thumbnail views dynamically, reflecting the current contents of the Metcast database. This option should alleviate the pain of making and adjusting the pages by hand: changing the names of areas , their sizes and modification dates. All the thumbnails shown are guaranteed to be current and with the corresponding data.

Like FNMOC pages, clicking on a thumbnail downloads the products and launches JMV. Unlike FNMOC pages, Metcast retriever is rather flexible as to how to process downloaded products: it may launch JMV, it may launch a specialized viewer for satellite data, it may launch powerpoint, it may play a QuickTime movie, it may play a sound, or simply dump the data in a specified folder.

Unlike FNMOC's WebJMV, Metcast channel can be set to watch for updates. This way, whenever one or several products in the set represented by a thumbnail are updated, the new versions are automatically downloaded and processed, without further user interaction.

Unlike FNMOC static sets of products, the user of Metcast thumbnails always has an option to see what products are "behind" a particular thumbnail (without downloading them), how big they are, who created and modified them. In addition, the user can change the product list by deleting or adding new products, thus creating his "personalized" thumbnail. This subscription (a bookmark) is downloaded to user's computer. The user can use this personalized thumbnail from that moment on without any need to customize it again.

Usage scenarios

The following are "snapshots" of two Metcast Channel demos I showed off to a fairly large group of people on Feb 12 and Mar 24, 1998. Server: nites-2, a HP-PA computer; client: a WinNT workstation with both Netscape 3.x and IE 4.0 installed.

Example 1

connect to the subscriber URL
search for products being published by FNMOC; subscribe to a "CONUS IR Satellite" and "text notes"
show off subscription, unsubscription, and getting info. Note that adding a channel already on the subscription list does not add it: The subscriber is moderately fool-proof.
the search can be done by keyword, by category, by publisher, or by any of the three or by all of the three in any combination. At least one search parameter should be given though.
show a context sensitive help (or at least, the opportunity thereof).
activate the subscription. Note a newly created folder (MetcastCh), with a retriever conf file, request file, and a bookmark. The first two files let you open the channel by hand. The channel normally starts when the subscription is activated. Once open, it runs independently of the browser: show that you can browse other web pages, or quit the browser altogether: this won't affect the channel a bit.
show the contents of the request file and mention that the subscriber service is a GUI for the request language.

Example 2

select a "text memos" channel, specify an update schedule (e.g., every minute) and activate the channel. Note that the retriever keeps running in the background, checking with the server for updates every minute
talk about a dropbox: a directory on a server machine being monitored by a special daemon. The contents of this directory is the "memo" channel itself. Whenever a new file is added to this directory, it becomes available for a channel to push; whenever a file is deleted, it disappears from the channel. Note, the file must be world-readable.
"drop" a few text files into the dropbox; for example,
```
 echo "hello world" > /w/metcast/TMEMOS/memo4
```
the file just dropped into the drop box appears on a client computer. If several new items were added into one or several subscribed channels within the update interval, they all will be downloaded, within the same update session.
mention that we can deliver satellite imagery using the same technique.

Example 3

click on the bookmark in the MetcastCh folder: it indeed opens a page that contains the current subscription.
search for "database" and show off a "database schema" channel.
Subscribe to it, activate the channel, and show a 11.sql file that contains the description of the Metcast Channel database. Thus the Metcast Channels service contains its own description
you can subscribe to this channel and receive updates whenever the database is modified, that is, when new channels or categories are added.

Example 4

open the subscription form again, search for "JMV"
show two found channels: JMV FAQ and JMV software.
select JMV FAQ, activate the channel and show the FAQ on the client, deposited into the MetcastCh folder
thus channels can be used to distribute anything, from satellite images to grids to text documents to PowerPoint presentations, to FAQ to software updates, to the database's own SQL description.

Example 5

Metcast emulating WebJMV: "a page with thumbnails". In IE, open a fnmoc_areas2.html page. It looks like the "standard" FNMOC areas page: a row of thumbnails with labels and buttons: compare to the corresponding page on a FNMOC web server. The former page was designed to look like a FNMOC thumbnails page; in fact, I simply swiped the corresponding .html file off a nites-2 WebJMV directory and modified the file a little bit, streamlining and embellishing HTML code in the process.

Although my page looks almost identically to the FNMOC areas page, the functionality is different. A click on a thumbnail now opens a channel. That is, it launches a retriever, which downloads a set of products the thumbnail represents, processes them and waits for updates. At the demo, "processing of the products" was merely storing the corresponding data (a satellite image, a JMV FAQ and some text document) in a special folder. It is very easy (as I showed as well) to do something more sophisticated, for example, launching JMV or an image viewer to show a satellite picture. As the channel keeps running watching for updates, a new version of the picture or any other satellite product would be downloaded automatically when becomes available, and would be automatically displayed (redisplayed) without any additional user intervention.

Thus after the user has clicked on a thumbnail, he doesn't need to touch the keyboard ever again. He can simply watch products and updates as they appear.

Example 6

Open the above fnmoc_areas2.html page in Netscape. I could have done this part of the demo using IE as well, but I wanted to show that Metcast Channels work in Netscape just the same. Instead of clicking on a thumbnail, which would open a channel with a predefined set of products, I clicked on a "Product List" link. This switched me to a regular Metcast Subscriber page (see above), which shows the products represented by the corresponding thumbnail. These products have been selected by a FNMOC web administrator. The user can see the products, can get a detailed information on them, and can unsubscribe a few of them and subscribe to a new one. Once he activates the channel, the retriever launches, downloads the current data and sets out to watch for updates as was instructed. The point of this part of the demo was to show that although a webmaster defines a set of products for a given area (thumbnail), the user may always customize this set for his own particular needs.

Example 7

Metcast Channel description in CDF

pull the subscriber page again
search for CDF or channel
select a channel "Metcast Channels in CDF", subscribe to it and activate it
when the channel is fired, locate the corresponding file in a download directory on a client. This file has a cute icon: a miniature "satellite dish", a standard icon for Internet Explorer's channels.
point and hold a mouse over the icon, and watch a "Metcast Channel" tool-tip appear
click on the file; note that WinNT recognized the file as an IE channel. "Subscribe" to it and show that the channel is added to the IE channels bar and the menu

The point was to show that I can create a CDF file that IE can recognize as its own. Thus I can play by IE rules. Not that I wanted to, actually: as I discovered, the channel definition format IE supports differs slightly from the format defined in a CDF draft submitted to the W3 Consortium, by Microsoft itself. Another case when Microsoft cannot abide by its own rules. Therefore, it is not a good idea to rely on IE channels to distribute all Metcast products. Moreover, the Metcast retriever has an additional advantage of being able to download a batch of products in one transaction.

CDF version of the Metcast Channels Table of Contents

Metcast Channels database has a special channel which describes the whole database -- provides a MChannels table of contents so to speak. If you request this document you can search through channel descriptions, MIME types, etc. With this document, any application can present user interfaces similar to that of the subscribe service -- but without a live connection to the database. Furthermore, if you subscribes to this TOC channel, you can learn of all the added or deleted channels, again, without being able to connect to the database.

This Metcast Channels Table of Contents -- in the Channel Definition Format (CDF) -- is created once the database first loads up. Whenever new channels, user groups, etc. are created, a special application had to be run to republish the updated table of contents. Fortunately, it is no longer needed. I re-wrote the application that produces CDF. The MChannels table of contents, an XML document, is now produced by a SQL procedure, stored in the database itself. It makes a lot of sense. Indeed, composing of the Dynamic Channel List requires scanning through a number of MChannelsDB tables -- a job particularly suited for a stored procedure. A stored procedure does not need to open a connection to a database: a stored procedure runs within the database engine itself. The procedure does not need to move data to a client and back as all processing happens on the server.

Furthermore, since the CDF publisher is a stored procedure, it can be activated by the database engine itself whenever a relevant table changes -- a new product type is added, a channel is deleted, or a description is updated. There is no need to periodically republish the Dynamic Channel List, as its is the database that keeps the list up-to-date. For all the practical purposes, we can consider the list to be a built-in attribute of the MChannels database -- an option of self-publishing of the database in XML. The latest offerings from Oracle and IBM have this option natively. A procedure to create the Dynamic Channel List is stored with the data, and is run with the data.

I must acknowledge Jim Harper for this idea. It never occurred to me that CDF/XML generation can be built into the database. Jim -- without knowing about my original implementation -- explained to me how he thought CDF publishing should work. His idea was so neat, so fitting and beautiful that I had to break my old code and rush to re-implement CDF in the right way.

As usual, an implementation is always messier than the idea; it was even messier than needed as I realized (after some two hours of experimenting and scouring through documentation) that Informix offers no facilities whatsoever to deal with BLOBs within stored procedures. I was very disappointed. I found a way around -- rather ungainly, but operational. It showed me clearly btw why isolation and lock modes do matter.

The Dynamic Channel List has been installed on metcast1 and spaceley. Whenever Jim Harper's Channel Administration makes any change to the MChannels Database -- adds or deletes a channel or a category, updates descriptions, etc. -- the Dynamic Channel List is automatically republished into Channel 14, as a part of an update transaction. The list is then distributed to any currently running Metcast Client, which shows an updated channel list when a user is about to publish or subscribe.

Setting up Metcast Channels under IE 4.x

The only issue that requires attention is making IE recognize a Metcast channel activation record and call an appropriate handler. It's a matter of setting up a proper association:

Launch IE4 and have it "explore" 'My Computer'
Pull down a menu item View|Folder Options...
In the tabbed dialog that appears, click on the "File Types" tab
Click on the "New Type..." button
A "Add New File Type" form appears. Fill it as follows:
- Description of type: Metcast Channel Activation record
- Associated extension: .car
- Content Type (MIME): text/x-metcast-car
- Default Extension for Content Type: .car
- uncheck "confirm open after download" checkbox
- under "actions", click on "new..." and fill out a "new action" form as follows:
  - action: open
  - application used to perform action:
    c:\subscribe\chan_act.bat "%1"
    adjust the path to chan_act.bat as necessary. The easiest way to do this is to click on a "Browse..." button on the "new action" form and select the chan_act.bat file in the standard Open File dialog. Afterwards, add space and "%1" (quotes are important!) to the path name that appears in the second slot of the "new action" form
  - click OK on the "new action" form
- click OK on the "Add New File Type" form
click OK on the "Folder Options" form

History

$Id: History-MChannels.html,v 1.8 2001/01/26 02:49:49 oleg Exp oleg $

oleg-at-okmij.org
Your comments, problem reports, questions are very welcome!