Development History of Metcast Channels

Notes on an early (prior to 4.0) version of the system

  1. Subscription service
  2. Metcast Channels vs. ...
    1. ...Metoc Virtual Library (MVL)
    2. ...WebJMV (aka Thumbnails)
  3. Usage scenarios (from a Metcast Channel presentation).
  4. Metcast Channels Table of Contents, in a Channel Definition Format (CDF)
  5. Setting up Metcast Channels under IE 4.x


Metcast Channel Subscription service

A subscriber service is one front-end user interface to Metcast Channels. A MetcastClient application is another. The subscriber lets a user scan or search through available Metcast channels, subscribe to selected channels, view and update his current subscription, and obtain more information about a channel. The subscriber, a CGI script, presents a self-explanatory form. Clicking on highlighted strings (e.g., "subscription schedule", "update interval") brings a specific help on that particular topic. Or at least, is supposed to. At present, the script simply shows the topic id, for which no help is written yet. Unimplemented the help service may be, it is designed to be rather universal: all help requests are routed to a single CGI script, which then selects the appropriate text document from its database and sends to the user. This way, no matter how elaborate the help system may become, no external links are in danger of being broken and in need of updating.

The subscriber script lets the user view all the channels of a particular category or published by a particular originator. In addition, the user may enter one or several keywords to search through channel descriptions. For example, the user may enter "satellite" to view all channels that carry satellite-related data; or "satellite visual" to restrict the search only to vis sat pictures. In addition, the user may further limit the search only to data produced by NASA. See usage scenarios below for more detail. The results of the search are presented in a scrollable list. The user may select one or several channels on this list and add these products to his current subscription. The user can obtain more information about the channel, or unsubscribe from it, by pressing corresponding buttons. Note that the subscription service works off a "live" database: whenever a channel administrator adds, updates, or deletes a channel, the change becomes visible to the subscription service instantly.

When the user has settled down on his subscription, he can press a button to activate it. This sends a special "channel activation record"; the user's browser is supposed to handle the document of this type (if not, the user would be prompted for the handler. He should select chan_act.bat file, for a WinNT/Win95 platform).

Processing of a channel activation record creates a new folder (usually named MetcastCh, unless specified otherwise). This folder would initially contain three files:

   retriever.conf, things.mbl, sub.html
The first two files have all necessary information for the Metcast retriever to request and deliver the data from subscribed channels. Specifically, retriever.conf contains the URL of a Metcast server and the request schedule. The file things.mbl is a DiscreteThings request itself, in the Metcast request language. These files are created automatically as a part of the subscription activation. Thus the subscriber service acts as a user front-end to the request language.

The last part of processing of the request activation record is launching of a retriever using the configuration files above. The channel thus opens and starts delivering things as requested. The channel (retriever) stays in the background to query for updates if the user has chosen that option. You can stop the channel and reactivate it later merely by launching the retriever and passing it the names of the two config files in the MetcastCh directory; you do not need to contact the subscriber service again. It's possible to configure your system to open channels whenever the system starts up or a particular user logs in.

The third file in the MetcastCh directory, sub.html, is a bookmark that contains the subscription. Whenever a user clicks on this bookmark, a browser launches and shows the subscription page exactly as the user has left it last time, with all selected channels and subscription schedule. The user may add or delete a channel, change the schedule, and activate the subscription again.

The title comments in subscriber's source code explain its functionality in full detail. The channel activation record is a multi-part MIME document, which is handled by the retriever itself. The last part of the document instructs the retriever to launch another retriever, this time as a channel. Thus the retriever is called tail-recursively. The channel activation record is a text MIME document, and can be sent via e-mail and processed at a later time, asynchronously.

Metcast Channels and a Metoc Virtual Library (MVL)

It is no coincidence that the subscriber web page looks very similar to a search page of MVL

The subscriber service has the same goal as that of MVL: to maintain a repository of products, present a list of products to a user, let the user browse and search through the list, select products of some particular interest, and download/view them. The Metcast Channel service goes beyond MVL in that:

Thus Metcast subsumes MVL.

The Metcast subscription service presents a bare-bone form that lets a user search for a product based on a variety of criteria. It is certainly possible to make a better looking and more customized form for a particular group of users, as a MTIS project clearly demonstrates. The MTIS search form looks far nicer, yet it uses the same Metcast subscription mechanism -- and the same CGI script, eventually. This is another example how Metcast tools can be used with a number of policies and end-user interfaces.

Metcast Channels vs. WebJMV (aka Thumbnails)

It goes without saying that all the products currently offered on the FNMOC pages can be registered with Metcast and offered on a "subscription" basis. This gives a user an advantage of an efficient search, an ability to receive automatic updates; Metcast also eases the burden of maintaining repository of products as the data are stored and described in one location (a database) within a simple and flexible schema; Metcast Web pages are generated on the fly and do not need additional storage.

You can easily enter all of the satellite imagery, special products, etc. into Metcast and distribute them right away. You can register grid products, by describing the corresponding areas of interest as channels or categories.

Thus Metcast Channels can be used to serve the sets of products that are defined by a FNMOC web administrator for geographical areas of interest, categories, etc. These sets of products are typically represented as "thumbnails". This is exactly how currently operates. In addition, Metcast Channels application can generate the thumbnail views dynamically, reflecting the current contents of the Metcast database. This option should alleviate the pain of making and adjusting the pages by hand: changing the names of areas , their sizes and modification dates. All the thumbnails shown are guaranteed to be current and with the corresponding data.

Like FNMOC pages, clicking on a thumbnail downloads the products and launches JMV. Unlike FNMOC pages, Metcast retriever is rather flexible as to how to process downloaded products: it may launch JMV, it may launch a specialized viewer for satellite data, it may launch powerpoint, it may play a QuickTime movie, it may play a sound, or simply dump the data in a specified folder.

Unlike FNMOC's WebJMV, Metcast channel can be set to watch for updates. This way, whenever one or several products in the set represented by a thumbnail are updated, the new versions are automatically downloaded and processed, without further user interaction.

Unlike FNMOC static sets of products, the user of Metcast thumbnails always has an option to see what products are "behind" a particular thumbnail (without downloading them), how big they are, who created and modified them. In addition, the user can change the product list by deleting or adding new products, thus creating his "personalized" thumbnail. This subscription (a bookmark) is downloaded to user's computer. The user can use this personalized thumbnail from that moment on without any need to customize it again.

Usage scenarios

The following are "snapshots" of two Metcast Channel demos I showed off to a fairly large group of people on Feb 12 and Mar 24, 1998. Server: nites-2, a HP-PA computer; client: a WinNT workstation with both Netscape 3.x and IE 4.0 installed.

Example 1

Example 2

Example 3

Example 4

Example 5

Metcast emulating WebJMV: "a page with thumbnails". In IE, open a fnmoc_areas2.html page. It looks like the "standard" FNMOC areas page: a row of thumbnails with labels and buttons: compare to the corresponding page on a FNMOC web server. The former page was designed to look like a FNMOC thumbnails page; in fact, I simply swiped the corresponding .html file off a nites-2 WebJMV directory and modified the file a little bit, streamlining and embellishing HTML code in the process.

Although my page looks almost identically to the FNMOC areas page, the functionality is different. A click on a thumbnail now opens a channel. That is, it launches a retriever, which downloads a set of products the thumbnail represents, processes them and waits for updates. At the demo, "processing of the products" was merely storing the corresponding data (a satellite image, a JMV FAQ and some text document) in a special folder. It is very easy (as I showed as well) to do something more sophisticated, for example, launching JMV or an image viewer to show a satellite picture. As the channel keeps running watching for updates, a new version of the picture or any other satellite product would be downloaded automatically when becomes available, and would be automatically displayed (redisplayed) without any additional user intervention.

Thus after the user has clicked on a thumbnail, he doesn't need to touch the keyboard ever again. He can simply watch products and updates as they appear.

Example 6

Open the above fnmoc_areas2.html page in Netscape. I could have done this part of the demo using IE as well, but I wanted to show that Metcast Channels work in Netscape just the same. Instead of clicking on a thumbnail, which would open a channel with a predefined set of products, I clicked on a "Product List" link. This switched me to a regular Metcast Subscriber page (see above), which shows the products represented by the corresponding thumbnail. These products have been selected by a FNMOC web administrator. The user can see the products, can get a detailed information on them, and can unsubscribe a few of them and subscribe to a new one. Once he activates the channel, the retriever launches, downloads the current data and sets out to watch for updates as was instructed. The point of this part of the demo was to show that although a webmaster defines a set of products for a given area (thumbnail), the user may always customize this set for his own particular needs.

Example 7

Metcast Channel description in CDF The point was to show that I can create a CDF file that IE can recognize as its own. Thus I can play by IE rules. Not that I wanted to, actually: as I discovered, the channel definition format IE supports differs slightly from the format defined in a CDF draft submitted to the W3 Consortium, by Microsoft itself. Another case when Microsoft cannot abide by its own rules. Therefore, it is not a good idea to rely on IE channels to distribute all Metcast products. Moreover, the Metcast retriever has an additional advantage of being able to download a batch of products in one transaction.


CDF version of the Metcast Channels Table of Contents

Metcast Channels database has a special channel which describes the whole database -- provides a MChannels table of contents so to speak. If you request this document you can search through channel descriptions, MIME types, etc. With this document, any application can present user interfaces similar to that of the subscribe service -- but without a live connection to the database. Furthermore, if you subscribes to this TOC channel, you can learn of all the added or deleted channels, again, without being able to connect to the database.

This Metcast Channels Table of Contents -- in the Channel Definition Format (CDF) -- is created once the database first loads up. Whenever new channels, user groups, etc. are created, a special application had to be run to republish the updated table of contents. Fortunately, it is no longer needed. I re-wrote the application that produces CDF. The MChannels table of contents, an XML document, is now produced by a SQL procedure, stored in the database itself. It makes a lot of sense. Indeed, composing of the Dynamic Channel List requires scanning through a number of MChannelsDB tables -- a job particularly suited for a stored procedure. A stored procedure does not need to open a connection to a database: a stored procedure runs within the database engine itself. The procedure does not need to move data to a client and back as all processing happens on the server.

Furthermore, since the CDF publisher is a stored procedure, it can be activated by the database engine itself whenever a relevant table changes -- a new product type is added, a channel is deleted, or a description is updated. There is no need to periodically republish the Dynamic Channel List, as its is the database that keeps the list up-to-date. For all the practical purposes, we can consider the list to be a built-in attribute of the MChannels database -- an option of self-publishing of the database in XML. The latest offerings from Oracle and IBM have this option natively. A procedure to create the Dynamic Channel List is stored with the data, and is run with the data.

I must acknowledge Jim Harper for this idea. It never occurred to me that CDF/XML generation can be built into the database. Jim -- without knowing about my original implementation -- explained to me how he thought CDF publishing should work. His idea was so neat, so fitting and beautiful that I had to break my old code and rush to re-implement CDF in the right way.

As usual, an implementation is always messier than the idea; it was even messier than needed as I realized (after some two hours of experimenting and scouring through documentation) that Informix offers no facilities whatsoever to deal with BLOBs within stored procedures. I was very disappointed. I found a way around -- rather ungainly, but operational. It showed me clearly btw why isolation and lock modes do matter.

The Dynamic Channel List has been installed on metcast1 and spaceley. Whenever Jim Harper's Channel Administration makes any change to the MChannels Database -- adds or deletes a channel or a category, updates descriptions, etc. -- the Dynamic Channel List is automatically republished into Channel 14, as a part of an update transaction. The list is then distributed to any currently running Metcast Client, which shows an updated channel list when a user is about to publish or subscribe.

Setting up Metcast Channels under IE 4.x

The only issue that requires attention is making IE recognize a Metcast channel activation record and call an appropriate handler. It's a matter of setting up a proper association:



$Id: History-MChannels.html,v 1.8 2001/01/26 02:49:49 oleg Exp oleg $
Your comments, problem reports, questions are very welcome!