Distributing Weather Products through an HTTP pipe

Last modified on February 12, 2004

General Introduction
  1. Features
  2. Metcast "API"
  3. Metcast vs. Marimba/Pointcast
  4. Metcast vs. NNTP (Network News protocol)
  5. Push Technology

Metcast Channels

  1. Introduction and Overview
  2. Request and delivery of things
  3. Publishing products into a channel
  4. Uploading files via a Metcast Channel
  5. Comparison of Metcast Channels with similar technologies
  6. Usage scenarios
  7. Metcast Channels' Table of Contents
  8. Authentication and Access Restrictions
  9. Metcast Channels Administration Protocol
  10. Implementation Details
  11. References: URLs, source code, executables, documentation

Weather Observation Definition Format (OMF)

  1. Motivation
  2. Structure of an OMF document
    1. Basic attributes and elements
    2. Reports Element
      1. SYN: land or sea surface weather report in a METAR, SPECI, SYNOP, SHIP or BUOY code
      2. UAR Report
      3. BTSC Report (Bathythermal and Salinity Observations)
  3. Aircraft Weather Advisories: (SIGMET, AIRMET)
  4. Terminal Aerodrome Forecasts (TAF)
  5. Plain-text WMO Messages
  6. Post-processing of OMF documents for JMV
    1. Computing of Flight Categories
  7. References (bibliography)
    • Total weather query: reconstruction and presentation of METAR, SPECI, SYNOP, and TAF reports using XSL stylesheets

Metcast Table of Contents (MTOC)

  1. Motivation and Principles
  2. Potential and Actual tables of contents
  3. Structure of a MTOC document
  4. Basic Elements and Attributes
  5. Metcast Channels Table of Contents (CTOC)
  6. Metcast Table of Contents of gridded products (MTOC-grids)
  8. Examples and sample documents

XML Markup for Joint Meteorology and Oceanography Grid Segment: JMGRIB

Technical Specifications and Notes

  1. Metcast Protocols
    1. HTTP as used by Metcast
    2. Request Language
    3. Metcast Table of Contents (MTOC)
    4. Authorization and a Trust-Management system
  2. Metcast Clients
    1. Retriever: a Metcast client application
    2. MIME helpers - decoders for weather content messages
    3. Retriever's configuration file
    4. mailcap: matching MIME messages with helpers
    5. A poor man Metcast Client: a web browser as a Metcast Client
    6. fetch-grid.pl and fetch-grid.py scripts
  3. Server
    1. The Metcast Server
    2. The Metcast Taker
    3. Use cases for handling of gridded data in Metcast
    4. Live demonstration of the Grid Datablade
  4. Brief HTTP 1.1 protocol reference
  5. References (bibliography)

The history of the project

  1. From JMV to Metcast
  2. The original JMV-TNG proposal
  3. The original WebNODDS proposal




A Metcast client submits a request for one or several products to a web server, via HTTP. The web server relays the request to a Metcast application server. The latter is usually hidden behind a firewall. The web server communicates with the Metcast server via a trusted connection: a dedicated hole in a firewall (good only for a specific pair of hosts). The connection could be done via ssh or a similar VPN technique. Furthermore, puncturing a hole in the firewall is not strictly necessary.

A product request is formulated in a a Domain-Specific request language, MBL. The language is similar to XML in spirit; however the language uses a shortened notation, which is easier to parse. The Request Language is not a generic query language: It is is impossible to formulate a SQL query (let alone an UPDATE or DELETE operation) in the Request Language. Therefore, even if the dedicated connection between a web server and the Metcast server is hijacked, the integrity of the Metcast or the database is not compromised. Furthermore, because the Request Language is so removed from SQL, the database query part is hidden from a client. A Metcast client does not know which database a server uses and what that database's schema is -- and if any database is being used at all. You can change the underlying database and its format -- and no Metcast client will ever notice.

A Metcast server replies with a single or a multi-part message, depending on the number of requested products. You can ask for a multitude of gridded, observational, imagery and text products. All these products will be delivered in a single (multi-part) message. Gridded products are sent in GRIB, imagery in TIFF or JPEG, and observational data in OMF XML. The content is accompanied by metadata, which tell the format of the data (grib, tiff, etc.) as well as other attributes (e.g., last modification timestamp).

A Metcast request can be conditional: you can ask for a number of products that were modified -- appeared in the database -- after a specified time moment. The reply from the server will contain only the desired updates, if any.


Making ad hoc requests for any area of interest and any set of products
A user can define any area on the globe and specify any set of products he is interested in: various grid products, observation reports and forecasts, satellite imagery. A NODDS user would feel right at home, as Metcast is a superset of NODDS.
If a user wishes to trade the flexibility of defining his own area and selecting his own products for speed of executing his query, he may wish to try one of pre-canned area-of-interest requests (a.k.a. "thumbnails"). These pre-canned requests give the fastest response as a server maintains a special cache of the corresponding replies.

TCP/IP protocol
Any Metcast client relies on a TCP/IP stack, which is a standard fixture of almost any operating system. This frees us from worries about configurations, modem scripts, connecting to a server, tearing down the connection after a period of inactivity, reconnecting -- the operating system takes care of all this for us.
If a user can browse the Web, he can get Metcast weather products.

Proxies and Gateways
As Metcast uses the same HTTP protocol as the Web itself, Metcast can piggy-back on the extensive Web infrastructure. In particular, weather product traffic can propagate through the existing network of proxies and gateways. Metcast also stands to benefit from various caching schemes gateways frequently employ.

Replication and load-balancing
There are a few commercial services that analyze HTTP traffic patterns and perform automatic replication, redirection, and dynamic load-balancing. Since retrieving a weather product looks and feels to a Web server the same as retrieving an HTML web page, we can employ these replication/load-balancing services as well; no any additional programming or configuration is required.

Metcast also gets it for free. Metcast can take advantage of all authentication mechanisms already built into Web servers: limiting access based on a client's domain or an IP address; validating a user given an id, a password, or other token; establishing a secure transport.

E-mail delivery of products
An HTTP server can easily send weather products via e-mail as HTTP and e-mail rely on the same protocol (MIME) to deliver rich content. The data can be received, archived and "viewed" (processed) by any MIME-compliant e-mail client. We also get encoding/compression capabilities for free: any compliant e-mail client automatically and transparently decodes encoded/compressed messages before presenting them to a user or passing to a helper. See a discussion of transfer-encoding header in MIME [RFC2045] and HTTP [RFC2068] standards for more detail.

Universal decoding of weather products
The way a specific type of weather content is to be decoded and presented is specified by a standard mailcap file. This is the same file that all e-mail clients and web browsers use to find out how to handle a received message of a particular MIME type. Thus a weather product is processed and viewed in identical way no matter how it was sent or received: via e-mail, by a web browser from a web server, by a Metcast retriever from a web server, by a receiver from a Fleet broadcaster, or any other combination.

Broadcasting of weather products
falls right into place with no additional effort. To broadcast a set of products, one launches a regular Metcast retriever, but with a special mailcap file. This will route retrieved MIME-messages with weather content straight to a broadcaster, without any processing. The processing (decoding and presentation) would be done by a receiver on the other end of a broadcast channel, which would probably use metamail to make sense of the data.
Thus, switching Metcast from a request-reply mode to broadcast is as simple as swapping a few configuration files.

Bundling several related products in a digest
A set of weather products can be bundled together in a digest, and delivered as a single multipart MIME message. Any part of the digest can be encoded/compressed if necessary. Digest is the standard fixture of e-mail; the digest message can easily be created by a server on the fly.

Pull vs. push: channels
To receive updates to grids, observations, etc., the Metcast retriever pulls changes from a server. To a user, however, this may look like server 's push. The pull is more efficient, able to support many more concurrent clients, far more sparing on network resources, and is not deterred by firewalls.
Most of the commercial push services (PointCast, Marimba) actually perform a pull. See a footnote Push Technology below for more details.

Metcast "API"

Metcast API is an application of HTTP and, therefore, language neutral. MBL, a Metcast request language, is ASCII- based; it involves no complicated (and thus platform specific) data structures. Metcast is no more favorable to Perl or any other particular language than HTTP is. Any programming language that can read and write files and sockets, and can manipulate ASCII strings may be used to communicate with a Metcast server and receive products. That means literally any language; even Fortran will do. All major HTTP clients (web browsers, that is) are written in C or C++. The original Metcast retriever was written in C++; OmniRetriever is a Java application.

It is indeed possible to talk to a Metcast server from a command-line. For example, you can use fetch-grid.pl or similar script, or even the original Metcast retriever; the latter is a console Win32s application, to be used in a Win9x/WinNT command-line window or a .bat file.

Furthermore, you can talk to a Metcast server without any special application. You can use a regular web browser:

A poor man Metcast Client
MBL Cookbook

You can even telnet to a Metcast server (on port 80) and type in your request. You need to compute its size though to set a content-length header. This is the most cumbersome way. It can be used as the last resort or for trouble-shooting: a telnet client is usually present on any computer.

Metcast vs. Marimba/Pointcast

The following is the comparison between Metcast and other Internet push technologies

All these "push" services, including Metcast, consist of three major components:

So why then, do we need our own version?

   Marimba and Pointcast include proprietary viewers (applets and screensavers) to show headlines of received updates. The updates are typically HTML (web pages), movies, pictures, sound - the type of information one typically comes across when browsing the Web. Therefore, these push services often use a Web browser to display the received info in full.
Weather products are specialized kind of information, which cannot normally be viewed with Netscape. Therefore, no matter what kind of push service one uses, one needs JMV or a similar viewer to display the received weather products. Moreover, one needs a set of helpers to translate data from a "channel format" to something that that viewer can understand. This process is not part of a typical Marimba channel and would have to be developed.

   Pointcast uses a proprietary format for delivered data. With Marimba's SDK, one can send and receive data in any format, but packing and unpacking needs to be specifically programmed. Thus using Marimba does not save any development effort in this respect.
Metcast uses MIME protocol to tag and pack products; Metcast's retriever can unpack the received messages and call the appropriate helpers. This Metcast functionality is already implemented and tested: it works already, and it works with JMV, seamlessly.

   Because Metcast uses standard MIME messages to distribute data, the messages can be sent using HTTP or e-mail transport, using a great variety of commercial or free off-the-shelf HTTP or e-mail servers. This lets us take advantage of authentication/data replication/load-balancing/translation/caching etc. services that are built into HTTP and SMTP servers and agents. Marimba and Pointcast have to develop all this infrastructure from scratch. Marimba and pointcast use proprietary servers such as Castanet which must be licensed for each separate TEDS location.

   When a client polls a server for updates, the server needs to check its database and find out if there is anything of interest to this particular client. Again, since weather information is not a typical Web product, one has to develop code to parse a METOC-specific request and query a METOC database. Neither Marimba nor Pointcast save any development effort in this case either. Marimba offers SDK to help to write this server (in Java), still, one has to write the code. As this server has to use PointCast's or Marimba's proprietary communication channels and protocols, the server becomes vulnerable to API changes (which a vendor can do at any moment), bugs and design deficiencies (pretty common to immature products), etc. On the other hand, Metcast application server uses an industry-standard CGI protocol, which is well-documented and well-supported, and has been around for at least two or three years. At this moment, Metcast can already query custom ISIS/METAR/SAT databases. Code needs to be written to interface TEDS, but one has to write this code no matter what push service one uses.

   When we started the Metcast project we closely examined the other push technologies and tried to take the best parts of each. We wanted to remain as open and standard as possible so that we would not be tied to any single vendor. We believe the Metcast solution achieves this goal. We further believe Marimba and Pointcast use proprietary communication protocols to protect their commercial interests. Metcast does not have to make money from the commercial sector which frees us from constraining the product to protect our livelihoods.


Metcast vs. NNTP (Network News protocol)

"NNTP is a protocol for the distribution, inquiry, retrieval, and posting of news articles using a reliable stream-based transmission of news among the ARPA-Internet community. NNTP is designed so that news articles are stored in a central database allowing a subscriber to select only those items he wishes to read. Indexing, cross-referencing, and expiration of aged messages are also provided. " [RFC-977]

When NNTP is used only to let a number of clients obtain information from a central repository (or several independent repositories), this protocol, in scope, is identical to HTTP. However,

The complexity of NNTP comes from a need to distribute information among inter-related repositories ([RFC-977]):

A host desiring new news, or which has new news to send, will typically contact one or more of its neighbors using NNTP. First it will inquire if any new news groups have been created on the serving host by means of the NEWGROUPS command. If so, and those are appropriate or desired (as established by local site-dependent rules), those new newsgroups can be created.

The client host will then inquire as to which new articles have arrived in all or some of the newsgroups that it desires to receive, using the NEWNEWS command. It will receive a list of new articles from the server, and can request transmission of those articles that it desires and does not already have.

Finally, the client can advise the server of those new articles which the client has recently received. The server will indicate those articles that it has already obtained copies of, and which articles should be sent to add to its collection.

HTTP servers do not need to do any of these.

   It is claimed that "The main thrust will be to sort incoming/outgoing message traffic into newsgroups for specific ships, subjects, etc.". But this is the main NNTP weakness. Let's consider a client in Rota, Spain. It gets all the feed pertinent to the North Atlantic. Let's suppose a particular user wants to check some weather product specific for a particular area in the Pacific. He needs to ask an NNTP administrator (who in turn has to ask an administrator up the feed, etc) to activate the "newsfeed" for this area. After a set-up period, the Pacific weather "newsgroup" starts to trickle to Rota. The user can now obtain the necessary data. He does not need them on the permanent basis, however. Yet no matter if someone reads the incoming information or not, the Pacific feed keeps piling up. To turn it off, a system administrator has to go through a chain of command again.

   With Metcast, if a user needs a particular weather product, it connects to a corresponding server, gets the data, and that is it. If the user needs the data again, it will ask again. It is always up to a client which server to ask and how often. A Metcast server itself is stateless: it tries to satisfy requests of any properly authorized client.

   Metcast's model is the one of a TV broadcast: a user can tune in and out of any station (channel) it can possibly access. In contrast, NNTP's distribution model is magazine subscription. To receive information, one has to specifically subscribe to it. The information then keeps coming up, no matter if one reads it or not. To stop the flow, one has to specifically unsubscribe. Unlike a TV station, a publishing house has to devote resources to maintaining a list of subscribers, distribution schedules, distribution routes, etc.

   The upshot: HTTP, being in a limited sense a subset of NNTP, has the right amount of features for distributing weather information. NNTP's complexity and extra features waste resources and make development more difficult, without adding any additional value. Also, NNTP was designed to distribute only static, already created content, with a non-extensible request syntax that leaves out any geographical queries. In contrast, HTTP can carry a creatable content, built on the fly in response to arbitrary, ad hoc queries.

Push Technology

"After all, is it really a push from a Web site if your computer checks in periodically to detect changes and download them when found? Even if data isn't really being pushed, it can seem like it to users"
Push technologies. Changing the Web experience? Infoworld, Aug 4, 1997, p. 99.

"Push technology can be grouped into five categories, ranging from slightly intrusive to definitely pushy.

A user requests that information be sent automatically through a medium of choice at specified intervals
Using a user preference profile, a push service monitors sources, looks for matches, and forwards the information
Automated pull
Similar to profile, but whole Web pages are often sent based on user's preference configuration
Automated push
A user subscribes to a server such as Pointcast, which uses Web-polling software to send periodic requests for information. Broadcasts are often transmitted during a live Web connection
A user's desktop includes content channels (sport, financial, etc) that constantly broadcast"
Push, Infoworld, Aug 4, 1997, pp. 1 and 60.



$Id: Metcast-Intro.html,v 3.4 2004/02/13 01:07:58 oleg Exp oleg $

Your comments, problem reports, questions are very welcome!