Last modified on February 12, 2004
Weather Observation Definition Format (OMF)
Reports Element
SYN: land or sea surface weather report in a METAR, SPECI, SYNOP, SHIP or BUOY code
UAR Report
BTSC Report (Bathythermal and Salinity Observations)
Metcast Table of Contents (MTOC)
XML Markup for Joint Meteorology
and Oceanography Grid Segment: JMGRIB
Technical Specifications and Notes
The history of the project
A Metcast client submits a request for one or several products to a
web server, via HTTP. The web server relays the request to a Metcast
application server. The latter is usually hidden behind a
firewall. The web server communicates with the Metcast server via a
trusted connection: a dedicated hole in a firewall (good only for a
specific pair of hosts). The connection could be done via
ssh or a similar VPN technique. Furthermore, puncturing a
hole in the firewall is not strictly necessary.
A product request is formulated in a a Domain-Specific request language, MBL. The language is similar to XML in spirit; however the language uses a shortened notation, which is easier to parse. The Request Language is not a generic query language: It is is impossible to formulate a SQL query (let alone an UPDATE or DELETE operation) in the Request Language. Therefore, even if the dedicated connection between a web server and the Metcast server is hijacked, the integrity of the Metcast or the database is not compromised. Furthermore, because the Request Language is so removed from SQL, the database query part is hidden from a client. A Metcast client does not know which database a server uses and what that database's schema is -- and if any database is being used at all. You can change the underlying database and its format -- and no Metcast client will ever notice.
A Metcast server replies with a single or a multi-part message, depending on the number of requested products. You can ask for a multitude of gridded, observational, imagery and text products. All these products will be delivered in a single (multi-part) message. Gridded products are sent in GRIB, imagery in TIFF or JPEG, and observational data in OMF XML. The content is accompanied by metadata, which tell the format of the data (grib, tiff, etc.) as well as other attributes (e.g., last modification timestamp).
A Metcast request can be conditional: you can ask for a number
of products that were modified -- appeared in the database -- after a
specified time moment. The reply from the server will contain only the
desired updates, if any.
transfer-encoding header in MIME [RFC2045] and HTTP [RFC2068] standards for more detail.
mailcap file. This is the same file that all e-mail clients and web browsers use to find out how to handle a received message of a particular MIME type. Thus a weather product is processed and viewed in identical way no matter how it was sent or received: via e-mail, by a web browser from a web server, by a Metcast retriever from a web server, by a receiver from a Fleet broadcaster, or any other combination.
retriever, but with a special mailcap file. This will route retrieved MIME-messages with weather content straight to a broadcaster, without any processing. The processing (decoding and presentation) would be done by a receiver on the other end of a broadcast channel, which would probably use metamail to make sense of the data.
retriever pulls changes from a server. To a user, however, this may look like server 's push. The pull is more efficient, able to support many more concurrent clients, far more sparing on network resources, and is not deterred by firewalls.Metcast API is an application of HTTP and, therefore, language neutral. MBL, a Metcast request language, is ASCII- based; it involves no complicated (and thus platform specific) data structures. Metcast is no more favorable to Perl or any other particular language than HTTP is. Any programming language that can read and write files and sockets, and can manipulate ASCII strings may be used to communicate with a Metcast server and receive products. That means literally any language; even Fortran will do. All major HTTP clients (web browsers, that is) are written in C or C++. The original Metcast retriever was written in C++; OmniRetriever is a Java application.
It is indeed possible to talk to a Metcast server from a command-line. For example, you can use fetch-grid.pl or similar script, or even the original Metcast retriever; the latter is a console Win32s application, to be used in a Win9x/WinNT command-line window or a .bat file.
Furthermore, you can talk to a Metcast server without any special application. You can use a regular web browser:
A poor man Metcast Client
MBL Cookbook
You can even telnet to a Metcast server (on port 80) and type in
your request. You need to compute its size though to set a
content-length header. This is the most cumbersome way. It can be used as the last resort or for trouble-shooting:
a telnet client is usually present on any computer.
The following is the comparison between Metcast and other Internet push technologies
All these "push" services, including Metcast, consist of three major components:
Marimba and Pointcast include proprietary viewers (applets and screensavers) to show headlines of received updates. The updates are
typically HTML (web pages), movies, pictures, sound - the type of
information one typically comes across when browsing the
Web. Therefore, these push services often use a Web browser to display
the received info in full.
Weather products are specialized kind of information, which
cannot normally be viewed with Netscape. Therefore, no matter what
kind of push service one uses, one needs JMV or a similar viewer to
display the received weather products. Moreover, one needs a set of
helpers to translate data from a "channel format" to something that that viewer can
understand. This process is not part of a typical Marimba
channel and would have to be developed.
Pointcast uses a proprietary format for delivered data. With Marimba's SDK, one can send and receive data in any format, but
packing and unpacking needs to be specifically programmed. Thus using
Marimba does not save any development effort in this respect.
Metcast uses MIME protocol to tag and pack products; Metcast's
retriever can unpack the received messages and call the appropriate
helpers. This Metcast functionality is already implemented and tested:
it works already, and it works with JMV, seamlessly.
Because Metcast uses standard MIME messages to distribute data, the messages can be sent using HTTP or e-mail transport, using a great variety of commercial or free off-the-shelf HTTP or e-mail servers. This lets us take advantage of authentication/data replication/load-balancing/translation/caching etc. services that are built into HTTP and SMTP servers and agents. Marimba and Pointcast have to develop all this infrastructure from scratch. Marimba and pointcast use proprietary servers such as Castanet which must be licensed for each separate TEDS location.
When a client polls a server for updates, the server needs to check its database and find out if there is anything of interest to this particular client. Again, since weather information is not a typical Web product, one has to develop code to parse a METOC-specific request and query a METOC database. Neither Marimba nor Pointcast save any development effort in this case either. Marimba offers SDK to help to write this server (in Java), still, one has to write the code. As this server has to use PointCast's or Marimba's proprietary communication channels and protocols, the server becomes vulnerable to API changes (which a vendor can do at any moment), bugs and design deficiencies (pretty common to immature products), etc. On the other hand, Metcast application server uses an industry-standard CGI protocol, which is well-documented and well-supported, and has been around for at least two or three years. At this moment, Metcast can already query custom ISIS/METAR/SAT databases. Code needs to be written to interface TEDS, but one has to write this code no matter what push service one uses.
When we started the Metcast project we closely examined the other push technologies and tried to take the best parts of each. We wanted to remain as open and standard as possible so that we would not be tied to any single vendor. We believe the Metcast solution achieves this goal. We further believe Marimba and Pointcast use proprietary communication protocols to protect their commercial interests. Metcast does not have to make money from the commercial sector which frees us from constraining the product to protect our livelihoods.
"NNTP is a protocol for the distribution, inquiry, retrieval, and posting of news articles using a reliable stream-based transmission of news among the ARPA-Internet community. NNTP is designed so that news articles are stored in a central database allowing a subscriber to select only those items he wishes to read. Indexing, cross-referencing, and expiration of aged messages are also provided. " [RFC-977]
When NNTP is used only to let a number of clients obtain information from a central repository (or several independent repositories), this protocol, in scope, is identical to HTTP. However,
INN, a popular NNTP server to work, one has to adjust a number of configuration files, set up cron jobs, arrange for a feed, etc.
The complexity of NNTP comes from a need to distribute information among inter-related repositories ([RFC-977]):
A host desiring new news, or which has new news to send, will typically contact one or more of its neighbors using NNTP. First it will inquire if any new news groups have been created on the serving host by means of the NEWGROUPS command. If so, and those are appropriate or desired (as established by local site-dependent rules), those new newsgroups can be created.HTTP servers do not need to do any of these.The client host will then inquire as to which new articles have arrived in all or some of the newsgroups that it desires to receive, using the NEWNEWS command. It will receive a list of new articles from the server, and can request transmission of those articles that it desires and does not already have.
Finally, the client can advise the server of those new articles which the client has recently received. The server will indicate those articles that it has already obtained copies of, and which articles should be sent to add to its collection.
It is claimed that "The main thrust will be to sort incoming/outgoing message traffic into newsgroups for specific ships, subjects, etc.". But this is the main NNTP weakness. Let's consider a client in Rota, Spain. It gets all the feed pertinent to the North Atlantic. Let's suppose a particular user wants to check some weather product specific for a particular area in the Pacific. He needs to ask an NNTP administrator (who in turn has to ask an administrator up the feed, etc) to activate the "newsfeed" for this area. After a set-up period, the Pacific weather "newsgroup" starts to trickle to Rota. The user can now obtain the necessary data. He does not need them on the permanent basis, however. Yet no matter if someone reads the incoming information or not, the Pacific feed keeps piling up. To turn it off, a system administrator has to go through a chain of command again.
With Metcast, if a user needs a particular weather product, it connects to a corresponding server, gets the data, and that is it. If the user needs the data again, it will ask again. It is always up to a client which server to ask and how often. A Metcast server itself is stateless: it tries to satisfy requests of any properly authorized client.
Metcast's model is the one of a TV broadcast: a user can tune in and out of any station (channel) it can possibly access. In contrast, NNTP's distribution model is magazine subscription. To receive information, one has to specifically subscribe to it. The information then keeps coming up, no matter if one reads it or not. To stop the flow, one has to specifically unsubscribe. Unlike a TV station, a publishing house has to devote resources to maintaining a list of subscribers, distribution schedules, distribution routes, etc.
The upshot: HTTP, being in a limited sense a subset of NNTP, has the right amount of features for distributing weather information. NNTP's complexity and extra features waste resources and make development more difficult, without adding any additional value. Also, NNTP was designed to distribute only static, already created content, with a non-extensible request syntax that leaves out any geographical queries. In contrast, HTTP can carry a creatable content, built on the fly in response to arbitrary, ad hoc queries.
"Push technology can be grouped into five categories, ranging from slightly intrusive to definitely pushy.