Metcast Table of Contents (MTOC)

Motivation

A Metcast server can be asked to deliver a wide variety of MetOc products -- imagery, 2D and 3D grids, observation reports, warnings and advisories. To formulate a request however, a user needs to know what is available, and how to ask for it. In particular, the user has to know the names and categories of products as well as their request attributes -- levels, taus, bounding boxes, etc. A menu of data types, let alone the list of the currently available products, is specific to each particular server, and is subject to change.

Every Metcast server therefore offers a reflexive facility: an ability to describe what it can serve, both in principle and in reality. In other words, the server can send the comprehensive menu and indicate which products on this list are actually present in a Metcast database and thus available for immediate download. A server can also tell how to build a Request Language phrase to ask for a particular piece of information.

In response to a Describe query, a server sends an XML document -- a Metcast Table of Contents. The present web page specifies the format and semantics of that document.

A Metcast Table of Contents should describe available products both in human and computer terms. Each product should contain a descriptive textual label that may be presented to a user. On the other hand, a MTOC document should have an obvious mapping to the Request Language: when the user chooses a product off the list, a Metcast client should be able to easily build the corresponding request. Another requirement for a MTOC document is compactness. Indeed, MTOC may be the most frequently asked product. Since a database may contain hundreds of images and grids, the description of each separate product has to be as concise as possible.

Principles

Potential and Actual tables of contents

There should be two table of contents. One, potential TOC, lists the products that we have agreements to receive and store in the database. The potential TOC shows what could potentially be in the database -- although not everything will actually be. The potential TOC is the most useful for scheduling of product deliveries from a Metcast server. The potential TOC is not dynamic. In fact, it should be relatively static. It is only changed when we, for example, reached an agreement to take MM5 model data from USAF or started decoding international SIGMETs.

The second table of contents is the actual one. It is generated from the contents of the database at the time of the request. The actual TOC is the most useful for ad hoc and mission-planning requests.

Several levels of detail for the Potential TOC

The potential TOC can be retrieved at several levels of detail. At least two levels seem necessary. At the most coarse level, the TOC should merely list categories of products: grids, satellite imagery, remotely sensed data, synoptic reports. A user can then request a potential TOC of grids. In that case, he would get the description of all gridded products that we have agreements to store and re-distribute. Again, this list is mostly static; therefore, such requests should be infrequent and should not take too much of the bandwidth. Perhaps we can store two versions of the potential TOC: for low users and for cleared users. Since this is a potential TOC, the coarse granularity of the access control is perhaps sufficient. A Metcast server will check users' authorization when it comes to delivery of the actual data.

A user may also ask for several categories of products in one TOC request. The root element in the response TOC XML document will enclose the descriptions of the corresponding products.

Several levels of detail for the Actual TOC

The actual TOC should also be available at several levels of detail. We do want to make available more levels of detail and more associative queries. For example, a user should be able to see:

• if the server can deliver grids
• if any grids are actually available
• all the gridded products available for download, for this particular user
• all the products that are relevant for a given area of interest
• all the products whose name contain the word 'temperature'
• the list of all the models
• the products computed by a particular model with the valid time within a given range

Taus and forecast times

The potential TOC should list forecast times (aka 'taus'), which a grid model is configured to compute. In contrast, the actual TOC should list valid-times and reference-times for computed products. There should be no taus in the actual TOC (nor in actual products).

Compatibility with a Dynamic Product List (DPL)

No backwards compatibility with the current JMV and a so-called dynamic product list (DPL) is planned. If JMV wants to take products from the new Metcast server, it should learn to understand the new TOC. We have been talking about deficiencies of DPL for at least three years. It is high time we did something about it.

Output Formats

For gridded data, NetCDF (GIEF dialect) first, JMGRIB for small grids later. Wind and similar vector products are delivered in GIEF-W. Conversion to GRIB can be relegated to a client. All our potential clients seem far more comfortable with NetCDF than with GRIB. Satellite imagery should be available in NetCDF first and also in GeoTIFF later. METARs/TAFs etc. are in OMF.

Unit names

We will use UNIDATA's UDUNITS library [UDUNITS] and its unit naming conventions, including the conventions for time units and time intervals.

The format of timestamps

A single format for timestamps and time intervals proved to be difficult to agree upon. Therefore, any time quantity must be accompanied by an attribute that specifies the units, in the UDUNITS [UDUNITS] format. The time units attribute will be defined with an appropriate default value. It is unlikely that actual XML documents will need to specify this attribute explicitly.

For example, let a file Sample.DTD has the following content

     <!ELEMENT Tag1 ANY>
     <!ATTLIST Tag1 TStamp NMTOKENS #REQUIRED
                    TStamp-unit NMTOKENS 'seconds since 1970-01-01'>
     
     <!ELEMENT Tag2 ANY>
     <!ATTLIST Tag2 TStamp NMTOKENS #REQUIRED
                    TStamp-unit NMTOKENS ''>

In the following XML document

     <!DOCTYPE Tag1 SYSTEM 'Sample.DTD'>
     <Tag1 TStamp='1029557742'/>

On the other hand, the default timestamp format in the following document

     <!DOCTYPE Tag2 SYSTEM 'Sample.DTD'>
     <Tag2 TStamp='2002-8-16 23:12'/>

     <!DOCTYPE Tag1 SYSTEM 'Sample.DTD'>
     <Tag1 TStamp='2002-8-16 23:17:00.00 -07:00' TStamp-unit=''>
       <Tag2 TStamp='60' TStamp-unit='minutes since 2002-8-16 23:17:00.00'/>
     </Tag1>

Indeterminate time periods (e.g., forecast times) should also be accompanied by units, for example, 'hrs' or 'mins'

Any Metcast client must handle time values in units seconds since 1970-01-01 (that is, in Epoch seconds). For maximum flexibility, the client should use the UDUNITS library.

Structure of a MTOC document

The Metcast Table of Contents format is an application of the Extensible Markup Language (XML).

A MTOC document may contain the following elements:

• MTOC - the top level (root) element that describes a set of products available from a particular Metcast server
• imagery - describes available imagery products
• grids - lists 2D gridded products that can currently be requested
• syn-obs - synoptic reports of various kinds
• syn-forecasts - shiproute, aviation, hydrological etc. forecasts
• syn-warnings - warnings and advisories
• wmo-messages - WMO plain-text messages
• channels - "things," products in a Metcast Channels database
• rem-sensing - remotely-sensed data
• item - a minor element describing one product that does not have further query attributes

The following sections define the major elements along with the minor elements that are relevant to them. In each section, XML DTD declarations are provided for precise definition of elements and attributes. The collection of XML DTD declarations found in this specification can be arbitrarily extended to add new elements and attributes for new enhancements.

The compactness requirement specified in the previous section implies that large, repetitive, relatively static pieces of information should be separated from more volatile descriptions. For example, every grid product is characterized by levels and physical units. Rather than quoting their full names and descriptions with every product, it makes sense to factor out this information into separate "tables". A product description item would simply refer to the corresponding units and levels by their unique ids. This is quite similar to table normalizations so common in data modeling and relational database design.

Furthermore, the names of standard levels or common projections for gridded data change very infrequently. It seems appropriate to include these tables as General Entities within the MTOC DTD. When a Metcast server generates a MTOC reply, the server does not need to place these tables in the output document: the server may merely refer to them by their GE references. An XML parser in a Metcast Client will transparently substitute the entities (the table of units, for example) for their references, using DTD.

Some of MTOC elements and attributes are common. For compactness, they are defined in the following special section.

Basic Attributes

The following attributes provide basic annotation information. These attributes may (and often must) be present in appropriate elements describing products.

<!ENTITY % TStamp-type "NMTOKENS" >
<!ENTITY % TStamp "TStamp %TStamp-type; #REQUIRED" >
<!ENTITY % TStamp-OPT "TStamp %TStamp-type; #IMPLIED" >
<!ENTITY % TStamp-units "TStamp-units NMTOKENS 'seconds since 1970-01-01'" >
<!ENTITY % TRange-type "NMTOKENS" >
<!ENTITY % TRange "TRange %TRange-type; #REQUIRED" >
<!ENTITY % Size-OPT "Size NMTOKEN #IMPLIED" >
<!ENTITY % LatLon "LatLon NMTOKENS #REQUIRED" >
<!ENTITY % LatLons "LatLons NMTOKENS #REQUIRED" >
<!ENTITY % BBox-REQD "BBox NMTOKENS #REQUIRED" >
<!ENTITY % BBox-OPT "BBox NMTOKENS #IMPLIED" >
<!ENTITY % Priority "Priority (urgent|bulk|aux) #IMPLIED" >
<!ENTITY % HREF-redir "HREF CDATA #IMPLIED" >
<!ENTITY % Units "Units CDATA #IMPLIED" >

Name and Title attributes

An MTOC element describes either one product, a category, or a set of related products. The products or sets are identified by their symbolic ids; they also carry textual labels. The identifiers are to be used in MBL phrases sent to a Metcast server. Labels are meant to be presented to a user.

The start tag of an element is often the identifier of the product described by the element. When this is not the case, the product identifier is given in a Name attribute. The product id may be descriptive enough to be shown to a user. A Title attribute if present gives a more detailed, human-readable label. Often these attributes are mentioned only in the DTD, with the default values appropriate for a particular element. A MTOC document therefore rarely includes these attributes explicitly.

In presenting an MTOC element to a user, one has to follow the following rules:

• If an attribute Title is present, use its value as a textual label;
• Otherwise, use the value of the attribute Name, if present;
• Otherwise, the symbolic tag name of the element is assumed to be descriptive enough.

Basic Elements

Certain annotation information -- for example, a description of a product that can essentially be characterized merely by its name -- is rather common. The corresponding (minor) elements therefore occur frequently inside other elements. For convenience, definitions of these minor elements are collected in the present section.

item Element

An item describes a product that is specified solely by its name (and perhaps appearance within the context set by parent elements). For example, SHIP and BUOY reports can be requested just by their product name, SYNSEA.

An item element describes an atomic piece of information (a product), and therefore does not have any children elements, or any other content at all. The item's attributes tell the (catalog) name of the product and perhaps ancillary information (e.g., timestamp). Often the product name alone is descriptive enough. If the catalog name is too cryptic a Title attribute may be present to give a more readable descriptive string.

<!ELEMENT item EMPTY>

Attributes

<!ATTLIST item
	Name CDATA #REQUIRED
	Title CDATA #IMPLIED
	%TStamp-OPT;
	%Size-OPT;
	%Priority;
>

MTOC Element

This is the root element of an MTOC document. Its child elements describe particular categories of products supported by a Metcast server.

<!ELEMENT MTOC ( grids | imagery | METAR | UAR | TAF | BTSC | SYNSEA | AC-report |
          AC-advisory | syn-warnings | rem-sensing | wmo-messages |
          channels )+>

Attributes

<!ATTLIST MTOC
	%TStamp;
	HREF CDATA #REQUIRED
>

Gridded Products

The table of contents for the gridded products, represented by the MTOC element grids, is described in a separate document [MTOC-grids].

Metcast Channels

The table of contents of the Metcast Channels, represented by the MTOC element channels, is described in a separate document [CTOC].

Satellite imagery

The imagery element describes pictorial products currently available from a Metcast server. This facility is currently implemented as a dedicated Metcast Channel MCASTIMG. See the Metcast Channels [CTOC] for more information. The imagery element is currently provided for backward and forward compatibility.

<!ELEMENT imagery EMPTY>

Attributes

<!ATTLIST imagery
	%TStamp-OPT;
	Title CDATA 'Satellite imagery: see MCASTIMG Metcast Channel'
>

Synoptic Products

Elements METAR, TAF, AC-advisory, etc. describe synoptic reports, forecasts, warnings and advisories that are available from a particular Metcast server.

<!ELEMENT METAR EMPTY>

<!ELEMENT AC-report EMPTY>

Attributes

<!ATTLIST METAR
	%TStamp-OPT;
	Title CDATA 'Surface synoptic reports from land stations'
>

Attributes

<!ATTLIST AC-report
	%TStamp-OPT;
	Title CDATA 'Aircraft Reports: PIREP, AIREP, and AMDAR/ACAR'
>

Examples

Potential TOC, the coarsest level

     <!DOCTYPE MTOC SYSTEM "MTOC.dtd">
     <MTOC TStamp='1029557742' HREF='http://zowie/cgi-bin/oleg/server'>
       <!-- The following says that the server on zowie does not
            have imagery, but it knows that the server on tourmaline
            does. The client should inquire tourmaline -->
     <imagery HREF='http://tourmaline/cgi-bin/mcsrvr/server/MTOC/imagery/'/>
         <!-- the following attribute is the default. It is
              spelled out for clarity -->
     <grids 
         HREF='grids/'
      />
     &SYN-PRODUCTS;
     </MTOC>

References

[ISO8601] ISO 8601. Date and Time formats
< http://www.mcs.vuw.ac.nz/technical/software/SGML/doc/iso8601/ISO8601.html>
< http://www.cl.cam.ac.uk/~mgk25/iso-time.html>

[UDUNITS] UDUNITS: A library for manipulating units of physical quantities
< http://www.unidata.ucar.edu/packages/udunits/>

[CTOC] Metcast Channels Table of Contents (CTOC)
< CTOC.html>

[MTOC-grids] Metcast Table of Contents of gridded products (MTOC-grids)
< MTOC-grids.html>

[MTOC-DTD] MTOC DTD
< http://www.metnet.navy.mil/Metcast/XML/MTOC.dtd>

[MTOC-sample] A sample MTOC document
< PTOC-sample.xml>

[MTOC-ABS] The list of all products potentially available from FNMOC
< PTOC-ABS.xml>

[Metcast-Intro] Introduction to Metcast
< http://www.metnet.navy.mil/Metcast/Metcast-Intro.html>

[MBL] A Guide to MBL - a Metcast Request Language
< http://www.metnet.navy.mil/Metcast/Request-Lang.html>

Last updated May 22, 2003

Converted from SXML by SXML->HTML

$Id: MTOC.scm,v 4.4 2003/05/23 02:11:22 oleg Exp oleg $