Metcast Table of Contents (MTOC)

  1. Motivation
  2. Principles
    1. Potential and Actual tables of contents
    2. Several levels of detail for the Potential TOC
    3. Several levels of detail for the Actual TOC
    4. Taus and forecast times
    5. Compatibility with a Dynamic Product List (DPL)
    6. Output Formats
    7. Unit names
    8. The format of timestamps
  3. Structure of a MTOC document
    1. Basic Attributes
    2. Name and Title attributes
    3. Basic Elements
      1. item Element
    4. MTOC Element
  4. Gridded Products
  5. Metcast Channels
  6. Satellite imagery
  7. Synoptic Products
  8. Examples
    1. Potential TOC, the coarsest level
  9. References



A Metcast server can be asked to deliver a wide variety of MetOc products -- imagery, 2D and 3D grids, observation reports, warnings and advisories. To formulate a request however, a user needs to know what is available, and how to ask for it. In particular, the user has to know the names and categories of products as well as their request attributes -- levels, taus, bounding boxes, etc. A menu of data types, let alone the list of the currently available products, is specific to each particular server, and is subject to change.

Every Metcast server therefore offers a reflexive facility: an ability to describe what it can serve, both in principle and in reality. In other words, the server can send the comprehensive menu and indicate which products on this list are actually present in a Metcast database and thus available for immediate download. A server can also tell how to build a Request Language phrase to ask for a particular piece of information.

In response to a Describe query, a server sends an XML document -- a Metcast Table of Contents. The present web page specifies the format and semantics of that document.

A Metcast Table of Contents should describe available products both in human and computer terms. Each product should contain a descriptive textual label that may be presented to a user. On the other hand, a MTOC document should have an obvious mapping to the Request Language: when the user chooses a product off the list, a Metcast client should be able to easily build the corresponding request. Another requirement for a MTOC document is compactness. Indeed, MTOC may be the most frequently asked product. Since a database may contain hundreds of images and grids, the description of each separate product has to be as concise as possible.




Potential and Actual tables of contents

There should be two table of contents. One, potential TOC, lists the products that we have agreements to receive and store in the database. The potential TOC shows what could potentially be in the database -- although not everything will actually be. The potential TOC is the most useful for scheduling of product deliveries from a Metcast server. The potential TOC is not dynamic. In fact, it should be relatively static. It is only changed when we, for example, reached an agreement to take MM5 model data from USAF or started decoding international SIGMETs.

The second table of contents is the actual one. It is generated from the contents of the database at the time of the request. The actual TOC is the most useful for ad hoc and mission-planning requests.


Several levels of detail for the Potential TOC

The potential TOC can be retrieved at several levels of detail. At least two levels seem necessary. At the most coarse level, the TOC should merely list categories of products: grids, satellite imagery, remotely sensed data, synoptic reports. A user can then request a potential TOC of grids. In that case, he would get the description of all gridded products that we have agreements to store and re-distribute. Again, this list is mostly static; therefore, such requests should be infrequent and should not take too much of the bandwidth. Perhaps we can store two versions of the potential TOC: for low users and for cleared users. Since this is a potential TOC, the coarse granularity of the access control is perhaps sufficient. A Metcast server will check users' authorization when it comes to delivery of the actual data.

A user may also ask for several categories of products in one TOC request. The root element in the response TOC XML document will enclose the descriptions of the corresponding products.


Several levels of detail for the Actual TOC

The actual TOC should also be available at several levels of detail. We do want to make available more levels of detail and more associative queries. For example, a user should be able to see:


Taus and forecast times

The potential TOC should list forecast times (aka 'taus'), which a grid model is configured to compute. In contrast, the actual TOC should list valid-times and reference-times for computed products. There should be no taus in the actual TOC (nor in actual products).


Compatibility with a Dynamic Product List (DPL)

No backwards compatibility with the current JMV and a so-called dynamic product list (DPL) is planned. If JMV wants to take products from the new Metcast server, it should learn to understand the new TOC. We have been talking about deficiencies of DPL for at least three years. It is high time we did something about it.


Output Formats

For gridded data, NetCDF (GIEF dialect) first, JMGRIB for small grids later. Wind and similar vector products are delivered in GIEF-W. Conversion to GRIB can be relegated to a client. All our potential clients seem far more comfortable with NetCDF than with GRIB. Satellite imagery should be available in NetCDF first and also in GeoTIFF later. METARs/TAFs etc. are in OMF.


Unit names

We will use UNIDATA's UDUNITS library [UDUNITS] and its unit naming conventions, including the conventions for time units and time intervals.


The format of timestamps

A single format for timestamps and time intervals proved to be difficult to agree upon. Therefore, any time quantity must be accompanied by an attribute that specifies the units, in the UDUNITS [UDUNITS] format. The time units attribute will be defined with an appropriate default value. It is unlikely that actual XML documents will need to specify this attribute explicitly.

For example, let a file Sample.DTD has the following content

     <!ELEMENT Tag1 ANY>
                    TStamp-unit NMTOKENS 'seconds since 1970-01-01'>
     <!ELEMENT Tag2 ANY>
                    TStamp-unit NMTOKENS ''>

In the following XML document

     <!DOCTYPE Tag1 SYSTEM 'Sample.DTD'>
     <Tag1 TStamp='1029557742'/>
the timestamp is understood to be (UNIX) Epoch seconds. All timestamps in OMF currently adhere to this convention. The TOC documents may adopt the convention too.

On the other hand, the default timestamp format in the following document

     <!DOCTYPE Tag2 SYSTEM 'Sample.DTD'>
     <Tag2 TStamp='2002-8-16 23:12'/>
is ISO8601. See ISO 8601:1988 and ISO 8601:2000 for more detail [ISO8601]. The default time units can always be over-written by the author of a document, for example:
     <!DOCTYPE Tag1 SYSTEM 'Sample.DTD'>
     <Tag1 TStamp='2002-8-16 23:17:00.00 -07:00' TStamp-unit=''>
       <Tag2 TStamp='60' TStamp-unit='minutes since 2002-8-16 23:17:00.00'/>

Indeterminate time periods (e.g., forecast times) should also be accompanied by units, for example, 'hrs' or 'mins'

Any Metcast client must handle time values in units seconds since 1970-01-01 (that is, in Epoch seconds). For maximum flexibility, the client should use the UDUNITS library.


Structure of a MTOC document

The Metcast Table of Contents format is an application of the Extensible Markup Language (XML).

A MTOC document may contain the following elements:

The following sections define the major elements along with the minor elements that are relevant to them. In each section, XML DTD declarations are provided for precise definition of elements and attributes. The collection of XML DTD declarations found in this specification can be arbitrarily extended to add new elements and attributes for new enhancements.

The compactness requirement specified in the previous section implies that large, repetitive, relatively static pieces of information should be separated from more volatile descriptions. For example, every grid product is characterized by levels and physical units. Rather than quoting their full names and descriptions with every product, it makes sense to factor out this information into separate "tables". A product description item would simply refer to the corresponding units and levels by their unique ids. This is quite similar to table normalizations so common in data modeling and relational database design.

Furthermore, the names of standard levels or common projections for gridded data change very infrequently. It seems appropriate to include these tables as General Entities within the MTOC DTD. When a Metcast server generates a MTOC reply, the server does not need to place these tables in the output document: the server may merely refer to them by their GE references. An XML parser in a Metcast Client will transparently substitute the entities (the table of units, for example) for their references, using DTD.

Some of MTOC elements and attributes are common. For compactness, they are defined in the following special section.


Basic Attributes

The following attributes provide basic annotation information. These attributes may (and often must) be present in appropriate elements describing products.

  Time Stamp unsigned integer or varchar
The time stamp is expressed in the units specified by the TStamp-units attribute
Example: 2001-04-17T1457, 1046218808
  Required Time Stamp attribute
TStamp-OPT ( TStamp)
  Optional Time Stamp attribute or omitted
  The units of time within the element where this attribute occurs. varchar
The default time units is UTC seconds since the Epoch, 00:00:00 Jan 1, 1970 UTC. This is the value returned by a POSIX function time(2).
  The range or interval between two time stamps two unsigned integers or varchars
The time stamps are expressed in the units specified by the TStamp-units attribute
Example: 1046218808 1046219808
  Required time range attribute
Size-OPT ( Size)
  The size of a resource number or varchar or omitted
The size is a number (optionally followed by a letter K, M or G).
This attribute tells either the exact or an estimated size of the corresponding resource, in bytes (or KB, MB or GB). A client can use this information to estimate download time or warn the user of a potentially bulk product.
  Specification of a point on the globe two numbers
LatLon is a string of a form "aaa.bbb ccc.ddd" where aaa.bbb and ccc.ddd are signed floating point numbers of the latitude resp. longitude of a point on the globe, in whole and fractional degrees.
The numbers are positive for Northern latitudes resp. Eastern longitudes, and negative otherwise.
The range of the numbers is [-90.0, 90.0] for the latitudes, (-180.0, 180.0] for the longitudes.
  Specification of a sequence of points on the globe list of numbers
LatLons is a string of a form "lat1 lon1 lat2 lon2 latn lonn" where lat1, lon1, etc. are signed floating point numbers telling the latitude resp. longitude of a point on the globe, in whole and fractional degrees.
See the LatLon attribute above for more details.
BBox-REQD ( BBox)
  Bounding box, a specification of the latitudal and the longitudal spans of an area of the globe four numbers
BBox is a string of a form "lat-N lon-W lat-S lon-E"
Here lat-N is the latitude of the Northern-most point of the area, lat-S is the latitude of the Southern-most point, lon-W is the longitude of the Western-most point of the area, and lon-E is the Eastern-most longitude.
The numbers lats and lons are signed floating-point numbers, in degrees. See a LatLon attribute above for more details.
It is required that lat-N >= lat-S. The left longitude ( lon-W) may however be greater than the right longitude ( lon-E). For example, a range of longitudes [-170,170] specifies the entire world but Indonesia. On the other end, the range [170,-170] includes Indonesia only. By the same token, [-10,10] pertains to a 21-degree longitude strip along the Greenwich meridian, whereas [10,-10] specifies the whole globe but that strip.
BBox-OPT ( BBox)
  Optional Bounding box four numbers or omitted
  A degree of importance associated with a product enum or omitted
Certain products (for example, tropical cyclone warnings) may be labeled by a Metcast server as very important. A Metcast client should consider always subscribing to these products. When submitting a request, the client should specify urgent products first. On the other hand, some other products may be too bulky; the client may tell that to the user, and request them lastly. A product may be deemed essential or valuable in forecasting, or auxiliary.
HREF-redir ( HREF)
  A re-direction URL varchar or omitted
The value of this attribute is a URL to a Metcast server. This attribute, if present in a particular MTOC product element, specifies that all requests regarding the product should be sent to that Metcast server.
  The units of measurement varchar or omitted
The units of measurements should be specified in the format of the [UDUNITS] package.

<!ENTITY % TStamp-type "NMTOKENS" >
<!ENTITY % TStamp "TStamp %TStamp-type; #REQUIRED" >
<!ENTITY % TStamp-OPT "TStamp %TStamp-type; #IMPLIED" >
<!ENTITY % TStamp-units "TStamp-units NMTOKENS 'seconds since 1970-01-01'" >
<!ENTITY % TRange-type "NMTOKENS" >
<!ENTITY % TRange "TRange %TRange-type; #REQUIRED" >
<!ENTITY % Priority "Priority (urgent|bulk|aux) #IMPLIED" >
<!ENTITY % Units "Units CDATA #IMPLIED" >


Name and Title attributes

An MTOC element describes either one product, a category, or a set of related products. The products or sets are identified by their symbolic ids; they also carry textual labels. The identifiers are to be used in MBL phrases sent to a Metcast server. Labels are meant to be presented to a user.

The start tag of an element is often the identifier of the product described by the element. When this is not the case, the product identifier is given in a Name attribute. The product id may be descriptive enough to be shown to a user. A Title attribute if present gives a more detailed, human-readable label. Often these attributes are mentioned only in the DTD, with the default values appropriate for a particular element. A MTOC document therefore rarely includes these attributes explicitly.

In presenting an MTOC element to a user, one has to follow the following rules:


Basic Elements

Certain annotation information -- for example, a description of a product that can essentially be characterized merely by its name -- is rather common. The corresponding (minor) elements therefore occur frequently inside other elements. For convenience, definitions of these minor elements are collected in the present section.


item Element

An item describes a product that is specified solely by its name (and perhaps appearance within the context set by parent elements). For example, SHIP and BUOY reports can be requested just by their product name, SYNSEA.

An item element describes an atomic piece of information (a product), and therefore does not have any children elements, or any other content at all. The item's attributes tell the (catalog) name of the product and perhaps ancillary information (e.g., timestamp). Often the product name alone is descriptive enough. If the catalog name is too cryptic a Title attribute may be present to give a more readable descriptive string.



  Name under which the product is cataloged in a particular Metcast database string
  A textual description of the item intended for a user string or omitted
If omitted, the value of the Name attribute is assumed descriptive enough.
  Time Stamp of this product  
This attribute is present only if a Describe request specifically asked for timestamps of every available product.
  An exact or estimate size of the product  
This attribute is present only if a Describe request specifically asked for sizes of every available products.
  A degree of importance associated with a product  

<!ATTLIST item


MTOC Element

This is the root element of an MTOC document. Its child elements describe particular categories of products supported by a Metcast server.

<!ELEMENT MTOC ( grids | imagery | METAR | UAR | TAF | BTSC | SYNSEA | AC-report |
          AC-advisory | syn-warnings | rem-sensing | wmo-messages |
          channels )+>


  Time Stamp of this MTOC document (when it was created)  
  A URL of the server that generated this product list. varchar



Gridded Products

The table of contents for the gridded products, represented by the MTOC element grids, is described in a separate document [MTOC-grids].


Metcast Channels

The table of contents of the Metcast Channels, represented by the MTOC element channels, is described in a separate document [CTOC].


Satellite imagery

The imagery element describes pictorial products currently available from a Metcast server. This facility is currently implemented as a dedicated Metcast Channel MCASTIMG. See the Metcast Channels [CTOC] for more information. The imagery element is currently provided for backward and forward compatibility.

<!ELEMENT imagery EMPTY>


  Time Stamp of the moment this description was created  
This attribute may be omitted, in which case a TStamp attribute of the parent MTOC element is assumed.
  A textual description of the item intended for a user string

<!ATTLIST imagery
	Title CDATA 'Satellite imagery: see MCASTIMG Metcast Channel'


Synoptic Products

Elements METAR, TAF, AC-advisory, etc. describe synoptic reports, forecasts, warnings and advisories that are available from a particular Metcast server.



  Time Stamp of the moment this description was created  
This attribute may be omitted, in which case a TStamp attribute of the parent MTOC element is assumed.
  A textual description of the item intended for a user string

	Title CDATA 'Surface synoptic reports from land stations'


  Time Stamp of the moment this description was created  
This attribute may be omitted, in which case a TStamp attribute of the parent MTOC element is assumed.
  A textual description of the item intended for a user string

<!ATTLIST AC-report
	Title CDATA 'Aircraft Reports: PIREP, AIREP, and AMDAR/ACAR'




Potential TOC, the coarsest level

     <MTOC TStamp='1029557742' HREF='http://zowie/cgi-bin/oleg/server'>
       <!-- The following says that the server on zowie does not
            have imagery, but it knows that the server on tourmaline
            does. The client should inquire tourmaline -->
     <imagery HREF='http://tourmaline/cgi-bin/mcsrvr/server/MTOC/imagery/'/>
         <!-- the following attribute is the default. It is
              spelled out for clarity -->



[ISO8601] ISO 8601. Date and Time formats

[UDUNITS] UDUNITS: A library for manipulating units of physical quantities

[CTOC] Metcast Channels Table of Contents (CTOC)
< CTOC.html>

[MTOC-grids] Metcast Table of Contents of gridded products (MTOC-grids)
< MTOC-grids.html>


[MTOC-sample] A sample MTOC document
< PTOC-sample.xml>

[MTOC-ABS] The list of all products potentially available from FNMOC
< PTOC-ABS.xml>

[Metcast-Intro] Introduction to Metcast

[MBL] A Guide to MBL - a Metcast Request Language

Last updated May 22, 2003

Converted from SXML by SXML->HTML

$Id: MTOC.scm,v 4.4 2003/05/23 02:11:22 oleg Exp oleg $