The following message was sent on Jun 2, 1995 (sic!) to a large group of people. A few abbreviations may appear confusing: it was whole two years before a word "Metcast" was first uttered; even JMV was called differently. All we had was NODDS. This message -- although written so many years ago -- appears still valid and up-to-date. For example, a proposed AIPS communication protocol seems rather close to what I suggested back then. Furthermore, a few old proposals of mine are yet to be implemented.
Web appears so hot that it's burning its way into the Navy, too. I think that it's a wonderful idea, to weave the Web over NODDS. But this will also mean an almost complete redesign/reprogramming of Nodds, albeit the outward appearance would change little.
Let me first tell how I imagine the new WebNodds work. You start with a map of the world or of the U.S.A., whatever, like in the Xerox Map server (http://pubweb.parc.xerox.com/map). You can zoom in/out etc. until you nail down the area you're interested in. Alternatively, you may also want to pick among a number of predefined areas. Then you can select weather products from a list of products or product groups. Click on the submit button, and enjoy what you've got in a standard "weather map" view. Sounds familiar? Hang on... The server may have recognized you (or your site) and, instead of that generic "define area" fluff, it can give you the list of your personal requests that you have defined/asked for before. So, you can skip on all this area definition and communication, and simply say, "I want my usual". Also, it's very easy for the server to determine if there are new products showed up in the database since the last time you talked to the server. If there are, and you want to take a look, the server will give you a list. Because the server knows who it is talking to, it may also hide some products which are "for your eyes only".
How's it different from what NODDS already does? On the surface of it, very little. But looking under the hood, a lot. For one thing, product lists and other configuration information is now stored on the server, not on a client site. So, the information always stays fresh. Also, there is no need to waste bandwidth broadcasting product etc. updates to all clients, even those who already got updates, or who aren't interested. Secondly, defined areas are also stored on the server (cataloged under site's "name"). This arrangement cuts down on the communication time: the only information sent between the client and the server during the initial dialog is just the names of areas. If you know what a name stands for, and you want the corresponding request, you just tell the server its name. There is no need to send the whole list of products being requested to the server, very often all the same list, and all over again: the list of products (with all needed conf info and preferences) is already on the server. Keeping it there also helps in billing (should such feature be necessary) and accounting.
The current NODDS server returns fetched information in the
form of a packed.dat file. And here comes my biggest gripe: packed.dat
is not self-contained. To display the data it carries, you need a defined
NODDS area, because packed.dat itself contains no geographical coordinates in itself; it does not even carry the name of the area it was made for. When a NODDS client receives packed.dat, it matches it to an area based on request.mfr: the possibilities of mismatching and impossibilities of concurrent requests are abound. For another thing,
packed.dat unpacker needs the list of what has been requested, just to
find out which requested products are missing or were substituted by
the server. It is really too contrived: isn't it simpler to get the
server to include a message in its reply if it failed to execute a
request exactly as asked, than for the client to play seek-and-match
game? To correctly display packed.dat, you also need a list of all
products, because you may need some parameters in that list (say, for
unit conversion). But again, if the server knew which units this
particular client prefers, it could've sent data in the "right" form
in the first place. BTW, right now the server (the TEDS database,
actually) performs unit conversions anyway. It seems apparent to
me that once a user has chosen his preferred set of units, he doesn't change this set frequently; this information therefore can be stored in a
"user profile" on the server. As far as display of EOF data is
concerned, there is a hell of a lot of other configuration files that
are necessary to properly show 3D grid data: I mean files with weird
abbreviations, keys, geographical information, as well as a file
containing ocean depths.
Earl has come up with a temporary fix to the problem of
incompleteness of packed.dat: he includes the area folder right into
packed.dat. This "defined area" is rolled out when the packed.dat is
being unpacked. I'm not quite sure however what happens if there is
already a folder with the same name. At any rate, this solution fixes some
of the problems I clamored about above, allowing the user keep the existing
NODDS client. Still, it is far more fitting to store all the required geographical information inside packed.dat, and
forget about NODDS areas altogether. Besides holding user's request, a NODDS area also caches server's reply. However all Web browsers, Netscape for one, do
their own caching, and do it quite efficiently and customizably. You
can tell a browser where to keep the cache, how often to purge it
and how long to keep data in it (for a single session only, or for
certain number of days, etc). I think it behooves us to rely on
the standard features found in a variety of available products, and take
the maximum advantage of them.
Communication is the bread of Web browsers and servers, they're intended, made and optimized for that. Therefore, all the communication code in NODDS, all DCS table setup, etc. would be completely gone in WebNODDS, and good riddance! This also greatly improves portability of WebNodds, as networking is a very particular thing and varies greatly from platform to platform, and takes a lot of effort to get it right and maintain.
It strongly appears to me that it is GRIB that should be made a standard for weather-type communication. Well, it's already the standard, alas not as far as NODDS is concerned. In my opinion, the server should send a GRIB file in reply to a request. I'm proposing a Mosaic helper, a viewer for the GRIB data type. It can even be incorporated in some version of Mosaic (free, like chimera, or commercial if we can get one of the guys at Spry or Netscape or IBM or Adobe interested). Most of the current NODDS::display functionality would be (and should be) used in the GRIB viewer.
BTW, the latest flurry with secure HTTP protocols etc. suits us extremely well. Much less work for us, should security /authentication be necessary in WebNODDS.
Using GRIB in WebNODDS has another unexpected advantage. Since
both http and regular e-mail (MIME to be precise) share the same
(file/session-level) communication protocol, it becomes possible to deliver
and view weather information asynchronously, via regular e-mail. The same
viewer that helps out Mosaic when GRIB comes in can be used to make
sense of a e-mail message with a GRIB attachment. The mailers take
care of ASCII encoding if necessary, they do it transparently and
automatically! Again, there is no need to do packed.dat unpacking,
ASCII to binary conversion, and messing with .G and .F files: the
standard products (any good e-mail front-end or a Web browser) do this as
a matter of routine. BTW, MIME employs a better ASCII-binary encoding
scheme than do uudecode or OTH-gold. Using mail to send out weather
info has a huge advantage: you're no longer limited to TCP/IP or any
other particular communication protocol or media, you can send a mail
from a FIDONET site to a mainframe, from a PC on a Novell network to a
Mac, etc. One can also take advantage of all the e-mail frills amassed
out there like mailing lists, automatic broadcasting, mail aliases,
mail exchangers, etc. etc.
I should also mention that http, smtp are the standards (smtp and mime are highly regarded non-proprietary open standards). So they are already a common open platform, which FNMOC, SPAWAR, etc, commands in the Navy, in the Air Force, etc could sign on without much haggling. As I understand, WebNODDS is better for "readiness" and "alertness": it helps to rely on standard protocols and equipment, rather something proprietary and custom-made, which cannot be easily replaced.
Although this may come as a surprise, WebNODDS as I imagined
it is not that far-fetched. For one thing, there are already a few
primitive weather map Web sites out there, both commercial and
free. Of course, none of them have the wealth of information (and,
importantly, forecasts) that FNMOC does. For another thing, my NODDS
server has already built-in functionality to heed user's profiles, to
store "areas" on the server, etc. It should not be a too difficult to reply
in GRIB rather than in packed.dat. I have also incorporated my image
compression into Web: you can click on a LaplacianPyramid
compressed image and get it delivered, unpacked, and displayed. From
user's point of view, this looks no different than retrieving a JPEG
image. I also can perform compression interactively: you type the
value of a quality parameter and click on "Compress Now" button, which sends a request to a HTTP server. It compresses the image and sends the
compression ratio, compression time, and the compressed data back to
the client, so you can see what you've got. It already works! On my
SunSparc20/Solaris 2.4/Netscape/HTTPD. BTW, my decompressor-viewer can
be used as it is to view encoded images sent by e-mail.
So, I offer to do all that, to conjure up what I've imagined... Interested? I can get by mostly by myself (well, I really count on Earl). As I said, the server side doesn't worry me too much, I can get the server to spit GRIBs in a month or so, in favorable conditions. Much of the NODDS::display capability can be reused; although of course I'd re-write everything in C++ or other language if it be my wish. Sure a lot of NODDS stuff should be thrown away and rewritten. Nevertheless it should not take more than several months to accomplish everything, with the first demonstrable version running in a matter of weeks. I have a strong hunch that it'll take longer to work out contract and other formal arrangements; I guess if I start now, it'll be ready long before a contract and job assignment etc. get signed :-)
My biggest worry is the ever evolving TEDS database. It's available only at FNMOC, which ties my hands in trying things. I may even consider writing my own SQL glues (and then any database, Empress, Ingres, Sybase etc. would do) or using ODBC. Anyway, having Maxion handy (handy for me, I mean) would help. And there is one thing I really really want: a standard C++ compiler. I'm pissed off wasting my time adjusting my code to idiosyncrasies of Maxion's gcc or trying to fix bugs in it, just because boneheads at Concurrent can't install gcc properly. They can't even get their system to deal with longer file names. Even Windows-95 support long names, up to 256-char long! I can confidently say, gcc 2.5.7 on Maxion won't compile my new code, I need gcc 2.6.3 or any standard C++ compiler (anything which abides by ARM or the draft C++ standard). BTW, I may try to write some parts (or all parts?) of WebNodds in Scheme or SmallTalk (they're more portable, and btw, there is an IEEE Scheme standard).