HTTP Virtual File System

The First Announcement

This is to announce an HTTP Virtual File System for Midnight Commander (MC). It is designed to "mount" any directory on a remote site, and view and copy files to/from/between remote or local directories. This is similar to a familiar NFS, or MC's own mcfs and ftps. Yet this VFS works via HTTP, requires minimum installation on a remote site (only a single, easily configurable sh-script), can work through firewalls, and has a number of other advantages, see below.

This article was posted on comp.os.linux.development.system, comp.infosystems.www.misc, comp.os.linux.misc, comp.os.research newsgroups on Monday, Feb 2 1998.

Note. The HTTP file system has been considerably extended and developed since this announcement. The system is no longer limited to the Midnight Commander, and can be used with any POSIX application or code. Consult an HTTP Virtual File System page for up-to-date information.

Usage example
Skeleton and guts
Comparison with FTPFS, MCFS and NFS
Security considerations and flexibility of access control
Where to get the thing and how to install it
Browser is the OS? OS is the browser!
Further development
Article headers
HTTP File System Page

As an example, here is what you can do:

cd http:day-2^DeepestRoot^tmp
View the listing of a /tmp directory of a remote (HP-UX) computer day-2 in a MC pane
select a file and view it with MC's built-in viewer
copy the file to a local directory (supposedly listed in the other MC pane)
edit the file there
copy the file back to the day-2:/tmp directory
cd http:mday-1
view the listing of a DocumentRoot directory of another HP-UX box, mday-1 (on the other side of a firewall)
copy a selected file (or files) from the day-2:/tmp directory listed in one MC pane onto mday-1 listed on the other pane.

Thus directories of remote computers act and feel as if there were local folders on a box running a MC session. This scenario actually works, you can see it for yourself (see below about availability and installation). A remote host can be anything that has an HTTP server and a sh; it may even be a WinNT server. Your local box can also be anything that runs the MC; although if it is not Linux386, you have to compile a vfs-client from a given source code.

The HTTP Virtual File system is built from the following layers:

a Midnight Commander itself, which asks its virtual file system for list of files and directories;
an MC's extfs Virtual file system, which translates these orders into calls to external executables (passing them parameters and files to transmit/receive);
an http vfs-client, one such executable, which further translates MC's orders into HTTP GET or PUT requests;
a VFS-server.sh, a CGI script running on a remote computer that executes the requests: lists directories on its own server, sends files, and accepts new content for old or newly created files.

Very extensive comments in the source code (in particular, VFS-server.sh and vfs-client.cc) tell precisely how orders get translated and executed, which HTTP requests are issued, and how they are replied to and acknowledged. BTW, the comments tell all about the DeepestRoot.

Comparison with FTPFS, MCFS and NFS

The HTTP VFS is rather similar to a FTPFS, a virtual file system built into MC to access remote FTP sites. There is also a striking similarity with NFS and MC's own version of it, MCFS. There are however, a number of differences:

HTTPFS operates through TCP channels using HTTP, a simple stateless reliable protocol (simpler than ftp).
HTTPFS can talk to any host that runs an HTTP server and the VFS-server.sh CGI script (which can be adjusted to run on WinNT and even on a Mac, if re-written in MacPerl).
HTTPFS works transparently through firewalls, HTTP proxies and Web caches; HTTPFS also stands to benefit from various caching schemes web gateways frequently employ as well as load-balancing and replication facilities.
HTTPFS can rely on authentication mechanisms already built into Web servers. In addition, it can do its own, see below.
HTTPFS can serve "files" and list "directories" that are made up on the fly. In particular, with HTTPFS, one can browse a remote _database_ as if it were a local filesystem!
HTTPFS can fire up triggers and call hooks whenever a file or directory get accessed or modified (something that is very difficult to do with ftp).
HTTPFS can use advanced HTTP 1.1 features, namely, transmission of only selected parts of files (Range:), and If-Match:, Keep-Alive, etc. headers.

See http://zowie.metnet.navy.mil/~spawar/JMV-TNG/ for a design of another system that builds upon HTTP riches (and which also works).

Security considerations

The VFS-server script obviously opens up the file system of a host computer to the entire world. If the HTTP daemon runs as a root, then any remote vfs client can do absolutely anything with this host's files. Clearly this is not often desirable. Therefore, one may want to limit access to this script (via .htaccess or conf/access.conf) to trusted hosts/users, and/or demand authentication, in a regular HTTP way. These access restrictions are the responsibility of a httpd; HTTPFS doesn't need to do anything special about them (and doesn't even need to be aware of them).

In addition, the VFS-server.sh script may implement its own access control. For example, it may refuse PUT requests. This effectively makes exported file systems read-only. The VFS-server.sh script may permit modification/listing of only certain files. The script may disallow DeepestRoot and ".." in the file paths, thus restricting user access only to a specific part of the file system tree. The possibilities are endless.

Availability and installation

[obsolete; see the HTTPFS page for up-to-date information]

An INSTALL file tells how to get things to work; HTTP-vfs.dr tells what the other files are for. Note the package includes an implementation of TCP streams, and C++ code to parse (multi-part) MIME entities, HTTP headers and configuration files of several flavors. The latter are built on an efficient implementation of dictionaries and collections of strings (also included, along with the validation code).

After all files are placed in the proper places, you can launch mc, and enter

cd http:myhost.com

cd http:myhost.com^DeepestRoot^tmp

where myhost.com is the remote host where you've installed a VFS-server.sh CGI script. You can browse now a remote directory as if it were a local folder, view the files there and copy files to a local file system or other VFS. To copy files into a http VFS, you need to recompile Midnight Commander with a new version of extfs.c (included in the present distribution).

Please mail comments, questions, trouble reports to me directly at

oleg-at-okmij.org

Browser is the OS? The other way around!

The slogan "the browser is the OS" has long become a cliche. Netscape has been touting this idea for at least two years, while Microsoft seems to have taken the best advantage of it. The Active desktop lets you view local files and directories as if they were (remote) Web pages. The idea is not new: I myself mentioned it back in the Summer of 1995, way before Microsoft got itself Internet-enabled:

http://okmij.org/ftp/DreamOS.html

However, the HTTP VFS proves that it is all the other way around: the OS is the browser!

Indeed, consider what happens when you enter "cd http:remote.host" in the MC, highlight an index.html file and try to view it. The OS (MC in this case) opens a connection to the remote.host, fetches the file, caches it locally, and launches lynx in the local mode to interpret the HTML content of this local file. Thus a back-end of a web browser that deals with networking, caching, etc. becomes a part of the OS itself. Just as FTP, DNS, DHCP are all a part of the "standard TCP stack", so should HTTP. An HTML formatter, the front-end of a web browser, becomes therefore a mere viewer of a particular (HTML) content, just as xv is a viewer of the image/gif content, and less is a default PAGER of text/plain files.

Yet to do

As you will note, while you can view a remote file "inplace", you cannot edit it inplace. As one of the MC's tips says, "Shell commands will not work when you are on a non-local file system"; file editing is currently implemented in MC as a shell command. The original MC4.1 extfs wouldn't even let you copy files into an "archive" (let alone create new files there). I had to tinker with extfs.c to make "archives" truly updatable. I have submitted the changes to MC development team.

Also, I have to think of http-vfs as a part of the OS itself. I have a few ideas how to do all this. What I don't know is, if anything of what I wrote in this article is of any interest, appeal or even use; or I'm simply wasting mine and everyone else's time...

Article Posting Headers


From oleg Mon Feb  2 12:07:11 1998
Subject: Announce: HTTP file system. Browser := OS; Midnight Commanding remote files
Date: Mon, 02 Feb 1998 11:11:09 -0600
Keywords: HTTP, MC, VFS, CGI, browser, Web, OS, NFS, virtual file system, remote file system
Message-ID: <886438688.518848522@DEJANEWS.COM>
Newsgroups: comp.os.linux.development.system,comp.infosystems.www.misc,comp.os.linux.misc,comp.os.research
Summary: an HTTP Virtual File System under Midnight Commander
X-Article-Creation-Date: Mon Feb 02 16:58:08 1998 GMT

Last updated June 4, 1999

This site's top page is http://okmij.org/ftp/

oleg-at-okmij.org or oleg-at-computer.org
Your comments, problem reports, questions are very welcome!