HTTP Virtual File System

The First Announcement

This is to announce an HTTP Virtual File System for Midnight Commander (MC). It is designed to "mount" any directory on a remote site, and view and copy files to/from/between remote or local directories. This is similar to a familiar NFS, or MC's own mcfs and ftps. Yet this VFS works via HTTP, requires minimum installation on a remote site (only a single, easily configurable sh-script), can work through firewalls, and has a number of other advantages, see below.

This article was posted on comp.os.linux.development.system, comp.infosystems.www.misc, comp.os.linux.misc, comp.os.research newsgroups on Monday, Feb 2 1998.

Note. The HTTP file system has been considerably extended and developed since this announcement. The system is no longer limited to the Midnight Commander, and can be used with any POSIX application or code. Consult an HTTP Virtual File System page for up-to-date information.

 
 

As an example, here is what you can do:

Thus directories of remote computers act and feel as if there were local folders on a box running a MC session. This scenario actually works, you can see it for yourself (see below about availability and installation). A remote host can be anything that has an HTTP server and a sh; it may even be a WinNT server. Your local box can also be anything that runs the MC; although if it is not Linux386, you have to compile a vfs-client from a given source code.

 

The HTTP Virtual File system is built from the following layers:

Very extensive comments in the source code (in particular, VFS-server.sh and vfs-client.cc) tell precisely how orders get translated and executed, which HTTP requests are issued, and how they are replied to and acknowledged. BTW, the comments tell all about the DeepestRoot.

 

Comparison with FTPFS, MCFS and NFS

The HTTP VFS is rather similar to a FTPFS, a virtual file system built into MC to access remote FTP sites. There is also a striking similarity with NFS and MC's own version of it, MCFS. There are however, a number of differences: See http://zowie.metnet.navy.mil/~spawar/JMV-TNG/ for a design of another system that builds upon HTTP riches (and which also works).

 

Security considerations

The VFS-server script obviously opens up the file system of a host computer to the entire world. If the HTTP daemon runs as a root, then any remote vfs client can do absolutely anything with this host's files. Clearly this is not often desirable. Therefore, one may want to limit access to this script (via .htaccess or conf/access.conf) to trusted hosts/users, and/or demand authentication, in a regular HTTP way. These access restrictions are the responsibility of a httpd; HTTPFS doesn't need to do anything special about them (and doesn't even need to be aware of them).

In addition, the VFS-server.sh script may implement its own access control. For example, it may refuse PUT requests. This effectively makes exported file systems read-only. The VFS-server.sh script may permit modification/listing of only certain files. The script may disallow DeepestRoot and ".." in the file paths, thus restricting user access only to a specific part of the file system tree. The possibilities are endless.

 

Availability and installation

[obsolete; see the HTTPFS page for up-to-date information]

An INSTALL file tells how to get things to work; HTTP-vfs.dr tells what the other files are for. Note the package includes an implementation of TCP streams, and C++ code to parse (multi-part) MIME entities, HTTP headers and configuration files of several flavors. The latter are built on an efficient implementation of dictionaries and collections of strings (also included, along with the validation code).

After all files are placed in the proper places, you can launch mc, and enter

cd http:myhost.com
or
cd http:myhost.com^DeepestRoot^tmp
where myhost.com is the remote host where you've installed a VFS-server.sh CGI script. You can browse now a remote directory as if it were a local folder, view the files there and copy files to a local file system or other VFS. To copy files into a http VFS, you need to recompile Midnight Commander with a new version of extfs.c (included in the present distribution).

Please mail comments, questions, trouble reports to me directly at

oleg-at-pobox.com or oleg-at-okmij.org or oleg-at-computer.org

 

Browser is the OS? The other way around!

The slogan "the browser is the OS" has long become a cliche. Netscape has been touting this idea for at least two years, while Microsoft seems to have taken the best advantage of it. The Active desktop lets you view local files and directories as if they were (remote) Web pages. The idea is not new: I myself mentioned it back in the Summer of 1995, way before Microsoft got itself Internet-enabled:
http://pobox.com/~oleg/ftp/DreamOS.html
However, the HTTP VFS proves that it is all the other way around: the OS is the browser!

Indeed, consider what happens when you enter "cd http:remote.host" in the MC, highlight an index.html file and try to view it. The OS (MC in this case) opens a connection to the remote.host, fetches the file, caches it locally, and launches lynx in the local mode to interpret the HTML content of this local file. Thus a back-end of a web browser that deals with networking, caching, etc. becomes a part of the OS itself. Just as FTP, DNS, DHCP are all a part of the "standard TCP stack", so should HTTP. An HTML formatter, the front-end of a web browser, becomes therefore a mere viewer of a particular (HTML) content, just as xv is a viewer of the image/gif content, and less is a default PAGER of text/plain files.

 

Yet to do

As you will note, while you can view a remote file "inplace", you cannot edit it inplace. As one of the MC's tips says, "Shell commands will not work when you are on a non-local file system"; file editing is currently implemented in MC as a shell command. The original MC4.1 extfs wouldn't even let you copy files into an "archive" (let alone create new files there). I had to tinker with extfs.c to make "archives" truly updatable. I have submitted the changes to MC development team.

Also, I have to think of http-vfs as a part of the OS itself. I have a few ideas how to do all this. What I don't know is, if anything of what I wrote in this article is of any interest, appeal or even use; or I'm simply wasting mine and everyone else's time...



 

Article Posting Headers


From oleg-at-pobox.com Mon Feb  2 12:07:11 1998
Subject: Announce: HTTP file system. Browser := OS; Midnight Commanding remote files
Date: Mon, 02 Feb 1998 11:11:09 -0600
Keywords: HTTP, MC, VFS, CGI, browser, Web, OS, NFS, virtual file system, remote file system
Message-ID: <886438688.518848522@DEJANEWS.COM>
Newsgroups: comp.os.linux.development.system,comp.infosystems.www.misc,comp.os.linux.misc,comp.os.research
Summary: an HTTP Virtual File System under Midnight Commander
X-Article-Creation-Date: Mon Feb 02 16:58:08 1998 GMT


Last updated June 4, 1999

This site's top page is http://pobox.com/~oleg/ftp/

oleg-at-pobox.com or oleg-at-okmij.org or oleg-at-computer.org
Your comments, problem reports, questions are very welcome!