From oleg Wed Jan 24 09:33:06 CST 1996 Newsgroups: comp.unix.programmer,comp.unix.admin,comp.unix.internals,comp.unix.shell Subject: scripting daemons through pipes; e.g.: newsreader in sh? (yes!) Summary: sh takes place of a human talking to interactive programs Followup-To: comp.unix.programmer Organization: University of North Texas, Denton Keywords: pipe, nntp, telnet scripting, agents Status: OR There are many applications designed to be UNIX-interactive (i.e., which listen on stdin and reply to stdout/stderr). Even many system daemons (ftpd, sendmail, lpd, httpd, etc) could be cajoled into maintaining a human dialog. Oftentimes a sysadm (or a regular mortal) needs to carry on a conversation vicariously, through an agent (a shell/perl script). This involves piping/redirecting stdin (and often stdout) of a program, or telnet (when talking to a daemon). There is a pitfall in piping stdin, though. This posts shows how to get around it in the cheapest and _portable_ way. A simple example: find out who really is a postmaster (there are more elaborate examples below). The obvious solution (echo "vrfy "; echo "quit") | telnet localhost 25 unfortunately, works only on SVR4-based systems (Solaris 2.4 to be precise). On most BSD-based systems (SunOS 4.1x, Digital UNIX, even FreeBSD 2.0.5) the above script fails: somehow, the first thing telnet gets from its redirected input (pipe) is EOF, which forces it to quit right away. An embellished version that uses a named pipe: mknod mypipe p telnet localhost 25 < mypipe & echo "vrfy " > mypipe fails always. On BSD-systems it quits just as instantly as the version with the regular pipe. On STREAM-based systems (Solaris 2.4) it leads to a complete disaster: try it *only* if you're adventurous enough (but be prepare to kill that telnet subprocess, probably from another terminal) What gives? A pipe is a gadget for connecting a writer to a reader. When the pipe is freshly created and the reader makes the first call (as it always should) it blocks: no data. Suppose the writer now sends 100 bytes down the pipe (while the reader had had requested only 70). The reader gets its 70; the next request for 70 bytes reads the remaining 30 bytes. After that, _any_ read request gives EOF. This is the catch: while the pipe contains no more data >from the writer, the reader would _not_ be blocked: any read request would return immediately, with the EOF condition. A workaround I stumbled upon was to make the reader reopen the pipe once it got the first EOF. Thus after digesting all data the writer has put so far, the reader would _wait_ again for more. This simple idea is implemented in a _way_ elementary C code: NAME exec_with_piped - execute a shell command with its input redirected from a named FIFO pipe SYNOPSIS exec_with_piped Named-pipe-name "Command-to-execute" DESCRIPTION Named-pipe-name is the name of a special file (FIFO pipe), which can be created by /etc/mknod File-name p Command-to-execute is to be run by /bin/sh. exec_with_piped launches "Command-to-execute", relaying data it reads >from the "Named-pipe" to command's standard input. When there is no more data in the Named-pipe, exec_with_piped re-opens the pipe and waits for the next portion of data to arrive. exec_with_piped exits when the Command it runs is completed. I have tested exec_with_piped on SunSparc Solaris 2.4 and SunOS 4.1.1, DEC Alpha/Digital Unix V3.2, FreeBSD 2.0.5, and Sequent S81/Dynix V3.1.4. The original code was written on SGI IRIX 4.x. The code has always worked. This can let you do a few less trivial things, like the ones below. Ex1: get a "root" page from an httpd and index it #!/bin/sh http_server=replicant.csci.unt.edu mknod /tmp/mypipe p exec_with_piped /tmp/mypipe "telnet $http_server 80" | indexing-code > index-file & echo "GET / HTTP/1.0" > /tmp/mypipe echo "" > /tmp/mypipe wait By the same token, one can get any other page from the httpd (or download all the pages from a web site). Ex2: /etc/mknod FIFO-PIPE p exec_with_piped FIFO-PIPE "rsh some.host Mathematica" & echo "Mathematica-command-1" > FIFO-PIPE ... see the result on the screen ... echo "Mathematica-command-2" > FIFO-PIPE ... see the result on the screen ... .... cat file-with-Mathematica-commands > FIFO-PIPE ... see the result on the screen ... echo "Quit" > FIFO-PIPE This was actually the reason I got into this plumbing business many years ago. Mathematica had a disgusting line editor. With exec_with_piped, I was able to use tcsh's history/line-editing functions... The most elaborate example, with a not-so-trivial client-server interaction: print the subject fields of the first 5 articles of a given newsgroup, talking to an NNTP server via telnet. Here, a client has to actually listen what the server is saying in reply to client's questions. The client then has to tailor the next question using the information from the previous replies. That is, the example shows off a really _two-way_ communication between "agents". Although a /bin/sh script nntp_scripting.sh that implements the example is rather small, it's still longer than 5 lines. It can be downloaded (for free), see below. Before running the script, please don't forget to adjust the name of the NNTP host (that is, a host running an NNTP server). Again, to get all this talking-agents functionality, you need only to compile one simple C code file. You don't need to install Tcl (/Expect) etc. BTW, it's easy to add a few lines to nntp_scripting.sh to ask the user which of the articles he likes. The script can push "head selected-number", "body" to a NNTP's pipe, to retrieve the article's text and display it. This would make nntp_scripting.sh work like rn. But it's a merely Bourne sh script! The code nntp_scripting.sh and exec_with_piped.c can be downloaded from http://pobox.com/~oleg/ftp/packages/pipe_scripting.shar http://pobox.com/~oleg/ftp/ tells what else is available from that FTP site. If you have any comment, question, suggestion, etc. please e-mail. I'll appreciate that!