Plumbing and Other Utilities

Rob Pike

Bell Laboratories

Murray Hill, New Jersey 07974

ABSTRACT

Plumbing is a new mechanism for inter-process communication in Plan 9, specifically the passing of messages between interactive programs as part of the user interface. Although plumbing shares some properties with familiar notions such as cut and paste, it offers a more general data exchange mechanism without imposing a particular user interface.

The core of the plumbing system is a program called the plumber, which handles all messages and dispatches and reformats them according to configuration rules written in a special-purpose language. This approach allows the contents and context of a piece of data to define how it is handled. Unlike with drag and drop or cut and paste, the user doesn’t need to deliver the data; the contents of a plumbing message, as interpreted by the plumbing rules, determine its destination.

The plumber has an unusual architecture: it is a language-driven file server. This design has distinct advantages. It makes plumbing easy to add to an existing, Unix-like command environment; it guarantees uniform handling of inter-application messages; it off-loads from those applications most of the work of extracting and dispatching messages; and it works transparently across a network.

Introduction

Data moves from program to program in myriad ways. Command-line arguments, shell pipe lines, cut and paste, drag and drop, and other user interface techniques all provide some form of interprocess communication. Then there are tricks associated with special domains, such as HTML hyperlinks or the heuristics mail readers use to highlight URLs embedded in mail messages. Some systems provide implicit ways to automate the attachment of program to data—the best known examples are probably the resource forks in MacOS and the file name extension ‘associations’ in Microsoft Windows—but in practice humans must too often carry their data from program to program.

Why should a human do the work? Usually there is one obvious thing to do with a piece of data, and the data itself suggests what this is. Resource forks and associations speak to this issue directly, but statically and narrowly and with little opportunity to control the behavior. Mechanisms with more generality, such as cut and paste or drag and drop, demand too much manipulation by the user and are (therefore) too error-prone.

We want a system that, given a piece of data, hands it to the appropriate application by default with little or no human intervention, while still permitting the user to override the defaults if desired.

The plumbing system is an attempt to address some of these issues in a single, coherent, central way. It provides a mechanism for formatting and sending arbitrary messages between applications, typically interactive programs such as text editors, web browsers, and the window system, under the control of a central message-handling server called the plumber. Interactive programs provide application-specific connections to the plumber, triggering with minimal user action the transfer of data or control to other programs. The result is similar to a hypertext system in which all the links are implicit, extracted automatically by examining the data and the user’s actions. It obviates cut and paste and other such hand-driven interprocess communication mechanisms. Plumbing delivers the goods to the right place automatically.

Overview

The plumber is implemented as a Plan 9 file server [Pike93]; programs send messages by writing them to the plumber’s file /mnt/plumb/send, and receive messages by reading them from ports, which are other plumber files in /mnt/plumb. For example, /mnt/plumb/edit is by convention the file from which a text editor reads messages requesting it to open and display a file for editing. (See Figure 1.)

Figure 1. The plumber controls the flow of messages between applications. Programs write to the file send and receive on ‘ports’ of various names representing services such as edit or web. Although the figure doesn’t illustrate it, some programs may both send and receive messages, and some ports are read by multiple applications.

The plumber takes messages from the send file and interprets their contents using rules defined by a special-purpose pattern-action language. The language specifies any rewriting of the message that is to be done by the plumber and defines how to dispose of a message, such as by sending it to a port or starting a new process to handle it.

The behavior is best described by example. Imagine that the user has, in a terminal emulator window, just run a compilation that has failed:

% make

cc -c rmstar.c

rmstar.c:32: syntax error

...

The user points the typing cursor somewhere in the string rmstar.c:32: and executes the plumb menu entry. This causes the terminal emulator to format a plumbing message containing the entire string surrounding the cursor, rmstar:32:, and to write it to /mnt/plumb/send. The plumber receives this message and compares it sequentially to the various patterns in its configuration. Eventually, it will find one that breaks the string into pieces, rmstar.c, a colon, 32, and the final colon. Other associated patterns verify that rmstar.c is a file in the current directory of the program generating the message, and that 32 looks like a line number within it. The plumber rewrites the message, setting the data to the string rmstar.c and attaching an indication that 32 is a line number to display. Finally, it sends the resulting message to the edit port. The text editor picks up the message, opens rmstar.c (if it’s not already open) and highlights line 32, the location of the syntax error.

From the user’s point of view, this process is simple: the error message appears, it is ‘plumbed’, and the editor jumps to the problem.

Of course, there are many different ways to cause compiler messages to pop up the source of an error, but the design of the plumber addresses more general issues than the specific goal of shortening the compile/debug/edit cycle. It facilitates the general exchange of data among programs, interactive or otherwise, throughout the environment, and its architecture—a central, language-driven file server—although unusual, has distinct advantages. It makes plumbing easy to add to an existing, Unix-like command environment; it guarantees uniform handling of inter-application messages; it off-loads from those applications most of the work of extracting and dispatching messages; and it works transparently and effortlessly across a network.

This paper is organized bottom-up, beginning with the format of the messages and proceeding through the plumbing language, the handling of messages, and the interactive user interface. The last sections discuss the implications of the design and compare the plumbing system to other environments that provide similar services.

Format of messages

Since the language that controls the plumber is defined in terms of the contents of plumbing messages, we begin by describing their layout.

Plumbing messages have a fixed-format textual header followed by a free-format data section. The header consists of six lines of text, in set order, each specifying a property of the message. Any line may be blank except the last, which is the length of the data portion of the message, as a decimal string. The lines are, in order:

The source application, the name of the program generating the message.

The destination port, the name of the port to which the messages should be sent.

The working directory in which the message was generated.

The type of the data, analogous to a MIME type, such as text or image/gif.

Attributes of the message, given as blank-separated name=value pairs. The values may be quoted to protect blanks or quotes; values may not contain newlines.

The length of the data section, in bytes.

Here is a sample message, one that (conventionally) tells the editor to open the file /usr/rob/src/mem.c and display line 27 within it:

plumbtest

edit

/usr/rob/src

text

addr=27

5

mem.c

Because in general it need not be text, the data section of the message has no terminating newline.

A library interface simplifies the processing of messages by translating them to and from a data structure, Plumbmsg, defined like this:

typedef struct Plumbattr Plumbattr;

typedef struct Plumbmsg  Plumbmsg;

struct Plumbmsg

{

    char            *src;       /* source application */

    char            *dst;       /* destination port */

    char            *wdir;  /* working directory */

    char            *type;  /* type of data */

    Plumbattr   *attr;  /* attribute list */

    int         ndata;  /* #bytes of data */

    char            *data;

};

struct Plumbattr

{

    char            *name;

    char            *value;

    Plumbattr   *next;

};

The library also includes routines to send a message, receive a message, manipulate the attribute list, and so on.

The Language

An instance of the plumber runs for each user on each terminal or workstation. It begins by reading its rules from the file lib/plumbing in the user’s home directory, which in turn may use include statements to interpolate macro definitions and rules from standard plumbing rule libraries stored in /sys/lib/plumb.

The rules control the processing of messages. They are written in a pattern-action language comprising a sequence of blank-line-separated rule sets, each of which contains one or more patterns followed by one or more actions. Each incoming message is compared against the rule sets in order. If all the patterns within a rule set succeed, one of the associated actions is taken and processing completes.

The syntax of the language is straightforward. Each rule (pattern or action) has three components, separated by white space: an object, a verb, and optional arguments. The object identifies a part of the message, such as the source application (src), or the data portion of the message (data), or the rule’s own arguments (arg); or it is the keyword plumb, which introduces an action. The verb specifies an operation to perform on the object, such as the word ‘is’ to require precise equality between the object and the argument, or ‘isdir’ to require that the object be the name of a directory.

For instance, this rule set sends messages containing the names of files ending in .gif, .jpg, etc. to a program, page, to display them; it is analogous to a Windows association rule:

# image files go to page

type is text

data matches ’[a-zA-Z0-9_\-./]+’

data matches ’([a-zA-Z0-9_\-./]+)\.(jpe?g|gif|bit|tiff|ppm)’

arg isfile $0

plumb to image

plumb client page -wi

(Lines beginning with # are commentary.) Consider how this rule handles the following message, annotated down the left column for clarity:

src plumbtest

dst

wdir    /usr/rob/pics

type    text

attr

ndata   9

data    horse.gif

The is verb specifies a precise match, and the type field of the message is the string text, so the first pattern succeeds. The matches verb invokes a regular expression pattern match of the object (here data) against the argument pattern. Both matches patterns in this rule set will succeed, and in the process set the variables $0 to the matched string, $1 to the first parenthesized submatch, and so on (analogous to &, \1, etc. in ed’s regular expressions). The pattern arg isfile $0 verifies that the named file, horse.gif, is an actual file in the directory /usr/rob/pics. If all the patterns succeed, one of the actions will be executed.

There are two actions in this rule set. The plumb to rule specifies image as the destination port of the message. By convention, the plumber mounts its services in the directory /mnt/plumb, so in this case if the file /mnt/plumb/image has been opened, the message will be made available to the program reading from it. Note that the message does not name a port, but the rule set that matches the message does, and that is sufficient to dispatch the message. If on the other hand a message matches no rule but has an explicit port mentioned, that too is sufficient.