CS248 Menu Buttons

UNIX Programming

"Chapter Eleven - Inter-process Communication: Pipes"

Chapter Outline

Lecture Notes

Inter-process Communication: Pipes

Now, we look at pipes which allow more useful data to be exchanged between processes.

Here are some of the things you need to understand.

What is a Pipe?

We use the word pipe when we connect a data flow from one process to another.

Shell commands can be linked together so that the output of one process is fed straight to the input of another.

For shell commands, this is entered as:

The shell arranges the standard input and output of the two commands, so that:

The shell has reconnected the standard input and output streams so that data flows from the keyboard input through the two commands and is then output to the screen.

Process Pipes

Perhaps the simplest way of passing data between two programs is with the popen and pclose functions. These have the prototypes:

popen

The popen function allows a program to invoke another program as a new process and either pass data to or receive data from it.

pclose

When the process started with popen has finished, we can close the file stream associated with it using pclose.

Try It Out - Using popen and pclose

Having initialized the program, we open the pipe to uname, making it readable and setting read_fp to point to the output.

At the end, the pipe pointed to by read_fp is closed

When we run this program on one of the author's machine, we get:

How It Works

The program uses the popen call to invoke the uname command. It read some information and prints it to the screen.

Sending Output to popen

Here's a program, popen2.c, that pipes dta to another. Here, we use the od (octal dump).

Try It Out -Sending Output to an External Program

Have a look at the following code, even type it in if you like...

When we run this program, we get the output:

How It Works

The program uses popen with the parameter w to start the od -c command, so that it can send data to it. The results are printed.

From the command line, we can get the same output with the command:

Passing More Data

Multiple fread and fwrite can be used to process more data.

Try It Out - Reading Larger Amounts of Data from a Pipe

Here's a program, popen3.c, that reads all of the data from a pipe by using multiple fread.

The output we get, edited for brevity, is:

How It Works

The progran uses popen with an r parameter, so it continues reading from the file stream until there is no more data available.

How popen is Implemented

The popen call runs the program you requested by first invoking the shell, sh, passing it the command string as an argument.

This has two effects, one good, the other not so good.

1. invoking the shell allows complex shell commands to be started with popen.

2. Each call to popen invokes the requested program and the shell program. So, each call to popen then results in two extra processes being started.

We can count all the lines in example program by cating

the files and then piping its output to wc -1 , which counts the number of lines.

On the command line, we would use:


Try It Out - popen Starts a Shell

This program uses exactly the command given above, but through popen so that it can read the results:.


when we run this program, the output is:

How It Works

The program shows that the shell is bing invoked to expand popen*.c to the list of all files starting with popen and ending in .c and also feed the output from cat into wc.

The Pipe Call

The pipe function has the prototype:

pipe is passed an array of two integer file descriptors. It fills the array with two new file descriptors and returns a zero.

Some errors defined in the Linux man pages for the operation are:

Any data written to fijle_descriptor[1] can be read back from file_descriptor[0].

Try It Out - The pipe Function

Here's a program, pipe1.c, that uses pipe to create a pipe..

When we run this program, the output is:

How It Works

The program creates a pipe using the two file descriptors file_pipes[]. It then writes data into the pipe using the file descriptor file_pipes[1] and reads it back from file_pipes[0].

Try It Out - Pipes across a fork

1. This is pipe2.c. It start rather like the first examples, up until we make the call to fork.

2. We've made sure the fork worked, so if fork_result equals zero, we're in the child process:

3. Otherwise, we must be the parent process:


When we run this progra, the output is, as before:

How It Works

The program creates a pipe with the pipe call. It then uses the fork call to create a new process. The parent writes to the pipe and the child read from the pipe.

Parent and Child Processes

The child process can be a different program than the parent.

Try It Out - Pipes and exec

Here we have a data producer program and a data consumer program.

1. For the first program, we adapt pipe2.c to pipe3.c. The lines that we've changed are shown shaded:


2. The 'consumer' program, pipe4.c, that reads the data is much similer:

Remembering that pipe3 invokes the pipe4 program for us, when we run pipe3, we get the following output:

How It Works

The pipe3 program uses the pipe call to create a pippe and then using the fork call to create a new process.

pipe4 receives the descriptor number of the pipe as an argument.

A call to execl is used to invoke the pipe4 program. The arguments to execl are:

Reading Closed Pipes

A read on a pipe that isn't open for writing will return 0, allowing the reading process to avoid the 'blocked forever' condition.

Pipes used as Standard Input and Output

We can arange for one of the pipe file descriptors to have a known value, usually the standard input, 0, or the standard output, 1.

The advantage is that we can invoke standard programs, ones that don't expect a file descriptor as a parameter.

There are two closedly related versions of dup, that have the prototypes:


File Descriptor Manipulation by close and dup

The dup always returns a new file descriptor using the lowest available number.

By first closing file descriptor 0 and then calling dup, the new file descriptor will have the number zero.

Try It Out - Pipes and dup

1. Modify pipe3.c to pipe5.c, using the following code:


The output from this program is:

How It Works

The program creates a pipe and then forks, creating a child process.

The parent and child have access to the pipe.

We can show the sequence pictorially. After the call to pipe:

After the call to fork:

When the program is rady to transfer data:

Named Pipes: FIFOs

We can exchange data with FIFOs, often referred to as named pipes.

A named pipe is a special type of file that exists as a name in the file system, but behaves like the unnamed pipes that we've met already.

We can create a named pipe using the old UNIX mknod command:

However, it is not in X/Open/ command list, so we use the mkfifo command:

From inside a program, we can use two different calls. These are:

Try It Out - Creating a Named Pipe

For fifo1.c, just type in the following code:

We can look for the pipe with:

How It Works

The program uses the mkfifo function to create a special file.

Accessing a FIFO

One very useful feature of named pipes is that, because they appear in the file system, we can use them in commands where we would normally use a file name.

Try It Out - Accessing a FIFO File

1. First, let's try reading the (empty) FIFO:

2. Now try writing to the FIFO:

3. If we do both at once, we can pass information through the pipe:

NOTICE: the first two stages simply hang until we interrupt them with Ctrl-C.

How It Works

Since there was no data in the FIFO, the cat and echo programs blocks, waiting for some data to arrive and some other process to read the data, respectively.

The thrid stage works as expected.

Opening a FIFO with open

The main restriction on opening FIFOs is that a program may not open a FIFO for reading and writing with the mode O_RDWR.

A process will read its own output back from a pipe if it were opened read/write.

There are four legal combinations of O_RDONLY, O_WRONLY and the O_NONBLOCK flag. We'll consider each in turn.

In this case, the open call will block, i.e. not return until a process opens the same FIFO for writing.

The open will now succeed and return immediately, even if the FIFO has not been opened for writing by any process.

In this case, the open call will block until a process opens the same FIFO for reading.

This wil always return immediately, but if no process has the FIFO open for reading, open will return an error, -1, and the FIFO won't be opened.

Try It Out - Opening FIFO Files

1. Start with the header files, a #define and the check that the correct number of command-line arguments have been supplied:


2. Assuming that the program passed the test, we now set the value of open_mode from those arguments:

3. We now check whether the FIFO exists, create it if nmecessayr, open it and give it output, wait, and close it.

How It Works

This program allows us to specify on the command line the combination of O_RDONLY, O_WRONLY and O_NONBLOCK that we wish to use.

O_RDONLY and O_WRONLY with no O_NONBLOCK

Let's try out a couple of combinations.

It allows the reader process to start, wait in the open command and then both programs to continue when the second program opens the FIFO.

Here is another combination:

This time, the reader process executes the open call and continues immediately, even though no writer process is present.

Reading and Writing FIFOs

Using the O_NONBLOCK mode affects how read and write calls behave on FIFOs.

A read on an empty blocking FIFO will wait until some data can be read.

A write on a full blocking FIFO will wait until the data can be written.

A write on a FIFO that can't accept all of the bytes being written will either:

Try It Out - Inter-process Communication with FIFOs

To show how unrelated processes can communicate using named pipes, we need two separate program, fifo3.c and fifo4.c.

1. The first program is our producer program. This creates the pipe if required, then writes data to it as quickly as possilbe.



2. Our second program, the consumer, is much simpler. It reads and discards data from the FIFO.


When we run these porgrams at the same time, using the time command to time the reader, the output we get is:

How It Works

Both programs use the FIFO in blocking mode. fifo3 is started first and wait for the FIFO to open. When fifo4 is started, the pipe is unblocked and data transfer occurs.

Advanced Topic: Client/Server using FIFOs

Now let's build a client/server application using named pipes.

We want to allow multiple client processes to send data to the server, using a single pipe.

Returning data to a client requires one pipe per client.

Try It Out - An Example Client/Server Application

1. First, we need a header file, cliserv.h, that defines the data common to both client and server programs.

2. The server program, server.c, creates and opens a pipe


3. We perform some processing on the data just read form the client.

4. Then we send the processed back back, opeing the client pipe.

5. Here's the client, client.c. It opens the server FIFO and creates a client FIFO.


6. For each of the five loops, the client data is sent to the server. Finally the server FIFO is closed and the client FIFO is removed from memory.

The following shell commands run a single copy of the server and several clients, to test out the programs.

This starts one server process and five client processes. Here is the output.

How It Works

The server creates its FIFO in read-only mode and blocks.

The client opens the FIFO for writing and createsits own uniquely-named FIFO for reading back from the server.

For a real server process that needed to wait for further clients, we ould need to modify it to either:

The CD Application

Now we make the CD application a simple client/server system. It takes serveral programs.

Here is the Makefile to show how the programs will fit together.


Aims

Our aim is to split the part of the application that deals with the database away from the user interface part of the specification.

Implementation

In the earlier, single process version of the application we used a set of data access routines for manipulating the data. There were:

These functions provide a convient place to make a clean separation between client and server.

In the single process implementation, we can view the application as having two parts:

In the client server implementation, we want to logically insert some named pipes and supporting code between the two major parts of the application.

This is the structure we need:

In our implementation, we choose to put both the client and server interface routines in the same file.

We have six .c files

First, we'll look at cliserv.h. This file defines the client/server intrfaces. It is required by both client and server implementtions.

Try It Out - The header File, cliserv.h

1. First, we have a #define that includes the feature test marco _POSIX_SOURCE.

2. There follows the required #include headers:

3. We define the named pipes:

4. We implement the commands as enumberated types, rather then #defines.

The first typedef gives the type of request being sent to the server, the second the server response to the client.

5. Next, we declare a structure that will form the message passed in both directions between the two processes.


6. Finally, we get the pipe interface functions that perofrm dta transfer.

Client Interface Function

Now we look at client_if.c. This provides 'fake' versions fo the database access routines.

Try It Out -The Client's Interpreter

1. This file implements the nine database functions protyped in cd_data.h.

The file starts with #include files and constants:

2. The static variable mypid/B> reduces the number of calls to getpid that would otherwise be require.

3. The database_initialize and _close routines are still called but now use pipes.

4. The get_cdc_entry routine is called to get a catalog entry from the database, given a CD catalog title.


5. Here's the source for the function read_one_response that we use to avoid duplicationg code:

6. The other get_xxx, del_xx, and add_xxx routines are implemented in a similar way to the get_cdc_entry.


7. Next, two functions for adding data, first to the catalog and then to the tracks database:



8. And lastly, two functions for data deletion:



Searching the Database

The function for search on the CD key is rather more complex. We arrange for the server to return all the possible matches to a search and then store them in a temporary file until they are requested by the client.

Try It Out - Searchng

1. This calls three pipe functions that are used in the next section, send_mess_to_server, start__resp_from_serever, and read_resp_from_server:

2. Here's the first call to search, i.e. with *first__call_ptr set to true.

3. Now there's this three-deep condition test, which makes calls to functions in pipe_imp.c.

4. The next test checks whether the search had any luck. Then the fseek call sets the work_file to the next place for data to be written.

5. This checks whether there are any matches left.


The Server Interface

The server side needs a program to control the (renamed) cd_access.c, now cd_dbm.c.

The server's main function os listed here

Try It Out - server.c

1. First, the usual headers and the macro _POSIX_SOURCE.

2. Now comes the main function.


3. Any client messages are fed to the process_command function.



Here is a diagram of the sequence of events that occur to pass data between client and server.

The Pipe

Here's the pipe implementation file, pipe_imp.c, whic has both the client- and server-side functions.

Try It Out - Pipes Implementation Header

1. First the #includes:

2. We also define some values that we need in different functions within the file:.

Server-side Function

Now we look at the server-side functions.

Try It Out - Server Functions

1. The server_starting routine creates the named pipe form which the server will read commands.

2. When the server ends, it removes the named pipe, so clients can detect that no server is running

3. The read_request_from_client function, shown below, will block reading the server pipe until a client writes a message into it:

4. In the special case when no clients have the pipe open for writing, the read will return 0, i.e. it detects an EOF.

Try It Out - Plumbing the Pipes

1. First, we open the client pipe:

2. The messages are all sent using calls to this function.

3. Finally, we close the client pipe:

Client-side Functions

Complementing the server are the client functions in pipe_imp.c. Very similar, except for the worryingly-name send_mess_to_server.

Try It Out - Client Functions

1. After checking that a server is accessible, the client_ starting function initiallizes the client-side pipe:

2. The client_ending function closes file descriptors and deletes the now redundant named pipe:

3. The send_mess_to_server function passes the request through the server pipe:


Try It Out - Getting Server Results

1. This client function starts to listen for the server response.

2. Here's the main read from the server which gets the matching database entries:

3. And finally, the client function that marks the end of the server response.

How It Works

The second, additional open of the client pipe for writing in start_resp_from_server,

is used to prevent a race condition.

Finally, here's the makefile to put it altogether:


Application Summary

We've now separated our CD database application into a client and a server.

Summary

This capter looked at passing data between processes using pipes.


CS 248 - UNIX Programming Web Site Menu
Information | Syllabus | Schedule | Online "Lectures" | Projects | Quizzes | Web Board



Copyright © 2001 by James L. Fuller, all rights reserved.