What was the point of separating stdout and stderr?

Question

I occasionally need to work with console commands and 100% of the time I want to redirect my output to a file I fail to redirect stderr as well as stdout the first time. I can't think of a single reason one would want the output and the error to got to different locations, especially since in most early programs what constituted error output and standard output was left entirely up to the programmer, and it's usage was wildly inconsistent.

I realize of course, I'm ranting in 2019. We've moved well past simply outputting to the console in most all respects; so much so it's very much the exception rather than the norm. In the early days of computing there might have been a good reason to separate stdout and stderr.

What, if it existed, was it?

Answer 1

You are thinking that all output is for human reading.

For instance, take the Unix cpio command. It writes the archive to stdout, which is always redirected to a device or file. It writes the archive with a header before each file that contains the size of the file, which lets it calculate the offset to the next file when reading it back.

If there was an error or warning thrown in there, it would basically make it unusable after that point due to the extra data thrown in the output. Plus you would want that warning to go to your screen or error log separately from the archive, otherwise you would never see it.

Answer 2

Because you might not want error messages in your output. According to computer scientist Stephen C. Johnson:

One of the most amusing and unexpected consequences of phototypesetting was the Unix standard error file (!). After phototypesetting, you had to take a long wide strip of paper and feed it carefully into a smelly, icky machine which eventually (several minutes later) spat out the paper with the printing visible.

One afternoon several of us had the same experience -- typesetting something, feeding the paper through the developer, only to find a single, beautifully typeset line: "cannot open file foobar" The grumbles were loud enough and in the presence of the right people, and a couple of days later the standard error file was born...

Answer 3

In UNIX, which was developed along with C, it is common to redirect the output of a command to a file or another program. For example, ls -a | more or ls -lr > index.

In these cases, you would not want error messages appearing in your list of files, and especially not if the output of one command is being used as an input to another, and inserting an error message would corrupt the file—or worse, be interpreted as a command by a device.

This solution was inspired by Fortran, where the output and error streams might be different physical devices. For example, you might want to see debugging messages on your console, but send ordinary output to a printer.

Answer 4

You may not be using these outputs to their full potential, or to their intended meaning. You never want to mix debugging & diagnostic output with actual results of your program.

Standard output: the intended result of the program, for example, list of files (ls), modified or processed text file (sed,grep,awk), a compressed file in response to input file (bzip,gzip), any stream-based filtering of processing that you might want to do, result of your calculation, and so on. You may be used to outputting directly into files, but that's a worse practice, because standard output is meant to be pipelined. You can have a long chain of tools that pass their output to the input of the next one via pipe |, eventually redirecting > the final output to the file of your choice, or even discarded (if the output is long, you probably don't want it to go directly into the terminal window). It's easy to redirect standard output to a file in the shell, it's much harder to convince a program that writes to a file, to talk to other tools.

Error output: This is meant so that the program can gossip about what it's doing. For example, outputting which file it's processing right now, percentage of completion, timestamps of when it started a certain stage of computing, warning, and error messages. During development, you usually output a lot to stderr, so that you catch any problems as soon as possible, and so you know where it gets stuck and deduce what went wrong. That's why stderr is not buffered: you want it to be seen immediately on screen, because if your program dies, any buffered text may never get to the screen. Of course, you may redirect this as well to a log file, and review it later, especially if you are not standing next to the terminal (if you are running your code on a cluster, for example).

Most programs are very chatty on stderr, even the ones you may not think of, such as GUI based programs (firefox, pdf reader,...).

Because the usage of both streams is completely different, you want an easy separation of the two. A few examples:

Observe what it's doing, and saving the result

programname arguments > outputfile

Discard the messages, save the result:

programname arguments > outputfile 2> /dev/null

Save the messages for later, look at the output on console to "preview" what you'll save on the next run:

programname arguments 2> logfile

Make a chain of tools that do something in a single line, save it (but observe all the diagnostics in real time):

programname arguments | filterprogram | anotherfilter > outputfile

Answer 5

Snips-n-snails probably has the correct answer, but I'll add something since you said:

I can't think of a single reason one would want the output and the error to got to different locations

Here's one: if they output is large, you want to be able to easily identify if there were any errors.

You're probably used to color output and formatting being available even on your console, but this wasn't the case for a long time.
Mono-spaced upper-cased characters were the only thing for a long time in the "good old days" (said with and without sarcasm). Spotting the error in your output could be difficult if your output didn't line up in neat columns.

Answer 6

To quote from ESR's "Unix Philosophy",

3. Rule of Composition: Design programs to be connected to other programs.

[...]

7. Rule of Transparency: Design for visibility to make inspection and debugging easier.

[...]

12. Rule of Repair: When you must fail, fail noisily and as soon as possible.

To interpret:

Programs will often be connected to other programs (or have their output saved in a file, sent over the network, etc.). However, software runs into unexpected problems all the time, and the later a problem is discovered, the more time and resources are wasted. So you want the user to be informed of problems as soon as they occur.

Of course, the user has the freedom to configure things differently, but a "sensible default" is to dump problems to the console regardless of where the "actual" output is being sent.

I should add that sometimes you simply want to "shut up" a program but still be notified of problems. For example when compiling code, you may not want to see the list of commands and files being run, but just any warnings or errors.

Do stdout and stderr give you a lot of fine-grained control? No, they're quick'n'dirty—which is very much in the spirit of UNIX. A lot of folks seem to agree that this makes engineers much more productive. If a different approach is what suits you and the tasks you perform, that's cool.

Answer 7

Lots of good answers here already. Let me just add some simple examples.

I can't think of a single reason one would want the output and the error to got to different locations

OK, consider the following shell command:

ls -l some/directory/that/might/not/exist > directory_content.txt

That does a long form listing of the specified directory (if it exists) and puts the content in a text file. If the directory doesn't exist, would it be more convenient for you to see the error on the screen or to find it in the file when you next look at the file?

Here's another one:

numFiles=`ls | wc -l`

That counts the files in the current directory and assigns the count to $numFiles. If there's an error (unlikely in this case, but still possible), would you like to see $numFiles assigned with 1 (because the error is one line long)? Or would you prefer to see the error on the screen.

What about this:

sudo find / -name someFile

That looks for a particular file on the whole file system. Unfortunately, on modern systems, it's likely to spew out lots of error messages for files even root shouldn't look at. If you do this:

sudo find / -name someFile 2> /dev/null

you can discard all the errors and see only the things you want.

Edit

There's a general principle at play here, which is that you should always distinguish error output from normal output and ideally keep it separate. Some mechanisms turn out to be better than others. For example, in C it is often the convention to use a special function return value to signify an error, often -1 for functions returning integers or NULL for functions returning pointers. This can cause problems if the caller fails to properly check the return value. Other subtle errors can occur too. Here's one that beginner C programmers often make

char c;
while ((c = fgetc(someFP)) != EOF)
{
    doSomethingWith(c);
}

I'll leave it as an exercise for the reader to identify the bug.

As a more spectacular example, consider Ariane flight 501. The inertial reference system had an exception due to the overflow of one value (velocity I think) due to it having been designed for Ariane 4. The IRS declared an exception and sent diagnostic data to the guidance system. The guidance system assumed the diagnostic data was actually legitimate attitude data and tried to correct the situation by fully deflecting the nozzles on the boosters. The rocket then began to break up and ultimately became an expensive firework.

Answer 8

There are already many good answers here, but I just wanted to add that Bash in version 4 introduced the |& and &> operators to "merge" the two streams.

• |& combines stderr and stdout and sends them both to stdout:

   $ curl -v example.com |& grep text/html
   < Content-Type: text/html; charset=UTF-8
       <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
  

• &> combines stderr and stdout and redirects them both to file:

   $ curl -v example.com &> /dev/null
  

Answer 9

Security.

Security and prevention of malicious attack has not been mentioned in any of the answers and should likely be included for completeness.

Deriving the nature of an error can assist an attacker in determining the effectiveness of a given attempt. This can be mitigated by separating error reports out and preventing them from reaching an unprivileged user.