38

I occasionally need to work with console commands and 100% of the time I want to redirect my output to a file I fail to redirect stderr as well as stdout the first time. I can't think of a single reason one would want the output and the error to got to different locations, especially since in most early programs what constituted error output and standard output was left entirely up to the programmer, and it's usage was wildly inconsistent.

I realize of course, I'm ranting in 2019. We've moved well past simply outputting to the console in most all respects; so much so it's very much the exception rather than the norm. In the early days of computing there might have been a good reason to separate stdout and stderr.

What, if it existed, was it?

  • Comments are not for extended discussion; this conversation has been moved to chat. – Matt Lacey Jul 3 at 3:43
  • Cleared this up because it was getting a little out of hand... Don't mean to offend anyone, and there were some great lines in there :) – Matt Lacey Jul 3 at 3:43
94

You are thinking that all output is for human reading.

For instance, take the Unix cpio command. It writes the archive to stdout, which is always redirected to a device or file. It writes the archive with a header before each file that contains the size of the file, which lets it calculate the offset to the next file when reading it back.

If there was an error or warning thrown in there, it would basically make it unusable after that point due to the extra data thrown in the output. Plus you would want that warning to go to your screen or error log separately from the archive, otherwise you would never see it.

  • 36
    I would emphasize more strongly in this answer that most unix tools produce output that is suitable to be piped to another program. So stdout by design is for data in a pipe, stderr is for errors or other messages to the user. – dirkt Jun 28 at 6:03
  • 4
    Even if it were only for human reading, you still don't want error/warning/status messages mixed in with the results. – OrangeDog Jun 28 at 10:48
  • 2
    @OrangeDog: ... and that's why C stdio defaults to line-buffering for stdout and unbuffering for stderr, if they're connected to a TTY. So they are unavoidably mixed on a text terminal unless you redirect one or the other, but at least with full-line granularity not usually in the middle of lines. – Peter Cordes Jun 28 at 17:55
125

Because you might not want error messages in your output. According to computer scientist Stephen C. Johnson:

One of the most amusing and unexpected consequences of phototypesetting was the Unix standard error file (!). After phototypesetting, you had to take a long wide strip of paper and feed it carefully into a smelly, icky machine which eventually (several minutes later) spat out the paper with the printing visible.

One afternoon several of us had the same experience -- typesetting something, feeding the paper through the developer, only to find a single, beautifully typeset line: "cannot open file foobar" The grumbles were loud enough and in the presence of the right people, and a couple of days later the standard error file was born...

  • 15
    Diomidis Spinellis provides more context in his blog post on the topic, based on Stephen C. Johnson’s email. – Stephen Kitt Jun 27 at 20:55
  • 2
    IOW, OP needs to stop looking at what's right in front of his nose. – RonJohn Jun 29 at 19:19
29

In UNIX, which was developed along with C, it is common to redirect the output of a command to a file or another program. For example, ls -a | more or ls -lr > index.

In these cases, you would not want error messages appearing in your list of files, and especially not if the output of one command is being used as an input to another, and inserting an error message would corrupt the file—or worse, be interpreted as a command by a device.

This solution was inspired by Fortran, where the output and error streams might be different physical devices. For example, you might want to see debugging messages on your console, but send ordinary output to a printer.

  • 5
    One very big example of this is the classic tar | gzip combo. If there's an error -- a file that can't be opened or something -- you don't want to corrupt the whole tarball with no indication to the user. – Nic Hartley Jun 28 at 17:49
  • 1
    @NicHartley Indeed! Or a daemon probably wants to tell the user at the console if it can’t start up, but send its routine messages to a log file. – Davislor Jun 28 at 17:54
  • Fortran didn't distinguish error and 'normal' as such, but at least on IBM, by mid-60s FIV/F66, it did distinguish unit 6 'line printer' intended to be read by humans and unit 7 'card punch' intended to be processed by another program, or another run of the same program (with updated data, better parameters, more resources, etc). 'printer' output might be more likely to contain error info but also very often contained normal non-error output. Programmers in those days were almost never near the console, not in the same room and often not the same building. – dave_thompson_085 Sep 16 at 23:58
12

Snips-n-snails probably has the correct answer, but I'll add something since you said:

I can't think of a single reason one would want the output and the error to got to different locations

Here's one: if they output is large, you want to be able to easily identify if there were any errors.

You're probably used to color output and formatting being available even on your console, but this wasn't the case for a long time.
Mono-spaced upper-cased characters were the only thing for a long time in the "good old days" (said with and without sarcasm). Spotting the error in your output could be difficult if your output didn't line up in neat columns.

  • 6
    Every time I do a find, I’m grateful for the opportunity to send stdout to a file, so I don’t have to scroll through a zillion permission denied errors looking for an actual file name – WGroleau Jun 28 at 13:56
  • @WGroleau May I introduce you to 2</dev/null? You two will get along great. – user1717828 Jun 28 at 17:26
  • 4
    @user1717828 I think you mean 2>/dev/null, unless you're doing something exceedingly strange. If you're going to be sarcastic, at least be correct. – Nic Hartley Jun 28 at 17:49
  • 3
    @PeterCordes I have been writing that one redirect incorrectly all these years, despite using file redirects (correctly) on a regular basis 🤣 I just typed find / -iname "*pdf" 2</dev/null on the command line four times in a row, in disbelief on myself, verifying that I have built up the muscle memory to use the wrong redirect in this one case. But yes, it works :-) – user1717828 Jun 28 at 18:15
  • 1
    @NicHartley: I assume most programs won't get stuck in an infinite loop of more error messages if their write to stderr fails. e.g. strace -o touch.trace touch foo/bar (a directory that doesn't exist) shows just write() of the normal error message returning -EBADF, no attempt to print an error about that error. Those stderr writes are one case where you probably don't want error checking. You seem to be claiming it wouldn't even work, but it most certainly does shut the program up because it makes stderr not writeable (instead of silent discard to /dev/null). – Peter Cordes Jun 28 at 18:25
12

You may not be using these outputs to their full potential, or to their intended meaning. You never want to mix debugging & diagnostic output with actual results of your program.

Standard output: the intended result of the program, for example, list of files (ls), modified or processed text file (sed,grep,awk), a compressed file in response to input file (bzip,gzip), any stream-based filtering of processing that you might want to do, result of your calculation, and so on. You may be used to outputting directly into files, but that's a worse practice, because standard output is meant to be pipelined. You can have a long chain of tools that pass their output to the input of the next one via pipe |, eventually redirecting > the final output to the file of your choice, or even discarded (if the output is long, you probably don't want it to go directly into the terminal window). It's easy to redirect standard output to a file in the shell, it's much harder to convince a program that writes to a file, to talk to other tools.

Error output: This is meant so that the program can gossip about what it's doing. For example, outputting which file it's processing right now, percentage of completion, timestamps of when it started a certain stage of computing, warning, and error messages. During development, you usually output a lot to stderr, so that you catch any problems as soon as possible, and so you know where it gets stuck and deduce what went wrong. That's why stderr is not buffered: you want it to be seen immediately on screen, because if your program dies, any buffered text may never get to the screen. Of course, you may redirect this as well to a log file, and review it later, especially if you are not standing next to the terminal (if you are running your code on a cluster, for example).

Most programs are very chatty on stderr, even the ones you may not think of, such as GUI based programs (firefox, pdf reader,...).

Because the usage of both streams is completely different, you want an easy separation of the two. A few examples:

Observe what it's doing, and saving the result

programname arguments > outputfile

Discard the messages, save the result:

programname arguments > outputfile 2> /dev/null

Save the messages for later, look at the output on console to "preview" what you'll save on the next run:

programname arguments 2> logfile

Make a chain of tools that do something in a single line, save it (but observe all the diagnostics in real time):

programname arguments | filterprogram | anotherfilter > outputfile

  • 4
    especially GUIs are chatty, because they assume their noise there isn't going to bother the user. Traditional Unix programs for interactive use on a terminal are typically silent by default (except for actual errors). But some newer ones, like 7zip, are also somewhat chatty, e.g. defaulting to progress bars – Peter Cordes Jun 28 at 18:06
  • 1
    Again to quote from ESR's Unix Philosophy, "Rule of Silence: When a program has nothing surprising to say, it should say nothing." – Artelius Jul 1 at 3:03
6

To quote from ESR's "Unix Philosophy",

3.  Rule of Composition: Design programs to be connected to other programs.

7.  Rule of Transparency: Design for visibility to make inspection and debugging easier.

12. Rule of Repair: When you must fail, fail noisily and as soon as possible.

To interpret:

Programs will often be connected to other programs (or have their output saved in a file, sent over the network, etc.). However, software runs into unexpected problems all the time, and the later a problem is discovered, the more time and resources are wasted. So you want the user to be informed of problems as soon as they occur.

Of course, the user has the freedom to configure things differently, but a "sensible default" is to dump problems to the console regardless of where the "actual" output is being sent.

I should add that sometimes you simply want to "shut up" a program but still be notified of problems. For example when compiling code, you may not want to see the list of commands and files being run, but just any warnings or errors.

Do stdout and stderr give you a lot of fine-grained control? No, they're quick'n'dirty—which is very much in the spirit of UNIX. A lot of folks seem to agree that this makes engineers much more productive. If a different approach is what suits you and the tasks you perform, that's cool.

2

Security.

Security and prevention of malicious attack has not been mentioned in any of the answers and should likely be included for completeness.

Deriving the nature of an error can assist an attacker in determining the effectiveness of a given attempt. This can be mitigated by separating error reports out and preventing them from reaching an unprivileged user.

1

Lots of good answers here already. Let me just add some simple examples.

I can't think of a single reason one would want the output and the error to got to different locations

OK, consider the following shell command:

ls -l some/directory/that/might/not/exist > directory_content.txt

That does a long form listing of the specified directory (if it exists) and puts the content in a text file. If the directory doesn't exist, would it be more convenient for you to see the error on the screen or to find it in the file when you next look at the file?

Here's another one:

numFiles=`ls | wc -l`

That counts the files in the current directory and assigns the count to $numFiles. If there's an error (unlikely in this case, but still possible), would you like to see $numFiles assigned with 1 (because the error is one line long)? Or would you prefer to see the error on the screen.

What about this:

sudo find / -name someFile

That looks for a particular file on the whole file system. Unfortunately, on modern systems, it's likely to spew out lots of error messages for files even root shouldn't look at. If you do this:

sudo find / -name someFile 2> /dev/null

you can discard all the errors and see only the things you want.

Edit

There's a general principle at play here, which is that you should always distinguish error output from normal output and ideally keep it separate. Some mechanisms turn out to be better than others. For example, in C it is often the convention to use a special function return value to signify an error, often -1 for functions returning integers or NULL for functions returning pointers. This can cause problems if the caller fails to properly check the return value. Other subtle errors can occur too. Here's one that beginner C programmers often make

char c;
while ((c = fgetc(someFP)) != EOF)
{
    doSomethingWith(c);
}

I'll leave it as an exercise for the reader to identify the bug.

As a more spectacular example, consider Ariane flight 501. The inertial reference system had an exception due to the overflow of one value (velocity I think) due to it having been designed for Ariane 4. The IRS declared an exception and sent diagnostic data to the guidance system. The guidance system assumed the diagnostic data was actually legitimate attitude data and tried to correct the situation by fully deflecting the nozzles on the boosters. The rocket then began to break up and ultimately became an expensive firework.

  • fgetc returns an int – Polluks Jul 2 at 21:20
  • @Polluks Correct but why is that a bug? – JeremyP Jul 3 at 8:39
  • EOF is -1, so out of char range. – Polluks Jul 4 at 10:02
  • @Polluks Yes but only if char is unsigned. If char is signed, which it often is, then EOF and 0xff are the same thing and 0xff is a legitimate character in some character sets and always legitimate in a binary file. – JeremyP Jul 4 at 10:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.