When and why did hexadecimal representation become more common than octal for displaying and printing out multi-bit binary fields?
Addressing the "why" part of the question - from my point of view as an assembly-code programmer on PDP-11 and VAX, the "standard" radix is most usefully chosen to match the instruction layout.
PDP-11 had 8 registers and 8 operand-mode indicators. Its double-operand instruction layout was
1 bit generally byte/word indicator (b) 3 bits opcode (o) 3 bits source mode (s) 3 bits source register (r) 3 bits destination mode (d) 3 bits destination register (R)
making octal the perfect way to express it:
The VAX, on the other hand, had 16 registers and 16 bits for operand mode (though some combinations were used for short literals). A basic operand specifier, in the variable-length instruction format was
4 bits mode (m) 4 bits register (r)
thus hex was perfect to express these.
Of course, the larger address space used on VAX gives other advantages to hex: fewer characters in an address. This might have some bearing on "when".
The when and why of use of hexadecimal over octal representation is intimately tied in with where and what: the use of one over the other depended greatly on environmental factors, as well as the machine itself, with programmer preference mainly being developed by the influence of these.
As Raffzahan points out, IBM 360 environments used hexadecimal from their start in the early 60s. This did not particularly influence other computing environments (particularly minicomputers) because IBM at that point was still its own separate world within computing for both technical reasons (e.g., using EBCDIC instead of ASCII) and because, despite the 1956 consent decree, they were still successfully using various measures to stifle compatibility and competition. (This wouldn't start to change in a serious way until the various lawsuits started pouring out in 1968 and 1969.)
Octal was a good fit for DEC minicomputers and early "micro-sized" machines (such as the PDP-8) because of their word sizes (18, 36 and 12 bits). Existing systems were naturally used to help develop new platforms, and thus new platforms after these would tend to inherit the octal versus hexadecimal choice from the existing ones. Famously, early versions of Microsoft Basic were assembled on a PDP-10 using MACRO-10, which didn't support hexadecimal, and so the source code used octal throughout where the bit values were considered important to the programmer, despite hexadecimal being the preferred number base for all of the first three CPUs (8080, 6800, 6502) for which it was written.
The same was true of the PDP-11, which was DEC's first major computer where the various word sizes were all multiple of 4 bits rather than 3. Even though software development rapidly moved to being done on the PDP-11 itself, DEC environments up to that time had been entirely octal, and so with programmers of DEC equipment mostly being more comfortable with that, it "stuck." (While other-dave points out that the bitfields in the instructions tended to be 3 bits wide, I don't buy this as a major factor; by 1970 hand-assembly was rare for machines of minicomputer size and up, in which category the PDP-11 indubitably fell.)
My guess is that the "it's what I'm used to" effect was at least part
of the reason for the front panel of the original Altair 8800 being
clearly organized for octal, despite Intel using hexadecimal
throughout the 8080's documentation. However, there were a
number of instructions, particularly
MOV, with three-bit fields
aligned with octal, so perhaps they did also have in mind that people
might be hand-assembling to octal for this reason. (The typical Altair
owner was unlikely to have another, larger computer handy for
(Source: Wikimedia Commons.)
That said, the awkwardness here is apparent if you open the image in a new tab and look at the upper set of markings ("SENSE SW" and "DATA") underneath the switches themselves; the split between the two 8-switch halves falls in the middle of one of the groups of three. (The owner of this system was obviously annoyed enough by the octal layout that he added tape to the front panel to visually indicate four groups of four!)
The 6800-based Altair 680B, released barely a year later, had already switched to panel markings that grouped the switches in fours, rather than threes.
By the late 70s the 8080/6800/6502 and their various progeny were starting to dominate the computing landscape, at least as far as numbers goes, and because they in particular tended toward use of hexadecimal it became more likely that's what a programmer would be comfortable with if he were not equally comfortable with both. But as fadden points out in a comment, even into the 90s there were still environments where octal was the norm and hexadecimal was generally rejected.
This continues to this day in certain areas where it's unlikely ever
to change because octal is significantly easier to read than
hexadecimal. For example, the POSIX file permission bits are a group
of two bits for sgid/suid followed by three groups of three bits for
owner/group/other permissions of read/write/execute: these are
normally and most easily specified (both for command line tools such
chmod and when programming) as four octal digits.
So if you're going to try to treat the computing community as monolithic, and just go on the number of programmers that preferred one over the other, I'd agree with Chromatix that the "shift" happened in the mid- to late-70s. But I think that is probably too simplistic a way of looking at it to be a particularly valuable insight.
When and why
That is quite close tied to the IBM /360 and its introduction in 1964. The /360 is based on the use of an 8 bit byte, 32 bit word (16 bit half word) and 24 bit address. Thus all basic memory items were multiples of 8 bit units - which are, without any remainder, best be displayed in hex. In addition displaying bytes in hex correlates well with the main reason to use 8 bit bytes as well: the usage to store two BCD in one byte (*1). A hex dump will show them as easy readable numerals - while the rest is code anyway. The same way it works for (half, double) words. In a dump they can be read and decoded as what they are without any additional concern (*2). In octal a word displayed as bytes, like in a dump, will get a different numeric expression than displayed as a (half, double) word (*3). Effectively the same reasoning why decimal isn't used in the first place (*4).
Before that size of bytes, half words and words were (more often than not) multiples of 3, which works quite fine with octal. After all, grown up with decimal it's way less mental work to not use two numbers, than to learn six more. It seams more natural, doesn't it?
After that next to all new designs switched to 8 bit bytes to allow easy data exchange with IBM mainframes. This happened even faster for mini computers as they where usually supplementary systems to (/360ish) mainframes.
Additional thought about the transition and it being different due environment.
Another-dave made the point that he'managed to cope with octal [on a PDP-11]', noting the prevalence of octal in an DEC envirionment even way later. Thinking of, the PDP-11 is a rather late comer to his as being from 1970. At that time the 6 to 8 bit byte was done history - likewise the switch to hex - for major suppliers. In fact, the PDP-11 marks that spot in DEC history. But DEC as a company (and its machines users) made a rather smooth transition from 6/12/36 bit computers to 8/16/32 bit - DEC did not only continue to support their existing line, but also continued to sell and even develop new models of their 36 bit line well into the 1980s.
In contrast with IBM the switch was done in a brutal manner as the /360 not only almost killed the company but in turn replaced any other (mainframe) architecture. After that 8 bit (and hex) was rolled out across all new products and everything else eliminated quite fast. Some 'islands' remained with existing supercomuter installations into the early 1970s but got moved either to /360 or non-IBM (often CDC/Cray). Some of the later carrying on for more years with octal representation.
Almost anyone entering the market after 1964 did start with 8 bit bytes and hex notion for binary data. This is (as Curt notes) especially true for micro-computers, as next to all of them started with 8 bit bytes and their OSes/UIs preferring hex notation.
So while the general turning point it quite clear and fixed in time with IBMs choice for an 8 bit byte holding two BCD, adoption did vary across manufacturers, its user base and application areas.
Adoption of changes by humans are always gradual .. keep in mind, even after 100+ years of road regulation and harmonisation, there are still countries with right-hand traffic :))
See also this question about the rational of 36 bit designs. While not a true duplicate, it's quite related here.
*1 - Previous machines (often) stored one decimal per 6 bit byte, thus wasting more than 1/3 of memory when handling decimal (which was a major task - after all, accountants always got the computers first, not scientists). By switching to 8 bit bytes, holding two nibbles with one decimal each, memory usage was close to optimum without resorting to base 100 (7 bit byte) or base 1000 (10 bit byte).
*2 - Beyond byte sex that is
*3 - For example the Number
0401(Hex Word) and
04 01 (Hex Byte) but
002001 (Oct Word) and
004 001 (Oct Byte)
*4 - BASIC programmers on home computers offer an additional nice example here, as many of them used assembly as series of PEEKs and POKEs expressed in decimal numbers. I met more than one who was fluent in 6502 assembly, still I had a hard time understanding their all decimal opcode numbers :))
Minicomputers and mainframes typically used octal, as many early mainframes had word sizes that were a multiple of 3 bits, and so did some minis. Operators and engineers within those environments became used to this, so even power-of-two word size minicomputers kept using octal.
Microcomputers, however, almost always had power-of-two word sizes for both address and data buses (or at least, a multiple of four bits), and there was a whole new generation of users who were not mentally locked into the mainframe/mini way of thinking. It was thus natural to start using hexadecimal instead.
You'll probably find, therefore, that hexadecimal rose to prominence about when microcomputers did, in the mid to late 1970s.
As others stated, of course this is related to word size on your machine.
I started learning higher programming languages on a WANG 2200T in 1977, and everything was hexadecimal there. My "Junior Computer" in 1980 was programmed in 6502 assembly, and everything was hexadecimal there as well. The first time I heard about octal notation was when I learned C with the K&R book in 1982. And my understanding was that on older machines you would prefer octal to hexadecimal.