Short Answer:
Motorola used it for their 6800 and MOS (6500) inherited it from Motorola. After all, the 6500 team members came out of the 6800 project so they were already used to it.
Long Answer:
If the 0x prefix clearly predates the 8-bit revolution,
It doesn't, really. Motorola used the $ prefix already with the 6800 of 1974. Unix had only recently (1973) been rewritten in C, which added the 0x convention around that time, and shown outside of AT&T.
then (1) why was $ adopted so widely?
Because of Motorola's 6800/02/09 line as well as MOS' 6500 series used in many lower-end micro (home) computers. Most notably by Acorn, Apple, Atari and Commodore. All of them were quite successful in the mass market. Naturally, everyone using them/writing software for these machines/CPUs would use its notation and create a strong sense for being the 'natural' way.
Later on, it got amplified by the popularity of the 68000 used, for example, with Atari ST and Commodore Amiga line of machines.
And the opposite, (2) having become ubiquitous for a whole generation of 8 bit programmers,
Now this does very much depend on your 'processor bubble', as it's only true for users of 65xx/68xx(x) based machines. Whoever grew up with an x80 (8080, 8085, Z80) or x86 based machine, like Altair (S100), Tandy TRS-80 and the whole MSX world, will disagree. They'll be quite firm in that a H suffix was the only valid way and everything else is quite exotic :))
why was it abruptly dropped?
It wasn't. There is no worldwide ruling body outclassing it. After all, using either prefix (or suffix) isn't inherent to hex code, but defined by the language used. Motorola/MOS assemblers want a $ prefix, Intel a H suffix, C a 0x prefix and others again want a more mathematics-like prefix of 16# (Ada) or 16r (Smalltalk).
The going out of style is thus more due x86 taking over and C bing the prime (low level) Language.
(Not that it matters, but I still use $ when commenting code or scribbling on dead trees.)
And so do many other (*3).
Bottom line: the language one uses defines the notation - and, as always, the first leaves the deepest impression.
Advantage of Prefix over Postfix
Classic mathematics uses a postfix notation of having the radix trailing a number. Which Intel followed with their nnnnH notation. For compiler writing, this brings the downside that a whole word (constant) has to be read first, before it could be decided how to interpret it. This means a buffer is needed holding the whole string, as it can only be converted after advancing until the postfix.
By using a prefix, it's clear from the beginning of a number sequence how to interpret the following characters. Thus there's no need to buffer (*2). This gives an advantage in parser design and does simplify the assembler a lot. This was especially helpful for fast creation of early cross assembler tools as first bootstrap.
Some History
Hex notation was rather uncommon in the early years. Instead, octal was the thing, as most machines were build to a multiple of 3 word size, as well as characters were usually handled as 6 bits.
Of the few machines/systems that very early on offered hex, many used various letters/symbols, like the last 6 in the alphabet UVWXYZ (*2), often even non-continuous.
IBM introduced a prefixed string format (X'nnnn') with the /360 in 1965
Datapoint used octal with a prefixed Zero
Intel's 4004 used hex with a suffix of h
Intel's 8008 used all octal in manuals and documentation, written as nnnO
By 1973, both assemblers were made to use the same B/O/H suffixes.
*1 - Caring for a few bytes of buffer might seam strange today, where the whole source is usually loaded into memory, but RAM was a scare resource back then. As a result, algorithms were preferred that could read data, like from paper tape, and directly work on each symbol read without buffering.
An optimal assembler allows to read any item a character at a time and process it right away with no need to look ahead. Using a prefix supports this.
*2 - A convention often used in engineering when needing symbols.
*3 - Here on RC.SE I try to use notation according to each topic, so $for 65/68 related, H for Intel, and so on...and when there is no clear relation, I'll fall back to /370 notation x'nn', as that was what I used most, despite having used $ first.
12Hfor hex, while Motorola used$12, so that explains why it was adopted widely for 8-bitters. But I am not sure if Motorola invented it, or copied it from somewhere else.