I was at Intel from the late 80’s to the early 00’s first as a CPU architect then running various design teams and research teams. Prior to Intel I was an architect/designer of systems using 68K, x86, and custom CPU’s.
Readers of this forum are mostly technologists and will tend to answer from a technology-centric perspective (as the saying goes: To a carpenter with a hammer, everything looks like a nail). The reality is that market outcomes are the result of multiple inter-dependent factors. Technologies are only one factor and frequently not the most important. These factors include market participants (buyers and sellers), market structure (supply chains, sales channels), as well as technologies. All of these factors rest on a foundation of organizational execution and economic behavior.
When assessing market outcomes one needs to understand what is being purchased, who is making the purchase decision, and what their value propositions are. The question was about CPU’s but buyers weren’t buying “CPU’s”, they were buying “boxes” containing CPU’s and other components such as DRAM, disks, and operating systems.
Why were they buying boxes? To run application software – which over the time period in question was bought as a product (a binary), not as a service (SaaS). Binaries were compiled to a particular CPU instruction set and OS API – changing either would require buyers to pay for a new binary. To maximize the return of acquisition cost these binaries were re-used over multiple box/CPU generations.
In approximate order of units shipped, the ecosystems using microprocessors in the late 80’s were: Wintel PC, Apple, Sun Microsystems workstations, other workstation vendors (e.g. Apollo, SGI), and a smattering of microprocessor-based mini-computers. Except for the Wintel PC ecosystem, all used 68K.
In 1986 Sun replaced the 68K with SPARC, their home-grown RISC. In 1994 Apple replaced the 68K with PowerPC. In the 90’s SGI moved to MIPS. Other 68K vendors were either acquired or went out of business. Each vendor changed CPU’s for different reasons. Sun wanted to move to RISC – no doubt influenced by Bill Joy who joined Sun from Berkeley where a RISC project was ongoing. By the 90’s, Apple had become worried about Motorola’s ability to keep pace with Intel in both performance and power (Apple threw a bone to Motorola by including them the AIM Alliance making PowerPC’s).
As the installed base of an instruction set or OS API increases, the incentive for software developers to target their binaries to these interfaces also increases – to amortize their upfront cost of development over a larger number of units shipped. This instantiates a feedback loop: Increasing ecosystem volume attracts more apps which increases demand for that ecosytem. Exacerbating this feedback loop was the fact that x86 volume was overhwelmingly concentrated on a single OS API (DOS then Windows) while 68K volume was split over multiple, incompatible, OS API’s (Apple, Sun, et al).
The profit from CPU generation N needs to pay for all the components of generation N+1: CPU development, fab process development, and at least one fab. The underlying economics of fabs are such that, as soon as a new fab process is ready, there should be a new CPU design ready to take advantage of the higher transistor budget enabled by that process and begin amortizing the very large upfront cost of building a fab. Adding to this was the fact that Intel’s yields were significantly higher than other CPU vendors.
One of the largest causes of 68K’s demise was Motorola’s execution relative to Intel. It was clearly the case that over the course of the 90’s the cadence at which Motorola shipped new CPU’s fell relative to Intel. These CPU execution cycles destructively (Motorola), or constructively (Intel), resonated with the underlying economics of silicon fabs, ecosystem platform volumes, and app development. Motorola’s CPU execution descended a vicious economic cycle while Intel’s ascended a virtuous economic cycle.
These are the broad strokes. There’s more detail here but this answer is already getting long.