Why did manufacturers of home computers avoid using the 6809 CPU?
I can't really see that anyone 'avoided' it. There have been many successful machines using the 6809. Beside the mentioned Tandy's CoCo there where other computers for the general audience, like
(Not exhaustive, there might be many more, as it's just from memory)
They were quite successful in Japan throughout all of the 1980s, and somewhat strange, the FM7 also in Portugal. The Fujitsu machines even featured two 6809, the second one operating as independent graphics subsystem.
Then there was Thomson as a major French player with the
These machines had good sales and a strong following (still today) in French speaking countries. In other places they were rather rate (*2). The MO6 was also OEM'ed by Olivetti as Prodest PC128.
(For all these machines it might be more useful to read the corresponding French/Italian/Japanese Wiki pages than the English ones :))
Another successful machine was the British Dragon 32/64 series of 1982. They are often attribute as Tandy clones, but that's rather due the fact that both use Motorolas SAM chipset. Compared to the CoCo, they offer a better Keyboard and an on board parallel interface.
And lets not forget the Vectrex (1982), and the fact that the MacIntosh prototype, developed around the same time, was also 6809 based.
I realize that the Z80 and 6502 had a 3- or 4-year head start in availability. But once it did become available in 1978, I don't understand why designers of new computers didn't choose it.
For one, above examples do show that it has been used, but it takes some time to decide, design and market a new machine. So while the raw CPU may have been available in late 1978, above examples show that it took roughly 3 years for computers to show up using this new CPU. Comparable to the Atari Series, only showing up 4 years after the 6502 CPU was available.
It doesn't surprise me that it was not used in any business computers. New = unnecessary risk to many of those potential customers. But most home users didn't care much about having a well-known operating system and large preexisting library of professional applications.
It might be less simple here. Professional users don't care about the machine or its CPU. They care about certain application. If a manufacturer supports its applications after switching the CPU, they gladly buy the new, incompatible one.
Now, with third party software it becomes more complicated. If a manufacturer can convince them about the new system and its future sales, they will support it and users will follow. Otherwise it's playing safe and make the new machine compatible. Back in the 1970s and early 1980s, professional software was rather closely tied to computer manufacturers, so switching CPUs wasn't uncommon. Their decisions were hardware driven and supported by good profit margins, allowing them to spend large amounts on software ports.
On the fast moving home computer market, margins were rather small, and changing a machine design that would result in a complete rewrite was unaffordable. That's why Commodore stayed that long with the 6502. It was less expensive to patch some parts of the Kernel for a new video controller while keeping the same old CPU.
Pagetable has just released a nice work showing how Commodore kept reusing code in the Kernal.
The 6809 had inherent advantages. From the Wikipedia [...]
I guess that and for sure the usage of full 64 KiB where the main reasons for the University of Waterloo Computer Systems Group's development of a 6809 daughterboard for the PET - what later became known as SuperPET after Commodore bought the design in 1981 (*3).
In fact, the SuperPET has been one of many 6809 add on cards for existing machines. Like The Mill for the Apple II or the 6809 Tube Modul for the BBC.
It seems strange to me that the engineers of manufacturers would have been attracted to these advantages, and would have striven to build machines utilizing it.
At the time the 6809 became available the game was no longer played by some lone engineer starting a new computer, but bigger companies and driven by much more than just curiosity for a new chip. Still, a wide usage of 6809 systems as CPU in other systems, from knitting machines to street lights and telephone systems does show that engineers did appreciate the additional abilities.
Also, and maybe even more important, 16-bit CPUs (8086, 68k, 32k) became available about the same time as the 6809. And the Mac is a great example, that switching over to 16-bit did bring even more advantages, especially in terms of memory, than just using a more advanced 8-bit unit. Kind of a 'too little, too late' case.
Conclusion: I don't think the 6809 was avoided. There have been many successful systems. But it was already too late to successfully compete with upcoming 16-bit systems.
*1 - Thomson somewhat screwed their own success by making the MO5 not fully compatible. While the Hardware is quite similar, they mixed up the memory map, thus making exchangeable programs less common.
*2 - Keep in mind, most machines had their home markets and respective companies were rather niche players in other parts of the world. For example Tandy was a big number in the US, but never really a thing in continental Europe. Much like Thomson machines were big in France, Belgium and Italy, but exotic in other parts of Europe. Interestingly they were somewhat successful in Britain. Similarly, Japan had a complete separate ecosystem.
*3 - Reading the SuperPET history reveals that the original 6809 choice even came from IBM(!) as part of the MICROWAT program they developed for the University of Waterloo CSG.