As I understand it, PC was not really competitive with Home computers until the 90’s. The PCs rise in popularity in the 80s must have been mainly in the business world.
Yes and no. Unlike often implied, the line between business and home/private is not only blurry but in fact never really existed. Even machines that were clearly meant for one 'side' ended up on the 'other'. For example, I knew several people during the early 80s who had outright professional systems at home as hobbyists. Like a 15 year old who got himself, in 1985, a multi-user Unix system to 'play' with - mind you, that thing was a large box with character based terminals. Real bulky ones, not emulations.
Or ZX81 systems that have been used (and sold!) as POS systems for small shops. Similarly, there was a lot of professional use of computers like C64 or CPC. Just think Open/Libre Office - a package that started out as a collection of office application (starting with a text processor) for the Amstrad/Schneider CPC series.
The most cunning example to me was a quite successful software (considering the special and small market) for wood farmers to mange their 'crop'. One may think this the typical environment for some mid-sized software house to deliver a custom solution. Well, no, it was C64-based and written by a farmer's son. The story goes that the farmer bought his sons a C64, and after a few weeks he started complaining that they should not play games so much but make something useful with their time. So one of the boys started to write a simple calculation program to add up harvested wood, from the usual measurements they brought home, to print out a sum sheet. This small BASIC program soon evolved to an application managing what's necessary to handle the woods as well as sales and, most important, accumulating and preparing all the numbers needed for reporting to various trade organizations and government offices - it's really hard to imagine the bureaucracy around something as seemingly simple as growing trees. That software soon became a standard application, and farmers were buying C64s all the way until they vanished to run that software. It got ported (and rewritten) to MS-DOS in the early 1990s, but there have been users known asking for service with their C64 version as late as 2012.
Bottom line, it is almost impossible to draw a rigid line between business or leisure based on the machine used. And even when one thinks there could be one, it gets busted when looking across country borders.
For example during the second half of the 1980s, Amiga and Atari ST scored worldwide, but the usage differed greatly. In the US, both machines where mainly home computers with a clear leaning toward games. In Europe there was a clear differentiation. While the Amiga was mostly a home/game machine, The Atari ST was seen as a business computer, challenging not only the IBM in the small business market, but also Apple for desktop publishing.
It's important to keep in mind that the term "Personal Computer" never was about IBM's PC in the first place. It describes a general purpose machine for personal use, even though the clever attempt by IBM to tie the term to its architecture did twist the views we have. It's just too convenient to see the IBM PC as a synonym to Personal Computer. Isn't it?
So during that time, what other options did people have for business computers?
As today: whatever suits your needs - and is available.
The offerings back then were much more diverse, as well as the expectation was much lower. At that time it was no computer vs. some computer. Having any computer for the job had much more impact than which computer it was.
Not only were computers quite different and evolving back than, users were as well. In the early 1980s potential users still had to be convinced that a computer could be helpful. A 'what do I need a computer for' mindset was commonplace. People could not imagine what it means to have one (*1,2). While the word processing challenge was basically solved by micros in 1980, typewriters still outsold microcomputers by large margins. Heck, even IBM stayed in the typewriter business until 1991, when it finally sold the typewriter division (including small printers and keyboards), which in turn became Lexmark.
So, it must be asked what 'that time' is supposed to mean. 1977 or 1989? It's just 12 years, but the changes during that time where greater than during the following 30 years.
Was it mainly the Z80-CP/M machines, or were there decent 16 bit options?
Due to the issues described (time, region and personal computer as generic machine), there can be no simple answer. Lets try to cover it by time. Also, I will exclude upper-end offerings like HP's 9100 or 9800 machines or IBM's 5100, as I think the scope of this question is rather about microcomputers and such available to a broader audience - especially less well funded ones :))
Also it's helpful to keep in mind that this can only be a rough scratch thru what was available.
Pre 1980:
Next to any microcomputer could have been used as Business system. There was no clear distinction at all.
For one, 8080/Z80 based S100 bus evolved quickly from a hobby concept into a viable line of open hard- and software. Much like the IBM PC and MS-DOS, S100 plus CP/M became a cornerstone of the market. Not as prevalent, but especially in the US a major force. Then again Apple with its Apple II (and ill fated III), Commodore with their PETs and Tandy's machines were major players. Similarly many (today lesser known) companies like Ohio Scientific, The Digital Group or Exidy had considerable shares 'on the Business Side'. Each with their own OS and their own software. Choosing 'the right' system was rather random, even when picking one of the larger firms.
Outside the US the market was even more diverse. In (Western) Europe, Commodore is eventually the only one with a strong market impact. The PET/CBM computers were seen as iconic business machines, rarely sold as home computers. Others like the NASCOM 1/2, the KISS developed by SKS (later sold by Triumph-Adler as Alphatronic P1/P2), the British PSI Comp 80 or Luxor's ABC-80 made their way as well, together with individual imports of US machines.
Further east there was (not least due to the nonexistence of individual small business) a wasteland and way far east Japan had their own culture of upcoming business computers. Eventually with Sord as the most prominent early manufacturer with its M100/200 series, as well as Sharp with their MZ machines. NEC started into the SOHO market as well its very successful PC-8000 series.
1980-1984
If the late 70s were the real punk age of computers (everyone able to hold a soldering iron started a computer company), the early 80s brought commercialization of punk. Punk computers where still everything but gray, just now made to business plans, not by enthusiasts growing into a business. In general Apple, Commodore and Tandy widened their sales and extended their proprietary machines. Compatible computers started to pop up. Not only the sheer endless number of Apple II clones, led by Franklin, but also TRS-80 clones like Lobo's Max-80 or EACA's Video Genie. All serious candidates for business use. Interestingly, where Commodore made its name in Europe as business computer, Apple gained an incredibly strong foothold in the US in education.
In Europe Apple, in the form of Taiwanese clones, made some inroads among university students, especially in engineering. Professional microcomputer systems got continued and expanded, like Luxor's ABC-800, or new upper end developments like Kontron's PSI-80 (*3,4) became available. Many well known office (e.g. Olympia with their BOSS line) or engineering companies (Kienzle, Diehl etc.) entered the market. Notable here may be Philips with their P2000 series. While the P2000T was targeted as a home machine, the P2000M was meant as a small business computer with its colour adaptor replaced by a 80x24 B&W. In addition the 1983 P2000C was a portable business machine much like the IBM or Compaq Portable of the same time frame - except being an exceptionally fast dual Z80 CP/M computer.
While late 1981 also marked the introduction of the IBM PC, it wasn't by any means set as a future success. Sure, a big brand, but there were many others, especially outside the US, competing with their machines. A notable competition was the Sirius 1 - at least in Europe - where it not only made IBM postpone the introduction of its PC, but also outsold the PC well into 1984.
1984 can be seen as a turning point. For one it brought the (in Europe) quite successful CPC series which operated at the lower end of SOHO - not at least to the green monitor offered by default. It has been already mentioned, that the origin of Star Office, which later became Libre Office, was on the CPC. And machines like the Apricot F1, notable for its attempt to be compatible to the IBM and advanced at the same time - not to mention its style. In the US Tandy finally turned their business onto the IBM PC track with their Tandy 1000 series (After doing the 2000 in 1983).
Oh, and then there was the Mac :))
1985-1990
1985 marks maybe the last time non-compatible machines were not only introduced but managed to get a considerable marketing share. Starting with Amstrad's PCW/Joyce, an all-in-one design based on a Z80 with 256 or 512 KiB Ram running CP/M 3.0 with floppies and printer and built-in screen. The machine was such a success in the small business market that it became the second best selling PC in all of Europe in 1985 and 1986 with a market share of 20% (IBM ~33%). In England it might even have been the absolute #1. Its success led to production until 1998(!) - a 4 MHz Z80 competing until a time when a 400 MHz Pentium II was state of the art in PCs (*5). The success was not at least due to the extremely competitive price. Usually less than a third of a similar configured PC-clone. Seems like it was a great choice for business.
1985 is also the year of the Atari ST. While this machine may have been more of a home computer in the US (*6), it was an instant success as a business machine in Europe. With the (for back then) incredibly good B&W screen, while at the same time way lower priced than a PC, but able to read/write its formats (*7), it became a first class choice for small business, not at least for text processing with software like 1st Word and ROM-Papyrus up to full figured desktop publishing with titles like Calamus and Pagestream (*8). Not to mention that several Autodesk products of today had their origin on the Atari during the 1980s, like 3dsMax and Animator.
After 1990
1990 brought something like a last push for different machines. Amstrad revitalized the CPC and Joyce line, Atari got the TT, Acorn the Archimedes ... and IBM the PS/1. All great machines for the SOHO market. Still, with the success of VGA and its successors, as well as with 486 systems, 1990 marks the point where IBM-compatible PC hardware had reached or surpassed the advantage of other designs, while at the same time prices, mostly due to Taiwanese competition, were driven so low, that the price advantage as well vanished.
As a result, competing architectures went into a decline. New business was more and more made with compatible PCs, while non compatible mostly got sold to an existing user base for replacement of older generations. Usually to save existing user experience and (data) compatibility. Since many standard applications got ported to the PC during that time, this became less of an issue ... and the rest is history :))
Well, or not:
After 2000
The market for 'incompatible' systems isn't as dead as it may seem. Just think about Chromebooks, ARM/Linux based tablets and not at least phones and other handheld devices for business needs.
*1 - Somewhat reminiscent of the typewriter, when people complained that it'll be so much more expensive and less flexible than handwriting :))
*2 - In a 1984(!) interview strategic director Steve Leininger of Tandy admitted that "As an industry we haven't found any compelling reason to buy a computer for the home. Word processing, yes. Real business type applications, financial modelling (aka Spreadsheets), maybe". A statement that quite well reflects the general attitude toward microcomputers in the early 1980s.
*3 - Not to be confused with the aforementioned PSI Comp 80.
*4 - Their OS is a quite interesting one. While offering a CP/M 2.2 compatible API, the OS itself was multitasking and multiuser based on hard time slices and optimized for real time response.
*5 - The incredible success of the PCW/Joyce also made media suppliers to compete in the market for 3" disks. When Amstrad adopted the 3" for the CPC, supply of disks was rather meagre and only became reliable after 1985.
*6 - And a unique success among musicians due to the built in MIDI. Today's Apple Logic Pro is directly based on Creator/Notator.
*7 - After finding a PC with 3.5" disks that is :))
*8 - A notable mention should go to Signum!, a word processor used still today via ST emulations on PCs, as it offered an unparalleled ability to write 'odd' and custom scripts. It has built-in functionality to define characters and fonts, something really handy for theologists who may need to cover several variations of long outdated letters within one script.