The Z80 is notoriously limited when it comes to mathematical calculators.
Not more than CPUs used in calculators before. Like any other device a pocket calculator is about sufficient speed, not maximum speed. A Z80 is more than capable to do several hundred up to a few thousand flotingpoint operations per second - way more than the average calculator user ever can use.
Other contemporary processors such as the 68k were fully capable of integer multiplication and division
Which doesn't make a big difference in real life application. Producing a square root after 1/1000 second instead of 1/100 second (or a two seconds) doesn't matter for the user, does it?
Not to mention that a 68000 wasn't much faster than a Z80 with comparable technology. The main advantage of 16 (or 32) bit processors wasn't about speed, but addressable memory.
and floating point arithmetic could be implemented with reasonable efficiency even with no FPU.
Not so much better than a Z80. Especially when considering the increased price.
Be serious, would you - and more important millions of parents - have been willing to pay like triple the price, just to have a different CPU, one the user (normally) never sees? For a speed increase 99.99% of all users never experience in real life?
Why did early TI calculators, especially the TI-8x series, use the Z80?
Seriously, why shouldn't they? The Z80 is a wide available CPU with many variations, has great tool collection, and most importantly is available for ASIC integration. Which is an incredible argument for system builders (like TI). Integration is the main source of cost savings. Including CPU, RAM, ROM, I/O and drivers in one package saves a fortune.
In fact, when looking at this new generation of TI calculators using a standard - and non-TI - CPU the question is rather
Why did they switch to a standard CPU based design
Until then TI calculators used CPUs based on special calculator designs. Well proven, all in house developed and handled. Diverting from such a well proven path is a rare decision.
For TI the answer might have been that it was maybe no longer possible to deliver the increased requirements with sometimes 20+ year old designs. To make it work replacement had to be not only capable but also cheap. Dirt Cheap. A complete restart like this costs a lot of money for development and production ramp up. And the resulting product must of course meet customer expectations - not the least of them will be the price. Sure, a new, way more capable model can fetch more money, but not a premium, as it still must be within school budget for most parents.
As a result the component must be not only available but also low priced. Selecting a well proven CPU that is available at a real low price per unit (when going for the expected sales) is a must. Sure, it must be capable to do the job. But only the job. There is no budget for something fancy only engineers may dream of.
In case of these machines, using a Z80 might have beaten even the use of CPU cores TI already produced at the time. That alone should be telling, shouldn't it?