The 8087 math coprocessor for the 8086 (and descendants) nominally added floating point and transcendental (trigonometric and logarithmic) instructions to the 8086. Contrary to naive expectations, the 8087 didn't "augment" the main CPU. It was a separate and independent chip, and the coupling with the main CPU was minimal: the 8086 literally knew nothing about what the 8087 was.
All the floating point instructions were really (as far as the main CPU is concerned) an ESC (ESCAPE) instruction followed by standard x86 parameter encoding. In an ESC, the main CPU did little more than act on the memory access operands (if any) and discard any fetched data. The 8087 could peek over the 8086's memory fetch stream, recognize the ESC instruction and execute the floating point instructions when it recognized them. Additional instructions on the main CPU were often needed to stall it, until the 8087 finished doing whatever it was doing.
That leaves an intriguing door open. The coprocessor architecture was not committed to the 8087 math coprocessor. It would be possible to plug something other than a 8087 into the coprocessor interface. You could theoretically make a coprocessor that did anything you wanted.
Were there ever (from Intel or third-party providers) coprocessor chips different from the 8087 math coprocessors?
I'm not talking about cheaper clones; I'm 99% there were some of those in the market. I'm talking about either a math coprocessor using a completely different instruction set (not merely an enhanced version of the x87 instruction set); or even better, another chip with a different purpose (not merely a floating point and transcendental operations chip): maybe some sort of physics engine, a vector computation engine, a crypto engine, interface gateway to another CPU/device, or something more clever than what I can think of?
If there were no actual products, were there any known plans for such a chip even if they didn't come to fruition?
I don't expect there were any mass market applications, but I can imagine something in a narrow field (Industrial? Scientific?)
(Starting with the 486DX, the floating point circuitry was embedded in the 486 die. It started as a mere physical integration of the two chips, but I imagine by now the integration is very deep. The original meaning of the ESC instruction has probably been lost and I don't expect that such a replacement coprocessor is possible on a modern CPU)