A standalone processor, such as the ones you have listed, would usually only be able to perform some basic operations on integer data. This would typically be add and subtract because those operations can be trivially implemented with minimal amounts of logic (see Ben Eater's series on building a CPU on a breadboard as an example).
Earlier versions of the 68000 (for example) could only perform multiply or divide on "words" - that is 16 bits. A math co-processor would enable the CPU to perform multiply and divide on "longs" (32 bits) or perhaps even bigger numbers. If the math co-processor also included an FPU (likely), then it would be able to perform such operations on floating point numbers too.
A math co-processor is intended to help a standalone CPU perform more, or more complex operations than are implemented in its silicon, and without one, you are left to perform these operations in software which is naturally much slower.
A modern GPU, which is built to be a number crunching powerhouse, can very likely do a lot of what a math co-processor can do, but it is not necessarily considered a general purpose math co-processor. People have used GPUs to do math intensive work in specialised applications (e.g. crypto currency mining), but that is not an every day thing to do. In general, they are built to process large amounts of numerical data in a graphical sense to draw textures on screen, stretch and contort them around objects, apply visual effects, etc. If you like, you could consider them an application specific math co-processor, offloading all of that intensive computation from the CPU.
Math co-processors these days are more or less a thing of the past - all of the functionality they provided is now built-in to CPUs, so there is no need for external hardware to provide this kind of functionality.
As for 8-bit era hardware and how it worked, this is going to differ wildly from maufacturer to maufacturer, just as computer systems in general did. Back then, it was anyones guess as to how the computing landscape would evolve, and what would eventually become "normal". A lot of techniques and architectures were tried, some succeeded and others didn't.
But given the capabilities of technology at the time, the operations the hardware performed was likely very simple, limited to applying colour palettes to sprites and positioning sprites on the screen, perhaps with some simple collision detection, and drawing to essentially a bit mapped screen.
You're likely to find some videos on YouTube explaining how to use this hardware (I know there are some channels like The 8-bit Guy who have made their own modern games for things like the Commodore 64) which may give away some clues as to how it works, but finding definitive and specific information on how it actually functions beyond what datasheets are available is going to be difficult.