I did 3d on both and optimizations were different on both machines; I didn't do very advanced stuff at the time, mostly rotating simple objects, etc but no game or complex scenes.
On the Amiga, you would use the blitter to draw lines and fill the bitplanes. The different bit planes were not interleaved but sequential, so the size of the blitter pass you had to do was depending on the colors used.
You would draw the same lines in all the planes needed and then do a blitter fill, but filling plane 0 and plane 3 in two passes was slower than filling plane 0 and 1 in a single pass which was slower than filling only plane 1 for example.
The blitter really sped up the filling, but the bitplane layout forced color palette organization in order to make sure the largest surfaces of your objects were the fastest to fill.
On the Atari ST, it was a different story: you had to draw your picture as you were building your triangles, so you needed to sort the vertices vertically and then, as you were doing line interpolations, you were filling up the screen.
The edges were tricky to work with, but large surfaces could be filled in a single pass.
In short, the ST required more CPU work, but drawing the picture could be done in a single pass. On the Amiga, the process was generally faster, but the memory layout generated a lot more memory traffic.
Despite these differences, the two came remarkably close in performance.