But the first systems didn't run an OS like that?
Well, they did, some kind of proprietary OS.
So how graphics were programmed to these machines?
Using functions of said OS, or bare bone.
Did the companies developed and implemented their own graphics APIs (like OpenGL)?
Exactly. Just usually a lot less advanced than that.
It might be useful to keep in mind, that early systems didn't have as much computing or rendering power as PCs of the same time. It's only rather recent, that console CPU and GPU are for most parts on the same functional level.
Much like 3D GPUs for PCs did take quite some time to accumulate all functionality known today. At first they did only support certain steps of a rendering pipeline. Similar interfaces like DirectX did evolve in parallel to this, not only moving more and more functionality into hardware, but also adding more. By now this development has slowed down a lot and allowed consoles to catch up in hardware (CPU+GPU), thus allowing the use of 'standard' solutions.