I think 3D (read: gaming) just wasn't their target market.
The thing about Matrox cards was that they were always a bit more expensive, targeted at workstations and "high reliability" environments such as control rooms, security monitoring, emergency dispatch, power grid, traffic and light control, process automation and medical sector. Really any place that has a "reliability component" which requires a lengthy testing and validation process and where components can't fail.
AMD/ATi and nVidia didn't target this market in their 3D components because history has shown that they had no problem releasing cards that sounded like a vacuum cleaner and needed a ridiculously powered power supply. Matrox couldn't release these kinds of things and maintain their standing for reliability. It isn't to say that Matrox couldn't have (it could possible cheapen their brand image) nor is it to say that AMD/ATi and nVidia didn't release more reliable cards (they have), but the real cutting edge 3D gaming stuff is pushing the limits without regard for failure.
As for the turning point in the late 90's for "king of the hill" 3D GPUs, the thing that really separated the cards at the time was color depth and various 3D features (filtering, etc.). 3Dfx was the top, but ATi and nVidia ate their lunch simply by offering more features. I recall that reviews (e.g. Maximum PC) would state how Unreal looked so much better with the nVidia Riva TNT's 32-bit color in comparison to 3Dfx's 16-bit.
For what it's worth, the problem with 3Dfx and a bit of its downfall was that the TNT was a similar cost all-in-one solution which had equivalent performance to the 3Dfx Voodoo2. Yes, the Voodoo3 was an answer to that as a faster all-in-one card, but the problem is that it used the same chipset as the older Voodoo2. It competed on performance, but there were no new features. I don't recall enough of 3Dfx's history, but it might simply be when they started addressing color depth and other things, they had already lost the race. It is possible that they had been focusing too much their work with Sega as they were supposed to be the chipset for the Dreamcast, but that deal fell through as Sega went with NEC's PowerVR in the end.