Can anyone explain why we started using this other socket shape?
It's the power rating. Or more correct, a changed power rating
DIN/IEC did original rate C14 at 10 Ampere and C6 at 2,5 Ampere. But the US standards and testing organisation UL (Underwriters Laboratories) (*1) does considere the usage of C6 with up to 10 Ampere (C14 as 15A). Thus UL certified devices/power supplied may use C6 instead of C14.
The reasoning behind is in part based on the wattage of the device connected.
This wouldn't be a big gain in itself, as the IEC standard also defines that the C5 socket has to have an inset with the same square footprint as the C14. Here as well UL did diverge and certified the usage of sockets (*2) without the inset, resulting in a somewhat smaler footprint, preferable for small external power supplies.
*1 - Well, it wasn't just them, but a combined action with the Canadian and Mexican standard bodies to create NAFTA wide standards.
*2 - IEC standard conforming cables will still fit both versions and where used by manufacturers. This resulted in a somewhat huge plug for a small socket. Soon the 'IEC part' of the plug got reduced to a slim shield covering the rectangular shape and finally (mostly) eliminated.