AFAIK, in Windows 3.11 it was possible to enter all kind of subnet masks, e.g. 255.255.255.1. Nowadays, the bits of the mask have to be consecutive.
However, I could imagine that such a unusual subnet mask actually worked as long as it was applied to all computers consistently. An algorithm using bitwise operators would come to a result and be able to conclude whether a packet is targeted or not.
Did this actually work?
Let's consider typical home use hardware of that time such as hubs and switches on Ethernet level, not routers or gateways which might have a different OS.