There was Lotus ScreenCam, which was initially released for Windows 3. It used its own SCM format, which only recorded events and optional audio. Later versions (I’m not sure which, or whether this was ever available in a Windows 3- or Win32s-compatible version) could convert SCM videos to Flash. Here’s a screenshot of ScreenCam 2, which is included in Lotus SmartSuite 4.0:
This review of ScreenCam 2.0 gives a good overview of its capabilities.
As others have mentioned, Windows 3-era computers weren’t powerful enough to record full-screen videos in real time — they lacked the CPU power to compress in real time, and the disk capacity to store uncompressed video.
Your suggestion to use DOSBox is probably the simplest option nowadays!
(ScreenCam is still available for current versions of Windows, published by SmartGuyz.)