Why do the OOB (out-of-bounds) areas of Doom look so unusual? As seen in the image above, the Doom engine not only incorrectly renders the scene, but also renders slices of other scenes that aren't even loaded in the map. This is not the case in other versions of doom. Just the MS-DOS original. Why does this happen and how is it possible?
This is known as "Hall of Mirrors" effect.
The Doom Wiki states:
When the game renders the level to the screen, it draws it into a buffer, an area of memory. During gameplay, the previous contents of the buffer are overwritten by consecutive frames. However, if a player travels outside the level, there are no walls to draw, so Doom draws nothing. Instead, the previous contents of the buffer are displayed, left over from previous frames.
If a player turns around and looks back toward the level, they can see through the walls and inside the level. The floors of the level stretch outwards toward the screen edge in vertical columns. This is because the Doom floor and ceiling drawing system is like a flood fill algorithm: because there are no walls to bound them, they simply "bleed" down to the edges of the screen.
The old MS-DOS release of DooM renders the scene differently than newer source ports do. There are some good resources online that cover your question better than I could, but this answer may give you an idea on where to start looking :)