I can't remember where I saw or put it, but sometime in the last couple of months I came across an installation for an early IBM system somewhere online, in scanned PDF form (maybe in the Bitsavers documentation archive?), most likely their first "RAMAC" hard-drive using rig plus all the associated access and computing hardware for using it at the heart of an electronic accounting system.
It's some pretty heavy engineering. The first parts of it lay out the minimum allowable, and recommended dimensions for the room that the drive, the computer, the operator console, their various power supplies and glue logic racks will be installed into, in plan form, mapping out their positions (a multi-cabinet U-shape with the free-standing operator console nestled within) plus the floor-level cabling runs and the space needed for operators to get around the back of the machines to access service panels etc that would otherwise be sandwiched against the walls. I can't recall the minimum dimensions, but the recommended space wasn't far off the entire floor area of my apartment (which is about the same size as several previous workplaces, encompassing different offices and even medical examination rooms), pegging it around a square 25ft / 7-and-a-bit metres on each side. The access space around the "back" of the cabinets was no more than 4ft even so.
And that's for a relatively minimal system. Bear in mind that the term "mainframe" comes from an old term for the central processor of a computer - each of the system's primary components (CPU, memory, I/O handler, power regulator, etc) would be built into one or more of those large "frames", of which the processor (made up of a great many individual, interconnected rackmount cards stuffed full of discrete components or, much like HAL 9000, small-scale integrated circuit packages) was of course the "main" example. The singular encompassing the plural, it came to imply any complete computer system built along similar lines, regardless of size...
And of course as well as the plans there were photos and an artist's-impression birds-eye view of the machine in use. The two or three humans in the picture didn't look ever so big compared to the computer, but it was still sort of human-scale, instead of being a monstrous factory-filling item.
Of course, that's not literally filling the room, but it is taking up enough of it, in awkward enough positions, that you can't really make practical use of the space that's left unless you install the machine in the corner of a considerably larger room and employed the alternative L-shaped layout instead.
A "minicomputer" setup, by comparison, would have been one small enough to only occupy one or two such frames, or within a wider but lower piece of furniture analogous to a mainframe's operator console, with the processor, memory, offline storage devices, power supplies, and maybe even the user interface (where it wasn't a separate compact terminal or a teletype) all fitting within that limit.
Incidentally, you make reference to mainframes being "like" rackmount servers... actually the comparison is more direct than you might first imagine. The format of 19-inch communications and server (etc) rackmount frames is pretty much exactly the same as that of, particularly, IBM mainframes and/or DEC minicomputers. I can't remember if it's a direct heritage or just convergent evolution, but in either case it's the same environmental limitations that moulded them - to whit, they have to be able to fit through a standard doorway without getting stuck or damage being caused to equipment or building fabric, even if that may mean temporarily taking the door off its hinges. And it also needs to get around tight corners in corridors, stairwells, etc whilst in transit to the target room and being turned to fit through the door. But at the same time you want to have the maximum amount of useful space within each frame to minimise the floor area taken up by it, to maximise the amount of tech you can cram inside each box, and minimise the number of boxes and thus the building and shipping costs. So the format quite rapidly converges on the largest rectangular box that will fit through doorways and can be moved easily through buildings made to a common human scale. I think at one point it was even commonly called IBM Frame Size. But in general it's about 24 inches wide, a shade under 7ft high, and about 30 inches deep...
(I've even seen distinctly non-mainframey equipment clearly built to similar guidelines - like the back-projection mobile interactive whiteboards made by Smartboard back before short-throw lenses were good enough to make slimline front-projection models practical. They were quite a bit wider across the beam of course, but measured just less than a doorway's width front to back - a little less than a mainframe in fact, so you didn't have to take the doors off - were just short enough to fit under the lintel when lowered to their minimum height, and the castors and V-shaped integrated trolley compensated for the width by making it somewhat easier to "lever" around difficult corners and wiggle into rooms from narrow corridors... I expect all-up size was something similar to the largest part of the RAMAC setup, ie the massive, ~36-inch, horizontal-spindle hard disc unit proper, which needed a crane to lift on and off of trucks and planes and sometimes demanded remodelling of buildings to get it installed... like, holes would have to be made in walls and then bricked back up again afterwards, floors preemptively reinforced, etc. With the console being almost as bad, but at least being lower-rise and a sensible weight. Though the Smartboard was way lighter and moveable by one person, whereas even the pure logic parts of a mainframe usually needed a team of two or more...)