A/the correct solution would be to set up some sort of small local PBX. In the spirit of retro you can probably score an analogue PBX for not very much money and the mechanical ones are arguably even more fun to play with than retrocomputers because you can watch them working, but if you would like that bit to be somewhat smaller and more reliable than a clattering Strowger exchange and aren't intimidated by Linux, a couple of VoIP ATAs ("Analog(ue) Telephone Adaptor") and a machine running something like bare Asterisk or a full FreePBX system would be how I'd do it. Or rather, how I did do it back in ~2005 for a substantially similar project to yours.
For testing the PBX you may want to start with a pair of phones until you're satisfied that calls work. Then it's a case of plugging in the modems and calling from one to the other using a pair of terminals or terminal emulators. This may need some further tweaking of VoIP settings since high-speed modems are more picky than humans about sound quality, but for a local-only configuration it should work so long as you're using the G.711 codec.
With VoIP, you should be able to easily configure a software "tap" and get a call recording in perfect digital quality. A more advanced analogue PBX should offer a tap feature, otherwise you're going to just have to dig out the croc clips and look for somewhere plausible to attach a tape recorder.
If you're OK with something a bit jankier than going the full PBX route, connecting two phone sockets back-to-back and using a PP3 battery to provide (barely) line voltage is apparently sufficient to convince modems that they're connected to a phone line. You'll need to tell the caller to ignore the lack of dial tone and force the receiver to send "ATA" (in this context, that's a Hayes modem command) because there will be no ringing voltage to tell it there's an incoming call, but this may be enough. Again, some ingenuity may be required to get a recording.
From here, if you want to use it rather than just record the tones, it should be simple enough to install a BBS application on one machine and dial into it from the other. This is perhaps as far as a non-expert may wish to go.
For the full late-90s dialup Internet experience, you get the fun(?) job of setting up a mini-ISP, probably on a Linux machine—doing this on the same machine that's running Asterisk should be fine—with at least some sort of (faked?) RADIUS login and PPP. Apart from the details regarding physical serial ports, this now falls outwith the scope of retrocomputing since modern broadband still involves PPP, RADIUS, etc, and in some devices you even get the pleasure(?) of using obscure, possibly even undocumented, Hayes modem commands to configure the connection.
Note that you'll only be able to get up to V.34(bis) with a pair of analogue modems. V.90 and other "56K" technologies requires a digital connection at the ISP end. VoIP is that digital connection, but when I last looked into this a couple of years back, I could not find a suitable softmodem which could answer V.90 calls. (I briefly considered writing one, but it's a lot of work for little reward.) Previous work to record V.90 involved finding somebody with a decades-old ISDN modem bank and a suitable PSTN connection. I think they had to make international calls to Australia in the end, and the extra latency of going half way round the world meant that the tones sounded a bit off compared to memories of local dialup calls back in the day.
One advantage of setting this up using VoIP rather than analogue is that it's much easier to connect to the wider world by signing up for a VoIP service and bridging the networks. Other interested people can then dial in to your system, or you can dial into theirs.
If you are really into the idea of using an analogue PBX instead, @gewt on Twitter (profile tagline "cat lady, but the cats are phones") is a fellow fan of them and should be able to offer advice or point you at suitable units.