Note: I'll ignore the payphone aspect of your question for the moment: I'll just treat it as wanting to connect together a number of old, pulse-dial telephones for internal communication.
Essentially what you need is a small PBX (Private Branch Exchange). These come in all sizes from around four-lines up to several thousand and are used in offices of all sizes to provide internal phone calls (extension to extension) and to reduce the number of external lines needed to provide incoming/outgoing external calls (by sharing access to a limited number of "real" phone-lines among all users). Originally PBXs were all "copper based" – each extension around an office would be wired back to the main PBX unit using dedicated telephone wires. These days, the internal routing (at least) is likely to be network based – extensions plug into the internal TCP/IP network alongside PCs.
In your case, you would probably just be using the PBX to route calls between the phones: in my (limited) experience1, external lines would not be necessary.
Your main problem is finding a small PBX that supports both "copper" telephone lines (doesn't appear to be too difficult) and supports pulse-dialing (instead of DTMF or "tone dialing" that modern phones use). The latter requirement is likely to be more demanding.
Note: This is where my personal experience ends... you would need to do some digging of your own to confirm whether any of the resources mentioned below would work for you!
Starting with a search for "pulse-dial pbx" led to the following question on the UK Vintage Radio forum:
Pulse dial exchange system?
Hi, I have a few old telephones and have always wanted to set up three or so around the house. I just wondered whether there is an exchange system that can support pulse dialing? I really don't know much about phones so please do excuse my ignorance!
Skimming just a few of the responses (all from 2015, so quite recent; emphasis mine):
I have set up a similar system earlier this year. I used a BT Revelation wired from the house NTE5 box with a plug and socket so that all the old phones could be disconnected leaving the modern system as was. This allayed my wife's concern! The BT Revelation box was easy and cheap to source from e-bay. All the documentation is available FOC on the web. If you search this site you will find the relevant threads. I was a novice to old phones but found this straightforward to set up.
Some years ago, I had the same "problem" and ended up with buying from UK a Panasonic KX-T616 and one programming telephone for this. Later I have seen many other options, but still the KX-T308 or KX-T616 seems to be the most reliable and easy to set up system accepting pulse and tone dialing.
A BT Revelation PBX would be ideal for connecting your pulse dialling phones to however they do seem to be increasing in price these days. One advantage of the Revelation is that it will convert the pulse train to DTMF if you wish to use your phones to make calls via the PSTN.
If you are only interested in using the phones "internally" a Minimaster 3 would fit the bill but these are becoming quite rare now.
This suggests there are systems still in existence that can handle pulse-dialing. From further web-searches, I've seen second-hand / refurbished examples of both the BT and Panasonic models mentioned on sale for around £60 (but caveat emptor). Also note that, at least in the case of the Panasonic system, you would need a "modern" phone of some description to "program" the system. Other searches turned up a PDF Installation Manual for the KT61610 and a PDF BT Revelation Quick Reference Guide.
Another post, further down the thread suggests a slightly different approach using an old PC (emphasis still mine):
It's worth knowing that the Digium TDM410P FXO / FXS card for Asterisk supports pulse-dialling, so you could actually build your own telephone exchange with an old PC. There are vendors on the usual site who sell work-alikes of the Digium card.
One such card with three FXS modules (to ring an analogue telephone) and one FXO module (to connect to an analogue exchange line) will form the basis of a respectable system that can be expanded in future. And you can configure it to respond in any way you like, to any dialled number; this makes it very easy to simulate the old clicky-clicky exchanges (I have configured mine so I can call my parents using just their old 4-figure number) or set up your own services such as dial-a-disc or bedtime stories.
And as before, a web-search shows examples on sale around the £60 mark. If you actually want to make use of the payphone nature of your devices, something like these cards is probably the direction to go.
All the "payphone magic" (i.e. not connecting a call unless or until money had been inserted) is handled by the exchange (the General Post Office, latterly British Telecom or BT in the UK; AT&T or Bell Telephones etc. in the US). The exchange will know that the line is a "pay" line, and refuse to connect you (except to the operator) unless some money is inserted. The payphones let the exchanges know when this happens by sending "beeps" of specific frequencies down the wire (see Red box (phreaking)2 on Wikipedia). When the paid-for time runs out, it is the exchange that emits the loud, rapid beeps (at least in the UK) that prevent the conversation continuing until more coins are inserted.
To the best of my knowledge, your payphones should operate as normal telephones if connected to a normal PBX as there will be nothing to block the calls. This will allow "retro phone calls", but won't require that coins be inserted first (although it might be possible to hear the "beeps" if you do).
I am also "almost certain" that no "standalone" PBXs will have facilities to impose "payphone mode". However, it may be possible to recreate this with the Digium/Asterisk cards. The post above implies that call-handling is configurable / programmable: providing there is a way of detecting the coin-insertion tones (either direct in hardware, or possibly in software), it seems at least plausible that you could only connect calls after coins have been inserted, and to issue beeps or a message when a timer runs out. (However, this is speculation on my part).
1 Twenty-five plus years ago (so I think it qualifies as "retro"!) I wrote software to send faxes through a fax-modem. The company had an 4-line PBX that it had (only recently) outgrown. By connecting my computer via a fax-modem to one extension, and a real fax-machine to another, I could test the software without having to use (and the company pay for) external calls. This worked happily without any external connections.
2 The term "Red Box" comes from the purported colour of boxes designed to mimic these "coin-inserted" tones (along with boxes of other colours to perform other abuses of the phone system). However, as Phreaking boxes on Wikipedia notes, use of these colours owes more to folklore than hard evidence.