Virtually every telephone modem in existence runs at a data rate that's a multiple of the Bell 103A's 300 bps. Why was the base 300 bps chosen in the first place?
10 CPS/110 Baud was the maximum rate these signals could be sent with acceptable sidebands using an all-mechanical system. 300 was 3 times the teletype speed, and that limit is set by the 4 kHz maximum bandwidth of a phone line and the allowable harmonics.
300 baud is exactly 3 times 110 Baud, measured in characters. The teletype standard was 110 bps with 1 start and 2 stop bits. That, plus 8 data bits (7 plus parity) equals 11 bits per character. 110/11 = 10 CPS. 300 Baud used 1 Start and 1 Stop, plus 8 data = 10 bits per character. 300/10 = 30, and 30 cps is 3 time 10 cps.
Mechanical teletypes such as the ASR33 sent 10 CPS. They had a rotary wheel that spins when a key was pressed. It had 11 contacts. The first was wired to break the current flow in the communications wire. This was a current loop, sent over a phone line back to the Central office, powered by a 48V DC battery. The last one or two contacts were wired so that the current was always on.
The 2nd through the 8th contacts were wired to a 8 switches that were pushed by a matrix that encoded ASCII from the TTY key that was pressed. Pressing a key released the motor clutch, the rotary contact wheel would revolve, and the make/break of the rotary switch would send the signal over miles of wire back to the central office at 110 Baud. This yields exactly 10 CPS. This had a start bit, a stop bit, and either one or two extra stop bit(s), a spacer, to allow the system relay to re-energize when no more characters were being sent.
Later FSK systems could run at 300 Baud, and not being mechanical, had no need for the 11th bit (the second stop bit).
300 Baud systems were the first electronic systems, and could stop without an extra stop bit, which increased speed by nearly 10%. The signals were modulated by FSK between two frequencies for transmit, and two for receive. Switching signals between these frequencies generates harmonics, which have to be kept within the 4Khz bandwidth of the phone system to prevent crosstalk.
The odd 11-bit 100 baud standard versus all other bauds being multiples of 10, (300, 1200, 2400) are caused by the differences in mechanical and electronic FSK/PSK systems that evolved.
With PSK (Phase Shift Keying) the amplitude and phase were changed. This fit within the same 4KHz signal bandwidth with the same sidebands. Each additional amplitude or phase shift doubles the number of bits sent per Baud, thus we ended up with 300, 1200, 2400, 4800, 9600 and so on.
The signalling rate (baud) is constrained by a few things. Probably most important is the maximum signalling rate (roughly, how many changes a second) of the path the signal takes (i.e., POTS wires). How many bits can be represented by one signalling change gives us the bits per second.
I think the early Bell modems were 110 baud and used frequency-shift keying (FSK). This gives us one bit per signal change, or 110 bps. Similarly the Bell 103 had a signalling rate of 300 baud with FSK giving us 300 bps.
110 and 300 baud were chosen at the time primarily because both modems were intended to be used over copper wire and "unconditioned" telephone lines, with at least one part of the connection going through an acoustic coupler. The worst-case for acoustic couplers talking to carbon microphones is somewhere around 300 baud. Since this is a worst-case, this is what we get.
(I recall 110 baud was related to reliable half-duplex teletype comms, but I might be wrong about that. @jameslarge points out that 110 baud/bps was the fixed, unbuffered rate common teletype terminals supported. 110 was probably chosen for many of the same reasons discussed here. e.g., robustness and reliability on dodgy copper and carbon connections.)
This could theoretically be increased, but reliability suffers.
A natural improvement is to increase the number of bits that can be transferred for each signalling rate, which is what newer modulation techniques like PSK, Trellis, etc. gave us. The baud rate can stay the same so that it is within safe parameters for unconditioned lines, and the bit-rate can be increased.
As phone lines improved and lines could be counted on to be conditioned (and we knew there wouldn't be a carbon-microphone step in there somewhere) and advances in modulation error correction and error detection, baud rates increased. This led to increases in bit-rate. I think the last telephone modems had a baud rate of 8000, and modulations that allowed for 56/46 kbs as a result.
300 bits per second has the advantage that it is the lowest common multiple of both 50 and 60. These made it easier to use the power line frequency (50 Hz in Europe, 60 Hz in USA) to synchronise the bit timing circuits. This was long before Quartz locked circuits became cheap enough to include in teletype equipment.
A very good first answer however I would also like to note that any data rates above 300bps could not be acoustically coupled and were direct connect modems only. And anything 33.6K and above basically demands at least one digital endpoint. By that I mean that the modem access concentrator would connect to the Public Switched Telephone Network via data-grade T-1 or larger (see also AT&T T-Carrier) data trunks. The DS-1 that rides the T-1 is channelized and the DS-0s are 64K clear channels and thus suitable for data rates up to 57.6K plus some protocol overhead and error correction. The end user has significantly lower upload speeds because they're using analog (voice) lines. This is also the genesis of today's asymmetric residential data model.
But it all comes back to baud rate. Bit rate is a measure of the number of data bits transmitted in one second. Baud rate is the number of times a signal in a communications channel can change state in one second. Regardless of the techniques used to encode bits in a carrier, the baud rate was always a limiting factor and was determined by the dynamic range of the signal processors available in the PSTN at the time. 110 Baud was safe. 300 Baud was pushing the limits of signal processing available in the 60s and 70s. And from there we were able to stack new methods of line coding and framing on top of that 300 Baud to make it almost up to 64Kbps. Amazing, huh?
A slight tangent but my first computer, a clone of the Ohio Superboard II called the UK101, had a cassette interface which was 300b/s using the "Kansas City" method of FSK. The computer had a simple UART whose TX connected to the frequency control of a simple modulator, and RX connected to a crude frequency detector and thus derived the binary stream from the "warble" of the FSK.
With a higher quality cassette recorder it was possible to run the tape interface at 600 baud (I can use baud and bps interchangeably in this context) or even 1200, but the latter was very supsceptible to the slightest tape glitch.
Or, you could build an RS232 level converter and connect the UART to a 300 baud modem.