Some other device (probably a computer) which we shall call "the terminal" to connect to your old device. On DOS or Windows, the device file for serial devices are called
COM4. On Unix/Linux systems, they in the form
/dev/ttyS*. You can use the following command to discover the names of your serial devices:
sudo dmesg | grep 'serial|tty'
You may get some false positives for
Some kind of software on the terminal which you can configure the serial settings, send typed characters, and display the received characters. Many Unix/Linux distributions include
cu, which simply copies standard input/output to the serial line -- there are no extra features -- but is sufficient for most of the instructions below. The gold-standard program is
minicom. It sounds like you're using PuTTY, which will also do just fine. Whatever program you pick, read its
man page or documentation to find out how to configure and use it.
(Optional) A voltmeter or oscilloscope will help you check voltages and polarities.
(Optional) A storage oscilloscope or logic analyzer will help you fully analyze the signals.
Although standards such as RS-232, RS-422, and RS-485 existed, you can't rely on any device to follow them. Standards were often incomplete, devices often evolved faster than the standards, and many developers simply ignored standards. Thus, there are a small number of devices for which these instructions won't work. (In fact, one of the many advantages of USB is the thoroughness of the standard leaves little room for variations, simplifying support for devices.)
Another consideration is the protocol being used. Serial can be used as a way for humans to interact with a device (e.g. a remote login). It can also be used for one device to send results to or control another device. Using a text-based protocol makes it easier for debugging; a developer can simply "talk" to the device through a terminal rather than writing a special program to do that. However, other devices use a binary protocol, as it is easier to program and more efficient of bandwidth. Furthermore, some devices (such as point-of-sale terminals) have protocols that are intentionally kept secret.
Without documentation, binary protocols are difficult to work with, and are beyond the scope of this answer.
Check the physical connector on the old device. If it is labelled "Serial" or "RS-232", continue below. If it is a 9- or 25-pin D-shaped connector, it might be a proper serial port. Your chances are better for consumer devices, but be warned that scientific and industrial devices often used the same connectors for non-serial uses. Older Macintoshes had a circular DIN connector which could and was used for RS-232 and RS-422. If you are tapping signals off a circuit board (with or without a header), there is a chance that it is only a logic-level signal (see next step).
(Optional) Using a voltmeter or oscilloscope, with the old device powered up, check the voltage. Connect the minus lead of your meter/scope to the ground pin of the device, and the plus lead to the transmitted data pin. If you get a negative voltage, you are good to go. If you get +5 V or +3.3 V, these are logic-level signals, and you will need a level shifter (like a MAX232/233) to get the correct voltages. Furthermore, (rarely) some logic-level need to be inverted before they are level-shifted.
Turn off hardware flow control on your terminal. Flow control was used in early systems that would otherwise lose characters if they could not keep up; for example, a printer might need time to move the print head during a carriage return, and would lose any characters that would be sent during this operation. Systems that are sophisticated enough to buffer incoming characters (most computers by 1990 and increasingly in peripherals as time progressed) usually had flow control disabled (although often still available). By turning off hardware flow control, this should assert the outgoing flow control signals and ignore the incoming flow control signals.
We will avoid sofware flow control by not sending any control-S (binary 0001 0011, DC3, XOFF, STOP) or control-Q (0001 0001, DC1, XON, QUIT) characters. Also, if your terminal has an option for software flow control, disable it so that recieved ^S and ^Q are treated as normal characters.
A device's response to invalid parity and stop bits (framing errors) can vary. There are various flavors of errors, which some devices distinguish, but doesn't matter here. Some devices drop all characters with framing errors. Others ignore the error and process the character normally. Some have a visual indicator or log the event to a file. Set your terminal to ignore all framing errors. (If there is no such option, don't worry about it.)
At this point, try 9600-8-N-1. That is, 9600 baud, 8 data bits, no parity, and 1 stop bit. This was a very common default among equipment.
While the terminal is connected and running, reboot your old device by powering it up or pressing its reset button. What is printed on the terminal when you perform the reset?
(a) If it is readable text, congratulations -- you have have found the correct baud rate and may proceed to step 10.
(b) If something garbled appears, repeat this step with each of the other baud rates. Should readable text appear, you have found the correct baud rate and may proceed to step 10.
(c) If every possible baud rate gives you garbled text, it is likely that this port uses a binary protocol. You may try the other steps below, but be aware that you may fail to find the correct settings.
(d) If no text ever appears at any baud rate, the old device probably doesn't output text upon reboot. Don't worry, continue to the next step.
If you still haven't found the correct baud rate, set 9600-8-N-1 and try to send several space (binary 0010 0000) or 'U' (0101 0101) characters. I have seen hardware that uses either character as an automatic baud-rate detection.
Still haven't found the baud rate? Send '0' (binary 0011 000) and '?' (0011 1111) at each baud rate, until either is echoed back correctly. If this still fails, you will have to give up.
At this point, the only things left are the data bits, parity, and stop bits. By the 1990s, most equipment were 8 data bits (decline of wierd word sizes and an increase in international character sets), no parity (serial was reliable enough not to need it) and 1 stop bit (parity and extra stop bits are a waste of bandwidth). Even if your old device uses something different, it won't matter unless your old device catches and handles framing errors. So, you are probably good to go.
If you really want to know the data, parity, and stop bits, you will need a storage oscilloscope or logic analyzer. You must make the device produce a known output (such as a boot message), and then analyze the signal produced.
Linux does have a programming interface for serial errors. To my knowledge, no terminal software makes use of it. You could try writing your own program, if so inclined. Otherwise, you could also play around with sending/receiving characters and then
cat /proc/tty/driver/<driver_name>, but I think this is extremely tedious.
The Liebert RAC2-8 is a unit to monitor and control heating and air conditioning in a building. Sounds like a cool project! (Sorry for the shameful pun.)
I got the manual here. There are two 25-pin RS-232 serial ports; you want the one labelled "Terminal", not the one labelled "Modem".
It appears that 1200 baud is the factory setting. Start there, and work your way up if things fail.
There is no mention of other serial settings, but the modem port requires Hayes-compatibility, and it sounds like it sends a status message when a terminal connected. The latter is probably the cause of the white squares you are seeing, and is possibly triggered by sensing the flow-control signals. My best guess is therefore 8 bits, no parity, one stop, and flow control on.
A terminal connected to an active RAC2-8 displays alarm and status messages when they occur. The prompt then asks the operator to press <space>, <S>, or <A>.
<space> displays Main Menu after a password is entered.
<S> displays present status.
<A> displays a list of alarms (up to 20).
If you don't know the password, it can be changed from the menu system on the physical front panel. Good luck!