Disclaimer: This answer is not known to be accurate.
From the information you've provided so far it's tricky to give complete advice, but here's a start:
Removing corrosion
The low CMOS battery error might be caused by battery leakage; from what I've seen of leaky batteries, they do appear rusty. This can cause significant damage to components. To neutralise the battery acid (which is actually alkeline), pour vinegar on the affected areas and leave it to soak for ten to fifteen minutes.[1]
To remove rust and corrosion, use isopropyl alcohol and something soft (e.g. cotton bud, toothbrush, other soft brush).[1][2][3][4] It can be more effective to soak the board, but you must ensure that the alcohol does not enter any speakers or other sensitive components.[2] It's possible that there could still be dissolved or loose residue on the board after cleaning; distilled water can be used to remove this (either by soaking the board in it or by pouring the water onto the board).[2]
Replacing the battery
From your description of the battery, it is a standard 3.6V rechargeable Ni/Cd battery.[3] For these instructions, I'm assuming that it looks something like this:

This is tricky to replace. The first step is to remove the battery. This can sometimes be done with side cutters,[3] but you can also de-solder the battery if this is too hard.[3][4] If there's glue still attaching the battery to the motherboard, gently rock the battery from side to side until it becomes loose.[4] Once the battery has been removed, you might want to clean the area some more so that the new battery isn't damaged by remaining leakage. Assuming that your board doesn't have a replacement battery slot, you can solder two wires to the gaps and connect them to a replacement NiCd battery that remains external to the board, so that the board won't be damaged when it leaks again.[3] As Richard Downer's answer says, ensure that you don't wire a non-rechargeable battery in such a way that it charges. It's probably best to go with a rechargeable instead, just in case.
Replacing capacitors
Your floppy drive controller might be failing because the capacitors have failed. This is bad, not only because they require replacement but because they leak a corrosive substance that can damage the board.[5] To replace them, you need to de-solder them from the board. First, check whether they're actually bad. You can do this by testing the components, but it is sometimes easier to check visually (though not all capacitors fail visibly![8]). See if you can spot the failed capacitors in this image:

After you've identified the faulty capacitors, you need some replacements. Find, buy, borrow, salvage or otherwise get your hands on some capacitors that have the same "stats" as on the existing ones. It's best to get a capacitor with the same capacitance and at least the same voltage,[7] but a higher capacitance also may work for many capacitors on a mainboard. A higher voltage rating is no problem, but don't go for less or it may explode![11]
On to de-soldering! There are a few ways to go about doing this; the one that I've found easiest (with other components) is melting the solder on one leg, pulling it out as far as it will go without breaking, then repeating on the other pin, alternating pins until it comes out. You might see some gunk near or underneath the capacitors; this can probably be cleaned off using the same method as outlined in the above "Removing corrosion" section, sans vinegar (I'm not sure whether this would neutralise the electrolyte or make it worse). If this doesn't work, gently scrape it off the board, taking care not to damage everything further.[7]
Often, solder will remain in the holes that you took the old capacitors from.[6][7] This can make it hard to push the new capacitors in without damaging them (don't try). There are a few different methods to remove the solder; they are listed here in one recommended order:[6]
- Melt the solder (adding a bit of new solder in the process if neccesary[10]) then suck it off the board with a solder sucker.[6][10] Do not do this more than twice; you risk damaging the board[10] (which is surprisingly easy to do in my experience).
- Place a de-soldering wick over the hole, put the soldering iron on top, and watch as the solder disappears.[6] This has the disadvantage of applying lots of heat for a prolonged period of time, so it's best not to use this if there are any components nearby.[6]
- Find a steel[6] / stainless steel[10] pin at least the size of the capacitor legs. Either heat the bottom of the pin whilst pushing it through[6] or melt the solder before pushing the pin through the hole.[10] You can twist the pin as the solder sets if you like,[6] but this should work without doing so. After the solder is hard, twist and pull the pin out of the hole, then brush away any remaining solder to leave an open hole.[6][10]
If the capacitor legs still refuse to go in for whatever reason (solder attached to the legs, the legs being too big, etc.) you can clean them with sandpaper so that they fit.[7]
When replacing the capacitors, make sure that they go the right way round, otherwise Bad Things™ may happen.[7][8] With any luck, your board will look like this:

Usually, the negative side has dashed lines and the positive side is marked with a +.[6][9] However, there are some non-standard manufacturers who put the dashed lines on the positive side,[9] so make sure to check which side is which. Once you've identified this, put the capacitors into the holes the right way around. Capacitors are often marked with negative signs on the negative side[6][9] and the positive leg may be slightly longer.[6] After they are in the holes, trim the legs with side-cutters[9] so that they stick about 2mm out of the board,[6] then solder. If you solder then cut unnecessary stress may be put on the board,[6] but this is probably not enough to cause significant damage.
It is recommended to remove all capacitors of each type before replacing them.[6][10] It is also recommended to remove all capacitors of each type if one has failed, because they're probably from the same batch. It is also recommended not to do this unless you really know what you're doing.[6][9][10]