I know that nowadays Tantalum is very common in computer components, specifically capacitors, due to its desirable electrical properties. I'm curious when it was first used specifically in the context of computing, and for what purpose? Bonus: did the computer spur the use of Tantalum (that is, was Tantalum useless before computers took off) or were computer engineers looking for something that had desirable properties and found that Tantalum, being used elsewhere, would do the job?
It is not used so much as it used to. It tends to be replaced by ceramic and polymer electrolytics as tantalums have their limitations. But back in the 1970s the electronic calculators needed small capacitors with high capacitance and low ESR (impedance) for power supply bypass capacitors, and tantalum capacitor technology had been improving so much over the decades that those properties were met.
Tantalum is a good conductor which can easily form a thin layer of oxide on its surface, which in turn acts as a good dielectric. This makes it a very good material for constructing electronic capacitors. In this form it was probably used as soon as semiconductor diodes and transistors became the principal computing medium.
It is also used for making high-power resistors, and as part of a "getter" to maintain the vacuum in thermionic valves. So it's entirely probable that early computers (1940s-50s) based on thermionic valves also used tantalum in either or both of these components. Valves using tantalum were first made for HF/VHF radio applications.
I am no expert in this topic, but the other comments and answer lack the purpose of the capacitor usage OP is asked for, so here it is:
If you look at any digital circuit it contains gates. The gates are like a power switch - they turn on/off. The problem is that if you have a complicated circuit with many gates (like CPU) with very fast speeds, then to propagate signals through all gates during small time you need very sharp edges of the signals, hence relatively big current impulses for short time.
The more gates and the less time, the bigger current spikes we need. Ah frequencies above 10 KHz the lines on a PCB are behaving more like a distributed transfer circuit constructed from capacitors, coils and resistors instead of simple wire. And the energy needed to transport from power supply to the final gate gets distorted, diminished, reversed, lost ...
On top of all this, big current changes over small time will create induction to nearby circuitry causing problems (because the PCB lines behave also like antennas).
The solution to all these problems is to add blocking and filtering capacitors. What they do?
Blocking capacitors shortcut very fast frequencies so they do not jump to another circuit nearby. You can look at them as antenna dampers.
Usually, several capacitors in parallel are used with different technologies very close to power supply pins of each digital IC. Each blocking capacitor has a different capacity (each filtering different frequency). Usual values are
33pF + 1nF + 10nF + 100nF in parallel. For low frequency or less noisy ICs, usually only 100nF is used.
Filtering capacitors provide temporary power storage to cover the current spikes without propagating the spikes through power lines all over the nearby circuitry. These caps have to be fast and have enough capacity in storage. They simply convert the fast discrete current pulses to slow continuous small current from power supply.
Technology of capacitor affects its capacity and also usage as they differ in behavior too (not just capacity) like durability, current response, localised frequency response ...
Ceramic caps are the best as they last forever, can handle current in both directions, but they have only small capacities so they used for blocking.
Tantalums can have much bigger capacities (1µF and more) and also last forever but can not handle reversed current (exploding, offsetting voltage, making mess). They are ideal for medium power filtering, but are sometimes also used for blocking too.
Electrolytes can have big capacities (1000µF and more), have limited lifespan and also can not handle reverse current (exploding, leaking), on top of which they are very susceptible to higher temperatures. They are ideal for power hungry supply filtering.
Rolled capacitors are not very often used in digital circuitry, as they are too big in comparison to ceramic/tantalum counterparts doing the same job.
There are more technologies out there like polymer, supercap,... but those were not usable in the Retro era.