If electromechanical computers count, then it would be remiss to ignore Konrad Zuse's relay-based computers, built just before and during WW2. These were, as I understand it, programmable digital computers, and can therefore be considered alongside the Heath Robinson and the Colossus as the first practical examples of such a thing. I don't believe they were ever used by the German war machine, for which we can probably be grateful.
Though not what we'd now call a programmable computer, the Bombe machines used to help crack Enigma were sophisticated electromechanical calculators, devoted to a specific (and quite vital) task which modern computers can perform only slightly faster. Rather than relays, the active components were rotors with circular arrays of electrical contacts, very similar to the Enigma machine itself but on a much bigger scale. The "menus" they were configured with could be considered programming of a sort.
The oldest still-extant digital computer is the Harwell Dekatron, which uses then-standard Post Office relays for sequencing control, but which however uses cold-cathode and thermionic valves for the high-frequency (about 100Hz) pulse circuit, memory and some calculation circuits. Although much slower than many of its contemporaries due to its heavy reliance on relays (Post Office relays are very big and slow), it was much more reliable and therefore useful, and probably cost much less to build and run.