Hmm, an interesting question to be sure. It certainly would have been possible to make something like a 4004 style microprocessor from TTL chips. In fact, when Intel made their microprocessor, the first in the world, they chose not to pursue a patent for it, because they felt that there was no invention there; it was obvious for someone to go and combine the workings of a processor from several ICs into a single IC.
That said, it would have been rather pointless for the home enthusiast to make a processor, because before the microprocessor there also were little or no semiconductor memories that he could program with the code that the processor runs. It's rather an unfulfilling excercise to make a stored program computer when you don't have a memory to store the program into. [Edit]: semiconductor memories were becoming available just prior to the microprocessor, and seems there was a time period when this was actually happening, please see the answer regarding the KENBAK computer!
Instead, people were using TTL chips much like programmers are using programming languages today. If you're making a controller for your model railroad, you'd pick a latch here and a flip-flop there. You'd pick up a monostable multivibrator instead of calling Delay(). The hardware design approach worked very well and people were blissfully unaware of the problems and complexities that would later come with software projects.
Building computers was a large industry in the 1960's. But the processor was not the main design problem in those computers, it was the interface to memory. It's rather more difficult to use something like a steel drum memory, even if it couples with a standard 1/2 inch shaft 6000 RPM electric motor, than, say, an EPROM.
Along with the 4004 microprocessor, Intel made also the companion chips 4001, 4002 and 4003: a 2Kbit ROM, a 320 bit RAM, and a 10 bit shift register that works as an interface to the outside world. Without the 4001, 4002 and 4003, the 4004 would have been as useless as the home TTL processor.
More info: http://www.intel.com/Assets/PDF/Manual/msc4.pdf
Interestingly, the 4001, 4002, 4003 and 4004 all were the brainchilds of a single man, a young design engineer at Intel, Federico Faggin. He was the lead engineer on all four and, even with one failed production run, he managed to make all the four chips in just 9 months. So I think those chips should be called his brain quadruplets!
Faggin is an interesting man, to be sure. Even younger, before coming to Intel, he also invented the self aligned silicon gate: the single most important IC manufacturing key technology after photolithography. He's still going strong and his interview from 2014 is one of the more interesting views in YouTube: https://www.youtube.com/watch?v=hugZii_eX30