I am fairly sure that this clicking wasn't just because my computer had issues, although that would be interesting to find out. I distinctly remember that the first computers I owned would make a rapid clicking noise on powerup. These were always desktops (I didn't own a laptop until about 2004, and it did not click on powerup). My current desktops haven't made that sound for more than a decade, so I presume that things progressed. Still...what was clicking?
Its hard to tell exactly from just that brief description, but I think you're probably referring to the sound of the hard drive head seek (repositioning to different areas of the disk) as data is read from the drive. That's the repetitive ticking you can hear in this video when the Windows 95 splash screen is showing and the OS is being loaded from the hard drive (starting at 30 seconds into the video).
Modern spinning hard drives drives still make clicking noises on seek, just at a much lower sound level. Generally you can only hear it if you put your ear close to the drive and the sound is different because seeks occur much faster so the distinct clicks tend to blend into a single buzzing or humming sound.
Most likely the floppy drive and/or initializing on power up. The drives had to seek a few times to ascertain what track they were on or if there was a disk present. Most machines don't have floppies or indeed any spinning media, so this motor sound is absent from many new computers.
None quite as loud as the Apple II “machine gun”, though.