I have a very similar desire as you. However, mine goes much deeper. I have a desire to create a competitor for the following:
Apple II, etc.
In 2018, I created, designed and produced my first SBC that could have been built in 1979 and would have competed with the Apple II.
So I have a little bit of experience but I'm no expert.
These days it's so easy to grab an entry-level FPGA and just create the entire computer. That's fine for some people but not for me. I also believe the Commander X16 is clearly David's "dream computer" and not anything that would have existed in the 80's. He even calls it his "dream computer".
The computer you mentioned would be tough to beat in 1979. But you said a competitor. Not a "killer". In 1979, a competitor to the Atari 800 would have been the TI-99/4(a), TRS-80, PET and Apple II+.
So what would it take to create a computer during that time? I assume your fictional company wouldn't have had deep pockets to create custom chips. Only Atari and TI could do that in 1979 (at least for the home market).
Having said all of that, I would assume the following specs for an Atari 800 competitor:
- Color Graphics (TMS9918)
- At least 32KiB of RAM
- Audio (AY-3-8910)
- BASIC (EhBASIC)
Now, the next step is to decide how authentic you want to be. If you want real authenticity, you would only use parts that were manufactured back then. This is certainly possible but I think you limit yourself and it makes the design much harder. In my SBC (search for Potpourri6502 on the 6502 forums), I only used parts that are still made today.
One example I'm talking about is the RAM. It is MUCH easier to just use a static RAM chip (SRAM) in the quantities we're talking about than try to design a DRAM equivalent. Most 8-bit computers at the time used dynamic RAM (DRAM) because it was cheaper. SRAM DID exist. But it was very expensive. It still is. But for < 128 KiB, SRAM is only a few dollars. Your fictional computer could just pretend it used DRAM if you like. :-)
This means I think it's OK to take a few modern conveniences and still pretend you're competing with Atari. This is all just for fun anyway.
So let's start with graphics. I would suggest using the TMS9918. It's not made any more but they are very easy to obtain (don't buy from China...most are fakes or defects). The TMS9918 isn't as powerful as the ANTIC/GTIA solution from Atari. But it's very popular for the day and it's pretty easy to work with. However, it requires its own RAM and that RAM is normally DRAM. However, there are circuits out there that can use SRAM with it. It's just a little more involved (my GitHub repo has an example here https://github.com/cbmeeks/TMS9918/blob/master/SRAM%2BReplacement%2Bfor%2BTMS99x8%2BVRAM.pdf).
Next is the CPU. I would pick the 65C02. Sure, the Z80 is a good choice too. I'm just partial to the 6502. BOTH are still produced today. But remember, you're making a competitor. There were Z80 based machines with the TMS9918 in the 80's. You want something different. Definitely get the 65C02 and visit the guys at the 6502 forums. They have excellent information!
Next is the RAM. I would shoot for 32KiB. The TMS9918 has it's own RAM which is 16KiB. So your system could be advertised as "48KiB" system. Which is what the Atari 800 had. If you go with the 65C02, you can follow the EXCELLENT 6502 primer here: http://wilsonminesco.com/6502primer
Next would be audio. I would recommend the AY-3-8910. Like the TMS9918, it's not made anymore but obtaining them is very easy. The 8910 also has two I/O ports that could be tied to joystick ports. The 8910 isn't as good as the Pokey in the Atari 800. But, you could use two of them. Giving your computer 6 voices (channels) over the Atari's 4. Or heck, use four of them like the Phasor did for the Apple II.
Next would be I/O. I suggest using a 65C22 VIA to control the audio chips and provide a simple parallel or serial port.
Next would be serial (UART). The 65C22 VIA can do serial and parallel communications. But I'm talking about true RS-232 communications with other computers. A good example is the 65C51. Which is also still produced (despite a nasty bug). This chip is simple and easy to work with. Just not very powerful. Just keep in mind that if you want to connect to a real RS-232 port, you will need level converters such as the MAX232 (or variants). Again, the folks at 6502 forums and primer can help with this.
Finally, software. I recommend the excellent EhBASIC. It's easy to get going.
All of these components would make an excellent computer. If this computer came out in 1979 or 1980 and was actually made in quantity (with a reasonable price), it would have ran toe-to-toe with an Atari 800.
I hope this helps. Feel free to find me on the 6502 forums (cbmeeks) if you want to discuss further.