I have a strange desire to make an imaginary competitor to the Atari
So there are a couple of places one could build a better 800 and still be entirely within the 1979 (well actually, 1978) tech tree. There's also some things that might not have been possible. So here goes...
The Atari was odd (unique?) in that it did not interleave memory access between the CPU and ANTIC. Instead, it ran both close to the maximum performance of the memory and allowed the ANTIC to pause the CPU.
In contrast, systems like the C64 used down-times in the 6502's clock to grab memory. Now since the memory was generally 2 MHz (although other posts here suggest this was not absolutely the case) that means if you go the interleave route, generally your CPU is going to be 1 MHz, or there won't be enough time left over for the display to grab data while the CPU is off doing its own thing.
The upside to the Atari design is that while the ANTIC had to pause the CPU to grab data, in common display modes, like the 40-column GRAPHICS 0, the CPU only needs to be paused about 35% of the time (IIRC). So overall performance was like a machine running at perhaps 1.25 MHz, and one could recover all 1.8 MHz by turning off the display. It seems like this was a better solution than the interleave option, which meant you were stuck at 1 MHz even with the display off. This of course assumes that the price of the 6502B model wasn't much higher than the A model, otherwise your price goal might get hit.
So two things to consider: was the 6502C model (no, not Sally, the real C) actually available in quantity and at a similar price point? And was memory also available at 4 MHz with the same constraints? It was for the BBC in 1981, but perhaps not 1978/9? If both are "yes" then the most obvious change would be to simply double the speed of the CPU and let the ANTIC pause it longer (in relative terms, same absolute). Since the pausing doesn't happen during VBI and margins (vertical at least) this would result in a significant performance boost (well, duh) with basically no other changes.
Beyond that, the most obvious low-hanging-fruit of that design was the sprites in the GTIA. These were, simply, terrible. Although moving them horizontally was trivial using a timer triggered on the HBI, vertical positioning required moving the sprite map in memory! This not not something the 6502 is known for, and doing so in BASIC... shudders. It also meant every sprite took up 1k of RAM!
Neither is good enough, this is, IMHO, the platform's most significant flaw.
From a programming standpoint, eight sprites of 8-bits width with separate vertical positioning registers would be infinitely preferable. One could expand this, as did the C64, to provide more bits of width or multi-color options. These, however, require more time during the HBI to read, and thus the CPU would spend more time paused. This is a small price to pay. One could further offset that cost by expanding GRACTL to use the upper four bits to turn on or off sets of sprites - say two sprites per bit - and thus allow the programmer to easily tune performance with minor bit-fiddling. You might need a few more color registers, depending on how many colors you allow per sprite and in total, but even that seems trivial.
The rest of the possibilities seem rather more questionable. POKEY's sound was not stellar, but it worked and did so with a very low gate count, and having heard enough C64 sound in my life I'm not convinced the SID was an order of magnitude better. Sadly, stereo TV sound was not yet a thing (even in theory I believe) so that's not something you could have added at that time.
On the hardware side, obviously a RS232 and printer port were absolutely required on the 800, and SIO could easily be a 8P8C (with MOTOR on CLOCK OUT), but overall the platform was pretty solid, especially for the era. Besting it is not trivial, the C64 did, barely, but only after another three years and two Moore's.