John Carmack almost certainly was the first to use the hardware scrolling capabilities of the EGA specifically, together with efficient tile and sprite drawing and erasing algorithms to create a slick, full-screen scrolling 16-colour console-like game engine on the IBM PC. "Adaptive Tile Refresh" was one part of this. He did so through experimentation, reading documentation and learning about the EGA and drawing/masking/rotating techniques through Michael Abrash's articles online, in Dr. Dobbs Journal and in the book Power Graphics Programming.
(And note that Mega Man (1990) for DOS came out very close (I believe 4 color CGA), and other earlier DOS games like License to Kill (1989) and Spy Hunter (1984) had scrolling as well - this wasn't in all directions and probably a full-screen redraw. I've also found a Quora post attributed to the developer of the DOS port of Golden Axe that says that game predates Keen.)
--
No, Carmack wasn't the first person ever to implement a system where tiles upon a moving background are redrawn row-by-row, or to devise algorithms for efficiently erasing and drawing blocks of tiles or sprites based on their motion (or lack of).
I've opened another question to discuss what parts of the Keen engine 'Adaptive tile refresh' refers to - I believe it's an optimization available when using hardware scrolling on a software-drawn tilemap to reduce the frequency and number of tiles that need to be redrawn as the screen scrolls. Looking at Chocolate Keen (reverse engineered from a disassembly of Keen 1-3) sprite handling also falls under ATR - when a sprite is drawn, it invalidates all tiles it overlaps on the tilemap on that display page, which are then redrawn on the next tilemap redraw (that is, refresh) pass.
As you stated in your question, drawing only incoming tiles on the edge of a scrolling tilemap would be a well-known technique for arcades or consoles where hardware scrolling was available, regardless of whether they used a bitmapped/bitplaned display or a tilemap. The Sega Master System and Intellivision (see Advanced Dungeons & Dragons) both have tilemap layers that are exactly the width of the displayed screen, so there's no other way to implement scrolling other than to draw the oncoming edge into the one-tile-wide region that becomes available by setting a flag to mask away one column of the screen with the border colour.
The ZX Spectrum has no hardware scrolling or sprites, only a 1-bit unscrollable display, chunked up in to 8x8 regions colored by 'attribute' values. Less-experienced ZX Spectrum programmers writing arcade ports such as Space Invaders-clones would clear the entire screen and draw the sprites, but more-experienced programmers would erase and redraw any moving elements or even redraw only their edges, or use XOR to draw-undraw elements if the resulting visual garbage when sprites overlapped other sprites or the background was acceptable. (XOR was known - mentioned in this 1976/8 patent, ZX Spectrum BASIC's drawing primitives expose XOR drawing mode as the statement OVER n.)
In short, people were familiar with similar tile and sprite drawing techniques or able to come up with their own algorithms suited to the target platform all through the 80s.
You say the PC had 'no specialized graphics controllers' but in fact it's the EGA card's PEL Panning register that made Keen's fine scrolling possible. The CGA has a configurable start address value which allows page flipping, vertical motion and coarse horizontal motion, but the EGA has an additional fine-panning register allowing motion within a memory byte/word - together these allow panning to any single pixel in the video RAM. The same registers are used in VGA Mode X (and a similar mechanism is used for scrolling on Amiga chipsets).
Carmack's innovation was to devise 8086 real mode algorithms like the ones in use on other systems for tiles and sprites, and having the engine be stable and fast across multiple competing first- and third-party EGA card implementations and monitors*. To get a full engine in place would need lots of experimentation and iteration. Adaptive Tile Refresh would be one useful part of this.
EGA programming is obtuse: bitplane selection, barrel shifting - lots of fields that can be set to ensure bitplaned graphics land in the correct 'slot' within a byte, etc. Descriptions of adaptive tile refresh also note that the drawing buffer is wider than the displayed region to allow for an off-screen region to scroll into (similar to the Sega example I mentioned), this means Carmack would have to also set the register containing the address increment per scanline (also known as the 'stride') to a non-standard value, and so on.
You asked Did Carmack figure all of this out on his own?
In the foreword to Michael Abrash's Graphics Programming Black Book, Carmack writes:
I learned the dark secrets of the EGA video controller [in Abrash's Dr. Dobbs articles], and developed a few neat tricks of my own. Some of those tricks became the basis for the Commander Keen series of games, which launched id Software.
Romero:
[Power Graphics Programming] is the book I gave @ID_AA_Carmack to read before he started on Catacomb PC in 1990. This led to Commander Keen scrolling breakthrough.
Asked on Twitter 'Did you know any of the antecedents from other machines?'
Carmack:
no, I didn't. Hard for people today to appreciate how scarce info was back then; Foley&Van Damn[sic] was no help for low level hacks
(Foley & Van Dam refers to the textbook 'Fundamentals of Interactive Computer Graphics', 1982 )
I had hacking/phreaking files from BBS, but never any good "real" programming info
So although tile refresh strategies for scrolling games on other systems existed previously, Carmack believes he came up with the complete strategy used in Keen in isolation.
*On VOGONS, you can find threads of people studying patching all the Keen games still, as certain titles misbehave in certain ways on certain cards even within a trilogy where one might expect the games to be the same.
I'm guessing the OP was an Amiga owner (like me) familiar with Ballistix and Shadow of the Beast (1989, both before Keen). Both these games have full-colour 50Hz gameplay with scrolling and (apparent) sprites. From the perspective of the 'home micros' (ZX Spectrum, C64, Amiga, Atari ST, etc.) as opposed to the business-focused IBM PC, the confusion as to what the fuss is about is understandable. In the PC world in 1989, EGA scrolling was 'cutting edge' stuff, though Apogee's claim that Keen "Makes an IBM outshine an Amiga" with its chunky world, beepy sounds and gaudy colours is a little bit ambitious. :)
