It wouldn't have been outside the realm of consideration at the time, but your guys would face threefold problems in terms of cost and practicality:
As others have stated, the cost of DACs and other high speed circuitry with which to build the image out of memory, especially if you want to combine high horizontal resolution AND high vertical resolution AND a decent scan rate. This is not insurmountable, real-world manufacturers found ways around the issue, but usually with methods that caused their own limitations on video modes.
Ditto, cost and availability of high frequency scanning monitors. The vast majority of everything, even in applications wholly divorced from broadcast video like DEC's terminals, was just based off regular TV innards (either colour or b/w) as they were cheap and plentiful. Which put a hard limit on usable vertical resolution of around 200~240 lines progressive and 400~480 interlaced (for 60hz; 50hz rises to 256~288 and 512~576), and why you see those numbers so often. Bear in mind that 60hz progressive was only really tolerated on medium-persistence-phospor colour monitors because it was the least flickery mode possible on common equipment; 50p was sort of bearable but 50/60hz mode switchers were common on 16-bit computers for more than just the stretched out image height. 60hz interlace was a flickery mess, and 50hz even worse. IBM's MDA was something of a watershed in its introduction of a non-TV-compatible 18khz scan rate and general use of radar-like long-persistence phosphor in the image tubes to make its 50hz scan appear more stable (something they later did with the original XGA, which had an 87hz scan interlaced to 43.5hz effective and would have otherwise been unusable). So if you want to do 640x480 you're going to either have to put up with interlace flicker (how many people used the interlace mode of their Amiga long-term without a flicker fixer and high-frequency monitor? not many...), or have an expensive HF (and probably dual-frequency, which makes it even pricier) monitor, or at least specify long-persistence phosphors for the inside of the tube (which then have their own problem of unavoidable afterglow and "traces" across the screen with any moving image).
MEMORY. RAM was expensive in 1980. 32kb wasn't simply a small indulgence, a luxury paid for by those who wanted a small upgrade to their computer. It was one of if not the most expensive component out of the whole machine, and a prime reason why, at the time, if you had a 16k computer you were quite the flash Harry. Atari, Apple and Commodore's 8-bit offerings offered typically 4 or 8k, the Sinclair ZX80 had just launched with ONE kb onboard (which the ZX81 would then replicate a year later, but with the option of a 16k upgrade cartridge that cost essentially as much as the entire main computer), and 16k wasn't really established as the bare minimum until '81-'82 when that became the baseline for the IBM PC, Sinclair Spectrum, and various other 8-bits... Commodore then causing a riot with the C64 at the end of '82, which they could only afford to do both because the price of memory had taken a severe tumble over the previous 18 months, and also because they owned a chip manufacturer (MOS tech). Giving over a good 32kb to video (which is the more likely thing that anyone trying to make a display of the class you describe would do, instead of 37.5kb - so, the then-common 320x200 and 640x400, not x240 and x480, though there's an alternative in the guise of the wide-pixeled 256x256 and 512x512 modes) would have been unheard of. IBM were pushing the boat out somewhat by cramming 16k onto their CGA board (which seemed a sensible alternative to installing 32k into the PC and then having half of it eaten by video), but their budget offering was the MDA with only 4k.
I mean, it would still have been doable. As others have mentioned, there are existing "graphics workstations" (and more commonly, graphics terminals ) with that kind of definition and memory space. But they were expensive, specialist things. Sort of like the IBM PGC in '82, which was a full-on graphics terminal crammed into a triple-width expansion card, complete with a humongous 320kb RAM... and a $5000+ price tag just for the graphics adapter, never mind the rest of the computer (oh, and the external expansion bay the card had to go in).
Your guys basically are caught in an odd position. On the one hand, the thing you're imagining them "inventing" already existed. On the other, it only exists in the guise of ruinously expensive pieces of niche, technical, corporate (or more likely institutional) equipment. But the price tags on them aren't really all that gougey... a surprisingly large slice of the cost went directly into a) the memory buffers needed to support that large a bitmap (which is why vector based and high-resolution text-mode terminals were about as common - they could give as good a display or better, for the tasks to which they were optimised, as a bitmap model with probably 8~10x the RAM), b) the fancy high-resolution and/or long persistence display needed to display that many raster lines at a high enough speed to avoid flickering, and c) the uncommonly high frequency, wide data path electronics needed to connect the other two components.
...
But, hey, let's think about how the circle can be squared a little.
First up, choice of modes. You've essentially preempted the Atari ST, AT&T 6300, unexpanded EGA card, etc. Or a double RAM, double frequency CGA. But added extra scan lines to make it more VGA (or MDA)-like.
Well, let's say, first step... ditch the extra lines. You don't need that extra 20% height to impress anyone, and if you do, you can compromise on horizontal resolution a little without going full VIC20. So we have modes along the lines of 320x200/640x200 (or 320x400)/640x400 (may as well have the medium-resolution, it can be useful), 288x224/576x224 (or 288x448)/576x448, or 256x256/512x256/512x512, with 4 bits, 2 bits, and 1 bit per pixel, fitting neatly within 32kb in all nine (eleven?) cases. That makes a whole load of things a hell of a lot easier from a 1980 electronics perspective, particularly limiting how many address lines are needed, and keeping a sensible lid on the capacity and number of RAM chips and on the required bandwidth (which also has implications for how fast the memory has to be, and/or how wide a data path you need to pull data out of it quickly).
Next up, the tube itself, or more accurately its drive circuitry. The faster you go, the better quality parts you need in order to deal with maintaining sharp, steady deflection yoke voltage waveforms and stronger drive voltages (including for the electron gun as it needs to be more intense due to moving faster side to side), as well as dealing with the usual effect of higher frequencies in electronics burning more power due to inductive filtering. If we were after full VGA (or PGC) resolution, and that same 60hz refresh rate, we'd need double the normal scan frequency, or about 31khz instead of 15-1/2. And there'd also be the issue of supporting two entirely different scan rates for the high and low line counts, as well as a pretty high pixel clock (25mHz in the VGA and PGC).
However, the limiting of the memory size and thus pixel dimensions has a nice side effect; we don't need to go all the way up to 31k (or the unusually high 36k of the ST's high-resolution which was possibly one reason that the original NEC Multisync tapped out there). Nor will we have to interlace, or have quite as high a pixel clock. We will need to exceed the 18kHz (and 16.3MHz) of MDA, though, and even the 22kHz of both EGA (1984 I think?) and the original Mac and Lisa (also all ~16MHz), because we need a higher line count and/or refresh rate than those variously supported. But, there's a middle ground that is exhibited by a now largely forgotten brace of early to mid-80s computers, especially those hailing from Japan (which needed high bitmap resolution to show Kanji alongside graphics) and 50hz areas (where a slightly less flickery image was desired, in combination with even more scan lines) as well as a lot of arcade machines and the Mac LC... the "25kHz" monitor (actually anywhere from 24.5 to 26.5k IRL, but the name stuck). 1980 is maybe a little early for the introduction of an affordable example of the breed, but, heck, your guys can be the ones to introduce the idea maybe.
25kHz horizontal (nominal...) is just enough to squeeze in 400 active lines at 60hz, with the remainder between that and the 24kHz you might think it'd require disappearing into a fairly minimal blanking and sync/flyback area. You can also get away with a somewhat reduced pixel clock, in the region of 19 to 24MHz depending on how much blanking your horizontal sync circuitry requires. Realistically, though, it's a 21MHz minimum for safety; it might work to go for 21.48M, which is 6x the NTSC colourburst and so part of a fairly common and widely co-opted line of clock crystals, but maybe a cuspier compromise - on the edge of tolerance for everything involved, just from different directions depending on part - to use 20MHz, which would still give as much blanking as VGA and the Amiga or ST on full overscan, and bring it within the spec of a greater range of existing hardware including the very fastest Motorola CRTCs, if used with a set of quick 8-bit shift registers and slightly over-driving some then-cutting-edge 250ns memory chips.
Thing is that then also causes problems if we want to use the lower resolutions with more colours; the lines have to be doubled up (by simply reading the same memory block twice) to preserve the sync rate. Which as it also doubles the necessary pixel clock at the same Hrez defeats the general tactic of having a steady data rate but dividing it up differently, trading off a lower pixel clock for more colour. For example in the CGA and more particularly PCjr we have 14.3MHz for 640x200 in 1bpp, 7.2MHz for 320x200 in 2bpp, and later 3.6MHz for 160x200 in 4bpp. For the ST it's similar with 32/16/8MHz and 640x400/640x200/320x200, with the different syncs required being covered by the requirement to use either two wholly separate monitors, or a multisync (or at least a dual sync, which is what the EGA and MCGA do). Other machines that have varying vertical resolutions but only one hsync rate suffer the same problem, which is why the VGA only has two colour depths (and it depends solely on the Hrez - 4bpp for 640, 8bpp for 320), and the AT&T exhibits similar just at one quarter the depth unless you add an expander card that progressively widens the effective data bus width with every additional 32k bank you install.
So that would make it look like we either need to rearrange the memory to form a wider bus (using a larger number of lower density chips, though our practical minimum is 16x (single-bit) 2kb 4116s anyway; maybe we could use 32x 4108s, which were their half-functional low-cost cousins?), to provide a doubled 320x200 or raw 320x400 in 4bpp, as well as 640x200/400 in 2bpp instead of mono, or that we'd have to provide no better than 4 colours, or opt for an expensive dual-frequency monitor (which, hey, worked for IBM... several years later).
An alternative does exist, however, in the form of installing a line buffer for the 200-line modes. A small 256-byte SRAM (or 512-nibble; preferably 160 or 192-byte/320 or 384-nibble if they exist... or 2x 128-byte in order to give 16-bit transfer, which then could even just be surplus Atari 2600 RIOT chips), either dual ported or double speed with circuitry providing interleaved R/W access, could work as a very high speed FIFO in a chase fashion, with data being written into it at half the speed it's read out on an approx half line delay (so data for the next line starts being written into the buffer just after the second repeat of the previous one starts being read out, and data for the second half finishes just before the first repeat of the next finishes, with the writes being jittered to start slightly earlier or later depending on whether it's an even or odd line). No idea how much it would cost in 1980, but I do know a lot of early micros used SRAM for their main memory instead of DRAM, so adding a quarter K of it shouldn't be prohibitive, and as far as the main data reading circuitry is concerned little has changed... its output is just being switched down a different path on the way to the actual output stage (essentially either going through the buffer, or bypassing it; similarly the output stage receives its data from a different source but one that's still addressed in much the same manner), and the counter that governs when each line starts being read out has its "2" bit flipped on or off dependent on whether the line counter's "1" bit is active.
This is something I've been thinking of as an addition to an imaginary reworking of the Atari STe, and I think the Amiga included something similar in its own video chip (though only buffering a single line, building it up from a combination of bit planes, sprites and copper commands, and using the small dead area where the system was involved with memory refreshing on each line to start reading out the prepared line just before the next one started being built)... though that was itself a somewhat pricey machine in 1985, and IBM never seemed to bother with the idea even in the VGA, even though that could have made for a 640x200 8-bit and a 320x200 direct-colour 16-bit mode as early as 1987, so I do wonder if there's some part of the construction that makes it impractical for 1980.
Anyway ... assuming that CAN be done, and we now have a base system that has sufficient memory, bandwidth, and buffering to be able to squirt data towards a monitor interface stage at a rate of approx 20Mbit/s (400 line unbuffered, 1 or 2 bits in parallel for an effective 20 or 10Mpixel/s switching rate) or even 40Mbit/s (200 line buffered and repeated, 2 or 4 bits in parallel, but the same 20 or 10Mpixel/s final rate). Where are we going to find a 20, or even 10MHz DAC from in 1980? At the basic level, converting binary data into analogue signal levels is a fairly simple affair, you just need a bunch of resistors in a ladder, but you have to make sure they're all switched at exactly the same time and with enough output power to create a relatively sharp edge to each pixel instead of being blurred by inductive filtering inherent to whole run of wiring between the DAC and the electron guns in the monitor (the same thing that puts an upper limit of about 350MHz on even the best VGA cables, and less than 100 on cheap ones or particularly long runs that don't include a booster & resynchroniser box, with everything higher rez than that suffering smearing and ghosting). At the time, there were precious few applications that demanded such a thing and the quality just wasn't there; even for broadcast video, your upper luminance bandwidth might be around 7 to 9MHz (in wholly analogue systems, equivalent to 14-18MHz digital... which itself was still barely managing half that frequency in the main), and colour information languished around the 4MHz realm. Plus with TV images you could get away with, if indeed you didn't straight up prefer, smoothly varying and smudged-together pixels, to give a more natural appearance to images. Great for photos or live video, not so good for sharp text or diagrams. Like the comparison between JPEG/MPEG and GIF.
But, again, we have a fairly simple answer, which was adopted by pretty much every IRL manufacturer of high-ish-end display equipment, including even the non-TV outputs of computers like the BBC Micro and, again, the IBM PC (including with CGA): TTL signalling, in Mono, RGB, RGBI, or RrGgBb mode. That is, we literally just send binary data out of the port, and let the monitor deal with it. With monochrome and straight 8-colour RGB (as used by the BBC, NEC PC88, and a few others), there's not even really anything to decode; the signal received is a simple gate saying "turn the beam on" or "turn the beam off". The signal received is interpreted by a flip-flop transistor gate inside the monitor as being either above a certain voltage threshold ("1", so "on") or below it ("0", so "off") and triggers the appropriate electron gun(s) accordingly, making for a very sharp image even with a relatively fuzzy input signal. So long as it's able to reliably get the output from one state to the other within about half a pixel time, and without too much variation on how long it takes to move from one level to the other, it'll appear on screen with sharp and acceptably straight (in the vertical dimension; the raster beam takes care of the horizontal) edges from one line to the next. Of course, this rather restricts the range of colours we can produce. A monochrome monitor is exactly that - a single beam that's either on or off. RGB colour only offers 8 colours including black and white. No greyscales, no intermediates.
Of course that wasn't really deemed acceptable so a little extra processing was off-boarded to the monitor's drive circuitry, in the form of the "intensity" bit, making most monochrome systems actually 2-bit, and most colour ones 4-bit. It wasn't exactly what you'd call an artist's palette (usually mono monitors only had three levels of brightness; 00 was off, 10 was "on", 11 was "even more on" with the intensity bit causing the beam's drive voltage to be set a bit higher, and 01 caused a cursor-speed blinking effect between normal-voltage on and off... colour used a variation of this where turning intensity on just added about 50% of "normal" voltage to all three guns at once, so you had a bunch of slightly dim primary colours joined by a similar number of slightly desaturated bright pastels, as well as there being two shades of grey below white... IBM opting for a variation on this where non-intensified yellow also had its green channel attenuated by 50% to make brown), but it did the job in an age where any colour at all was pretty good going (and CGA's output was still a little more versatile than, e.g. the analogue output of a Sinclair Spectrum, and it sort-of still worked over composite video even though the actual clarity with colour encoding turned on was rather less than 320 pixels). And you could even get away without needing a palette register, which would either demand extra silicon die area in your custom display chip, or an additional small SRAM on the board; this is the reason why CGA's palettes are so weird - they're the best combinations the designers were able to come up with within the limitation of directly feeding bits from memory into the output drivers; essentially, in low rez, the memory bits go straight into the R and G lines, with B and I being controlled by a combination of the palette choice (4 officially, but a fifth could be glitched in through a certain setting of other registers meant more for use with monochrome composite displays) and a single 4-bit-wide discrete-logic register that controlled the background colour (i.e. what appears when both memory bits are 0). In that way, they avoided having to find space for another 15 or even just 3 4-bit entries. Presumably if they'd had just a little more time, budget, or board space, they'd have included a mode which divided the clock down still further and put either 3 or 4 bits directly onto the RGB or RGBI lines to give a freer choice of colour at the expense of resolution on the TTL monitors (something that can be done over composite, but actually uses a tweaked version of the hi-rez mono mode)...
So if you don't mind losing the ability to choose from 64 colours, you can achieve 16-colour output rather more simply and elegantly by just sending the 4bpp data directly down four wires in the monitor plug, and maybe instead allowing the choice of any 4 of those 16 in the medium resolution modes (particularly useful to have 4 greyscales, 3 plus a highlight colour, or B/W plus 2 highlights, which is what a lot of interfaces on the ST and Amiga did in their 640x200 4c modes) with a relatively simple 4x4 bit SRAM or other register. Monochrome would most likely use a fifth wire, or re-use one (or two, in medrez or in a hirez text mode - possible, as demonstrated with the MDA and many older terminals, because pulling a character out of ROM at the same time as setting a few attribute bits for the whole cell is more practical than trying to do the whole thing as a bitmap) of the existing ones alongside a simpler signal wire that sets the monitor into mono or colour mode. Actual monochrome monitors of course could follow AT&T's example of, instead of operating exclusively in on/off (plus optional intensity) mode, being able to interpret incoming colour data into 4, 8 or 16 greyscales - which still needs a DAC tied to the electron gun, but it's a) only working on one channel, and b) has a much more direct line to the driven load, so can be engineered specifically to work well with it and doesn't have to deal with all the extra impedence of the wires between it and the video card.
If the 64-colour thing is a absolute deal-breaker for you, then you can go with the EGA setup; two wires per colour channel, essentially making each one into a separate monochrome TTL system, but in this case each bit contributes a certain amount of voltage to the connected gun (ie we have three very simple 2-bit DACs in the monitor, splitting the difference between the true monochrome or the RGBI systems which were on/off with a booster, and the multilevel monochrome reinterpretation of colour input), one twice that of the other, so it's either off, 33%, 67% or 100% - though each could still be quite discriminate over its own input and still produce a pretty sharp pixel without much blurring due to the cable. It's about the practical limit for TTL signalling, so for 1980 it'd be on the bleeding edge and you'd be unlikely to do better without brewing up your own high-speed, high-output DACs to connect to an analogue RGB monitor. You'd have a small advantage over EGA at least; whilst you'd still need a 16x6 register (or a 16x8 SRAM with two bits unused for each colour), and 6 lines in the cable (all being simple, robust TTL), you wouldn't need the additional circuitry the monitor contained to not only switch frequencies, but also change its colour mode according to said frequency... each EGA monitor contained both RrGgBb decoding circuitry, but also an RGBI stage, which emulated the CGA behaviour by putting certain bit patterns onto the RrGgBb part. For a connected monochrome monitor, we'd likely do away with the 64-colour palette and just use 4 (or 2, or 1) of the wires as straight intensity. Picking 16 greyscales out of 64 isn't the most useful thing in the world unless you really like doing smooth fades between different screens in your software, and even with only 16 greys you can produce remarkably photographic images with a bit of dithering, same as you can do the same for colour with only 4096 colours (and do an OK job with 64, though being limited to 16 out of those, or just 4 greys, does hamper you somewhat).
The last unspoken thing is actually having enough storage and processing power in your machine to deal with those high colour, high rez displays. It didn't really become practical to move beyond the CGA or even C64 (which is only 8kb per screen) level until proper 16-bit processors (like 7~8MHz 68000 or 8086, not the 5MHz 8088 of the PC and certainly not the 1-2MHz 6502s or 3-4MHz Z80s that would be your 1980 targets), large amounts of system RAM (at least 128k, preferably 256 or 512, as per the Mac and original design layouts of the Amiga and ST, and the base level of most XTs and juiced-up PC/XT clones), and guaranteed floppy disc availability for storage (again, that's about 1982/83 onwards, with the arrival of the XT, as Cassette was still an option for the PC). You just need too much processor time, and too much backing store (and bandwidth of such) plus temporary memory space to hold and manipulate the graphics. A halfway house could be stringy floppies (i.e. high speed medium capacity endless loop tapes) and putting twice as much memory on the video system itself to allow multiple pages, or just allowing a reduction in colour depth in the lower resolutions to allow double buffering within the 32k - or, most particularly, building in a character-based (ie, text) mode, which for any application that's mainly text rather than graphics focussed can offer a huge saving in terms of memory and bandwidth (e.g. the MDA has a quarter the memory of CGA, but it can produce the exact same amount of text in terms of characters x rows, and does so with higher quality, smoother looking lettering as the character cells are 9x14 pixels instead of 8x8, making for a total of approx twice as many pixels painted to the screen; thus, about an 8x saving)... you're likely going to want to include a character ROM of some kind anyway so it doesn't have to be loaded from disc into RAM each time, and the output will almost certainly be mediated by some kind of CRTC (which can do graphics, but would have almost certainly have been mainly engineered for text mode output), so the majority of the work will have been done for you, so you just need to set up the alternative way of addressing and translating memory.
But on the whole, what we're looking at here is something that would be installed into a minicomputer or a professional/institutional rackmount micro, such as the DEC PDP series, rather than something that's going to go into the home, or even into a premium professional desktop machine. All up I'd be surprised if you could put the entire system together for less than $10,000 in 1980 dollars, which would get you a pretty nice car at the time. You'd have to have been extremely committed to high quality computer graphics to consider buying such a thing.
I mean, maybe that's your protagonists' intended market. But like I said, if that's the case, there were already competing systems with similar specifications out there. They'd not have been so much inventing anything, instead they'd have had to try and undercut the existing big players with some kind of innovation that meant they could do everything for a much lower price. Like, less Apple or Intel, more Commodore/MOS...
