Basically I'm wondering what the maximum theoretical resolution is that can be recorded to film using this machine.
There is no resolution defined by dedicated pixels as picture generation (the tube) as well as picture storage (the film) is analogue. Any description in terms of modern digital imaging will miss the capabilities.
Quantisation in vertical is done by lines, which are distinct, but variable in size and distance. Considering that its made for TV signals a vertical resolution between 500 and 600 such lines can be assumed.
In Horizontal there are literally NO pixels at all. Changes in grey level, as well in colour can happen at any level and incline - the later being defined by signal bandwidth.
If you must, you may imagined a line consisting of a Bazillion pixels. A black to white change over a line can reach all values possible by quantum mechanics :) The caveat is that a hard change from black to white does take a certain time - defined by the signal bandwidth, something measured in the analogue world by 'lines', the maximum number of alternating black and white lines that can still be seen as full white and full black. A TV does anywhere between 150 and 300(*1), studio equipment can do up to ~500 lines (remember, that's vertical).
Now, being TV, this bandwidth is different for colour (*2). Bandwidth on colour is usually less than 1/3rd that of B&W, so less then 100 'hard' colour changes over a scanline (*3). This is another point where TV is made to fit human perception, as colour is just a bonus information in rough resolution overlayed to more detailed B&W basic data.
And finally there is of course the film's 'resolution' it is again measured in lines. A cheap ASA 100 colour film will deliver anywhere between 60 to 150 lines/mm. Depending on how the 35 mm film is used, the image area 18 x 24 mm (motion picture) or 24 x 36 mm (still/Leica) thus as worst case good for at least 1000 x 1500 lines (~2000 x 3000 pixel), but usually more like 4000 x 6000 (~8000 x 12000 pixel).
So that's what signal path and media can do. What of that can be reached with this device may be kinda blow.
*1 - If one must compare, double the number is what a digital picture format has to cover (as minimum) to create a picture of the same quality by putting up alternating black and white pixel.
*2 - At least when feed a non RGB signal.
*3 - Kinda like Amiga's HAM mode - in fact due the vary same reason: limited Bandwidth.