but I don't know how to convert the set of hex digits the user inputs into memory into the format required for the FAC
I have a strong feeling that this is not about hex, but binary integers, 16 bit or more.
I know about FRMNUM ($DD67) and PRNTFAC ($ED2E), but I can't find the subroutine that Applesoft uses to take a decimal number and convert it to the six-bit exponent-mantissa format.
You're already on the right track. For example
GIVAYF ($E2F2 *1) converts a 16 bit integer held in A/Y (High/Low) into a floating point number in FAC.
; Converting a 16 bit value to FAC
; IN: -
; OUT: $1234 as FP value in FAC
LDA #$12 ; 16 Bit $1234 in A/Y
JSR GIVAYF ; Convert it to FAC
Of course this is less handy if your numbers are not straight integers, but some format of your own - which brings us to what might be the primary issue:
Where is your data is originated?
Is it from some internal storage, or user input. If it's from user input why not skipping the task of reading a number, turning it into some two part integer and then convert it to FP and use the Applesoft input routines directly?
Which again takes us back to the already mentioned
FRMNUM, which fetches a numeric input from where TXTPTR ($B7) points to and evaluates it into a value in FAC. Of course
FRMNUM is a rather complex beast, doing all sort of things, including expression evaluation (*2) not just for numbers but strings as well.
A more down to the minimum task version would be
FIN ($EC4A), which reads a ASCII string and turns it into FAC. FIN expects again TXTPTR to point to the string to be converted, but as well the first character already read into A. It will use GHRGET to fetch additional characters. CHRGET reports colon (':') or end of line ($00) as terminators (*3). Numbers read are in the format of [sign]integer[.integer[E[sign]integer]]. Anything not fitting that format will terminate, so best use a colon or zero byte. It returns with the number in FAC, or an error. Calling it could look like this:
; Converting a numeric string to FAC
; IN: String in MYBUF
; OUT: FP value in FAC
LDA #<MYBUF ; Lets point to the number we've inputted
JSR CHARGOT ; Get the first byte into A
JSR FIN ; Convert it to FAC
MYBUF DC '123.456E-7',':'.
No, of course not. There's always more: A nice side effect of its use of CHRGET is that it not only moves TXTPTR along, but also ignores spaces, so one may go ahead and convert a series of numbers in sequence if they are separate by terminators (colon or NUL). This could easy be used to hold numbers between operations as text, easy for display and convert them on the fly for usage. Like holding a calculator stack in four fixed length fields.
This needs a way to turn FAC into a number again, which
FOUT ($ED34) conveniently will do - it's essentially the core code of
PRNTFAC, just without outputting the data.
Which brings us once again back to FRMNUM. By arranging numbers as above, but replacing the terminator bytes with applesoft token, it could be feed to FRMNUM, doing all the calculations right away. It might be a bit slower than handling everything as FP as long as possible, but then again, this might not really matter for a user input driven calculator.
Oh, possibilities are endless again :))
On a side note, it would be great if you could not only comment your source (*4), but as well add it not as picture but text. This does not only increase readability, but helps others a lot to reply with code sniplets you might love to get.
*1 - I always loved that address, it's o easy to remember: Enteger to Float too (yeah, I know, silly, can't help)
*2 - Sadly not as convenient as some other BASICs
VAL()functions, as it ecpects the operators to be tokenized. Which of course could be done, essentially turning you'r calculator into a single line program, but that's a different story.
*3 - For the workings of GHRGET/GOT you might want to take a peek at the Apple Assembly Line 1/12 - which is always a good read if you want to get into Assembly on the Apple II. IIRC the whole series has been republished recently as a book. Worth a read.
*4 - I would consider some header describing input/output/task a bare minimum to each and every function. Keep in mind, you want to read your code in 10 years from now as easy as it is today - without a need to dissassemble your own thoughts.