How can I detect signed integer overflow on the Game Boy?

Question

Unlike a normal Z80, the Game Boy has no CALL pe/po, JP pe/po or RET pe/po instructions. I've been trying to figure out how to detect integer overflow on the Game Boy. (It doesn't have those instructions for the sign flag either but that can be done with BIT 7.)

This is what I've been trying to come up with but it's an absolute mess. After doing a few calculations on paper, such as &81 - &02, &7F-&01, etc. I came to the (possibly faulty) conclusion that overflow is equivalent to sign change xor Carry Flag. It almost works but there seems to be a problem. If A and B have opposite signs but they subtract to less than 7F the compare is wrong. The second picture shows an incorrect result.

Like I said, the source code is spaghetti, but I couldn't come up with a better way.

;compares A to B signed.
    bit 7,a        ;returns nz if A is negative, z if positive.
    call GetZeroFlag  ;stores this result 
    sub b
    bit 7,a
    push af
        call TestForSignChange
        jr z,NoSignChange
    pop af
    ;at this point, if carry is clear, overflow occurred.
    jr c,NoOverflow
    jr OverflowOccurred
NoSignChange:
    ;at this point, if carry is set, overflow occurred.
    pop af
    jr nc,NoOverflow
OverflowOccurred:
    ;test bit 7 of A again. If A is positive, A < B. If A is negative, A >= B
    bit 7,a
    jr nz, GreaterThanSigned
LessThanSigned:
    scf         ;set carry if less than, just like an unsigned compare does.
    ret
GreaterThanSigned:
    or a        ;clear carry if greater than or equal, just like an unsigned compare does.
    ret
NoOverflow:
    ;Test bit 7 of A again. If A is positive, A >= B. If A is negative, A < B.
    bit 7,a
    jr nz, GreaterThanSigned
    ;I don't understand why this isn't "JR Z, GreaterThanSigned"
    jr LessThanSigned
    
    
GetZeroFlag:
    push af
        push bc
            push af
            pop bc           ;get the flags into the C register
            ld a,c
            and %01000000    ;zero flag is where the 1 is.
            ld (tempflags),a ;store the result in ram
        pop bc
    pop af
    ret
    
TestForSignChange:
;compares bit 7 of A before and after the subtraction.
;If the value of bit 7 of A changed, return nz, otherwise return z.
    push bc
        push af
        pop bc
        ld a,c
        and %01000000    ;ignore all but the zero flag
        ld c,a           ;store the current state of the zero flag in C
        ld a,(tempflags) ;get the state of the zero flag before the subtraction in A
        cp c             ;compare it to the state of the zero flag after the subtraction.
    pop bc
    ret

Answer 1

You can determine overflow for addition and subtraction of two signed bytes A and B as follows. First flip bit 7 of both A and B. This will not affect the result of (8 bit) addition or subtraction, nor of the sign flag, but it will affect the carry.

Then for addition compute A+B. The overflow flag will be (carry == sign). For subtraction compute A-B. The overflow flag will be (carry != sign).

Example for subtraction:

; subtract signed byte in addrb from signed byte in addra
; result in B, overflow in carry flag.

; flip bit 7 of addrb and store in B
ld A, (addrb)
xor 80h
ld B, A

; addra in A and flip bit 7
ld A, (addra)
xor 80h

; subtract B to compute (addra) - (addrb). move result to B
sub B
lb B, A

; test overflow: carry != sign.  carry flag holds result.
rla
adc A, 0
rra

A signed compare (addra) < (addrb) is much simpler:

; compare signed byte in addra with signed byte in addrb.
; carry set if and only if (addra) < (addrb).
ld A, (addrb)
xor 80h
ld B, A
ld A, (addra)
xor 80h
cp B

Answer 2

Overflow occurs on addition when two numbers with the same sign add up to a number with a different sign.

It occurs on subtraction when two numbers with different signs produce a result with the same sign as the second number.

So the general rule is: a number begins on one side of zero; the add or subtract should move it further away from zero; somehow it ends up on the other side.

So for addition of b and c, I guess something like:

; Compute result, store in h.
ld a, b
add c
ld h, a

; Test for the same sign in the two original numbers, store in l.
ld a, b
xor c
cpl
ld l, a

; Test for a sign difference with the result.
ld a, b
xor h

; Check the two sign differences.
and l
and $80

jp z, no_overfliow

That’s off the top of my head, untested and therefore possibly error-ridden, assuming all those instructions exist on the Game Boy’s moderately unique instruction set.

Alternatively, you could break it down into a series of sign tests; e.g. for subtraction the only possible causes of overflow are:

1. a positive number is subtracted from a negative number, and the result is positive; or
2. a negative number is subtracted from a positive number, and the result is negative.