I seem failing to explain this with 8080 compatibility,
But that's exactly the point here - that way the flag byte is fully 8080 compatible. No matter what any software does to these bits. It will make sure that all (automatic) ported 8080 software does not create side effects.
There is no reason not to use the upper 8 bits, since the 8086 flag word is straight 16 bit (read: not twice 8). Using any of the upper 8 can be done freely, as they are unknown to and unreachable by 8080 software.
as the latter has no OF at all, and a software wouldnʼt depend on spare bits.
But 'manually' creation of the flag byte could result in unintended (re-)setting them. Just imagine some software creating an 8080 flag byte, maybe to set various bits to return the result of a subroutine. no matter what default it uses, it would (re-)set OF as well.
Yes, this wouldn't matter in strict 8080 code, but what about the same software get enhanced at some point with 8086 code calling it? It opens all the problems of very nasty incompatibility issues. Keep in mind, this is all about continued use of large chunks of older software.
By keeping the lower 8 bit 100% 8080 compatible all of this is avoided at no cost.
This question opens a rather interesting follow up:
Why you assume it should have been moved into the lower 8 bit?
And what good would it have brought?