I would think any answer to this question will be, as lvd already noted, quite tied to the use case looked upon if not highly opinionated anyway. So I would like to underline the questions made with some general facts(*1), anyone can use to check the opinionated parts.
So the question is how similar were the two competing microprocessors?
Well, both are
- Accumulator based designs (like most others of the time),
- Offer a rich range of addressing modes (including indexing),
- Use memory mapped I/O,
- Use similar clock per instruction,
- Operate on a similar bus design,
- Using a similar memory access cycle,
- Available in similar speed grades
Was the performance the same?
CPU's with the same basic design, similar memory access cycle and similar cycles per instruction usually end at the same over all speed range. No matter if 6500, 6800 or others like the 2650 (*2).
How [dis]similar were the instruction sets?
Usually an Assembly programmer of one would be able to get familiar with the other within hours, as they are both memory dependent accumulator machines. Including the same basic instruction set using the same or quite close mnemonics.
The only notable difference are complex addressing modes which are handled different - and a few 16 bit instructions the 6800 offered. Nothing really hard to come by.
Where the 6800 is more of a straight school book CPU, the 6500 shows traits of optimization toward embedded use. Thus being less great to code for more dynamic tasks. But again, not a major difference for real world usage.
As well, could a 6501 be dropped into a 6800 socket and work with an existing motherboard despite not being binary compatible?
That was what it was meant for: a 100% compatible drop in replacement due using the same bus interface - only different software.
This is further supported by both using the same basic memory layout of RAM (and IO) at bottom and ROM at top of the address space. So unless the 'motherboard' is was a very special (and very early) embedded design (*3), chances are good. Even more if we, as motherboard suggests, talk about a more generic, PC like design.
*1 - It does not make any sense at all to compare on an instruction base or instruction sequences, as any noted difference will depend on the task to be performed and the selection used. Like having a multiply instruction may be yield superior local performance but not really matter over all.
*2 - Not to mention that for most part CPU speed was (and still is) memory dependent, so any CPU using a certain speed grade of memory will yield broadly the same real world performance, including complete different architectures - that is unless a lot of additional logic is thrown at, allowing very complex instructions.
*3 - Something one would have used an 6801 or 6802 anyway. They use a different pinout/bus interface, with no direct replacement offered by MOS.