(Only partly my answer, as the important list is a collection of what has been found and noted by others in comments to the question - I thought putting it in an answer would be helpful to others looking)
I know the Atari's FP package used BCD for rather dubious reasons,
I wouldn't call it dubious, but rather obvious. Using BCD based FP will always return the same result as done 'by hand' as there will be no binary artefacts. All roundings due to precision will be the same as by 'classic' methods, thus yielding the 'right' result.
but does anyone know of other examples of basic "operating system" level code on common platforms that used BCD?
I guess "operating system level" in this code covers what usual home computers of that time had in ROM. Right?
Now the list:
According to Tim Locke, systems using BCD-FP were
- Kyotronic 85 (Tandy M100/102, etc.) with a 8085 CPU.
- TI 99/4 (9900CPU)
- MSX (!) on Z80
All of them were 8 byte implementation with 14 digit precision (Atari was 6 byte/10 digit). MSX-BASIC also offered a 4 byte, 6 digit single format. MSX-BASIC is also is quite notable as it's based on Microsoft BASIC 4.5 which had binary FP. So there must have been an explicit request to change to decimal based FP (*1).
In addition Tim mentions the quite marvellous Kyan-Pascal, available for several 6502 machines including Atari 8 Bit, Apple II and Commodore C64. Its real format was also BCD-based with a length of 8 bytes and 13 valid digits (*2). This is especially remarkable as Kyan produced (in comparison) exceptional fast code.
I suspect BCD was added due to the history of using micros in calculator roles
Isn't any FP use a 'calculator' one?
but I am wondering if this is something that was common in the home computer role.
Rather not. It may be safe to assume it's just by intention due to the targets set for development. If the designers / customers thought of producing 'right' results as a design goal, then it was.
This is as well the reason why Microsoft used BCD-FP for Multiplan. After all, it'd be quite odd if some super expensive modern computer with even more expensive software would produce a result that differs from what the old worn-out mechanical calculator says - and is obviously wrong when checking (*3).
Thus, offering some package with 'right' precision was a valid sales argument. I remember many articles in (micro) computer papers back then musing about artefacts and errors introduced into mathematics by usage of binary FP and how to avoid them.
On a side note, Jeremy P mentions that Pet Space Invaders on the Commodore did use BCD for some calculation. Not OS level, but quite interesting.
*1 - I wouldn't be surprised if the FP handling was taken from their COBOL compiler (*4) or Multiplan, as both used BCD as well - BCD is a requirement for COBOL anyway, and it makes a quite sense for a spreadsheet.
*2 - In difference to all other BCD-FP here, the exponent was also stored in BCD - others used binary to increase range (works as it's a fixed point value)
*3 - Never bet on users not finding a hidden error - anyone remember the Pentium Bug?
*4 - Fun fact, MS-COBOL wasn't just some product for customers with old business code, but used a lot within (early) MS products for micros - take Sort as an example :))
SEDto be found. That feels like too superficial a test to roll into an answer though.