In the first case, I do not encounter any problem in the rest of my program.
That's because you are calling into ROM routines which set up everything for you.
In the second case I notice significant display problems, in particular if the display was in 80 columns before, and if the COUT routine is used after.
Well, you only switched the hardware. You didn't clear the screen, you didn't tell the COUT routine that it's now running in 80 column mode, etc.
What is the explanation of these different behaviors?
See above: you are executing a routine in ROM which does all the extra work for you, vs. you are switching the display mode without doing anything else.
How to use properly 80coloff switch in assembly language ?
"Properly" depends on the circumstances. If you use COUT, to do it "properly", you need to update the values in the zero-page COUT depends on. You also need to do housekeeping like clearing the screen. If you control the framebuffer directly, you don't have to do that.
What is the most reliable procedure to go from 80 to 40 columns (and vice versa) in assembly language ?
The most reliable is your variant one (as you've discovered yourself). Alternatively, call directly into the ROM routine that sets this up (
$11 just is a special character which is detected and triggers a jump to this part). Alternatively, read the routine in the ROM disassembly, and do all the work yourself.
The first variant has the advantage that it doesn't depend on the ROM layout.