How exactly does one send a new routine to the drive and execute it?
Usually by executing the Memory-Write command twice followed by UserN command, as described in chapter 8 of the floppy manual (*1).
Once to place the routine to be used and then to setup its address as user function (or whatever it is supposed to replace).
A suitable function to do so may look like this:
1000 REM Write to Floppy RAM
1010 REM AD -> Address to be written to
1020 REM DA$ -> Data (code) to be stored
1030 OPEN 15,8,15 : REM Open command channel
1040 PRINT#15,"M-W";CHR$(AD);CHR$(AD/256);CHR$(LEN(DA$)) : REM Send Memory Write command
1050 PRINT#15,DA$ : REM Send Data (code)
1060 CLOSE 15
1070 Return
Alternative, a Memory-Execute can be inserted, to execute the code right away:
1055 PRINT#15,"M-E";CHR$(AD);CHR$(AD/256)
(The code shown is made for C64/1541/DOS 2.6 and from memory, thus no warranty on correctness. Nonetheless the instructions used should work on next to all Commodore computers/drives. Of course you may want to look up the specific address layout for the drive you're using, as well as more advanced command if using a newer BASIC.)
The function can move any data to any place (*2).
If it's about extending DOS, then pointers within DOS needs to be updated. This can be done as well with the above function (without the M-E line). Of course addresses to be used for program, and how to patch pointers and where they are located, depends on the drive in use and the DOS installed. So you may want to take a look at the corresponding DOS disassemblies.
Going into all detail of Commodore DOS and possible usage is way too broad for this (or any single question), so here only a few hints. While they work on most drives, this will use the 1541 as base.
Using a file buffer is a great place to store a routine (best one were no other file operation is interfering). 1541 got 5 buffers, of 256 bytes each, located at $300.
If it's about making simple routines for parallel processing, then the User-Functions U3..U8 are made for that (*3). Their entry points are located at $500+((N-3)*3) (*4) and can be called by simply sending the User command ("Un") as often as desired.
Further parameters can be put in the command string - just keep it short as the command buffer is just 66 (?) bytes.
Since the default pointer for the User commands is in buffer 2, writing your code to $500 may set up "Un"-entry point and code in one operation.
In fact, all of that can be done in one operation if the code resides on floppy, Block-Execute ("B-F"), which loads a floppy block into a buffer and executes right there. Of course the block has to be written to disk first.
A more sophisticated way is to use explicit Block-Read operations to 'page in' code blocks, from disk, to be executed by User function (or Memory-Execute). Now even complex module/overlay solutions are possible.
The most simple way to get information back it reading some memory (like the very same buffer the code has been written to or parts thereof) with Memory Read ("M-R").
Otherwise writing to the error buffer can be used to return results as error messages
Or put it into file buffer of a file that has been opened previous and just read it. This may need a bit of fiddling with buffer internals, but may be the most rewarding way.
*1 - Isn't it funny how much pseudo secrets are running around this when everything needed is described in the manual?
*2 - Ofc, only useful if there's RAM and only if none of the locations are used by the parts of DOS involved with this command.
*3 - U1/2 points to otherwise useful functions and can't be redirected as easy.
*4 - Yes, that's right in buffer #2. But there's also a pointer somewhere that can be made to locate them else were