In his book The Silicon Jungle David Rothman mentions several times the term "modifying" or "customizing" a software program.
Just like today, usage of generic products have always needed customisation to be used: forms in word processors, queries in DB products and so on.
And then the consultant didn’t even supply instructions to operate and modify the software. You could customize the MDBS program in FORTRAN or BASIC.
MDBS is a database system - to make it useful, one has to add some DB application/queries/etc.
You could have WordStar customized for your machine; but why not instead buy a computer that works well with it from the start?
Wordstar is made to run on a wide variety of machines, which have different ways of accessing keyboard (*1) and screen (*2), so when buying a Wordstar, one had to select at least the configuration for your system - or create a new one if the target machine is not already listed. The whole process is similar to editing a Unix Termcap, which acts as a dynamic abstraction layer - except that with early programs it was done in a static way, with the configuration patched into the binary.
Other applications required the user to patch the binaries directly to adjust functionality. So there has been a wide variation of customization.
This has gone out of sight for most of the time today, as for one, keyboard hardware (encoding) is much the same everywhere (essentially PC style), but also OSes covering this by replacing hardware related encoding by symbolic layers.
What does he mean by this? Surely commercial programs like WordStar weren't sold as source code?
No, but they were made to be patched for customization.
In addition, it wasn't uncommon that software was also provided fully or partly as source. Depending on the product source code was seen as part of documentation, meant to allow the user to utilise every detail and/or to research details of operation not found in the 'written' documentation.
Is this modification and recompilation at the source code level? Or is he talking about "configuring".
All of that may apply. Software has to be configured, modified, and or supplemented to be used. Back then as well as today.
*1 - Keyboards (directly attached to computers as well as via terminals) had many different layouts and number of keys available, from straight TTY with no cursor and editing keys at all, all the way to luxurious monsters with special keys for everything imaginable. Also, different keyboards used different key codes for similar keys.
*2 - Screen control was (mostly) done by control sequences, but as with keyboards, different computers (and terminals as well) used different encodings for the same function.