Preface 1: There can't be a single answer for all the varieties that have been out there
Preface 2: It's important to keep in mind, that make didn't start out as the almighty build automation and installation tool it's seen as today, but as a utility to reduce compile time by only compiling files that have changed or that depend on changed files. Compiling was a resource intensive and slow task on machines back then. It wasn't 50 files per second, but rather 50 seconds per file. So saving every little step did count a lot.
Were there any real precursors for make
Yes, there were, but usually rather OS-, development environment- and project-specific tools.
or did make invent the concept of build automation?
Not really, it's rather that make is a solution for a problem that did not arise the same way on mainframes. Software structure (at least for reasonable sized projects) was way more modularized and built around (internal) APIs. Changing a module interface was rather frowned upon - and even more so using global variables and the like.
Access to interfaces and data provided was usually encapsulated by interfaces - much like today's idea of methods. Except, we didn't use all these various fancy names. There were interfaces and records (parameter blocks). These 'methods' were kept binary stable as long as possible to avoid the need to compile whole applications at once. Changing some basic structure on the fly and starting a make was seen as quite unprofessional - think before you code.
Software development was much more incremental, based on module concepts. Modules (and thus interfaces) were (could be) versioned. The task to handle this was often handled by, or at least done with great support from, the linker, handling dependencies including version matching (*1).
Bottom line, it was a different approach using a much more deliberate process.
I took part in development of a rather large (>1200 modules) mainframe software. In the mid 1990s the development process was stable for more than 10 years (in fact, even longer, predating this project) - when a major customer became interested in the development process used. The usual crap about quality. And the usual combination of outsider management with no real idea about software - or more exactly with about the knowledge of a weekend course about software - and some young graduate with 'fresh ideas' - as well just as fresh as his limited knowledge from university could carry - produced a request to change to a make style development - as our proven process is of course outdated by modern tools. Even worse, the whole team was developing into a single repository only separated by task and only versioned by delivery cycle. Yeah, right.
Also as usual, the largest customer got more say about things he should not care about than was good. There is no make for the mainframe OS we used, so we had to create our own. Even more so, we didn't have trees of weird source-and-alike files, but well defined libraries holding sources, macros and scripts including many rules about how to combine and evaluate them.
Long story short, it took us about a year (and about half a million dollars) to develop a new build process. We finally settled to a hybrid. The development was done as before, except now every developer had to have his own repository (which of course added way more errors and more management to handle them). When a release was up to be delivered everything was now pumped into the new build system and configurated in one huge almost-an-hour-long build run.
Of course that new system never did report any new error. But hey, modern ... if I only had killed it with the argument that make is way older than the system we had, back when they proposed it :))
*1 - Linking was also usually not done against a bunch of .o files, but libraries holding versioned binaries - which itself could be the result of a linking process.