Why was IBM's Scientific Subroutine Package superseded?

Question

It is a familiar fact that scientific software tends to do a lot of vector arithmetic and similar, that one does not want to keep rewriting the low-level code for such, so the usual practice is to use an off-the-shelf arithmetic library such as LINPACK.

I've been trying to track down the early history of such, and apparently the first such library was IBM's Scientific Subroutine Package (SSP), which was presumably bundled with IBM hardware in the early days (because IBM software in general was, prior to the great unbundling of 1969) and which seems to have been important and widely used in its day, until superseded by EISPACK in the early seventies.

An excellent, detailed account of the creation of EISPACK can be found at https://web.archive.org/web/20170702065825/http://history.siam.org/pdfs2/Dongarra_%20returned_SIAM_copy.pdf

Now, I'm trying to understand this history from the perspective of the people who were there at the time, and I keep trying to think of what I would do if I were a software manager at IBM, and it seems to me that I would consider it important to keep SSP up to date, keep it the number one choice, because that is something that could very well influence the purchase of computer equipment at academic institutions, which could later influence the purchase of equipment in business by students graduated from those academic institutions.

Was IBM's lunch simply eaten by a hungry and determined competitor? It doesn't quite look like that. From the above document, it seems that EISPACK was written by academics; in general, academics do not regard the writing of code is a high-value activity, but as something best avoided if reasonably possible; this does seem to have been the case here; from the above document:

That kind of software is always one of these funny things in terms of providing research funds. If you’re writing software, many people ask the question “where’s the research contribution in working on software, how does it affect us and why should we be paying for this?” So that’s been a constant source of problems from the political side in funding mathematical software on a level with some of the research that’s done in other areas. And sometimes we’d have to hide the fact that we’re writing software, so we’d say that we’re developing portable techniques...

It does, on reflection, feel a little surprising for the world's strongest IT company to be outcompeted by people who weren't even supposed to be spending their time writing code.

So what happened?

Did someone at IBM fall asleep at the wheel?

Did the EISPACK developers discover some new techniques so important that it was worth redoing a lot of programming work that had previously been done by a corporate team?

Did the EISPACK developers decide that portability was so important that it was worth redoing that work? If so, are there any statements on record from any of them explaining this decision and the reasoning behind it? The above document contains no indication of such.

Is there some other factor that I am missing?

Answer 1

Specifically concerning EISPACK. what happened was that James Hardy "Jim" Wilkinson in the UK (whose career as an applied mathematician started with practical ballistic modelling in WWII, working with Turing and other computing pioneers, and continued for the rest of his life at the UK National Physical Laboratory, not in some academic ivory tower) re-invented the entire foundation of computational linear algebra - or more accurately, gave it a practical foundation for the first time in the history of mathematics, with the new concept of "inverse" or "backward" error analysis.

Wilkinson's 700-page book from the 1960s "The Algebraic Eigenvalue Problem" is still a classic. The IBM SSP borrowed a couple of simple algorithms from it (plus another one invented by Jacobi way back in 1840s!) but EISPACK was "the whole nine yards." The EISPACK library only contained algorithms for eigenvalue problems. It had about 70 different algorithms, compared with four in IBM SSP.

EISPACK was not "written by academics" but created at Argonne National Lab in the USA, with the objective of being portable and free public domain software (And having cut his teeth working on EISPACK and LINPACK as a new graduate, Dongarra then went on to develop the LAPACK library which made both of them obsolete)

As an ironic footnote, Wilkinson was completely dismissive of a class of eigenvalue extraction methods now known as Arnoldi iteration, claiming (rightly) that they were of huge theoretical importance but (wrongly) could never be made to work in practical numerical analysis because of problems with rounding errors. Solutions to the numerical problems were discovered in the 1980s and these algorithms are now the "goto" standard methods for solving eigenproblems that are orders of magnitude bigger than anything Wilkinson could have imagined.

THE IBM SSP was also not competitive with commercial libraries such as IMSL (first released in 1970) which were much more comprenhensive (thousands of routines, not about 100) and not restricted to one computer manufacturer's products.

Answer 2

I used the IBM Scientific Subroutine Package (SSP) combined with the IBM 1130 single-user computer system, and Fortran, at Giannini Hall at U.C. Berkeley circa 1979-1980, and recall some strange features such as matrices were read or written in transposed form. When I started teaching Economics in New York in 1982, I migrated to Gauss using an IBM desktop computer, and recall that Gauss had inherited the transposed matrix protocol. Today, R and Statsmodel in Python include more sophisticated data analytics tools, such as robust statistics and better handling of outliers. Until the development of R, or propriety S-Plus before R, Gauss was my goto engine to calculate Eigenvalue in input/output economics research. Together with Fortran (replaced by Pascal before Python), SSP had become obsolete in the microcomputer environment.