Benchmark between Home computers? What can be a valid parameter for that?

Question

I'm writing my thesis about the home computer market between 1977 and 1984 (focusing on Commodore Inc.so PET, VIC-20 and 64 and competitors). I'm trying to plot a curve describing the technological innovation on the home computer market, and I am concerned about finding a good parameter to evaluate the progress of the industry in terms of effort in knowledge acquired and performance improvement.

Can I use MIPS? Is there any benchmark available for a list of products that were on the market in that period of time? Can anyone suggest a better indicator, considering the lack of data that sometimes occur in this kind of research?

That's what I found for now:

https://en.wikipedia.org/wiki/Instructions_per_second
https://web.archive.org/web/20141009144357/http://www.jcmit.com:80/cpu-performance.htm

Can anyone can come up with a more complete list or a better idea about the parameter to use?

Also, if it can be of any help to clarify the subject, my work is for an economics thesis, and it's based on the research of Foster and Christenen about the S shaped curve of technology

Answer 1

If your focus is on Commodore products from 1977 to 1984, then MIPS would likely be the wrong way to differentiate. I believe the whole line used a 1 MHz 6502 chip, meaning MIPS will be basically the same. The SuperPET added a 1 MHz 6809, but I don't believe the CPUs could operate simultaneously. It wasn't until 1985, with the Commodore 128, that the 8502 (upgraded 6502) could run at 2 MHz, and even could drive an 80-column monitor. (Even the C128 offered backwards compatibility with the C64, running at 1 MHz on a 40-column screen in that mode.)

So a better parameter might be "user experience", i.e. graphics, sound, and text modes. Note that not all users want or need the same thing. A typical business would greatly prefer an 80-column, easily read display over color graphics and multi-voice sound. A 10 year-old kid would prefer the latter, and his parents would appreciate that it used the television instead of a dedicated monitor.

Commodore's major competitors in the North American home included (no particular order):

• Apple (Apple ][ with a 1 MHz 6502)
• Radio Shack (TRS-80 with a 1.77 MHz Z80, roughly equivalent to 1 MHz 6502
• TRS-80 Color Computer with 0.9 MHz 6809
• Texas Instruments (TI-99/4, with TI's own 3 MHz 9900 "16-bit" CPU -- it had only three hardware registers, none of which were an accumulator, making it feel slow, especially it's BASIC language)
• Atari 8-bit computers (6502 at about 1.8 MHz)

The fastest in MIPS I think would have been Atari. Other major considerations would have been keyboard (chiclet, membrane, or normal), presence of lower case, characters per line (anywhere from 22 to 80), color video, sound, sprites, external storage, and game control method(s).

Maybe your thesis could feature a 2d graph, with "kidability" on one access and "professionality" along the other.

Answer 2

See the other answers re: potential issues with benchmarking the Commodores (i.e. the results will probably all be the same), but as to potential sources Personal Computer World ('PCW') had a standard set of BASIC benchmarks that it used to evaluate all new home computers back in the day.

Source code and results for the BBC, Amstrad CPC, IBM PC, Acorn Atom, Vic-20, Apple II, Dragon 32, both Orics, SVI-328, ZX81, Microtan, Spectrum, Atari 600 and TI-99/4a can be found here, and finding the review issues of PCW for other machines will likely give you their benchmark scores.

Spoiler: the Vic-20 actually does pretty well when overall, being beaten only by the two Acorns, the Amstrad and the IBM PC. The BBC comes on top. Interestingly it takes almost exactly half as long as the Vic-20 on average, and is built around a 6502 that runs almost exactly twice as fast as that of the Vic, but has an entirely independent implementation of BASIC.

The PCW benchmarks aren't ideal as a test of hardware alone, they're a test of the combination of machine speed and quality of BASIC, but most home users who weren't just looking for games would write their own software in BASIC so I'd argue that it's a valid era-appropriate benchmark.

See the difference between the Oric 1 and the Oric Atmos for a representation of how the quality of the BASIC interpreter affects the tests; those are identical hardware but the latter has version 1.1 of Tangerine's [Microsoft] BASIC ROMs whereas the original has only version 1.0.

Answer 3

For home computers, the most important criteria were generally:

1. does a particular model do what you need/want?
2. how much will it cost?

"can the computer do X?" was a much more important question than "how fast can the computer do X?" for most consumers, because they were limited to things the computer would do out of the box, things that could be done with commercially-available software, or things they could do themselves (if they chose to learn how to program the computer themselves).

Cost was also a big consideration because at that time prices were relatively high and the 'value' of a computer was not well established. In many households a computer was just a very expensive toy used to play games. A computer that could be connected to an existing TV had a big cost advantage over a computer that required you to purchase a monitor.

The most popular home computers were the most popular because they had a lot of software available, and were relatively cheap compared to their competitors. Their speed running benchmarks was not really of interest (except for after you had already purchased it, at which time benchmarks were an important part of the "my computer is better than your computer" discussions that inevitably broke out).

Answer 4

Comparing processor architectures is tricky because they are all very different. How fast something can be done depends not only on the clock speed, instructions available, RAM speed, and how many bus cycles were used by the video chip (which was usually affected by mode it was in), but also on how well the algorithm is coded. How a Z80, 6502 or 6809 is coded to do something can vary a lot. The 6502 has simpler instructions which execute quickly, while the Z80 has more complex instructions which take more cycles to execute. Which is better will vary depending on your task.

Something that can more easily be benchmarked is languages, especially within language families. The most common language was BASIC and the most common brand was Microsoft, but you can compare different brands of BASIC, and even different languages, across processors and ignore clock speeds, RAM speed and bus cycles. This gives a simple numeric result which would be more useful to a potential owner.

Commodore BASIC and Applesoft BASIC benchmark very similarly with the Commodore 64 and Apple II benchmarking almost identically due to them both using Microsoft 6502 BASIC at 1 MHz. The Atari 800 used the 6502 at 1.79 MHz but due to the graphics chip taking up cycles, and the non-Microsoft Atari BASIC not being coded very optimally, it was much slower than Microsoft BASIC on the C64 or A2.

The next question becomes, what aspect of the languages do you want to benchmark. The most commonly known benchmark is an Ahl's Simple Benchmark (Creative Computing, Nov 1983, page 260) which was commonly seen back in the 80's, and there are over 150 results available on page 8 of the July 1984 issue, but all it benchmarks is a language's ability to run a loop, square and square root numbers, and determine how fast and accurate the floating point routines are. For the most part, the accuracy of the floating point calculations depends on how many bytes were used to store them. Unfortunately, I am not aware of a more comprehensive benchmark.

Answer 5

Can i use MIPS?

Well, of course you may use MIPS, but is it meaningful?

After all, like with mainframes in the 60s and minis in the 70s, it wasn't about number crunching power. The first stage when a new technology becomes available is fundamental enabling, not speeding up. It's about that tasks that couldn't have been done before now are within reach. Take word processing. Just being able to backspace a whole sentence gives a completely new way of writing, compared to typewriters and whiteout. It didn't matter how slow and clumsy that machine was.

After that fundamental change, everything else is just gradualy again.

Is there any benchmark available for a list of products that were on the market in that period of time?

The Ultimate Benchmark might be a good start.

While developed just recently, the tools used are contemporary with early microcomuters. Simple benchmarks, like MIPS or Drystone only test a certain, rather limited, range of capabilities. But especially in the early time not all CPUs where (mostly) equal, but implemented way different strategies (if at all) to handle certain tasks. The Ultimate Benchmark now tries to switch from such simple setup to a more all around one, by inserting the abstraction layer of Forth and running a variety of more real world like tasks, much like SPEC.

This allows for general and specific optimization of the Forth system toward a certain CPU, so that its specific advantages could be used. All while still running the same source task.

Of course even this benchmark focuses on the CPU specific side, but then, extremely different graphics and sound hardware is even harder to compare.

Can anyone suggest a better indicator, considering the lack of data that sometimes occur in this kind of research?

Now, isn't that the task you're told to solve? SCNR :))

Answer 6

Early personal computer magazines published several tiny benchmarks. One of the earliest portable benchmarks was Byte Sieve, published in Byte Magazine in multiple articles circa 1981 thru 1984. Byte magazine published several articles using this benchmark to compare different programming languages, as well as to compare the relative performance many personal computers of that era. The value of this benchmark is the existence of multiple performance reports from the earliest 8-bit personal computer era, providing a good baseline appropriate for the personal computing world.

A simple prime number sieve is an interesting benchmark in that it measures mostly basic arithmetic and array access instruction execution time, and the array size can be scaled both down and up for a range from tiny embedded devices up to larger contemporary processors. It is not extremely different from McCalpin's Stream benchmark, which at one time was a popular supercomputer performance metric. Close-to-linear scaling factors can be estimated and applied to Sieve sizes for comparison purposes. Unlike Ahl's later benchmark from Creative Computing, it does not spend most of it's time executing unknown-quality vendor-written math library routines.

Here it is in Basic: https://github.com/hotpaw2/hello-world-1/blob/master/sieve.bas

Running this benchmark on in Applesoft Basic on an Apple II with 48k of RAM requires that the array size be reduced to around 7000. IIRC, it took around 12 to 13 minutes to run.

A VAX 11/780, which DEC marketing claimed to be capable of "1 MIP" in general performance, took between 2.5 to 3 seconds to run the Sieve benchmark (size 8192) when compiled from a C version (source code can be found in many places online).

And here is the Sieve benchmark in Swift: http://www.nicholson.com/rhn/files/benchmarks/sieve.swift

On my iPhone X, after scaling to 1 iteration of size 8192, it would take around 18 milliseconds in interpreted HotPaw Basic, and around 0.2 milliseconds compiled from Swift -Ounchecked .

Answer 7

Rather than a software benchmark, how about a physical one: transistors/USD

Software benchmarking the 8-bit Commodores will produce a very clustered set of results. They all used a (roughly) 1 MHz 6502 until the C128 came along, and they all ran almost the same MS Basic interpreter. Most of the value in owning a computer was not running benchmarks, but rather what use you could make of it.

But comparing the Pet 2001 to the C64C on number of transistors (in processor, support chips, ROM and RAM), the capabilities go up massively while the cost drops. With more memory and better support chips for graphics and sound (hence, more transistors) the more useful the computer became.

Of course, you'd have to correct for the roller-coaster inflation rates of the 80s to make this meaningful. Commodore's an interesting case of starting off buying whatever components would work cheaply (compare two Pet 2001s, and they'll have slightly different power cords, internal fasteners, etc) to growing large enough to become a price-maker for electronic components.