5

I'm writing my thesis about the home computer market between 1977 and 1984 (focusing on Commodore Inc.so PET, VIC-20 and 64 and competitors). I'm trying to plot a curve describing the technological innovation on the home computer market, and I am concerned about finding a good parameter to evaluate the progress of the industry in terms of effort in knowledge acquired and performance improvement.

Can I use MIPS? Is there any benchmark available for a list of products that were on the market in that period of time? Can anyone suggest a better indicator, considering the lack of data that sometimes occur in this kind of research?

That's what I found for now:

https://en.wikipedia.org/wiki/Instructions_per_second
https://web.archive.org/web/20141009144357/http://www.jcmit.com:80/cpu-performance.htm

Can anyone can come up with a more complete list or a better idea about the parameter to use?

Also, if it can be of any help to clarify the subject, my work is for an economics thesis, and it's based on the research of Foster and Christenen about the S shaped curve of technology

  • Benchmarks were very important for business and scientific computing, where the machines were intended to run custom programs to accomplish specific tasks and there was a need to measure "value" for different systems. But for home computing, benchmarks turned out to be less useful because consumers were much more focused on price. For example, being able to avoid the purchase of a monitor by attaching a computer to an existing TV was an important feature not measured by benchmarks. – Ken Gober Nov 16 '17 at 15:41
  • 2
    If you use information from this site, please cite this as a source. You can do this by clicking the share link below an answer if you want to cite a specific answer. (Not that I don't think you'd cite your sources, but I think this is the first time somebody's mentioned citing this site in a thesis.) – wizzwizz4 Nov 16 '17 at 22:14
  • 1
    @wizzwizz4 my friend, i will surely cite you, this site and everyone that helped. I will do more if it's possible, i will upload here the whole research once i'm done :) – Cicomameli Nov 17 '17 at 16:43
  • @Cicomameli Just a link would be fine - this is a Q&A site, not a Q&A&T site! :-) – wizzwizz4 Nov 17 '17 at 16:57
  • 1
    @wizzwizz4 of course, i just thought this could be a nice way to say thank you :) actually as this is my first thread here and i wasn't expecting so much help, so at the moment i'm incredibly thankful and confident in the humankind :D I'd offer you a beer if i could! – Cicomameli Nov 17 '17 at 17:12
6

If your focus is on Commodore products from 1977 to 1984, then MIPS would likely be the wrong way to differentiate. I believe the whole line used a 1 MHz 6502 chip, meaning MIPS will be basically the same. The SuperPET added a 1 MHz 6809, but I don't believe the CPUs could operate simultaneously. It wasn't until 1985, with the Commodore 128, that the 8502 (upgraded 6502) could run at 2 MHz, and even could drive an 80-column monitor. (Even the C128 offered backwards compatibility with the C64, running at 1 MHz on a 40-column screen in that mode.)

So a better parameter might be "user experience", i.e. graphics, sound, and text modes. Note that not all users want or need the same thing. A typical business would greatly prefer an 80-column, easily read display over color graphics and multi-voice sound. A 10 year-old kid would prefer the latter, and his parents would appreciate that it used the television instead of a dedicated monitor.

Commodore's major competitors in the North American home included (no particular order):

  • Apple (Apple ][ with a 1 MHz 6502)
  • Radio Shack (TRS-80 with a 1.77 MHz Z80, roughly equivalent to 1 MHz 6502
  • TRS-80 Color Computer with 0.9 MHz 6809
  • Texas Instruments (TI-99/4, with TI's own 3 MHz 9900 "16-bit" CPU -- it had only three hardware registers, none of which were an accumulator, making it feel slow, especially it's BASIC language)
  • Atari 8-bit computers (6502 at about 1.8 MHz)

The fastest in MIPS I think would have been Atari. Other major considerations would have been keyboard (chiclet, membrane, or normal), presence of lower case, characters per line (anywhere from 22 to 80), color video, sound, sprites, external storage, and game control method(s).

Maybe your thesis could feature a 2d graph, with "kidability" on one access and "professionality" along the other.

  • 1
    Personally, I'd make it 3D with time as an additional factor. But then it's hard to display, so colours might be used to represent time instead. – wizzwizz4 Nov 16 '17 at 17:47
  • 2
    @wizzwizz4 I wasn't envisioning graphing a connected line. What I would do is offer a single quadrant with kidability along the x axis and professionality along the y. Then within the quadrant there would be several small icons, one for each computer. For example, the original PET, with its chiclet keyboard and limited graphics would be low in both parameters, The PET model with full keyboard and 40-col screen would be above that, and the 80-col model, even higher. The VIC, with its 22-columns screen, but with graphics and sound, would be lower than the chiclet pet, but further right. – RichF Nov 16 '17 at 20:17
  • 1
    @RichF Well, at least the comment about the TI is a bit off. for one, it wasn't a custom CPU, but a standard 9900. Second the CPU wasn't per se slow at all. Speed was quite comparable to a 6502 at the same clock rate, not at least due it's 16 bit nature. A major drawback of the TI99/4 was that the full figured 9900 CPU was made to emulate a never released 8 bit single chip implementation. Also, BASIC was only slow when compared with the limited capabilities of MS-BASICs. By using its abilities, one could easy write fast grahic games in BASIC withotu a single line of machine code.. – Raffzahn Nov 17 '17 at 16:27
  • 2
    @Cicomameli The variables you mention all seem applicable. Perhaps not focus on the number of sound chips, but the capability of whatever chip(s) were offered. As far as external storage, IIRC they all offered tape capability natively, with floppies and (eventually) hard drive storage optional. I never realized it until I was writing the answer, but 77-84 is actually a pretty good range. For those 8 years, there was no major advance in MIPS for home computers. The IBM PC and Mac did appear during that time, but costwise they were on upper end of what the home market would buy. – RichF Nov 17 '17 at 16:56
  • 2
    Commodore had a bunch of other major competitors outside of the US: Amstrad, Sinclair, Thomson, the combined weight of everybody manufacturing an MSX... – Tommy Nov 17 '17 at 19:34
4

See the other answers re: potential issues with benchmarking the Commodores (i.e. the results will probably all be the same), but as to potential sources Personal Computer World ('PCW') had a standard set of BASIC benchmarks that it used to evaluate all new home computers back in the day.

Source code and results for the BBC, Amstrad CPC, IBM PC, Acorn Atom, Vic-20, Apple II, Dragon 32, both Orics, SVI-328, ZX81, Microtan, Spectrum, Atari 600 and TI-99/4a can be found here, and finding the review issues of PCW for other machines will likely give you their benchmark scores.

Spoiler: the Vic-20 actually does pretty well when overall, being beaten only by the two Acorns, the Amstrad and the IBM PC. The BBC comes on top. Interestingly it takes almost exactly half as long as the Vic-20 on average, and is built around a 6502 that runs almost exactly twice as fast as that of the Vic, but has an entirely independent implementation of BASIC.

The PCW benchmarks aren't ideal as a test of hardware alone, they're a test of the combination of machine speed and quality of BASIC, but most home users who weren't just looking for games would write their own software in BASIC so I'd argue that it's a valid era-appropriate benchmark.

See the difference between the Oric 1 and the Oric Atmos for a representation of how the quality of the BASIC interpreter affects the tests; those are identical hardware but the latter has version 1.1 of Tangerine's [Microsoft] BASIC ROMs whereas the original has only version 1.0.

  • 1
    Ironically, the VIC-20 is faster (slightly) than the Commodore 64 and (and the 128 in 40-column mode, which is how most people used it), but the 64 is a much better computer than the VIC-20 (and the 128 even moreso). – Jim MacKenzie Nov 18 '17 at 14:48
3

For home computers, the most important criteria were generally:

  1. does a particular model do what you need/want?
  2. how much will it cost?

"can the computer do X?" was a much more important question than "how fast can the computer do X?" for most consumers, because they were limited to things the computer would do out of the box, things that could be done with commercially-available software, or things they could do themselves (if they chose to learn how to program the computer themselves).

Cost was also a big consideration because at that time prices were relatively high and the 'value' of a computer was not well established. In many households a computer was just a very expensive toy used to play games. A computer that could be connected to an existing TV had a big cost advantage over a computer that required you to purchase a monitor.

The most popular home computers were the most popular because they had a lot of software available, and were relatively cheap compared to their competitors. Their speed running benchmarks was not really of interest (except for after you had already purchased it, at which time benchmarks were an important part of the "my computer is better than your computer" discussions that inevitably broke out).

3

Comparing processor architectures is tricky because they are all very different. How fast something can be done depends not only on the clock speed, instructions available, RAM speed, and how many bus cycles were used by the video chip (which was usually affected by mode it was in), but also on how well the algorithm is coded. How a Z80, 6502 or 6809 is coded to do something can vary a lot. The 6502 has simpler instructions which execute quickly, while the Z80 has more complex instructions which take more cycles to execute. Which is better will vary depending on your task.

Something that can more easily be benchmarked is languages, especially within language families. The most common language was BASIC and the most common brand was Microsoft, but you can compare different brands of BASIC, and even different languages, across processors and ignore clock speeds, RAM speed and bus cycles. This gives a simple numeric result which would be more useful to a potential owner.

Commodore BASIC and Applesoft BASIC benchmark very similarly with the Commodore 64 and Apple II benchmarking almost identically due to them both using Microsoft 6502 BASIC at 1 MHz. The Atari 800 used the 6502 at 1.79 MHz but due to the graphics chip taking up cycles, and the non-Microsoft Atari BASIC not being coded very optimally, it was much slower than Microsoft BASIC on the C64 or A2.

The next question becomes, what aspect of the languages do you want to benchmark. The most commonly known benchmark is an Ahl's Simple Benchmark (Creative Computing, Nov 1983, page 260) which was commonly seen back in the 80's, and there are over 150 results available on page 8 of the July 1984 issue, but all it benchmarks is a language's ability to run a loop, square and square root numbers, and determine how fast and accurate the floating point routines are. For the most part, the accuracy of the floating point calculations depends on how many bytes were used to store them. Unfortunately, I am not aware of a more comprehensive benchmark.

  • 1
    Thanks a lot, i'm not much into programming but i was aware that different versions of BASIC can perform differently on the same hardware. Thanks a lot for your comment :) – Cicomameli Nov 17 '17 at 16:19
  • On 8-bit computers, Ahl's benchmark mostly measures how well the math routines in ROM were written, not just the raw CPU performance. Byte's Sieve benchmark is both earlier, and less dependent on the quality of the transcendental math library functions. – hotpaw2 Nov 17 '17 at 21:32
3

Can i use MIPS?

Well, of course you may use MIPS, but is it meaningful?

After all, like with mainframes in the 60s and minis in the 70s, it wasn't about number crunching power. The first stage when a new technology becomes available is fundamental enabling, not speeding up. It's about that tasks that couldn't have been done before now are within reach. Take word processing. Just being able to backspace a whole sentence gives a completely new way of writing, compared to typewriters and whiteout. It didn't matter how slow and clumsy that machine was.

After that fundamental change, everything else is just gradualy again.

Is there any benchmark available for a list of products that were on the market in that period of time?

The Ultimate Benchmark might be a good start.

While developed just recently, the tools used are contemporary with early microcomuters. Simple benchmarks, like MIPS or Drystone only test a certain, rather limited, range of capabilities. But especially in the early time not all CPUs where (mostly) equal, but implemented way different strategies (if at all) to handle certain tasks. The Ultimate Benchmark now tries to switch from such simple setup to a more all around one, by inserting the abstraction layer of Forth and running a variety of more real world like tasks, much like SPEC.

This allows for general and specific optimization of the Forth system toward a certain CPU, so that its specific advantages could be used. All while still running the same source task.

Of course even this benchmark focuses on the CPU specific side, but then, extremely different graphics and sound hardware is even harder to compare.

Can anyone suggest a better indicator, considering the lack of data that sometimes occur in this kind of research?

Now, isn't that the task you're told to solve? SCNR :))

  • 1
    You are definitely right about the enabling capacity, and sorry you are right ahah i wasn't asking to do my tasks, i'm just on tight schedule and my concerns about the fact that MIPS or any other pure perfomance indicator was not fitting my purpose. I will opt for a list of features and create a chart based on them, i guess it's better from any point of wiew. Thanks! – Cicomameli Nov 17 '17 at 16:24
  • Forth runs quite slowly on processors with limited stack handling. If you look down the results table, all the 6502-based machines do worse than you'd expect. – scruss Nov 18 '17 at 16:52
3

Early personal computer magazines published several tiny benchmarks. One of the earliest portable benchmarks was Byte Sieve, published in Byte Magazine in multiple articles circa 1981 thru 1984. Byte magazine published several articles using this benchmark to compare different programming languages, as well as to compare the relative performance many personal computers of that era. The value of this benchmark is the existence of multiple performance reports from the earliest 8-bit personal computer era, providing a good baseline appropriate for the personal computing world.

A simple prime number sieve is an interesting benchmark in that it measures mostly basic arithmetic and array access instruction execution time, and the array size can be scaled both down and up for a range from tiny embedded devices up to larger contemporary processors. It is not extremely different from McCalpin's Stream benchmark, which at one time was a popular supercomputer performance metric. Close-to-linear scaling factors can be estimated and applied to Sieve sizes for comparison purposes. Unlike Ahl's later benchmark from Creative Computing, it does not spend most of it's time executing unknown-quality vendor-written math library routines.

Here it is in Basic: https://github.com/hotpaw2/hello-world-1/blob/master/sieve.bas

Running this benchmark on in Applesoft Basic on an Apple II with 48k of RAM requires that the array size be reduced to around 7000. IIRC, it took around 12 to 13 minutes to run.

A VAX 11/780, which DEC marketing claimed to be capable of "1 MIP" in general performance, took between 2.5 to 3 seconds to run the Sieve benchmark (size 8192) when compiled from a C version (source code can be found in many places online).

And here is the Sieve benchmark in Swift: http://www.nicholson.com/rhn/files/benchmarks/sieve.swift

On my iPhone X, after scaling to 1 iteration of size 8192, it would take around 18 milliseconds in interpreted HotPaw Basic, and around 0.2 milliseconds compiled from Swift -Ounchecked .

2

Rather than a software benchmark, how about a physical one: transistors/USD

Software benchmarking the 8-bit Commodores will produce a very clustered set of results. They all used a (roughly) 1 MHz 6502 until the C128 came along, and they all ran almost the same MS Basic interpreter. Most of the value in owning a computer was not running benchmarks, but rather what use you could make of it.

But comparing the Pet 2001 to the C64C on number of transistors (in processor, support chips, ROM and RAM), the capabilities go up massively while the cost drops. With more memory and better support chips for graphics and sound (hence, more transistors) the more useful the computer became.

Of course, you'd have to correct for the roller-coaster inflation rates of the 80s to make this meaningful. Commodore's an interesting case of starting off buying whatever components would work cheaply (compare two Pet 2001s, and they'll have slightly different power cords, internal fasteners, etc) to growing large enough to become a price-maker for electronic components.

  • I had a laugh at "Commodore buying components cheaply" - The opposite was Sinclair with "Not buying any components, regardless how cheap they were" – tofro Nov 18 '17 at 19:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.