Every single ROM in my collection has the wrong SHA1 hash according to No-Intros

Question

I have a collection of ROM files for the NES, SNES, Nintendo 64 and others. These ROMs match exactly those games that I own or owned growing up.

Recently, I found out about the "No-Intros" database, and thus made a script to import all the filenames and hashes into my database. I then looped through all my ROM files and checked if they match their data.

None of them do.

I have very carefully double-checked that it's not a bug in my script. My database does contain the correct information from their database.

I do it like this:

1. Pick a ROM file, such as: Tetris (Europe).nes
2. Calculate the SHA1 hash sum for the file, in this case: 817169b819aadaae52cce6b3d8d2fc24270566d7
3. Check what SHA1 hash No-Intros claims that this exact file name has in my database: 66883b9eddec933e36b6bff0479cefd2434ffb40
4. Conclusion: They don't match!

Apparently, every single ROM I have is modified somehow.

Now, I vaguely know that many ROMs circulating over the decades have been "dirty" ones with modified headers (but otherwise identical game contents), messing up the hashes. That's likely what's happening here. I also vaguely know about programs that can scan and "repair" dirty ROMs to make them clean. However, I feel very, very scared about running software on my computer, so this would have to be a last resort, plus I'm not sure I can find any usable such software.

If possible, I would like to "clean" the ROMs myself with some simple logic or some very trustworthy self-contained script or something. But even more than that, I would like to know if my guess is even right at all.

I assume that No-Intros' data is correct, or at least is not so massively incorrect as to give the wrong hashes for every single ROM I own.

This makes me feel uneasy. I hate the thought of having some kind of "hacked" ROMs where some dude in 1997 or something modified the game slightly so that it runs differently. I want to be sure that I have only the exact data on the cartridges. Nothing more and nothing less.

This is all very confusing to me since my ROMs follow a very strict naming convention. For example: Zelda II - The Adventure of Link (Europe) (Rev B).nes. It seems unlikely (although definitely possible still) that somebody would go out of their way to rename bad ROMs to such a strict and specific naming scheme.

What do you suggest?

Answer 1

There are multiple header formats for NES and SNES ROMs. For SNES and N64 ROMs there are also byte ordering issues. Unless you know precisely what source ROMs they used to make the database you can only use trial and error to find the matching ripper settings.

The simplest option is probably to download No Intro ROM packs that others have assembled and discard the ROMs you do not wish to keep (e.g. for copyright reasons). BitTorrent is a safe and easy way to do that and the packs are easily available.

Otherwise the only option is to try every combination of ripper settings until you find the ones they were using. Looking at their web site they do not seem to have documented the settings anywhere, or even clearly which version of the games they ripped in some cases. It's not uncommon for there to be multiple retail versions of the game for a particular region.

Answer 2

To elaborate on what user said:

1. Have you tried the hashes from DAT-o-MATIC instead? Those are what I go by.

2. In my experience, NES hashes are typically calculated against the ROM data with the iNES header stripped off, so the hashes don't depend on header values which vary from dumper to dumper and may have been modified to automatically trigger emulator-specific compatibility tweaks.

   If I remember correctly, it's the first 16 bytes you need to skip on those.

   It's also possible your ROMs are in a different format, though less likely.

3. N64 ROMs can take four different forms, depending on how the dumper laid out the data it retrieved. (byte-swapped, word-swapped, both, or neither)

There's an open-source swiss army knife for ROMs named uCON64 which can serve pretty much any ROM reformatting needs you might have. (For example, see ucon64 --help --n64 for more info on byte-swapping N64 ROMs.)

(It does technically also do hash checking, but its built-in checksum verification and "rename to match filename defined in datfile" (-rdat) functions are archaic, so you'd need something like DatUtil (EXE and source) to convert more modern hash database formats to what it expects.)

Answer 3

I am changing my original comment to an answer...

Original comment: "It is my understanding that the "No Intro" data set hashes are hashes of only the ROM image data -- that is, any header, "trainer", and/or "trailer"/extra data is removed. The "defacto" header for NES carts is iNES, which is 16 bytes. Some files may have extra data at the end of the file (a trailer), such as the ROM cart name -- You can detect this by calculating the exact ROM size, and computing the hash for only that many bytes and ignore the rest."

What this means is that you cannot simply compute a full-file hash of the file and compare them. You will need to parse the header (unless it is a so-called "headerless rom") to determine the start and end of the actual ROM image to hash and compare only that. There are several tools you can find on the web which do this -- but do not replace your *.nes files with the stripped versions... The headers exist for a reason and contain information about additional hardware/circuits that the ROM image data cannot provide because it is not part of the ROM data. This is like a book cover is not part of the book text, and cannot be transcribed directly in to plain text. The best you can do is create a "header" that uses words to describe the cover.

Edit to add further help:

You will need to look up the format and information that the iNES header contains, in order to determine several things. This includes whether or not the *.nes/*.fc cartridge contains a 512 byte "trainer" before the PRG ROM data, the size of the PRG ROM data, and the size, if even present, of the CHR ROM data. Then hash only the PRG ROM and CHR ROM (if present) to compare with the No Intro hashes. This is part of why they are called "No Intro" -- they remove things like headers, "intros", trainers, and other things people and groups add to ROM dumps which are not part of the original ROM. Their website explains all this.