There are actually two software programs involved here: NintendoSpy running on your computer and a program that must run on the Arduino. The program running on the Arduino reads data from the spliced game controller cable and communicates with NintendoSpy on the PC to tell it what it saw.
I'm guessing that NintendoSpy has contained in it a version of this program compiled for an ATMega328 CPU, uploads it to the Arduino device when it's first connected to it via the COM port, and then starts it. After that NintendoSpy would talk with the running program as above.
Unfortunately, your RedBoard, though the same physical shape as an Arduino, is otherwise vastly different. The voltage levels are different, which means that you probably shouldn't connect it to the Nintendo controller cable without using a voltage converter (or you may damage your board), it has a completely different CPU (ARM Cortex M3 instead of ATMega328) and it doesn't even appear to use the standard Arduino protocols for uploading software to it.
Basically, your board is a certain physical shape so that it can use a few Arduino peripherals, but otherwise it seems not to be an Arduino. It's like a computer with IBM PC-shaped slots that is otherwise completely different from an IBM PC.
If you wanted to use this board you would probably need to:
- Modify the Arduino part of NintendoSpy code (under the
firmware/ directory) to run on your board. Since your board doesn't appear to be supported by the Arduino IDE, this could be a very large project.
- Build a new version of NintendoSpy that would upload your modified code to your board; since the communications protocols seem to be different this could also be a huge project. Alternatively, modify NintendoSpy so that it doesn't do the upload but instead assumes the board software is already running, and do the upload and start the software yourself before asking NintendoSpy to connect.
If you want to pursue this route, the Arduino StackExchange is probably a good place to post questions related to this. But given its difficulty, it seems worthwhile to spend the $10-15 just to buy an Arduino Uno clone to run this. You may also be able to use a cheaper Micro or Nano ($5-10, or even less), though if those are not directly supported by NintendoSpy you may have to tweak the firmware program just slightly.
Even if you do switch to an Arduino, the Adruino StackExchange is still probably better than here to post about difficulties connecting NintendoSpy to the Arduino itself, since they will know more about Arduino/PC connectivity. Once you've got NintendoSpy talking to the Arduino and have the cable pins plugged into the right sockets, questions about the actual game controller protocol would be appropriate here.