Say I want to hook up a Nintendo 64 console to an HDTV via HDMI or Component. What are my options for upscaling hardware to be able to play the system without low quality or blocky picture and with low controller latency without directly modifying the console?
You are asking how to convert the analog video output from a console for using it on a modern HDTV system. The usual answer is to use some kind of converter from the best video output option available in the console, to the best video input available in the TV. Chances are that your TV already has the right kind of input. In this case, you won't gain anything by using an external converter, which will do the same as the internal converter in your TV already does, usually performing worse than the internal converter, which should be best coupled to the electronics of your TV, and will surely provide you a "Game" profile in which the latency is reduced to a minimum.
If you own a TV designed for the european market, you are usually granted with composite video input, component input, RGB (scart), S-Video (sometimes it is included in scart) and HDMI, so you would use your RGB or component video output from your console and let the TV upscaler do the job.
If you own a TV designed for the american market, you won't have scart, but you may have component input. If the console outputs RGB and you don't have RGB inputs, use a RGB to YPrPb converter, which uses very little electronics so it won't add latency.
Generally speaking, any converter that doesn't need to change the timings and the domain of the video signal (i.e. it stays analog after the conversion) won't (shouldn't) introduce latency. Letting a external device to do the complete transformation, from analog to digital and from SDTV timings to HDTV timings will put you in the risk of suffering some latency.
In short: always that's possible, let the TV to be the one that does the internal upscaling, and do a conversion from the best video output you can get from your console to the nearest video input option on your TV.
It depends on whether you also want to modify the N64 to support RGB output.
Without modification, the best output to use from the N64 is S-video. Cables are commonly available on eBay for the N64 that support S-video, and there are upscalers that will convert S-video to HDMI 1080p. This unit from Amazon looks like it would do the job.
You can also get far better output if you modify the N64 to output an RGB signal, plus use a different converter to go from the RGB to HDMI. There are details for modding the N64 here, and it does look rather involved. If you do go this route, you still need a converter like this to connect the N64-to-SCART cable to your HDMI monitor.
Regarding up-scalers and scan conversion, I'm honestly not crazy about either of those Amazon products. My go-to scan converter is the Sony DSC1024, which outputs standard VGA at up to 1600x1200. The 4:3 aspect, multi-input switching, scan-doubling, NTSC/PAL compatibility, and aperture capability all combine to make it a great converter for retro-computers and consoles. You can still find them used at a good price, sometimes.
The N64 natively outputs Composite and S-Video, so without modifying the console, the best results will be through an S-Video cable like this fully shielded cable.
But your TV may not support S-Video at all, or it may add noticeable lag or poorly scale the image to the screen, resulting in a stretched and/or blurry image. A Micomsoft XRGB-mini Framemeister will correct these issues.