The Internet Archive shows a few mentions of a product called Noteworks (or NoteWorks), capable of providing some form of instant messaging with pop-ups under DOS. It is mentioned for example in Newton’s Telecom Dictionary:
Pop-Up Electronic Mail An electronic mail system that runs as a terminate-and-stay-resident program (typically within DOS) and can be popped up inside any application to send or read mail. Our office, we have a TSR electronic mail program that pops up. It is called Noteworks, and we really love it.
and in Small Business Server 2000 Best Practices:
This company […] kept its old NetWare server around in part so it could run the NoteWorks popup messaging service (this was kinda similar to the chat applet in Windows For Workgroups 3.11 a generation ago). Dick, the founder and president, claimed that NoteWorks historically met his needs in two ways. First, if he was already on a telephone call, he could receive a popup note that told him a telephone call was waiting on line two or someone was in the lobby to greet him. Second, the ability to communicate in real-time across his physical plant, which includes a large and noisy machining area where “Hey you!” doesn’t cut it, was needed.
So it seems the program was indeed named Noteworks, as you remember. A 1993 ComputerWorld article names its publisher as Noteworks Corp., and gives a similar “call waiting” story as in the SBS2000 book.
NetWare itself provided built-in instant messaging. Under DOS, this relied on the NET SEND
command, and messages would show up (in text mode) in the top line of the screen. Under Windows, a pop-up window would display the message.