Once you have an operating system that provides services callable from programs, you need to provide a way for programs to request those services.
These days the general approach only requires a single instruction that will trap from user mode into kernel mode; it can either be an instruction intended for that exact purpose (like SYSENTER on some Intel x86) or a more generic instruction (like INT 2E on earlier Intel) that will serve the purpose. Such mechanisms generally don't look like the rest of the instruction set, e.g., the opcode does not itself uniquely say what operation is intended by the programmer; any operands have to be separately specified, rather than being part of the syscall instruction.
But an earlier fashion was to provide hardware support for making system calls look more like the rest of the instruction set. This typically included providing many different opcodes, so that (as with hardware) one opcode invoked one system function. Maybe any operand(s) of the instruction would have the usual processing (such as effective address computation - modification and indirection) performed before the kernel was entered, and so on. I'll name this approach the 'extracode' approach, to use the English term of the period.
Several systems I've used or read about adopted the extracode approach: Atlas (which coined the term), the FP6000 and its offspring the ICL 1900 range, the SDS 9xx series ("programmed operators"), PDP-10 ("unimplemented user operations"), and probably more.
To give a TOPS-10 (PDP-10) example: to look up a file, the LOOKUP instruction is used. The instruction format is "LOOKUP ac, addr" - this is a single instruction, one word (not some macro sequence). The opcode for LOOKUP (assigned by the kernel design) is not used for anything else. The 'ac' field contains a channel number, the 'addr' points to a filename. The instruction may also specify index and indirection in the usual manner. The hardware computes the effective address of the filename from addr, index, indirection just as it does for all instructions, including validity checking. Then the trap is taken.
This is in contradistinction to other systems where the system-call sequence requires loading (in this case) the effective address of the filename into an agreed place, an indication of what function we want to perform in some other place, and then issuing the trap instruction that is common to all system calls.
The assembly-level programmer is conscious of the difference.
So, why don't we do it that way any more?
My own guesses at an answer is that thinking about each system call as a separate machine instruction is tied to an assembly-language view of programming, a view that the kernel was providing user mode with a virtual machine that "improved" on the hardware, and also to an era when the service repertoire was quite small.
For this question, I'm not interested in the case of emulation of missing instructions - where lower-end machines implemented in software operations that were done in hardware (or microcode!) on higher-end systems. I also recognize that there might be a grey area around features such as Alpha's PALcode or Prism's epicode, which I'll view an adaptation layer between hardware provisions and OS kernel requirements. And there was a grey area in some systems - e.g., TOPS-10 used separate UUO opcodes for many different functions, but also multiplexed others on top of a single CALL/CALLI UUO. Nevertheless, I'm hoping that there's enough of a distinction to provoke a discussion here.