DEC's use of MIPS was only ever as basically a stop-gap.
Before they used MIPS, DEC had started work on a project called Prism. It was intended to be their first commercial RISC processor. In June of 1988, however, there was a meeting of senior executives. The PRISM project was producing some interesting technology, but didn't have a chip set available yet (like VAXen, it was going to be a multi-chip design, with FPU separate from the CPU). The executives decided they were too far behind the power curve, so they shut down the PRSIM project, and decided to start using MIPS chips instead.
At the time, this was seen solely as a workstation thing though--something completely different from the VAX. But, it was enough to get one of the executives to ask Robert Supnik to look into the question of whether this new RISC "stuff" could, perhaps, someday become a threat to DEC's VAX systems.
To answer that, Supnik formed what was called the "RISCy VAX" study group. They quickly concluded that yes, RISC could become a legitimate threat to the big machines, not just workstations. In fact, not only could become a threat, bt probably already was enough of a threat that they needed to respond. Further, based on the earlier PRISM work, they felt confident that DEC could produce a RISC processor they could sell.
They then considered a number of approaches to how to make a RISC processor that would run VMS efficiently enough to be successful. Approaches included a stripped-down VAX instruction set, some sort of hybrid RISC/CISC design, etc.
Around then, they considered just porting VMS to some RISC chip (such as MIPS) that was originally intended to run UNIX, but eventually decided that wasn't practical--that porting VMS to such a design would probably add something like 2 years to the schedule.
So, they decided on a pure RISC design, but with some sort of "trap door" to allow them to fairly cleanly add support for some VAX-like features upon which VMS depended (e.g., some parts of how it did its interrupt handling and paging). That resulted in the Alpha's PAL feature, with separate libraries of PALcode to support VMS and OSF1.
As others have mentioned, at the time DEC also saw 32-bit architectures reaching the end of their usefulness, so they decided that the new processor should be a 64-bit design from the beginning--where MIPS not only started out as 32-bits, but didn't have a 64-bit design until well after the Alpha.
They also looked at some of the existing RISC characteristics and decided against them. For example, they saw delay slots as scaling poorly (not obvious how they fit with a multiple-issue, out of order microarchitecture, among other things).
There's quite a bit more to it, of course, but I think that covers most of the "why not MIPS" question though.
References