There are many great answers here and it seems safe to assume that the given answers have established that bitness is about the data a CPU handles. After all, the task of a CPU is Data Processing not Address Generation.
Next to all relevant criteria have been mentioned, but I'd like to take it a step back to a more generic approach, showing that there are three basic ways bitness can be defined by
And it always works in this order.
And the first one is often forgotten(*1) or ignored (*2).
Noteworthy here is that it's not always about more is better - like Atari did by marketing the Jaguar as 64 bit. Motorola for example did originally market the 68k as being 16 bit (and so did many early users). At that time 8 bit CPUs were standard for microcomputers, with the exception of some 16 bit families (PACE et. al.), so 16 bit were seen as the next step. Going as sole 32 bit provider was seen as an uphill battle against every other offering, as customers would rather ask why spend money on something they don't need. So while some of the 16 bit age, like 9900 or 8086 were really 16 bit, Motorola, National and Zilog placed their new development as 16 bit - despite the fact that we today would see them as 32 bit.
Tech people often focus on the lowest level, implementation. What is the size of the data bus, internal or external, the size of the ALU, registers and so on. While this does have impacts on performance and for sure offers a good game of 'mine is better than yours', it is very specific on single implementations, making it hard to see the whole picture - after all, a NS32008 is an 8 bit CPU, thus not better than an 8080, isn't it?
ISA - Instruction Set Architecture
Taking the (abstract) ISA as guideline is eventually not just the middle of the list but also the middle ground here. An ISA describes how a CPU works at instruction level. It describes what the CPU can do using the resources visible to a program handled by the instructions available. It doesn't care if a CPU is implemented bit serial or full parallel or anything inbetween. It doesn't rely on how wide an external bus is either.
ISA vs. Implementation
An ISA can be implemented in a wide variety, and that has been used all over since the early days (of computer families). IBM /360 were not always 32 bit implementations - but they always present the same 32 bit ISA. Or take PDP-8/S, a bit serial implementation of the PDP-8 ISA. And not at least the 68008 vs. 68000 vs. 68020. While having many different bitnesses in terms of register, ALU and bus, they are all the same 32 bit ISA.
In the end, it's the ISA that decides what a CPU is. Implementation differentiates two chips in usage not more than using the same at variations of clock speed. I guess noone will state that a 2 MHz 6502 is a different CPU than a 1 MHz one.
Technology vs. Marketing
Last but not least it should be noted that all the bitness arguments are a thing of the past. And even back then not all that meaningful. It was (made) important during a phase of maturing, when going from barely usable architectures and implementations thereof toward today's all around capable ones. So trying to make sense from today's point of view does not always produce a valid result.
*1 - Martin made a great point by the XC2000
*2 - Especially here, as most of us are rather on the technological side than about pumping out funny phrases.