People used to think 0 wasn't a number. Then that the square root of 2 wasn't a number. Then that -1 wasn't a number. Then that 'i' wasn't a number.

There just isn't one official list of "what is a number". Real mathematicians will consider anything a number temporarily if it's convenient for them.

For anyone who cares, I recommend David Foster Wallace's book "Everything and More" for a great history of mathematicians' thoughts about infinity.