You are a little. In a properly RESTful API you simply don't construct URLs most of the time. Instead you follow links. If you change the URL structure you break the links. If that's what you want then a version number in the URL makes sense, but in a properly RESTful API that's probably not what you want.
I think for a lot of people the reason they don't see the benefit of things from REST is they try to use them in isolation. You say "this is useless for my RPC style API!" and you are right.
But, if you follow a link and that link is generated by the server, I still don't see how encoding a version number in the URL makes things any harder.
FWIW, I've made an effort to follow the purity of it, but the APIs I see that try to do true ReST are insanely obtuse. If you know of any real world case studies where a provider went from "fake" ReST to true ReST, I'd love to read it. The contrived examples are not helping me out in the comprehension department.
I think for a lot of people the reason they don't see the benefit of things from REST is they try to use them in isolation. You say "this is useless for my RPC style API!" and you are right.