There are different ways to be "good" at math. I have a few friends who are very good at contest type math (e.g. well-ranked on the Putnam) and I also have friends doing math PhDs at top schools. They are mostly not the same people. The contest winners are very quick; they know a lot of tricks and general problem-solving strategies and can think very efficiently about how to apply them and come to a clever solution. The researchers are not necessarily as quick, but they're more patient, willing to spend days or weeks focusing on a single problem and learning/developing the machinery necessary to solve it, and they have more of a pure intellectual curiosity about math -- they approach it as a source of new and interesting questions, rather than just a set of techniques to be learned or problems to be solved. Sure they all did well on the usual standardized tests (SAT, etc), but if you were going to use a test like the Putnam to identify who you wanted to hire as a mathematician, you'd make a lot of poor choices.

That said I basically agree with you that it's not reasonable to call someone "good" at math if they're unable to do simple calculations (with the caveat that lots of great mathematicians screw up simple calculations on a regular basis -- but they're still at least usually better than the average person). But there are certainly people who memorize enough formulas to pass the test, yet never acquire or don't retain any conceptual understanding. In that sense, passing a test is a necessary but definitely not sufficient condition for being good at math.