The original computers were big because they were mechanical, and they shrunk by being reduced to electric and due to more precise engineering.

QC are big because the energy levels are so high that they require complex equipment to focus energy and remove heat, akin to the tyranny of the rocket equation.