Not with quantum computers, which is the entire argument for moving to post-quantum crypto.
If we can manage to solve the scaling problem and can have, say, 2000 qubits of useful computation to break ES256, then we "only" need to scale to 3000 qubits to break ES384. Which is a lot less daunting than it seems, since we'll all but certainly need breakthroughs in error correction or coherence times to reach that threshold of useful applications of Shor's algorithm.
Quantum computers are like fusion in the 1940's, "it's only going to take 20 more years".
Using arguments in line with klebb, if quantum is linearly difficult, and each qubit takes 1 year, with your thought experiment's numbers that's 1000 years between breaking 256 and 384, let alone 512.
Early on, a decade ago, I was much more accepting of the quantum camp's arguments. But now in retrospect they've oversold, overpromised, and under delivered.
These hypothetical breakthroughs may not come to fruition.
I don't necessarily think that this is coming any time soon, but factoring a 5-bit integer is fundamentally different from factoring a 200-bit integer, which is why I said we probably need a breakthrough in quantum error correction or other methods such that we can actually entangle hundreds of qubits for long enough to perform meaningful computation.
I'm not convinced that we've made meaningful progress on this front in the past 20+ years since IBM first used Shor's algorithm to factor 15 back in 2001. (In 2012, they factored 21. In 2019, they failed to factor 35 due to accumulation of errors.)
If you're willing to accept the premise that quantum computers will exist that can break RSA-4096, then those same quantum computers can break elliptic curve crypto based on P256 or P384 or P521.
If you're not willing to accept that premise, then what's the point of doubling your key length? Wasting CPU cycles?
If we can manage to solve the scaling problem and can have, say, 2000 qubits of useful computation to break ES256, then we "only" need to scale to 3000 qubits to break ES384. Which is a lot less daunting than it seems, since we'll all but certainly need breakthroughs in error correction or coherence times to reach that threshold of useful applications of Shor's algorithm.