Ok since this is not really answered... Hypothetically, If I'm a maintainer of this project. I decided I hate the implementation, it's naive, horrible performance, weird edge cases. I'm wiser today than 3 years ago.

I rewrite it, my head full of my own, original, new ideas. The results turn out great. There's a few if and while loops that look the same, and some public interfaces stayed the same. But all the guts are brand new, shiny, my own.

Do I have no rights to this code?

You have all rights to the code that you wrote that is not "colored" by previous code. Aka "an original work"

But code that is any kind of derivative of code before it contains a complex mix of other peoples rights. It can be relicensed, but only if all authors large and small agree to the terms.

You have rights, but if it's a derivative, the original author might have rights too. If you made a substantial creative input, the original author can't copy your project without your permission, but neither can you copy theirs.

Hmm are we in a ship of Theseus/speciation area? Each individual step of refactoring would not cross the threshold but would a rewrite? Even if the end result was the same?

Let us also remember that certain architectural changes need to happen over a period of planned refractors. Nobody wants to read a 5000 line shotgun-blast looking diff

Afaik you can do whatever you like to GPL licensed code, you do not need a license to refactor it.

I understand you need to publish the source code of your modifications, if you distribute them outside of your company.

You can do anything except change the license, which ensures that right to do anything passes on to others in perpetuity. That's how it's designed.

You also can't relicense it to be less restrictive

Or more restrictive! There are certain exceptions permitting combinations of open-source code however.

The standard way of "relicensing" a project is to contact all of the prior code contributors about it and get their ok.

Generally relicensing is done in good faith for a good reason, so pretty much everyone ok's it.

Trickiness can turn up when code contributors aren't contactable (ie dead, missing, etc), and I'm unsure of the legally sound approach to that.

The legally-sound approach is to keep track of your actions, so you can later prove you've made "reasonable" efforts to contact them.

If a copyright holder does not give you permission, you can't legally relicense. Even if they're dead.

If they're dead and their estate doesn't care, you might pirate it without getting sued, but any recipient of the new work would be just as liable as you are, and they'd know that, so I probably wouldn't risk it.