There is a fixed amount of land available in a city, and a city is responsible for servicing it. If you've a property that's gobbling up some of the city's carefully-managed land and creating an eyesore that's a detriment to your neighbors, then it's perfectly reasonable for the city to take action.
That is part of what a government's reason for existence is. That's what taxes are for. To protect people with livelihoods and property from those who would deprive them of such. That's what makes us live in a first world society and not the world of Mad Max.
The modern patterns of land ownership in Britain are due, basically, to a Mad Max world of taking what you want and having "rights" only to what you can defend by force against all comers.
Perhaps a compromise in which property is protected but the protection comes in exchange for some limitations on how you can use it... isn't actually the end of the world?
