I had my home address set since couple years without turning history on.

This week I wanted to change address, but Google wouldn't let me unless I give access to all location, web and app activity.