It has a noticeable impact when your car is fairly efficient. I notice a roughly 10% increase in fuel consumption in the winter and I live in a mild Pacific coast climate where winter barely even happens.

Do you have electric heaters too? I have an older car, basically zero difference with long drives, but the car doesnt heat at all until the engine gets warm.

There is a slightly higher fuel consumption in city driving, because it takes longer for the engine to heat up, and the idle rotations are usually higher when the engine is still below optimal temperature.