Because when the application is breaking it's good to know why! Logs can be just as ephemeral as metrics -- in many cases, even more so. They're not even mutually exclusive.
Where exactly does this anti-logs sentiment come from? Is it because tools like datadog can be lackluster for reading logs across bunches of hosts?
For me, I don't use Datadog so it's not that $ParticularTool does not work with logs, it's all stuff I put in my original post, it's a ton of samples, filtering puts heavy strain on the systems and it's extremely brittle IME.
If you have good metrics, you can generally get much further not even logging aggregating outside tossing everything into STDOUT and checking on it when you have alerts.
My experience is that metrics may tell you something is wrong, but logs are required to tell you what went wrong and why.
A simple fixed-length rolling buffer can get you pretty far for logging, but it isn't something you necessarily want to get off-device except when something bad has happened.
Where exactly does this anti-logs sentiment come from? Is it because tools like datadog can be lackluster for reading logs across bunches of hosts?