Great question! As a team of software engineers and data scientists, we relied on observability tools like Datadog to build reliable infrastructure for products, but were really surprised that it didn't exist.
We're not 100% sure you can't just use Datadog for this, but it seems to be shoehorning a technology to a usecase its not designed for.
This is pretty neat. I like the positioning of a monitoring service for your data.
As someone who isn't a data engineer: How did you guys come across this problem? Is this something people have often?