If an application is flaky then I want to know: How frequently does it fail? How does this depend on combinations of configuration parameters? How does this compare between the stable, master, and next branches? etc.

The best way that I know for doing this is to write tests that are flaky because they expose the underlying flakiness in the application.

If an application is flaky and its test suite always runs 100% then I'd be pretty suspicious about that test suite being adequate.

> And if the application code is flaky

This is the only relevant factor. Forget the rest. Users don't experience your flaky tests just like they don't experience your messy Jira boards or your bad office coffee.

How do you know which is failing without exhaustive analysis?

See, once you know why the test fails and it's not the tested application, which is exceedingly rare in practice, you can just disable it or fix it. But only if you're actually sure, not before.

In my experience it is usually the test.

In my experience that's because tests are usually written in a timid style that tries not to provoke flaky behaviour from applications.

If your test suite can handle non-determinism then you can approach test-writing in a completely different - braver - way.