I can not stress how important it is to have 
  queryable web logs.

I'd love to read some (actionable) articles on how to set up some of the benchmarking / logging stuff that you guys have. For sure I'll check out .NET Memory Profiler and mini profiler linked in the article, but I'd be grateful if anyone wants to post more comprehensive info on where to start.