I'm a bit torn on this. I realize that you need to benchmark to really prove anything, and that optimization intuitions are often wrong, but on the other hand, you really need a wide variety of world benchmarks to approach any form of real world "proof". In the early implementation stages, I think clear thinking such as provided in this article is probably more important than spending 50 times as long setting up an array of benchmarks. Hopefully Salvatore has done some micro-benchmarks during development to guide his thinking, but even if not, it's hard to refute the thinking in this article.
If you think about the problem space of Redis vs Varnish, it's intuitively obvious that Varnish deals with a wide variety of general data without many opportunities to optimize beyond general purpose algorithms such as an OS provides. Whereas Redis has specific data types often with small footprints, and very careful attention paid to the details of optimization for memory and disk usage.
I think you're overstating the time it takes to come up with benchmarks to evaluate performance optimizations. Even micro-benchmarks tailored to showcase your performance under ideal circumstances are a start. The longer you go without doing any performance comparisons, the longer you go without knowing if your work was worth it.
I'm not trying to deride his work - it's a neat project, and I will probably read through his earlier entries more. I'm down with all of the reasons provided, but I recognize that as humans, we tend to believe in things we understand. Hence, we need to evaluate.
Perhaps I am, but why are we assuming he hasn't done any benchmarks? It seems quite likely that his opinion is informed by actual results. Does he need to draw up graphs and spend more time on a blog post to be taken seriously?
Well, yes. I take him "seriously," but I'm not yet convinced his techniques outperform the kernel. That's how systems work is done. If you want to convince people that your way is better, then you need data to back it up.
Remember that I started this discussion off by saying "This author makes compelling arguments for implementing application level paging." I'm down with his arguments - people have actually made them before, and I thought they were good arguments then, too. But I can't say "Yes, I agree, what you have done improves performance" until I actually see performance comparisons.
But that's only the first level point. The second level point is: are the optimizations worth it? That is, if you only improve performance by less than 1%, then it's probably not worth the hassle. These are the sorts of things that experiments can tell you.
If it sounds like I'm being pedantic: well, yes, I am. I do systems research. This is the same standard I hold myself and my peers to. If someone asked me to review a systems paper that claimed to improve something, but had no results, I'd reject it. I recognize this is a blog post and not an academic paper, but my standard for "do I accept that this is a better approach" has not changed. And I have seen plenty of blog posts with experiments.
If you think about the problem space of Redis vs Varnish, it's intuitively obvious that Varnish deals with a wide variety of general data without many opportunities to optimize beyond general purpose algorithms such as an OS provides. Whereas Redis has specific data types often with small footprints, and very careful attention paid to the details of optimization for memory and disk usage.