Most of the 'Power of two choices' I've read about assumes the presence of a global queue (http://www.eecs.harvard.edu/~michaelm/postscripts/handbook20...) -- there's a parallel variety, but they go light on details in that text.

I'm unaware of how Heroku does things. I'd guess they dropped the global queue because it's unpractical (failure prone, not scalable as it's a single point of contention).

I'm mostly surprised to see people happy being able to handle 1 or 2 requests in parallel per instance in general. That sounds absolutely insane to me.