The requests will not be served at the same time, that's the whole point. If a request is routed to a busy dyno, you will have to wait that the previous job finish before being able to start yours.
Thing is though, it says that a Dyno is a Ubuntu virtual machine. In what sort of horrendous configuration can an ENTIRE VM serve only a SINGLE REQUEST AT A TIME?!
That is utter madness, and the validity of the argument depends on whether it's the Heroku or this dude's fault that the VM is serving only a single request at a time (and it taking >1sec to handle a request).
But one VM can host multiple Rails instances, each on different port. That's what Passenger or Unicorn do, acting as proxy of group of locally spawn Rails instances.
I believe the article is saying that this wait will only occur with Rails due to it being single threaded? Reactor pattern frameworks like node don't suffer the same issue.
You are exactly right, but I would hope that someone wouldn't be using Node at all if CPU bound requests are even remotely commoneplace on their app. But of course, if they built their app to spawn processes when this happens, you end up with a case of a need for intelligent routing.
At some point you hit memory limits, disk IO limits, or simply a connection limit. It doesn't matter what limit:
If you have some requests that are longer running than others, random load balancing will make them start piling up once you reach some traffic threshold.
You can increase the threshold by adding more backends or increasing the capacity of each backend (by optimizing, or picking beefier hardware if you're on a platform that will let you), and maybe you can increase it enough that it won't affect you.
But no matter what you do, you end up having to allocate more spare resources at it than what you would need with more intelligent routing.
If you're lucky, the effect might be small enough to not cost you much money, and you might be able to ignore it, but it's still there.
> But no matter what you do, you end up having to allocate more spare resources at it than what you would need with more intelligent routing.
I think we have to remember that the "intelligent routing" in question here is actually marketing-speak for "one request per server." Are you saying that when your servers can only receive one request at a time, you will necessarily need fewer than if your servers can handle three requests at a time but are assigned requests randomly?
No it's not. "Intelligent routing" means giving each server exactly as many requests as it can handle, and no more. If your servers can handle 3 requests each, using intelligent routing to make sure they never get more than 3 will blow the latency of the random method out of the water.
