From what I gather the article seems to indicate an candidate is given access to their code base to 'do a fix' remotely. Is this a viable strategy initially? Learning the application takes time. Also there is some risk that the candidate will run off with the codebase.
I'm sure they've thought of this, but I can see a situation where you default to a 'standard' audition task in it's own sand-boxed environment just to mitigate any theft issues.
The article stated you would be working in the area you wish to work in, so if you are a developer you will most likely be working on WordPress or something relating to it (I may be wrong, just a presumption).
Considering that Wordpress is open source and any candidate applying at Automattic as a developer will most likely be familiar or know of it, the chance of having a developer not being familiar with it in one way or another are slim, even more so if it's someone actually wanting the job.
If I was looking to be hired by a company working with open source, I would most definitely do my homework up fornt and familiarize my self with their "flagship product" as it were.
I'm sure they've thought of this, but I can see a situation where you default to a 'standard' audition task in it's own sand-boxed environment just to mitigate any theft issues.