This is a lovely idea, but I'd like to share with readers an experience I had a while ago.

I was looking to outsource some coding, so I advertised and chose someone that seemed reasonable. I asked for some sample code doing similar things to what I wanted, they responded with some snippets that looked right, so we entered an agreement. I would provide some sample input and output, they would provide code that produced the right results. Once I verified that it did what was required, I would pay.

It was a problem domain that wasn't overly complex, and I thought it would be an ideal task to outsource. It was a lot more complex that the example I'll give here, but I want to give you a flavour of how it all went wrong. This is not the task, just something to explain how it went wrong.

I asked for code that would return "True" if the 2nd and 4th characters in a supplied string were the same, but otherwise all characters were different. So here are some examples of input and output:

    severity    True
    vagabonds   True
    unknowable  True
    unsnarled   True
    telegraphs  True
    robot       True

    faunas      False
    flimsily    False
    freighting  False
    rummaging   False
    mosaic      False
    shushes     False
    conflicting False

Remember, the actual task was described in full, was much more complex than this, and we were simply providing some example test cases.

What we got back was the equivalent of this:

    def test_string(x):

        if x == 'severity'    return True
        if x == 'vagabonds'   return True
        if x == 'unknowable'  return True
        if x == 'unsnarled'   return True
        if x == 'telegraphs'  return True
        if x == 'robot'       return True

        if x == 'faunas'      return False
        if x == 'flimsily'    return False
        if x == 'freighting'  return False
        if x == 'rummaging'   return False
        if x == 'mosaic'      return False
        if x == 'shushes'     return False
        if x == 'conflicting' return False

So to anyone who tries to specify their coding tasks by using tests - beware. There are coders who will strive to meet the exact letter of the requirements, and then demand payment.

For what it's worth, we never resolved this and eventually paid an agreed amount to make them go away. We took this as a reasonably expensive lesson about some of the potential pitfalls of out-sourcing.

Hey, thanks for the example. This is definitely something we want to avoid with this platform and have thought of ways to mitigate it.

One way is we will allow the buyer to further test submissions as black boxes to see if the output matches their expectations.

The second way is with a rating system as extra information for the buyer and an incentive for the coders to write good code.

How do you avoid this scenario:

  * Customer creates job spec
  * Customer creates white examples
  * Customer creates black examples
  * Coder bids
  * Customer accepts
  * Customer sends spec
  * Customer sends white examples
  * Coder delivers code
  * Code fails black examples
  * Customer refuses to pay
  * Coder claims code meets spec

??

The platform is weighted in favour of the buyers. What I mean by that is the coders compete to win the payout. The buyer doesn't see or own the source code until they pay. Many coders can submit solutions and the buyer can pick the best one to purchase. It's up to the buyer to query the submission fully as a black box and to trust the rating system.