Posted on May 21, 2011 @ 04:07 GMT in Projects
Recently, my fiancée was investigating the use of Amazon Mechanical Turk (AMT) as a means of paying people for participating in her latest empirical research. The study involves taking an online survey that she created at Qualtrics.com(her University has an account).
AMT has a built-in feature for creating surveys. Unfortunately, it is rather limited in its implementation:
- There are no options for customizing the structure of surveys (e.g. multiple pages, order of presentation, etc.)
- Customizing the look and feel requires knowledge of HTML and CSS. Skills that my fiancée and many others in her position do not possess.
If you are creating a very simple survey with no more than a few questions, and if you are not terribly concerned about the survey’s appearance, then the built-in survey feature of AMT is probably the best option for you. But if you want a more complex survey and more control over its presentation, then you may be better off choosing from one of the various providers of web-based survey software.
Unfortunately, AMT’s web interface does not seem to allow for creating HITs that redirect the user to an external website. It was at this point that my fiancée asked me to look into the issue to see if there was a possible “programming” solution to the problem.
My first stop was the AMT requester sandbox site where I created a developer account. I then ent through the same procedure on the AMT worker sandbox site. I now had access to the AMT sandbox as both a requester and a worker, which allowed me to create and test my own HITs.
First, I examined the “Design” section of the AMT requester site. I chose to work with the “Blank Template”, which proved to be the exact opposite of what its name implies. After removing the contents of the “blank” template, my first thought was to just insert an
<iframe> and point it to the URL of my fiancée’s survey.
<iframe width="100%" height="100%" src="YOUR_URL_HERE"></iframe>
Unfortunately, this solution has several major flaws. First, AMT’s web interface requires that your HIT template contain at least one question, i.e. an
<input> element with a name prefixed with “Q1”. Adding an
<input>element of type “hidden” satisfies this requirement, but it also leads to the next problem. Now that the survey meets the requirement of having at least one “question”, AMT inserts a “Submit” button at the bottom of the frame. The idea is that the user answers your “question” and then clicks the “Submit” button to claim their reward. In our scenario, the “question” that we want the user to answer is the survey in the
iframe. However, there is no obvious way of preventing the user from clicking the “Submit” button before completing the survey. In other words, there is no way to prevent the user from getting paid for doing nothing! Author’s note: If you know of a method to prevent this from happening, please let me know.
I did some digging around in the AMT developer resources, and I discovered that, through the MTurk API, it was possible to create an “External HIT”, which loads a specified URL as the contents of a HIT. Creating an external HIT has a major advantage over the previous solution in that it causes the “Submit” button to be disabled. Therefore, the user can not submit the HIT and get credit for doing nothing! Of course, after the user has completed the survey, they deserve to get paid, at which point we want to enable that button. To do so, we must send a signal to AMT when the survey is over. When, AMT loads the survey URL, there are three parameters appended to the URL’s query string:
workerId. To signal AMT that the survey has been completed and to allow the worker to claim their reward, we must send a GET or POST request to
https://www.mturk.com/mturk/externablSubmit that contains the same three parameters. Easy enough. Except, how do we determine when we’re at the end of the survey?
Qualtrics.SurveyEngine.OnEndOfSurvey, with my own function that signals AMT that the survey has been completed. I accomplished this by creating an invisible
<input> elements. This allowed me to extract the values at the end of the survey. It also has the side benefit of including those values in the survey data allowing the surveyor to compare them to the values reported by AMT.
All code is available on github.