-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
144 lines (132 loc) · 5.87 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
<!DOCTYPE html>
<html lang="en-us">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="">
<title>Crowdsourcing guidelines</title>
<!-- Latest compiled and minified Bootstrap CSS -->
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/css/bootstrap.min.css">
<!-- Custom styles for this template -->
<link rel="stylesheet" href="style.css" <!-- HTML5 shim and Respond.js for IE8 support of HTML5 elements and media queries -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/html5shiv/3.7.2/html5shiv.min.js"></script>
<script src="https://oss.maxcdn.com/respond/1.4.2/respond.min.js"></script>
<![endif]-->
</head>
<body>
<nav class="navbar navbar-inverse navbar-static-top">
<div class="container">
<div class="navbar-header">
<button type="button" class="navbar-toggle collapsed" data-toggle="collapse" data-target="#navbar" aria-expanded="false" aria-controls="navbar">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand" href="#"></a>
</div>
</div>
</nav>
<div class="container">
<div class="headline">
<h1>Crowdsourcing guidelines</h1>
<p class="lead">
Some tricks of the trade and common pitfalls to avoid.
</p>
</div>
<h2>Clear instructions are crucial</h2>
<p>
No matter how careful you are, you won't write the instructions clearly the first time.
There is no substitute for getting someone (a roommate or friend and eventually crowdsourced workers)
to try the task and provide feedback. (It's a good idea always to have someway that a worker can provide feedback, btw.)
</p>
<p>
Workers won't read your instructions, especially if they are long.
It's just human nature... they will skip ahead and try to start doing the actual work.
To combat this, insert quiz questions after each paragraph in your instructions to ensure they
got the point.
</p>
<p>
It's also a good idea (if you can) to make the UI so easy to use that the workers don't <i>have</i> to remember anything
(e.g., use hover or confirmation).
</p>
<h2>Some workers are spammers</h2>
<p>
On most crowdsourcing platforms you will need to perform some kind of quality
control to eliminate results from bad workers. Most approaches involve some kind of
redundancy: either asking multiple workers to perform the same task and looking for agreement,
or else randomly inserting <em>gold questions</em>, ones for which there exists a gold-standard
answer already known - and then blacklisting workers who get too many of these wrong. Note:
it is often useful to consider three kinds of answers to gold questions:
<ol>
<li>answers that are clearly right - if a worker fails to provide this answer, they aren't being diligent;</li>
<li>answers that are clearly wrong - if a worker provides this answer, they are also bad;</li>
<li>answers that are ambiguous or require some kind of judgement - where any response is ok from the POV of
estimating worker quality.</li>
</ol>
Many crowdsourcing tasks (e.g., image tagging) can require a mixture of these
types of answers.
</p>
<h2>Making results unbiased</h2>
<p>
Before seeing a question, a worker should be equally likely to provide any of
the responses one wishes to be unbiased.
</p>
<h3>Response defaults</h3>
<p>
A worker should have to explicitly make a selection. For example, when
requesting tags for an image, one should <emph>not</emph> default to "no" by
using checkboxes, as in this example:
</p>
<div class="example" style="width: 200px;">
<form>
<table>
<tr><td><input type="checkbox"></td><td>Person</td></tr>
<tr><td><input type="checkbox"></td><td>Animal</td></tr>
<tr><td><input type="checkbox"></td><td>None of the above</td></tr>
</table>
<input type="submit" value="Submit">
</form>
</div>
<p>
Instead, it is better to use radio buttons and explicitly make the worker
select "yes" or "no", as in this example:
</p>
<div class="example" style="width: 200px">
<form>
<table>
<tr>
<th>Yes</th><th>No</th>
</tr>
<tr>
<td><input type="radio" name="person" value="yes"></td>
<td><input type="radio" name="person" value="no"></td>
<td>Person</td>
</tr>
<td><input type="radio" name="animal" value="yes"></td>
<td><input type="radio" name="animal" value="no"></td>
<td>Animal</td>
</tr>
</table>
<input type="submit" value="Submit">
</form>
</div>
<h3>Response placement</h3>
<p>
Another way that results can be biased has to do with the order of possible
responses. People are more likely to select responses that are closer
to the top. Thus, when asking multiple choice questions, be sure to randomize
the order of the answers.
</p>
</div><!-- /.container -->
<!-- Bootstrap core JavaScript
================================================== -->
<!-- Placed at the end of the document so the pages load faster -->
<script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<!-- Latest compiled and minified JavaScript -->
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.1/js/bootstrap.min.js"></script>
</body>
</html>