Status Identity Issues in Crowd Sourcing: Transaction Costs or Reputation Effects? (mTurk vs. CrowdFlower)

Another interesting point about crowd sourcing - what does identity have to do with it? Now, due to the strong presence of scammers, many of the researchers in this area think that reputation is the best way to control quality with micro tasks. (E.g. http://behind-the-enemy-lines.blogspot.com/2010/08/reputation-integration-and-future-of.html) But I'm a contrarian sociologist, so I am not so sure that I would necessarily agree. To see why, I first need to point out the issues of social standing and transaction costs that factor into this labor market.

First, I started to think about transaction costs as a reason why micro tasking was popular for over qualified people like college students to make very little bits of money. (Leaving out the 'game-like quality' arguments here.) For example, as a PhD student, I can always go work part-time at something, but getting the job, training for it, and then settling into a routine of work that won't interfere too much with my research takes time and effort -- indeed, it would probably require a lot of searching with the current economy. On the other hand, I can work in little bits and pieces with Mechanical Turk, earning little money, but in a way that doesn't require a lot of startup costs. I can tell you from experience that motivation is there, and low barriers to entry (transaction costs) make it easy to sign up with mTurk and start working right away. Who doesn't already have an Amazon account?

However, should we also think about this in terms of a status identity-driven distinction? It might take a lot of effort to go and get a reasonable part-time job, but maybe the issue is also that it just feels wrong or demeaning to take a low skilled position. However, with mTurk, who of my friends are going to know that I'm doing this for extra money? Nobody, unless I tell them. Thus, this is work that the overqualified could do, easily, but without the negative reputation effects that could occur if, say, your friends knew you were working in fast food management instead of doing corporate law. Working on mTurk -- say if I were unemployed or staying home with kids or between jobs or just part time -- wouldn't necessarily have the same status identity hit. I could rationalize this to myself as "just earning some extra money" and it would be unlikely to generate negative reputation effects with my peers, because it is unlikely that they would find out I was doing this unless I told them.

And that, my friends, is why I think we can definitely find overqualified workers lurking on Mechanical Turk. (So long as they don't get burned too quickly by scammy requesters.) In other words, maybe we should be asking "to what extent are 'too good' workers using mTurk as a way to generate some extra money in ways that are a) easy and/or b) not identity-threatening/demeaning?" (This is also a nice explanation for why you might see overqualified people doing temp work, together with the ability to move into a good FT position...)

Testing Status Identity vs. Transaction Costs
A nice experiment to test this would be to see how well or quickly the same people completed a boring task using CloudCrowd vs. mTurk -- experiments anyone? The way that CloudCrowd works is to use a Facebook login to start working, which is what gave me the idea for this post in the first place, because I immediately went "eww.... I don't want to let my friends know I'm doing this low-status work."

CloudCrowd uses facebook and existing accounts to guarantee that people are 'real' by looking at facebook. But not only will this create the status identity issues I mentioned above, with one's friends finding out you were doing menial labor, but a corollary is that we might get in fact worse quality work from some people who would be just fine in the semi-anonymous mTurk scenario.

Here's why that might happen: for some people, requiring login with your facebook ID is just fine, and through social sanctions and reputation effects, there will be an incentive to provide high quality work. I imagine this works just fine for some workers. Using the issue of status identity, however, I can also see a quality-destroying incentive here: if I don't want to show my friends I'm doing this, I might make up a new facebook ID, and then with that "worker login" I would probably feel less pressure to do quality work. Heck, if I'm going to start creating fake IDs, I may as well make a couple of them, and maybe have a throwaway bank account as well, to make it easier to abandon if one was flagged.

So, to be academic about it, I would expect to see this the two-tiered quality effect, and it would become my testable hypothesis if I were going to do this kind of research. (But I'm not, so it's fair game, people!) Another alternative might be that overqualified workers would just not do this type of work if they had to go through facebook or another social media source that was linked to their visible personal profile.

Well, now I really need to stop messing around with mTurk and work on my real research...

The Weka API: Matrix class

Weka includes a matrix library in weka.core.matrix.Matrix. This library is adapted from the JAMA package, originally developed by NIST and the University of Maryland. The aim of the library is to support functions for numerical linear algebra.

Creating a Matrix Instance

To create a Matrix instance from jRuby, I use the following construction method. It takes a two-dimensional Ruby array as argument, and returns an instance of the Matrix class.

def make_matrix array_2d
  matrix = Matrix.new(array_2d.length, array_2d[0].length)
  array_2d.length.times do |i|
      array_2d[0].length.times do |j|
          matrix.set(i, j, array_2d[i][j])
      end
   end
   return matrix
end

For example: puts make_matrix([[1,2,3], [4,5,6], [7,8,10]])
displays

1 2 3
4 5 6
7 8 10

Solving a Linear System

The example within the Weka documentation solves the following problem:

Solving using jRuby is equally trivial:

A = make_matrix [[1,2,3], [4,5,6], [7,8,10]]
b = make_matrix [[0.1],[0.2],[0.3]]
x = A.solve b

Warning: although the matrices can be converted to strings using to_s, the result is limited in decimal places, which can be misleading. Better to display the solution using explict 'get' statements, such as:

puts "Solution: "
puts x.get(0,0), x.get(1,0), x.get(2,0)

Solution:
-0.0333333333333334
0.0666666666666667
-3.46944695195361e-17

Arithmetic on Matrices

The Matrix library supports several standard operations on matrices. It is easy to add, subtract, or multiply two matrices which are of the right 'shape'. For example, to find the 'residual' from the above solution: ||Ax - b||

r = A.times(x).minus(b)
puts r.normInf

(Here, 'normInf' is the infinity norm: the maximum value in the rows of the matrix 'r'. The library supports other norms. The popular "square root of sum of squares" is represented by 'normF', the Frobenius norm.)