current trends in development research

Research methods are conventionally divided into quantitative, qualitative and participatory each with differing underlying approaches, tools and techniques. Quantitative, qualitative and participatory approaches have different disciplinary origins, developed distinctive tools and each has developed its critique of the other approaches. In development research quantitative methods have typically been the main focus, with qualitative and participatory methods often relegated to desirable 'frills'. This is partly because of the overwhelming emphasis in many development agencies with economic economic growth and economic dimensions of poverty. In many development agencies the concern with quantification intensified over the 1990s with requirements for performance assessment and targets in logical frameworks. Pressures for quantification have further intensified to demonstrate progress on Millenium Development Goals, 'scaling up' of impacts and macro-level change.
Traditional disciplinary divides are however becoming increasingly breached. Moreover new tools and new solutions to shortcomings of old tools are continually being developed. Increasingly the emphasis is on developing an appropriate integrated mix of research methods.

Quantitative Methods

Quantitative methods as they are commonly conceived derive from experimental and statistical methods in natural science. The main concern is with rigorous objective measurement in order to determine the truth or falsehood of particular pre-determined hypotheses.

• the main focus is on measuring 'how much is happening to how many people'.
• the main tools are large scale surveys analysed using statistical techniques. Quantitative measurable indicators relevant to the pre-determined hypotheses are identified and combined into questionnaires.
• questionnaires are then conducted for a random sample or stratified random sample of individuals, often including a control group.
• causality is assessed through comparison of the incidence of the variables under consideration between main sample and control group and/or the degree to which they co-occur.
• in large-scale research projects teams are composed of a number of skilled research designers and analysts assisted by teams of local enumerators.

For easily accessible overviews of the strengths and pitfalls of different statistical techniques see the website for Statsoft
For access to many further resources see the quantitative methods, statistics and quantitative database sections on the University of Amsterdam 's SocioSite.


Qualitative Methods

Qualitative methods have their origins in the humanities: sociology, anthropology, geography and history. They differ from quantitative methods in aiming, not primarily at precise measurement of pre-determined hypotheses, but holistic understanding of complex realities and processes where even the questions and hypotheses emerge cumulatively as the investigation progresses.

    * typically focuses on compiling a selection of microlevel Case Studies investigated using a combination of informal interviews, participant observation and more recently visual media like photography and video.
    * questions are broad and open-ended and change and develop over time to fill in a 'jigsaw' of differing accounts of 'reality', unravelling which may be said to be generally 'true' and which are specific and subjective and why.
    * different sampling methods are combined: different purposive sampling techniques, identification of key informants and also 'random encounters'.
    * Causality and attribution are directly investigated through questionning as well as qualitative analysis of data. Computer programmes are used to deal systematically with large amounts of data.
    * typically requires long-term immersion of a skilled researcher in the field who engages in a reflexive process of data collection and analsysis.

For access to many further resources see the qualitative methods sections on the University of Amsterdam 's SocioSite

The Forum for Qualitative Research website brings together resources and debates in English and other European languages.

For overviews of computer analysis software see Lewins, Ann and Silver, Christina (2004) Choosing a CAQDAS (Computer-Aided Quatlitative Data Analysis) Package: A Working Paper

Participatory Methods

Participatory methods have their origins in development activism: NGOs and social movements. Here the main aim is not so much knowledge per se, but social change and empowerment - and this wherever possible as a direct result of the research process itself. In particular it seeks to investigate and give voice to those groups in society who are most vulnerable and marginalised in development decision-making and implementation.
The participatory process may involve small focus groups, larger participatory workshops or individual diaries and diagrams which are then collated into a plenary discussion. Participation (and hence sampling) may be open or carefully targeted to particular social groups. Larger meetings may be subdivided into what are assumed to be more 'homogeneous groups' or groups with complementary information.
Participatory research typically uses and adapts diagram tools from farmer-led research, systems analysis and also oral and visual tools from anthropology, though many commonly used tools have also been developed by NGOs and participants in the field. Use of diagram and oral and diagram tools makes both discussion and analysis accessible to non-literate participants and across language groups. Through sharing their different sources of information participants themselves may increase their understanding of development issues and the problems they face and develop solutions, as well as giving more reliable and representative information to researchers. In some cases local people themselves conduct research following initial design of specific tools and training. Some recent NGO innovations propose doing this on a large scale.

towards an integrated methodology

Wherever possible, most research will use an integrated methodology which builds on the complementarities between methods in order to build on strengths, crosscheck and triangulate the information which is most crucial for addressing the particular research questions concerned and also try to disseminate information in different ways for different audiences in order to ensure, as far as possible, benefits for outcomes for participants.
Participatory methods should play a central role at all stages from conception, through piloting and refinement to the research proper and then finally dissemination. Using participatory methods as the 'first port of call', has many advantages in terms of rapidity and reliability of collecting many types of qualitative as well as quantitative information, manageability in terms of time and resources and also its potential for contributing to the development process. Participatory methods are not necessarily a substitute for other methods, but enable much more cost-effective targeting of everyone's time and energy on those areas of the research for which participatory methods are rather more problematic. They are also generally an essential component of research dissemination to those participating in the research, a stage which is commonly ignored and omitted.

What is the Research Methods Knowledge Base?

The Research Methods Knowledge Base is a comprehensive web-based textbook that addresses all of the topics in a typical introductory undergraduate or graduate course in social research methods.  It covers the entire research process including: formulating research questions; sampling (probability and nonprobability); measurement (surveys, scaling, qualitative, unobtrusive); research design (experimental and quasi-experimental); data analysis; and, writing the research paper.  It also addresses the major theoretical and philosophical underpinnings of research including: the idea of validity in research; reliability of measures; and ethics.  The Knowledge Base was designed to be different from the many typical commercially-available research methods texts.  It uses an informal, conversational style to engage both the newcomer and the more experienced student of research.  It is a fully hyperlinked text that can be integrated easily into an existing course structure or used as a sourcebook for the experienced researcher who simply wants to browse.

RESEARCH METHOD

Planning is series of program plan used to obtairf the accurate result, by using a varied research method.
In line with this, Winarno Surachmad (1980:131) states that: “Method is the main way used to obtain one objective, for example to evaluate hypothesis truth, by using instrument and curtain technique”.
While Sri Adji Suryadi Prawiradihardja (1973 : 4) states that “Research is an action which means finding out and to evaluate something accurately. Accurately means as exact as possible coming near the turth, so the result comes near the truth”
Based on the two opinions above, it can be concluded that research draft is series of way to find out and manage, examine and evaluate the truth accurately so the result can be obtained to be suitable the objective. Because in this research the research draft in a technique used to find out evaluate the problem, which will be research.
According to characteristics, their research is a kind of action research, namely research aimed to increase the teaching learning process of reading comprehension through descriptive study of difficulties.

The carrying out of the research in held collaboratively namely the research involving another person beside the researcher that is both as participant and observer. This research uses step plot (planning, actuating, observing, and reflecting presented in three cycles) after being previous obtained the main problem about how to increase the ability of reading comprehension in the third year student of SMAN I Rambipuji Jember.
The research is planned to be carried out three cycles in the third year through steps illustrated as follows.

Social media, curation and watching tv these days

Lately I've been forming an opinion about the importance of categorization, comparables and curation on the web. I know everyone is keen on social media, and companies are scrambling to keep up, but perhaps we should think more about the contexts in which having reviews, comments, and other forms of interaction make sense. I am certainly not against social media, as it provides a very interesting counterpart to published information, and I have to confess that I am just the kind of nerd that loves to read amazon product reviews. But at other times, say when I am looking for a nature documentary on Hulu, I find the information, well... less than helpful. (Do I really need to see anti evolutionist comments when I'm looking for a nature documentary?) Let's think about this more.

The Hulu problem is that entertainment products such as TV don't appeal to all groups. I can easily love things that other people like and vice versa. At present, the type of user reviews we see online aren't distinguished by different marketing segments, and there are of course many people who think that others should think like them. Hulu is also a case where I'm often very aware of my own preferences, so in that case, I don't care for a long description. I'd prefer something like stars, expert reviews or really, nothing at all. People should save those opinions for dedicated fan / anti-fan sites.
Social scientist spoiler: It might also be interesting to get some observed behavior, rather than opinions, such as "total completed views" divided by "total views started" or something of this nature. Or if reviews were tagged with this information, I would be better able to see who has full information, and who is just writing a snap judgment based on 5 minutes of watching a complicated drama.

This problem of whose views count seems to be showing up now with the implementation of google news, as the company struggles to figure out which blogs are news, which are opinion, which are garbage, etc. Obviously, this is a problem of curation. When I read the New York Times, I know that the company is exercising judgment and restraint over its writers, and if something slips by, they will print a retraction or update. But what about if a prominent writer publishes a blog? What about scientists? (A favorite example is the Becker-Posner blog, which is very entertaining.) Actually, this example makes me think that perhaps academics realize this distinction much better than anyone else, because we know that what is published as peer-reviewed research in journals or as books with university presses will be held up to the highest standard. When we do a blog, it has many advantages: it can reach more people, we can speak in a conversational fashion, it will be updated quickly, and so forth. But this work is categorically different than the highest standards set by peer reviewed research journals. (I am not decided about open access journals, but certainly to grad students who wonder whether they should spend more time on their articles or blogs, the answer is clear. Having a lot of followers on your research blog won't get you tenure, and because it is not vetted by experts in your field, I am not sure that it should ever serve that function, even if you may eventually get points for public engagement.)

Another clear and big problem with curation involves my frequent hobby: watching TV. (Sadly, as an academic, reading feels too much like work, and so aside from the newspaper, I am not reading much outside of work, unless I have been on vacation for a while.) With factors like the digital switchover making it difficult to receive more than a few local channels at once without endless fussing and refinding channels, as well as the growing number of sites to watch shows online, we are moving away from the power of the major networks to provide a curated package of shows. This is no problem when you already know what you like -- just buy the show in physical or virtual form and watch it sequentially. This works great for me, especially with intricate shows that have a lot of character development, which I used to not watch until the dvds were out, due to my busy and erratic schedule. (Perhaps this correlates with why I liked one off episodes of shows like CSI so much?) 

On the other hand, now that I've watched most of the episodes of shows that I liked, and have followed series, and got rid of my tv in moving and digital hassles, I am finding it increasingly difficult to figure out which new shows to bother watching. My extreme frustration with networks for cancelling two shows I really liked--life on mars and defying gravity--is part of this problem, pushing me to go after older network shows and new cable shows, which have less chance of being cancelled just after I've started watching regularly. The question is how to find those things that I'd like to follow? There are also times when having someone else curate would be better, because I just can't be bothered to do a big long web search. The advantage of cable tv is that you can just turn the thing on and start watching a lineup of shows picked by the network that follow both a sequential and temporal order. This is one of the reasons why I like listening to dj sets and shows as a way of finding new music; although TV is somewhat different from music, there is still skill involved in that selection process, putting together a package of things that go together. 

Spending a lot of time finding new things is more of a problem for busy me, as opposed to being something fun for people who have a lot of time on their hands to find and discover things that you like, like teenagers and college students. The search process here is also likely to be related to their process of identity discovery, finding things that are popular or rare, current or historic. 

So in my view, the curation and collection are the true missing elements to both Hulu and Amazon, because although they do have certain types of collections, the collections are static as opposed to dynamic. The TV programming TV of old was dynamic, in that you had a relevant collection of shows at a certain time, which could seem out of date at a later time. Think of Christmas episodes of sitcoms. This temporal movement was also helpful to weed out things that I would watch, but didn't LOVE -- ok, I'll watch dancing with the stars because it's there, but I'm not going to go online to watch old episodes because it's not my thing. I wonder how much this "good enough" vs. blockbuster effect is to blame for changing viewing patterns with buying episodes ala carte? (A perfectly fine show, but I certainly wouldn't be inclined to pay for it... which I suppose relates to the arguments about bundling that are still being resolved around cable tv.)

This magical filtering and movement also relate to why watching tv was such a pleasure for me before - although I had to tune in for a favorite show, to just watch something took no more effort than a few clicks of the remote.  In switching to watching online, the seductive brain rot of old TV is replaced with an endless process of overfilled netflix queues, too many show subscriptions on Hulu, and the sense that I am somehow behind on things with my online tv viewing (!). 

To think about cable, perhaps what happened with the proliferation of cable channels was an ever splintering specialization that gave us channels that would fit a mood or a wish, but lacked the generality of the average network. This was a gain, but it was also a loss, because how many hours of programming could one possibly sit through with the cooking network or fitTV? Then the more successful of these networks generalized up again - witness the personality chefs, to take the place of talk show hosts or news anchors. But that's when they lost me, because I'm not a just the lifestyle cooker, who learns about products to buy them at whole foods, but someone who wants technical details and expertise. 

So I imagine that there must be a balance between specialization and generality in show content and viewing audiences. But this is where online content is brilliant, because the same shows can now be recollected into different packages now, for example with a customized daily watching of 2-3 hours for type 1) the lifestyle cook, type 2) the travel and food culture type or type 3) the technical shows watcher. 

Well, this post is getting long and I'm not sure where I've gotten to, but I can at least say that this seems to be what happened with the networks vs. cable channels and police dramas. So perhaps the model isn't as unfamiliar for big media as one might imagine, and the details are more in who curates and how well they are able to construct viewing sets that are updated frequently enough for daily watchers, while still being able to engage those of us who might check in once a week. Here's hoping that we can create playlists for TV shows, and eventually someone will put together a regularly updated collection or channel of shows for me to watch online, so that I can, for example, have high quality nature programs for relaxing evenings. I'd probably even sit through the advertising.

Research organization using OneNote

I think I can safely say that for the past 3 years I've been looking for the ideal way to organize my research materials. By the time I went to do my fieldwork in New York, I had already started on a big paper collection, and since that time it has only expanded. (Yes, how many people do you know that fly off with a suitcase of papers and journal articles. I am ashamed to say that has been me.)

In starting on the ethnographic work in earnest back in 2008, I did a survey of qualitative analysis software, and after testing Atlas.ti, MaxQDA and nVivo, I picked nvivo to work with. (I later ran across Qualrus, which some people like, but I had already bought nvivo so I didn't review it particularly.) Now nvivo is a very powerful tool for qualitative researchers and it definitely helps us to be better organized. But it wasn't quite the holy grail that I had been seeking for research organization, although I definitely will keep using this type of software.

I found that I wasn't using it quite as often as I would have imagined for a key reason: it takes a long time to load in XP, and having said no to Vista, I'm currently waiting to have the time to move over to Windows 7. So it wasn't the easy interface that I could just pull up to make notes with the same way as writing in a physical journal like I do when I'm out interviewing.

Now, to take on that function, I tried a few different things in the last few months - evernote, Mendeley, OneNote, back to Mendeley, and now back to OneNote. Now that I have been working with OneNote for a while, I have to say that I absolutely love it. It apparently was started as an initial technology for tablet PCs, and has great integration there (so I hear) but it is so much more. I suppose this is the kind of epiphany that mac users had when they stared using Devonthink, but for those of us who are on PC, there wasn't this great kind of option.

You can do an online search yourself and see all about onenote and why people like it, but let me tell you why I like it for research.
1. It's intuitive and easy to use
2. There are many more useful features once you get going with it
3. Easy searching in it
4. Easy to draw
5. Stays pretty organized by default
6. Easy way to hold all those notes and scribblings

Now, I won't say that there are no problems. It isn't great with PDFs, and does this silly 'print to OneNote' thing for those. If you make a table, you can't do an autosum, as one might expect. If you draw, there's no way to group the drawing objects like in word or powerpoint (sigh). There is no OneNote mobile for blackberry.

But regardless, it is making me so happy to use this and I feel that I'm now able to put together my research notes and quotes in a way that really makes sense, and is searchable. This is definitely worth the $80 or so that I paid for it, and to be quite honest, it should probably be included with all versions of Office, because it really is just that useful. But really, don't just take my word for it, download the trial version and give it a whirl.

More Suggestions for Your First Articles as a Graduate Student

Writing articles. This can be a pretty stressful task if you are in a research oriented program and not working on a project where you are expected to publish with your advisor or other faculty. I envy other graduate students who have had that kind of excellent apprenticeship. On the other hand, you already know how to write, so it is possible to learn how to write scholarly articles. 

One of the key things you should know is that publication is usually a very long process, so you should push yourself to start on it early in your graduate career. Unlike blogs or newspaper articles, going through one or more rounds of peer review will basically ensure that your work will be properly vetted. In fact, another advantage that of this longer process is that I have realized that it allows me to let research papers go out before I actually "feel ready" or feel that the writing is perfect. With long time horizons, there will still be plenty of time to make sure that I'm going to be happy with the final product. (Besides, almost all articles will require revisions to respond to the peer review comments.)

Another key problem is having data to write about. As a social scientist, it is usually not sufficient to just write a theory paper or a literature review, meaning that some kind of data must be processed and analyzed. You might be gearing up to do a large survey or international research for the dissertation project, but that doesn't mean you should wait to find publishable pieces. It would make sense to use a smaller project or class assignment for this purpose instead, and while that is easier for quantitative researchers to collect some useful bits of data, qualitative researchers can do this as well. One of the easiest ways to get some reasonable empirical data is to just do a quick IRB and a local participant observation project on some phenomenon; even without interviews, you could get something interesting with a reasonably low hassle or time factor. (In my own case, I went to Chicago auctions in the beginning and watched bidding, which was helpful for later observation on my larger project.) Just make sure to apply for IRB clearance beforehand -- although this is not required for a class project, it is required for publishing with human subjects research, cannot be applied for retrospectively, and it is another good thing to practice.

Once you have written a draft paper, based on data, for a class or otherwise, it might be a good idea to submit it to a conference, and/or present it at a scholarly workshop. This will help you to get some more input on the theory and methods of your study, and be sure to write up comments soon after so that you don't forget them.

Now comes decision time, and you should critically assess the level of quality of the work. Be honest with yourself about quality and think about the current paper length and most interesting findings to focus on. I don't advocate putting out sloppy research, but even if you realized that you had some major flaws, most studies should have some interesting parts that you could focus on, and feel free to round file the other parts. (Or save them for later, because you never know -- however, remember that you will need to keep up your IRB clearance if you later want to reanalyze human subjects data with identifiable information, otherwise you would need to strip out any names and identifiers, deleting/shredding any files that were used to keep a real names list of subjects.) Thinking about the parts of the research to use can be stressful, which is also why taking the work to a conference or workshop for peer feedback is helpful - helpful to help you realize what parts of your findings were interesting to other people and the types of related literature that you will need to draw from. 

Once you have a reasonable paper, with some interesting parts, and not too long, then you need to find a potential publishing venue and revise the work for submission. I think revision is some of the hardest work, but it does get easier, and it certainly improves the quality of the findings. This is also the stage where you should start on the focused literature review. One of the typical problems I see with student papers is when someone has done the literature review for a class, but then the actual findings of the data point to a completely different theoretical area. Don't fall into the trap of being lazy about this problem, because it can cause rejections, and besides, you might be relying too much on the theoretical literature and not focusing enough on your actual data. Fixing this problem by cutting down and refocusing the literature review is part of the process.

A typical timeline for a paper with a small dataset would be 1) gathering data over a few months, analyzing and shaping the paper, 2) revising this for a workshop or conference, and then presenting, which can add a few months to a year, 3) Respond to peer comments from the presentation, pick a potential venue and revise for submission over 2-3 months. Depending on the journal, you might get a rejection, in which case you return to step 3, incorporating comments and picking a new journal target. When you do get a revise and resubmit, 4) you should respond to reviewer comments systematically, reshape the article to make sure that theory and data match, and tighten it up to meet the word count. Make sure that you really pay attention to your writing at this point, because it may be the last stage where you can make major revisions. Then 5) you will get a page proof, where you can review to check for any glaring errors and make a few minor fixes. Hopefully this gets you to 6) scheduled to publish, although that could be an unknown time in the future depending on how long the article queue is for the journal. 

Clearly, we all want major articles in major journals, but I think that it is less paralyzing to start out by focusing on subfield journals with shorter articles. Although I had a... disappointing experience with one such journal, and ended up pulling the manuscript, in other cases, I found that working with these smaller journals was helpful in terms of time, getting great feedback from the editors, and I also had helpful comments from reviewers. These focused journals won't have the larger readership of a generalist journal, but it is a nice way to start building your scholarly reputation. (Another target would be a graduate student journal, but subfield journals are probably going to be higher impact.) 

In order to focus on learning the mechanics of writing up a research article, an easy filter would be to look for journals that have a 5,000 to 8,000 word limit. That type of limit is also an easy way to figure out whether you have "enough" for an article - more than likely, you'll find out that you have way too much for just one of these. Throughout this process, you should expect to do many revisions, and they can sometimes be painful, such as figuring out which parts to cut to reduce the word count, but the advantage is that you will be (or at least should be) increasing the clarity and focus of your work. 

As part of the process, one of the best tools that I've found is to use a reverse outline. This has a lot of potential uses. 

1. This is helpful for figuring out how to revise the work for the initial journal submission - diagram article structure from sample papers of the journal where you want to submit, and use that as a template. This will be a nice mechanical exercise, but will help you to become acquainted with the stylistic conventions of the journal and how other people write about their research. Depending on the length of the article, I start with a first pass of the different sections, then eventually do a few on a paragraph by paragraph basis, asking "what does this particular paragraph do in terms of the argument?"

2. In order to focus your writing, do a reverse outline on your own paper. This will be helpful to figure out which pieces can be cut, and if you need to rearrange for clarity. 

3. When you finally get to an R&R, many months later, you can diagram your own writing as a way to figure out what the heck you were talking about, and show you where you need to make revisions.

4. When you feel stuck -- do this, either diagramming a different article for the journal or doing this with your own paper. I find that it almost always helps me to move out of that terrible place of not knowing quite what to do next with the article, or feeling lost in a sea of details. 

A note about reviewer comments: regardless of whether you were accepted or not, a good practice is to make those into a to-do list -- you don't have to do them all, but this helps to depersonalize them and make them into action items. It is also good form to to do this for a revise and resubmit, because as you do the edits, you are making a list of changes. Then you can include this list and your comments about which changes you did or didn't do, to include with your revised manuscript. It introduces a nice element of accountability into the process. 

My EZ solution to get people to pay for online news

Sigh. I've been very disappointed with garbage SEO content from web searches lately, which only reaffirms my feeling that there will have to be a new type of shakedown with web providers, rankings, and so forth. (Lame SEO as the new spam? Maybe.)

The issue comes back to my previously mentioned problem of needing online curation methods. Which websites do we trust? Which provide the best* information?
*Yes, that's subjective, and I know plenty of people would take TMZ over NYT, but still.

 So how does this relate to the issue of newspapers and online content? Well, if you break the problem down to essentials, the reason you'd pay for a paper newspaper vs. just hearing your neighbor tell you about the warehouse fire down on the docks is an issue of quality and reputation. You pay for reputation. (Let's exclude the entertainment-purposed newz lite for simplicity.)

Online news provision changes the game though, now information can spread, it can travel, and it doesn't take an economist to figure out that people don't like to pay for things that they can get for free, or where they can get acceptable substitutes for free. Question is, how do you then monetize high quality news online then?

Well, I've got a solution. In an online age, where information travels quickly, monetizing the news is difficult, because there are so many alternatives. But you can certainly monetize community participation. Let's think about this a bit more...

What's the difference between reading the news at the New York Times and paying for some content, vs. finding an AP story elsewhere for free? Well, NYT is reputable, and hey, they don't have a bunch of flashing graphics ads that are distracting. (Yes, I would pay for no ads versions, but I think I'm in the minority on that one.)

Actually, let me change examples to a site where this will work a lot better, and use an example close to my heart (as an academic) to talk about The Chronicle of Higher Education. What the Chronicle offers, and you wouldn't find on other websites that might repost the same or similar content, is the comments section. Take the following randomly selected story about professors being required to be present in their offices for a minimum number of hours per week here: http://chronicle.com/article/Professors-at-U-of-North/124308/. As you might find from a quick skim through the rest of the website, the Chronicle requires subscriptions to view some articles, but not all of them. However, almost all articles now allow comments, and with many things, you get quite a healthy argument going under the fold.

Which brings me to my point - the Chronicle may be charging for some content, but I bet they would convert a lot more people by switching to charging for the right to participate in the conversation and making comments. Yes, moderation issues and TOS issues will probably have to be worked out to disallow spam, and of course with an audience of PhDs and university administrators, you usually get very high quality comments, which won't be the situation for many other news providers. But these caveats aside, I think it would be pretty effective to charge readers for the right to comment, rather than the right to read material. Basically, we've got it wrong - forums used to be where you could get free information, but maybe the future of news is that you only get to participate in forums if you pay, whereas the news itself is free.

Of course, this isn't a perfect solution, because it could have negative effects like shifting news providers toward comment-worthy or controversy heavy stories, it would make active management of forums essential, you would miss out on intelligent comments by people who wouldn't pay, it might be undemocratic by not allowing poorer people to participate,* the commentary might drive away some potential readers, and who knows, I could be wrong about the majority of people being willing to pay to comment. So it's probably not the panacea, but it would be an interesting experiment to see how well this would work to monetize high quality news and reporting, vs. the paywall schemes that are being rolled out lately.
* This could be mitigated by allowing editorial selection of some comments for free, the same way letters to the editor are sorted and chosen.
** Also, we do still need a viable micropayments system to capture the nonsubscriber / infrequent visitor issue, which decreases the 'barriers to purchasing' in technical terms. Hello, Google? Paypal? Why isn't one of the big boys working on this?

Mechanical Turk and Artificial Reviews

So I've been very interested in the whole mTurk phenomenon, with crowdsourcing small tasks. I decided to start doing tasks myself in order to see what things were best for this type of work, but frankly, I was appalled by some of the uses. I've seen solicitations for newsletter signups, which are bound to get someone spammed. I've seen solicitations for "please post 3 comments to the following blogs" to artificially get a conversation going. I've seen requests for Facebook follows/likes/etc., requests to break CAPTCHA's, requests to "download college material from this site," and "fill out a free credit report online." (If that last one isn't a serious identity scam, I don't know what is.)

Now I am able to see that this is part of the engine behind artificial website rankings and reviews, to pump up garbage, and game the ranking metrics. Here's a terrible one for academics: just today, I saw someone use it to artificially bump up his SSRN downloads, which I happened to report to both SSRN and to mTurk. I also 1-starred the book who's PR company was soliciting 5 star reviews on Amazon.

This brings me to an interesting point, though, because a whole community of people has evolved around mTurk, and some of them have moral stances, flagging questionable tasks like this. (See http://turkers.proboards.com/) The sad thing is though, not everyone has this kind of moral compass or perhaps they need the money/just don't care. I can understand those motives even if I don't condone it. So I have to say that this experience has really changed my view of what is "real" on various ratings and review sites. We might think those things are easy to moderate out, but some of the requests are really so specific as to require "at least 200 facebook friends" or other qualifications, and of course the sanctioning power of reputation effects doesn't matter if someone has already created an artificial online presence or several. (See what I mean by googling up "black hat SEO" where people will proudly talk about creating multiple artificial identities and "even using them to talk with one another.")

Ok, that last one is a little involved for the mTurkers, and could still happen even before the internet. But if there are people who will post bogus reviews for $0.05 each, at least there are a few people who will probably feel the moral outrage at this kind of thing, like myself, and post a negative comment on some of those same sites. At least that's what I'm hoping.

But I can easily see how this is going to become an increasingly difficult business to check and to moderate. It is going to require new and perhaps more sophisticated methods of doing marketing research, for example, because it is just so easy to post a "help me vote for funding for X medical disease online" task on mTurk--I actually saw this one--and if one just trusts the survey results, you might not be getting anything near a real sample. I'm just waiting until I see one along the lines of "buy one share of this company stock."

That said, mTurk does have some great potential, and I'll have more later on the useful and interesting tasks you can do with mTurk. However, I have come to the conclusion that if Amazon wants to police this better, they need to moderate requestors better (ala Amazon stores), and/or create an "approve this mTurk task" as an ongoing HIT to clear each and every request, which would work nicely with new or otherwise questionable requestors. I am sure that if they police things, new blackhat sites will spring up, but since Mechanical Turk is associated with the Amazon brand, they do need to be careful not to let it be the wild west, or they could suffer their own negative reputation effects.

 Read More
A nice academic blog on Mechanical Turk: http://behind-the-enemy-lines.blogspot.com/
Interesting labor regulation issues from the crowdflower blog: http://blog.crowdflower.com/2010/06/regulating-distributed-work-part-three-why-its-a-good-idea/

Status Identity Issues in Crowd Sourcing: Transaction Costs or Reputation Effects? (mTurk vs. CrowdFlower)

Another interesting point about crowd sourcing - what does identity have to do with it? Now, due to the strong presence of scammers, many of the researchers in this area think that reputation is the best way to control quality with micro tasks. (E.g. http://behind-the-enemy-lines.blogspot.com/2010/08/reputation-integration-and-future-of.html) But I'm a contrarian sociologist, so I am not so sure that I would necessarily agree. To see why, I first need to point out the issues of social standing and transaction costs that factor into this labor market.

First, I started to think about transaction costs as a reason why micro tasking was popular for over qualified people like college students to make very little bits of money. (Leaving out the 'game-like quality' arguments here.) For example, as a PhD student, I can always go work part-time at something, but getting the job, training for it, and then settling into a routine of work that won't interfere too much with my research takes time and effort -- indeed, it would probably require a lot of searching with the current economy. On the other hand, I can work in little bits and pieces with Mechanical Turk, earning little money, but in a way that doesn't require a lot of startup costs. I can tell you from experience that motivation is there, and low barriers to entry (transaction costs) make it easy to sign up with mTurk and start working right away. Who doesn't already have an Amazon account?

However, should we also think about this in terms of a status identity-driven distinction? It might take a lot of effort to go and get a reasonable part-time job, but maybe the issue is also that it just feels wrong or demeaning to take a low skilled position. However, with mTurk, who of my friends are going to know that I'm doing this for extra money? Nobody, unless I tell them. Thus, this is work that the overqualified could do, easily, but without the negative reputation effects that could occur if, say, your friends knew you were working in fast food management instead of doing corporate law. Working on mTurk -- say if I were unemployed or staying home with kids or between jobs or just part time -- wouldn't necessarily have the same status identity hit. I could rationalize this to myself as "just earning some extra money" and it would be unlikely to generate negative reputation effects with my peers, because it is unlikely that they would find out I was doing this unless I told them.

And that, my friends, is why I think we can definitely find overqualified workers lurking on Mechanical Turk. (So long as they don't get burned too quickly by scammy requesters.) In other words, maybe we should be asking "to what extent are 'too good' workers using mTurk as a way to generate some extra money in ways that are a) easy and/or b) not identity-threatening/demeaning?" (This is also a nice explanation for why you might see overqualified people doing temp work, together with the ability to move into a good FT position...)

Testing Status Identity vs. Transaction Costs
A nice experiment to test this would be to see how well or quickly the same people completed a boring task using CloudCrowd vs. mTurk -- experiments anyone? The way that CloudCrowd works is to use a Facebook login to start working, which is what gave me the idea for this post in the first place, because I immediately went "eww.... I don't want to let my friends know I'm doing this low-status work."

CloudCrowd uses facebook and existing accounts to guarantee that people are 'real' by looking at facebook. But not only will this create the status identity issues I mentioned above, with one's friends finding out you were doing menial labor, but a corollary is that we might get in fact worse quality work from some people who would be just fine in the semi-anonymous mTurk scenario.

Here's why that might happen: for some people, requiring login with your facebook ID is just fine, and through social sanctions and reputation effects, there will be an incentive to provide high quality work. I imagine this works just fine for some workers. Using the issue of status identity, however, I can also see a quality-destroying incentive here: if I don't want to show my friends I'm doing this, I might make up a new facebook ID, and then with that "worker login" I would probably feel less pressure to do quality work. Heck, if I'm going to start creating fake IDs, I may as well make a couple of them, and maybe have a throwaway bank account as well, to make it easier to abandon if one was flagged.

So, to be academic about it, I would expect to see this the two-tiered quality effect, and it would become my testable hypothesis if I were going to do this kind of research. (But I'm not, so it's fair game, people!) Another alternative might be that overqualified workers would just not do this type of work if they had to go through facebook or another social media source that was linked to their visible personal profile.

Well, now I really need to stop messing around with mTurk and work on my real research...