-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathscoring-platform.html
193 lines (188 loc) · 16.5 KB
/
scoring-platform.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
<!DOCTYPE html>
<!--[if lte IE 8 ]><html lang="en" class="js-off lte-ie8"><![endif]-->
<!--[if IE 9 ]> <html lang="en" class="js-off ie9"><![endif]-->
<!--[if (gt IE 9)|!(IE)]><!-->
<html lang="en" class="js-off">
<!--<![endif]-->
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0, shrink-to-fit=no">
<link rel="stylesheet" href="css/wmui-style-guide.min.css">
<title>Wikimedia Research - Projects - Machine Learning as a Service for Free Knowledge</title>
<script>
document.documentElement.className = document.documentElement.className.replace(/\bjs-off\b/,'js-on'); // no BEM notation thx to IE
</script>
<!--[if lt IE 9]>
<script>window.html5={'shivCSS':false};</script>
<script src="js/vendor/ie/html5shiv-3.7.3.min.js"></script>
<script src="js/vendor/ie/respond-1.4.2.min.js"></script>
<![endif]-->
<link rel="preload" href="fonts/Charter_regular.woff2" as="font" type="font/woff2" crossorigin>
<link rel="preload" href="fonts/Lato_regular.woff2" as="font" type="font/woff2" crossorigin>
<script src="js/fonts-loader.js" async></script>
</head>
<body class="page--scoring-platform page-parent--projects">
<header id="header" class="header" role="banner">
<div class="content-box">
<a href="#content" class="is-aural is-focusable">Jump to content</a>
<a href="#nav--main" class="is-aural is-focusable">Jump to navigation</a>
<h1 class="site__title"><a href="./"><span class="site__logo"></span>Wikimedia Research</a></h1>
<label class="btn--nav-main" for="trigger--nav-main" aria-hidden="true" title="Show main menu">
<i></i> <span>Menu</span>
</label>
<a href="https://twitter.com/WikiResearch" class="lnk--contribute" title="Follow Wikimedia Research on Twitter"><span>Follow </span>@WikiResearch</a>
</div>
</header>
<div class="page">
<div class="content-box">
<div class="col col--start">
<input type="checkbox" id="trigger--nav-main" class="trigger--nav-main">
<nav id="nav--main" class="nav nav--main" role="navigation">
<ol>
<li class="nav__item"><a href="index.html">About</a></li>
<li class="nav__item is-on"><a href="projects.html">Projects</a>
<ul class="nav__sub-items">
<li class="nav__sub-item"><a href="community-health.html">Community health</a></li>
<li class="nav__sub-item"><a href="increasing-diversity.html">Increasing diversity</a></li>
<li class="nav__sub-item"><a href="knowledge-gaps.html">Knowledge gaps</a></li>
<li class="nav__sub-item"><a href="recommender-system-ux.html">Recommender system UX</a></li>
<li class="nav__sub-item is-on"><a href="scoring-platform.html">Scoring platform</a></li>
<li class="nav__sub-item"><a href="structured-citations.html">Structured citations</a></li>
<li class="nav__sub-item"><a href="structured-multimedia-data.html">Structured multimedia data</a></li>
<li class="nav__sub-item"><a href="why-we-read-wikipedia.html">Why we read Wikipedia</a></li>
</ul>
</li>
<li class="nav__item"><a href="publications.html">Publications</a></li>
<li class="nav__item"><a href="news.html">News</a></li>
<li class="nav__item"><a href="events.html">Events</a></li>
<li class="nav__item"><a href="contact.html">Contact</a></li>
</ol>
</nav>
</div>
<div class="col col--end">
<main id="content" class="content" role="main">
<div class="page__parent-title">Projects</div>
<h1 class="page__title">Machine Learning as a Service for Free Knowledge</h1>
<p class="page__tagline">We are investigating the design of automated quality control in Wikimedia projects. We explore ways to enhance the impact of machine classifiers, while minimizing their detrimental effects.</p>
<img src="img/patterns/patrick-hendry-431197.jpg" title="image by Patrick Hendry" alt="scoring platform image" />
<section id="project-overview">
<h2>Project overview</h2>
<p>Wikipedia reflects a complex interaction between humans and technology. The technology used for Wikipedia has shaped its social environments. Its social environments have shaped its technology. The two have evolved together; changes to one, often affect the other.</p>
<p>Between 2004 and 2007, Wikipedia grew quickly, and its early contributors created novel technology to ensure quality in Wikimedia projects. This technology was used to classify and analyze every edit to Wikipedia and to predict whether the edit was "good" or "bad." The technology used machine classifiers, which were evaluated by human patrollers who were looking for evidence of vandalism on Wikimedia projects.</p>
<p>The machine classifiers made it easier to identify "bad" edits, but the technology did not account for new contributors who made edits in good faith with poor results. Their edits were treated like vandalism, and this led to a dramatic decline in Wikipedia contributors. Wikimedia Foundation researchers discovered this problem and shared their findings.</p>
<p>While the Wikimedia Foundation has invested in efforts to improve the newcomer experience, quality control tools have remained unchanged. The <a href="https://www.mediawiki.org/wiki/ORES">Scoring Platform project</a> was formed to evaluate this problem and to explore ways to enhance the positive impact of technology on Wikipedia contributors, while minimizing the negative effects of this technology on participation.</p>
</section>
<section id="project-updates" class="updates">
<h2>Recent updates</h2>
<ol class="list list--updates">
<li class="update list__col">
<a href="https://blog.wikimedia.org/2017/07/19/scoring-platform-team/" class="update__card">
<h3 class="update__card--title">Announcing the Scoring Platform team</h3>
<time datetime="2017-07" class="update__card--time">Jul 2017</time>
<span class="update__card--desc">The new Scoring Platform team will be working on democratizing access to AI, developing new types of predictions, and pushing the state of the art with regards to ethical practice of AI development.</span>
</a>
</li>
<li class="update list__col">
<a href="https://blog.wikimedia.org/2017/03/07/the-keilana-effect/" class="update__card">
<h3 class="update__card--title">Moving the needle on Wikipedia’s coverage of women scientists</h3>
<time datetime="2017-03" class="update__card--time">Mar 2017</time>
<span class="update__card--desc">Using an article quality classifier, we quantified the "Keilana effect": the impact of outreach initiatives started by Emily Temple-Wood and other women to bridge the gender gap in Wikipedia.</span>
</a>
</li>
<li class="update list__col">
<a href="https://blog.wikimedia.org/2016/10/27/wikipedia-quality-trends-dataset/" class="update__card">
<h3 class="update__card--title">New dataset shows fifteen years of Wikipedia’s quality trends</h3>
<time datetime="2016-10" class="update__card--time">Oct 2016</time>
<span class="update__card--desc">We’ve generated a dataset that tracks the quality of articles at monthly intervals over the entire 15-year history of Wikipedia across multiple languages—that’s 670 million assessments!</span>
</a>
</li>
<li class="update list__col">
<a href="https://www.wired.com/2015/12/wikipedia-is-using-ai-to-expand-the-ranks-of-human-editors/" class="update__card">
<h3 class="update__card--title">Wikipedia Deploys AI to Expand Its Ranks of Human Editors</h3>
<time datetime="2015-12" class="update__card--time">Dec 2015</time>
<span class="update__card--desc">"It turns out that the vast majority of vandalism is not very clever.": the launch of ORES featured in Wired.</span>
</a>
</li>
<li class="update list__col">
<a href="https://www.technologyreview.com/s/544036/artificial-intelligence-aims-to-make-wikipedia-friendlier-and-better/" class="update__card">
<h3 class="update__card--title">Artificial Intelligence Aims to Make Wikipedia Friendlier and Better</h3>
<time datetime="2015-12" class="update__card--time">Dec 2015</time>
<span class="update__card--desc">The nonprofit behind Wikipedia is turning to machine learning to combat a long-standing decline in the number of editors: ORES featured in the MIT Technology Review.</span>
</a>
</li>
<li class="update list__col">
<a href="https://blog.wikimedia.org/2015/11/30/artificial-intelligence-x-ray-specs/" class="update__card">
<h3 class="update__card--title">ORES service is officially launched</h3>
<time datetime="2015-11" class="update__card--time">Nov 2015</time>
<span class="update__card--desc">A new AI service gives Wikipedians X-ray specs to see through bad edits and handle some of the highest-volume crowdsourcing issues on the internet.</span>
</a>
</li>
</ol>
</section>
<section id="project-meta" class="project-meta">
<h2>Project team</h2>
<p><a href="https://meta.wikimedia.org/wiki/User:Halfak_(WMF)">Aaron Halfaker</a>, <a href="https://en.wikipedia.org/wiki/User:Nettrom">Morten Warncke-Wang</a></p>
<h2>Collaborators</h2>
<p>Sumit Asthana, Andrew Hall (University of Minnesota), Amir Sarabadani (Wikimedia Deutschland), Adam Wight (Wikimedia Foundation)</p>
<h2>Publications</h2>
<ul class="publications">
<li>R. Stuart Geiger and Aaron Halfaker. 2017. <a href="https://commons.wikimedia.org/wiki/File:Operationalizing-conflict-bots-wikipedia-cscw-preprint.pdf">Operationalizing Conflict and Cooperation between Automated Software Agents in Wikipedia: A Replication and Expansion of 'Even Good Bots Fight'</a>. In <em>Proceedings of the ACM on Human-Computer Interaction (CSCW)</em>, Vol. 1, Article 49 (December 2017), 33 pages. DOI: <a href="https://doi.org/10.1145/3134684">https://doi.org/10.1145/3134684</a></li>
<li>Diyi Yang, Aaron Halfaker, Robert Kraut, and Eduard Hovy. 2017. <a href="http://www.aclweb.org/anthology/D17-1212">Identifying Semantic Edit Intentions from Revisions in Wikipedia</a>. In <em>Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing</em>, pp. 1990-2000. 2017.</li>
<li>Aaron Halfaker. 2017. <a href="https://upload.wikimedia.org/wikipedia/commons/f/fa/Demonstrating_the_Keilana_Effect_%28OpenSym%2717%29.pdf">Interpolating Quality Dynamics in Wikipedia and Demonstrating the Keilana Effect</a>. In <em>Proceedings of the 13th International Symposium on Open Collaboration (OpenSym '17)</em>. ACM, New York, NY, USA, Article 19, 9 pages. <a href="https://doi.org/10.1145/3125433.3125475">https://doi.org/10.1145/3125433.3125475</a></li>
<li>Amir Sarabadani, Aaron Halfaker, and Dario Taraborelli. 2017. <a href="http://wikiworkshop.org/2017/papers/p1647-sarabadani.pdf">Building Automated Vandalism Detection Tools for Wikidata</a>. In <em>Proceedings of the 26th International Conference on World Wide Web Companion (WWW '17 Companion)</em>. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 1647-1654. DOI: <a href="https://doi.org/10.1145/3041021.3053366">https://doi.org/10.1145/3041021.3053366</a></li>
<li>Diyi Yang, Aaron Halfaker, Robert E. Kraut, and Eduard Hovy. (2016, March). <a href="https://www.aaai.org/ocs/index.php/ICWSM/ICWSM16/paper/viewPaper/13077">Who Did What: Editor Role Identification in Wikipedia.</a> In <em>Proceedings of the Tenth International AAAI Conference on Web and Social Media (ICWSM '16)</em> (pp. 446-455).</li>
<li>R. Stuart Geiger, and Aaron Halfaker. 2016. <a href="https://spir.aoir.org/index.php/spir/article/view/1383">Open algorithmic systems: Lessons on opening the black box from Wikipedia</a>. In: <em>Selected Paper of Internet Research 2016: The 17th Annual Conference of the Association of Internet Researchers (AOIR '16)</em></li>
<li>Morten Warncke-Wang, Vladislav R. Ayukaev, Brent Hecht, and Loren G. Terveen. 2015. <a href="http://www-users.cs.umn.edu/~bhecht/publications/qualityimprovement_cscw2015.pdf">The Success and Failure of Quality Improvement Projects in Peer Production Communities</a>. In <em>Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '15)</em>. ACM, New York, NY, USA, 743-756. <a href="https://doi.org/10.1145/2675133.2675241">https://doi.org/10.1145/2675133.2675241</a></li>
<li>Jodi Schneider, Bluma S. Gelley, and Aaron Halfaker</>. 2014. <a href="https://www-users.cs.umn.edu/~halfaker/publications/Accept_Decline_Postpone/schneider14accept.pdf">Accept, decline, postpone: How newcomer productivity is reduced in English Wikipedia by pre-publication review</a>. In <em>Proceedings of The International Symposium on Open Collaboration (OpenSym '14)</em>. ACM, New York, NY, USA, , Pages 26 , 10 pages. <a href="https://doi.org/10.1145/2641580.2641614">https://doi.org/10.1145/2641580.2641614</a></li>
<li>Aaron Halfaker</>, R. Stuart Geiger, and Loren G. Terveen. 2014. <a href="https://www-users.cs.umn.edu/~halfaker/publications/Snuggle/halfaker14snuggle-preprint.pdf">Snuggle: Designing for efficient socialization and ideological critique</a>. In <em>Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14)</em>. ACM, New York, NY, USA, 311-320. <a href="https://doi.org/10.1145/2556288.2557313">https://doi.org/10.1145/2556288.2557313</a></li>
<li>R. Stuart Geiger and Aaron Halfaker</>. 2013. <a href="http://opensym.org/wsos2013/proceedings/p0200-geiger.pdf">When the levee breaks: Without bots, what happens to Wikipedia's quality control processes?</a>. In <em>Proceedings of the 9th International Symposium on Open Collaboration (WikiSym '13)</em>. ACM, New York, NY, USA, , Article 6 , 6 pages. <a href="https://doi.org/10.1145/2491055.2491061">https://doi.org/10.1145/2491055.2491061</a></li>
<li>Aaron Halfaker</>, R. Stuart Geiger, Jonathan T. Morgan</>, John Riedl. 2013. <a href="https://www-users.cs.umn.edu/~halfaker/publications/The_Rise_and_Decline/halfaker13rise-preprint.pdf">The Rise and Decline of an Open Collaboration System. How Wikipedia’s Reaction to Popularity Is Causing Its Decline</a>. <em>American Behavioral Scientist</em>. Vol 57, Issue 5, 2013. <a href="https://doi.org/10.1177/0002764212469365">https://doi.org/10.1177/0002764212469365</a></li>
</ul>
<h2>Resources and links</h2>
<div>
<ol class="list list--resources">
<li class="resource list__col">
<span class="resource__title">Home page</span>
<a class="resource__link" href="https://www.mediawiki.org/wiki/Wikimedia_Scoring_Platform_team ">Scoring Platform Team on MediaWiki.org</a>
</li>
<li class="resource list__col">
<span class="resource__title">Research pages</span>
<a class="resource__link" href="https://www.mediawiki.org/wiki/ORES">Objective Revision Evaluation Service (ORES)</a>
<a class="resource__link" href="https://meta.wikimedia.org/wiki/Wiki_labels">Wiki labels</a>
<a class="resource__link" href="https://www.mediawiki.org/wiki/JADE">Judgement And Dialog Engine (JADE)</a>
</li>
</ol>
</div>
</section>
</main>
</div>
</div>
</div>
<footer id="footer" class="footer">
<div class="content-box">
<ul class="footer__list">
<li><a href="acknowledgments.html">Acknowledgments</a>
<li><a href="https://github.com/wikimedia/research-landing-page">Source code</a></li>
</ul>
<p>Text is available under the <a href="https://creativecommons.org/licenses/by-sa/4.0/" target="_blank">Creative Commons Attribution-ShareAlike 4.0 International</a>, additional terms may apply. <br>Code is available under the MIT license.</p>
<p><a href="https://wikimediafoundation.org/" class="lnk--wikimedia-project">A Wikimedia Foundation project</a></p>
</div>
</footer>
<!-- Piwik -->
<script type="text/javascript">
var _paq = _paq || [];
_paq.push(["setDomains", ["*.research.wikimedia.org"]]);
_paq.push(['trackPageView']);
_paq.push(['enableLinkTracking']);
(function() {
var u="//piwik.wikimedia.org/";
_paq.push(['setTrackerUrl', u+'piwik.php']);
_paq.push(['setSiteId', '13']);
var d=document, g=d.createElement('script'), s=d.getElementsByTagName('script')[0];
g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
})();
</script>
<noscript><p><img src="//piwik.wikimedia.org/piwik.php?idsite=13" style="border:0;" alt="" /></p></noscript>
<!-- End Piwik Code -->
</body>
</html>