diff --git a/404.html b/404.html index cf2f3df9..0bf94eff 100644 --- a/404.html +++ b/404.html @@ -145,14 +145,14 @@

Latest

diff --git a/category/teaching/index.html b/category/teaching/index.html index 1e34f1cd..5e1ee057 100644 --- a/category/teaching/index.html +++ b/category/teaching/index.html @@ -142,39 +142,9 @@

teaching

-

Social Computing - 2022/2023

+

Social Computing (2020/2021-2024/2025)

-Bachelor's Degree in Internet of Things, Big Data & Web at the University of Udine. Academic Year 2022/2023. Lectures: 2. Hours: 4 -
-
-
-

Social Computing - 2024/2025

-
-Bachelor's Degree in Internet of Things, Big Data & Web at the University of Udine. Academic Year 2022/2024. Lectures: 1. Hours: 2 -
-
-
-

Social Computing - 2023/2024

-
-Bachelor's Degree in Internet of Things, Big Data & Web at the University of Udine. Academic Year 2022/2024. Lectures: 2. Hours: 4 -
-
-
-

Social Computing - 2022/2023

-
-Bachelor's Degree in Internet of Things, Big Data & Web at the University of Udine. Academic Year 2022/2023. Lectures: 2. Hours: 4 -
-
-
-

Social Computing - 2021/2022

-
-Bachelor's Degree in Internet of Things, Big Data & Web at the University of Udine. Academic Year 2021/2022. Lectures: 6. Hours: 12 -
-
-
-

Social Computing - 2020/2021

-
-Bachelor's Degree in Internet of Things, Big Data & Web at the University of Udine. Academic Year 2020/2021. Lectures: 6. Hours: 12 +Bachelor's Degree in Internet of Things, Big Data & Web at the University of Udine. Academic Years 2020/2021 - 2024/2025
diff --git a/category/teaching/index.xml b/category/teaching/index.xml index aa1d99b4..7bccc3dd 100644 --- a/category/teaching/index.xml +++ b/category/teaching/index.xml @@ -1,78 +1 @@ -teaching | Michael Sopranohttps://michaelsoprano.com/category/teaching/teachingWowchemy (https://wowchemy.com)en-usSun, 18 Aug 2024 11:00:00 +0000https://michaelsoprano.com/media/icon_hu4525676fb5a133c3f276d8a3d210c954_1943161_512x512_fill_lanczos_center_3.pngteachinghttps://michaelsoprano.com/category/teaching/Social Computing - 2024/2025https://michaelsoprano.com/post/social-computing-2024_2025/Wed, 10 Jul 2024 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2024_2025/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>Hafska Akebli</strong> - Laboratory (Part 1-2)</li> -<li><strong>Michele Lizzit</strong> - Laboratory (Part 1-2)</li> -<li><strong>Michael Soprano</strong> - Laboratory (Part 2)</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, -Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, -theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by providing a lecture on Crowd_Frame, a framework I developed for the design and deployment of crowdsourcing tasks (2 hours). Additionally, I provided the students with -their second course assignment. The course is conducted in Italian.</p>Social Computing - 2023/2024https://michaelsoprano.com/post/social-computing-2023_2024/Tue, 10 Oct 2023 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2023_2024/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>David La Barbera</strong> - Laboratory (Part 1)</li> -<li><strong>Michael Soprano</strong> - Laboratory (Part 2)</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, -Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, -theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by providing two lectures on Crowd_Frame, a framework I developed for the design and deployment of crowdsourcing tasks (4 hours). Additionally, I provided the students with -their second course assignment. The course is conducted in Italian.</p> -<ul> -<li><strong>Lesson 4-5</strong>: <a href="https://www.dropbox.com/scl/fi/4rmiryatmpzhyrcmejuyb/SC_MS_4_Crowd_Frame.pptx?rlkey=1cd5jhx1n2fi491qvkagau3un&amp;dl=0" target="_blank" rel="noopener">Crowd_Frame</a></li> -</ul>Social Computing - 2022/2023https://michaelsoprano.com/post/social-computing-2022_2023/Mon, 10 Oct 2022 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2022_2023/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>David La Barbera</strong> - Laboratory (Part 1)</li> -<li><strong>Michael Soprano</strong> - Laboratory (Part 2)</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, -Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, -theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by providing two lectures on Crowd_Frame, a framework I developed for the design and deployment of crowdsourcing tasks (4 hours). Additionally, I provided the students with -their second course assignment. The course is conducted in Italian.</p> -<ul> -<li><strong>Lesson 4-5</strong>: <a href="https://www.dropbox.com/scl/fi/ez7aanl9b9vn4m6zt6ckw/22-23.Crowd_Frame.pdf?rlkey=95kkc3nrf9ejy6r17s5i44y7w&amp;st=b24tubx6&amp;dl=0" target="_blank" rel="noopener">Crowd_Frame</a></li> -</ul>Social Computing - 2021/2022https://michaelsoprano.com/post/social-computing-2021_2022/Sun, 10 Oct 2021 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2021_2022/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>Michael Soprano</strong> - Laboratory</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by teaching the practical applications of the concepts outlined by Professor Stefano Mizzaro in a total of six lectures (12 hours). Additionally, I assigned course assignments to the students. The course is conducted in Italian.</p> -<ul> -<li><strong>Lesson 1</strong>: <a href="https://www.dropbox.com/scl/fi/az4z8x0zu3ysajpjsfs0h/SC_MS_1_Python_Rest_Richiami.pptx?rlkey=ozxl5bwwpp56wjgp3ty01v7tf&amp;dl=0" target="_blank" rel="noopener">Python / Rest: Richiami</a></li> -<li><strong>Lesson 2</strong>: <a href="https://www.dropbox.com/scl/fi/xrtha37m67zob5k7sresz/SC_MS_2-Twitter_API_Tweepy.pptx?rlkey=i54k7eha11f59ies21pyn9u16&amp;dl=0" target="_blank" rel="noopener">Twitter API: Tweepy</a></li> -<li><strong>Lesson 3</strong>: <a href="https://www.dropbox.com/scl/fi/93umif7xa71woszewr5zo/SC_MS_3-EndpointV2_Req.pptx?rlkey=zp1vux6vjp1x6xc7evip6l0d4&amp;dl=0" target="_blank" rel="noopener">Twitter API: Endpoint V2</a></li> -<li><strong>Lesson 4</strong>: <a href="https://www.dropbox.com/scl/fi/d22udpapqnitwuc32ban4/SC_MS_4_Grafi_NetworkX.pptx?rlkey=3r949h14dqxsnzwoffprdgaz7&amp;dl=0" target="_blank" rel="noopener">Grafi: NetworkX</a></li> -<li><strong>Lesson 5-6</strong>: <a href="https://www.dropbox.com/scl/fi/tmgtqxt3ihnvtgj5r0150/SC_MS_5_Crowd_Frame.pptx?rlkey=prdahfpu3u0y7k48dzi2mc1fk&amp;dl=0" target="_blank" rel="noopener">Crowd_Frame</a></li> -</ul>Social Computing - 2020/2021https://michaelsoprano.com/post/social-computing-2020_2021/Sat, 10 Oct 2020 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2020_2021/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>Michael Soprano</strong> - Laboratory</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by teaching the practical applications of the concepts outlined by Professor Stefano Mizzaro in a total of six lectures (12 hours). Additionally, I assigned course assignments to the students. The course is conducted in Italian.</p> -<ul> -<li><strong>Lesson 1</strong>: <a href="https://www.dropbox.com/scl/fi/xpq3b6zhzf3zpczqkwn9g/SC_MS_1_Twitter_API_Intro_Ric.pptx?rlkey=trdwm6yxgohyktj79c0bu3xsx&amp;dl=0" target="_blank" rel="noopener">Twitter API: Introduzione &amp; Richiami</a></li> -<li><strong>Lesson 2-3</strong>: <a href="https://www.dropbox.com/scl/fi/rtdvumim9mzuihlu3zlxu/SC_MS_2-Twitter_API_Tweepy_Req.pptx?rlkey=t7rms0spo58a612oniuda1vey&amp;dl=0" target="_blank" rel="noopener">Twitter API: Tweepy &amp; Requests</a></li> -<li><strong>Lesson 4</strong>: <a href="https://www.dropbox.com/scl/fi/0e8iz71e45mxix4cmxiam/SC_MS_3_Grafi_NetworkX.pptx?rlkey=ab76koii4j1we6zpwatihhh10&amp;dl=0" target="_blank" rel="noopener">Grafi: NetworkX</a></li> -<li><strong>Lesson 4-5</strong>: <a href="https://www.dropbox.com/scl/fi/56egim1ixzm1bm4tn988x/SC_MS_4_Crowd_Frame.pptx?rlkey=yl0z8mqxb6q6n4u1h2etfuz5e&amp;dl=0" target="_blank" rel="noopener">Crowd_Frame: How To</a></li> -</ul> \ No newline at end of file +teaching | Michael Sopranohttps://michaelsoprano.com/category/teaching/teachingWowchemy (https://wowchemy.com)en-usSun, 18 Aug 2024 11:00:00 +0000https://michaelsoprano.com/media/icon_hu4525676fb5a133c3f276d8a3d210c954_1943161_512x512_fill_lanczos_center_3.pngteachinghttps://michaelsoprano.com/category/teaching/ \ No newline at end of file diff --git a/index.html b/index.html index 0fe9c97f..3af5980c 100644 --- a/index.html +++ b/index.html @@ -1630,151 +1630,6 @@

Academic Activity

Teaching

-
- -
-
-
-
- -
-
-
-
- -
-
-
-
- -
-
-
-
- -
-
-
@@ -1805,26 +1660,26 @@

Topics

-Amazon Mechanical Turk -Angular -bias +Amazon Mechanical Turk +bias conversational agent covid-19 -crowdsourcing -Data Analysis +crowdsourcing electronic health record explainability -fact-checking +fact-checking fairness few topics framework -misinformation -peer review -Prolific -public administration -Toloka -truthfulness -truthfulness assessment +human computation +misinformation +network analysis +peer review +Prolific +public administration +scholarly publishing +truthfulness +truthfulness assessment
diff --git a/index.json b/index.json index 3fe1147e..f347e431 100644 --- a/index.json +++ b/index.json @@ -1 +1 @@ -[{"authors":null,"categories":null,"content":"Benvignût! This is my personal webpage. I am a PostDoc at the University of Udine, affiliated with the Social, Media, Data \u0026amp; Crowd Laboratory, and I hold a Ph.D. in Computer Science, Mathematics, and Physics.\nMy main research interests include Human Computation, Crowdsourcing, and Information Retrieval. Currently, I am focused on leveraging crowdsourcing to address the growing challenge of misinformation spread.\nIn my free time, I enjoy discovering new literature and work towards being a decent runner, hiker, and skier.\n Download my Curriculum Vitae in English or Italian. Feel free to explore my Master's, Bachelor's, and High School theses (in Italian).\n","date":-62135596800,"expirydate":-62135596800,"kind":"term","lang":"en","lastmod":1728386301,"objectID":"2525497d367e79493fd32b198b28f040","permalink":"","publishdate":"0001-01-01T00:00:00Z","relpermalink":"","section":"authors","summary":"Benvignût! This is my personal webpage. I am a PostDoc at the University of Udine, affiliated with the Social, Media, Data \u0026amp; Crowd Laboratory, and I hold a Ph.D. in Computer Science, Mathematics, and Physics.","tags":null,"title":"Michael Soprano","type":"authors"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1726572600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1726576025,"objectID":"c4cd58d6905d915159aabedf08ff501c","permalink":"https://michaelsoprano.com/talk/deploying-complex-crowdsourcing-tasks-without-fear-an-introduction-to-crowd_frame/","publishdate":"2024-09-17T11:30:00Z","relpermalink":"/talk/deploying-complex-crowdsourcing-tasks-without-fear-an-introduction-to-crowd_frame/","section":"event","summary":"Demo Presentation - The 3rd Italian Conference on Big Data and Data Science (ITADATA 2024). Pisa, Italy.","tags":["demo","human computation","crowdsourcing","crowd_frame"],"title":"Deploying Complex Crowdsourcing Tasks Without Fear: An Introduction to Crowd_Frame","type":"event"},{"authors":["Jaspreet Singh","Michael Soprano","Kevin Roitero","Davide Ceolin"],"categories":[],"content":"","date":1725148800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1725184728,"objectID":"37de9c683bbb318b815214077838e5d7","permalink":"https://michaelsoprano.com/publication/conference-paper-misdoom-2024/","publishdate":"2024-09-01T00:00:00Z","relpermalink":"/publication/conference-paper-misdoom-2024/","section":"publication","summary":"This paper explores the use of crowdsourcing to classify statement types in film reviews to assess their information quality. Employing the Argument Type Identification Procedure which uses the Periodic Table of Arguments to categorize arguments, the study aims to connect statement types to the overall argument strength and information reliability. Focusing on non-expert annotators in a crowdsourcing environment, the research assesses their reliability based on various factors including language proficiency and annotation experience. Results indicate the importance of careful annotator selection and training to achieve high inter-annotator agreement and highlight challenges in crowdsourcing statement classification for information quality assessment.","tags":["crowdsourcing annotation","information quality assessment","argument type identification"],"title":"Crowdsourcing Statement Classification to Enhance Information Quality Prediction","type":"publication"},{"authors":["Michael Soprano"],"categories":["teaching"],"content":"Teachers Stefano Mizzaro - Theory Hafska Akebli - Laboratory (Part 1-2) Michele Lizzit - Laboratory (Part 1-2) Michael Soprano - Laboratory (Part 2) Aims The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, Twitter, etc.) and the crowdsourcing phenomenon.\nIn the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, theoretical foundations, and practical applications.\nTable of contents I contributed to the course by providing a lecture on Crowd_Frame, a framework I developed for the design and deployment of crowdsourcing tasks (2 hours). Additionally, I provided the students with their second course assignment. The course is conducted in Italian.\n","date":1720609200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1728390520,"objectID":"a8d39ed46e9bbcd000b3f17603822f9b","permalink":"https://michaelsoprano.com/post/social-computing-2024_2025/","publishdate":"2024-07-10T11:00:00Z","relpermalink":"/post/social-computing-2024_2025/","section":"post","summary":"Bachelor's Degree in Internet of Things, Big Data \u0026 Web at the University of Udine. Academic Year 2022/2024. Lectures: 1. Hours: 2","tags":["Amazon Mechanical Turk","Toloka","Prolific","Angular","Crowdsourcing","Data Analysis"],"title":"Social Computing - 2024/2025","type":"post"},{"authors":["Michael Soprano","Kevin Roitero","Ujwal Gadiraju","Eddy Maddalena","Gianluca Demartini"],"categories":[],"content":"","date":1720569600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1720705059,"objectID":"45b5e1627d200b8e3002b8d06787a02e","permalink":"https://michaelsoprano.com/publication/journal-paper-tsc-2024/","publishdate":"2024-07-10T15:00:00+01:00","relpermalink":"/publication/journal-paper-tsc-2024/","section":"publication","summary":"Crowdsourcing tasks have been widely used to collect a large number of human labels at scale. While some of these tasks are deployed by requesters and performed only once by crowd workers, others require the same worker to perform the same task or a variant of it more than once, thus participating in a so-called longitudinal study. Despite the prevalence of longitudinal studies in crowdsourcing, there is a limited understanding of factors that influence worker participation in them across different crowdsourcing marketplaces. We present results from a large-scale survey of 300 workers on 3 different micro-task crowdsourcing platforms: Amazon Mechanical Turk, Prolific and Toloka. The aim is to understand how longitudinal studies are performed using crowdsourcing. We collect answers about 547 experiences and we analyze them both quantitatively and qualitatively. We synthesize 17 take-home messages about longitudinal studies together with 8 recommendations for task requesters and 5 best practices for crowdsourcing platforms to adequately conduct and support such kinds of studies. We release the survey and the data at: https://osf.io/h4du9/.","tags":["longitudinal studies","crowdsourcing platforms","surveys","online sampling","amazon mechanical turk","prolific","toloka"],"title":"Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platforms","type":"publication"},{"authors":["David La Barbera","Eddy Maddalena","Michael Soprano","Kevin Roitero","Gianluca Demartini","Davide Ceolin","Damiano Spina","Stefano Mizzaro"],"categories":[],"content":"","date":1717113600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1717244786,"objectID":"a51d41f7a834b97be079ca417f213722","permalink":"https://michaelsoprano.com/publication/journal-paper-ipm-2024-2/","publishdate":"2024-05-31T15:00:00+01:00","relpermalink":"/publication/journal-paper-ipm-2024-2/","section":"publication","summary":"There is an important ongoing effort aimed to tackle misinformation and to perform reliable fact-checking by employing human assessors at scale, with a crowdsourcing-based approach. Previous studies on the feasibility of employing crowdsourcing for the task of misinformation detection have provided inconsistent results: some of them seem to confirm the effectiveness of crowdsourcing for assessing the truthfulness of statements and claims, whereas others fail to reach an effectiveness level higher than automatic machine learning approaches, which are still unsatisfactory. In this paper, we aim at addressing such inconsistency and understand if truthfulness assessment can indeed be crowdsourced effectively. To do so, we build on top of previous studies; we select some of those reporting low effectiveness levels, we highlight their potential limitations, and we then reproduce their work attempting to improve their setup to address those limitations. We employ various approaches, data quality levels, and agreement measures to assess the reliability of crowd workers when assessing the truthfulness of (mis)information. Furthermore, we explore different worker features and compare the results obtained with different crowds. According to our findings, crowdsourcing can be used as an effective methodology to tackle misinformation at scale. When compared to previous studies, our results indicate that a significantly higher agreement between crowd workers and experts can be obtained by using a different, higher-quality, crowdsourcing platform and by improving the design of the crowdsourcing task. Also, we find differences concerning task and worker features and how workers provide truthfulness assessments.","tags":["crowdsourcing","misinformation","fact-checking","truthfulness assessment"],"title":"Crowdsourced Fact-checking: Does It Actually Work?","type":"publication"},{"authors":["Michael Soprano","Kevin Roitero","David La Barbera","Davide Ceolin","Damiano Spina","Gianluca Demartini","Stefano Mizzaro"],"categories":[],"content":"","date":1706918400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1706992044,"objectID":"af6209263d25032cabcb8be029e4ac73","permalink":"https://michaelsoprano.com/publication/journal-paper-ipm-2024/","publishdate":"2024-02-03T15:00:00+01:00","relpermalink":"/publication/journal-paper-ipm-2024/","section":"publication","summary":"The increase of the amount of misinformation spread every day online is a huge threat to the society. Organizations and researchers are working to contrast this misinformation plague. In this setting, human assessors are indispensable to correctly identify, assess and/or revise the truthfulness of information items, i.e., to perform the fact-checking activity. Assessors, as humans, are subject to systematic errors that might interfere with their fact-checking activity. Among such errors, cognitive biases are those due to the limits of human cognition. Although biases help to minimize the cost of making mistakes, they skew assessments away from an objective perception of information. Cognitive biases, hence, are particularly frequent and critical, and can cause errors that have a huge potential impact as they propagate not only in the community, but also in the datasets used to train automatic and semi-automatic machine learning models to fight misinformation. In this work, we present a review of the cognitive biases which might occur during the fact-checking process. In more detail, inspired by PRISMA – a methodology used for systematic literature reviews – we manually derive a list of 221 cognitive biases that may affect human assessors. Then, we select the 39 biases that might manifest during the fact-checking process, we group them into categories, and we provide a description. Finally, we present a list of 11 countermeasures that can be adopted by researchers, practitioners, and organizations to limit the effect of the identified cognitive biases on the fact-checking activity.","tags":["cognitive bias","misinformation","fact-checking","truthfulness assessment"],"title":"Cognitive Biases in Fact-Checking and Their Countermeasures: A Review","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1698309e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1698394801,"objectID":"a4927be3f1353c53366ddd8350f71765","permalink":"https://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-healthcare/","publishdate":"2023-10-26T08:30:00Z","relpermalink":"/talk/towards-a-conversational-based-agent-for-healthcare/","section":"event","summary":"Workshop Talk - European Federation of Medical Informatics Special Topic Conference 2023 (EFMI STC 2023). Turin, Italy.","tags":["conversational agent","electronic health record","large language models","public administration"],"title":"Towards a Conversational-Based Agent for Healthcare","type":"event"},{"authors":["Michael Soprano"],"categories":["teaching"],"content":"Teachers Stefano Mizzaro - Theory David La Barbera - Laboratory (Part 1) Michael Soprano - Laboratory (Part 2) Aims The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, Twitter, etc.) and the crowdsourcing phenomenon.\nIn the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, theoretical foundations, and practical applications.\nTable of contents I contributed to the course by providing two lectures on Crowd_Frame, a framework I developed for the design and deployment of crowdsourcing tasks (4 hours). Additionally, I provided the students with their second course assignment. The course is conducted in Italian.\n Lesson 4-5: Crowd_Frame ","date":1696935600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1728390520,"objectID":"baeb88c7ed928274a96a381b4c283286","permalink":"https://michaelsoprano.com/post/social-computing-2023_2024/","publishdate":"2023-10-10T11:00:00Z","relpermalink":"/post/social-computing-2023_2024/","section":"post","summary":"Bachelor's Degree in Internet of Things, Big Data \u0026 Web at the University of Udine. Academic Year 2022/2024. Lectures: 2. Hours: 4","tags":["Amazon Mechanical Turk","Toloka","Prolific","Angular","Crowdsourcing","Data Analysis"],"title":"Social Computing - 2023/2024","type":"post"},{"authors":["Michael Soprano","Kevin Roitero","Vincenzo Della Mea","Stefano Mizzaro"],"categories":[],"content":"","date":1695168e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1697143748,"objectID":"343710264000293cf3f4d7227bc8744b","permalink":"https://michaelsoprano.com/publication/workshop-paper-ital-ia-2023/","publishdate":"2023-09-20T17:40:00+01:00","relpermalink":"/publication/workshop-paper-ital-ia-2023/","section":"publication","summary":"Conversational agents provide new modalities to access and interact with services and applications. Recently, they saw a backfire in their popularity, due to the recent advancements in language models. Such agents have been adopted in various fields such as healthcare and education, yet they received little attention in public administration. We describe as a practical use case a service of the portal that provides citizens of the Italian region of Friuli-Venezia Giulia with services related to their own Electronic Health Records. The service considered allows them to search for the available doctors and pediatricians in the region's municipalities. We rely on the use case described to propose a model for a conversational agent-based access modality. The model proposed allows us to lay the foundation for more advanced chatbot-like implementations which will use also alternative input modalities, such as voice-based communication.","tags":["public administration","electronic health record","conversational agents"],"title":"Towards a Conversational-Based Agent for Health Services","type":"publication"},{"authors":["David La Barbera","Michael Soprano","Kevin Roitero","Eddy Maddalena","Stefano Mizzaro"],"categories":[],"content":"","date":1691193600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694093922,"objectID":"3834b58b74a20f1cd91a82784e2f3b63","permalink":"https://michaelsoprano.com/publication/workshop-paper-iir-2023/","publishdate":"2023-08-05T15:30:30+01:00","relpermalink":"/publication/workshop-paper-iir-2023/","section":"publication","summary":"In this paper, we present our journey in exploring the use of crowdsourcing for fact-checking. We discuss our early experiments aimed towards the identification of the best possible setting for misinformation assessment using crowdsourcing. Our results indicate that the crowd can effectively address misinformation at scale, showing some degree of correlation with experts. We also highlight the influence of worker background on the quality of truthfulness assessments.","tags":["crowdsourcing","human computation","fact-checking","misinformation","truthfulness assessment"],"title":"Fact-Checking at Scale with Crowdsourcing Experiments and Lessons Learned","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1685318400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1698394826,"objectID":"68aec2d0cd76ae46198120e98755b94a","permalink":"https://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-health-services/","publishdate":"2023-05-29T00:00:00Z","relpermalink":"/talk/towards-a-conversational-based-agent-for-health-services/","section":"event","summary":"Workshop Talk - Terzo Convegno Nazionale CINI sull'Intelligenza Artificiale (Ital-IA 2023). May 29th, 2023. Pisa, Italy.","tags":["conversational agent","public administration"],"title":"Towards a Conversational-Based Agent for Health Services","type":"event"},{"authors":["Kevin Roitero","David La Barbera","Michael Soprano","Gianluca Demartini","Stefano Mizzaro","Tetsuya Sakai"],"categories":[],"content":"","date":1684713600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1686216050,"objectID":"6486499d1482393e35a34e97f1c3935d","permalink":"https://michaelsoprano.com/publication/journal-paper-tois-2023/","publishdate":"2021-09-15T15:00:00+01:00","relpermalink":"/publication/journal-paper-tois-2023/","section":"publication","summary":"To scale the size of Information Retrieval collections, crowdsourcing has become a common way to collect relevance judgments at scale. Crowdsourcing experiments usually employ 100-10,000 workers, but such a number is often decided in a heuristic way. The downside is that the resulting dataset does not have any guarantee of meeting predefined statistical requirements as, for example, have enough statistical power to be able to distinguish in a statistically significant way between the relevance of two documents. We propose a methodology adapted from literature on sound topic set size design, based on t-test and ANOVA, which aims at guaranteeing the resulting dataset to meet a predefined set of statistical requirements. We validate our approach on several public datasets. Our results show that we can reliably estimate the recommended number of workers needed to achieve statistical power, and that such estimation is dependent on the topic, while the effect of the relevance scale is limited. Furthermore, we found that such estimation is dependent on worker features such as agreement. Finally, we describe a set of practical estimation strategies that can be used to estimate the worker set size, and we also provide results on the estimation of document set sizes.","tags":["relevance judgments","statistical analysis","crowdsourcing"],"title":"How Many Crowd Workers Do I Need? On Statistical Power When Crowdsourcing Relevance Judgments","type":"publication"},{"authors":["Michael Soprano"],"categories":["teaching"],"content":"Teachers Stefano Mizzaro - Theory David La Barbera - Laboratory (Part 1) Michael Soprano - Laboratory (Part 2) Aims The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, Twitter, etc.) and the crowdsourcing phenomenon.\nIn the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, theoretical foundations, and practical applications.\nTable of contents I contributed to the course by providing two lectures on Crowd_Frame, a framework I developed for the design and deployment of crowdsourcing tasks (4 hours). Additionally, I provided the students with their second course assignment. The course is conducted in Italian.\n Lesson 4-5: Crowd_Frame ","date":1665399600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1728390520,"objectID":"156dbcb531908818085ce9d75e4bb499","permalink":"https://michaelsoprano.com/post/social-computing-2022_2023/","publishdate":"2022-10-10T11:00:00Z","relpermalink":"/post/social-computing-2022_2023/","section":"post","summary":"Bachelor's Degree in Internet of Things, Big Data \u0026 Web at the University of Udine. Academic Year 2022/2023. Lectures: 2. Hours: 4","tags":["Amazon Mechanical Turk","Toloka","Prolific","Angular","Crowdsourcing","Data Analysis"],"title":"Social Computing - 2022/2023","type":"post"},{"authors":["Davide Ceolin","Giuseppe Primiero","Michael Soprano","Jan Wielemaker"],"categories":[],"content":"","date":1659052800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"f5bea931bb7158f139afb104d61db308","permalink":"https://michaelsoprano.com/publication/journal-paper-is-2022/","publishdate":"2022-07-29T15:00:00+01:00","relpermalink":"/publication/journal-paper-is-2022/","section":"publication","summary":"Review scores collect users’ opinions in a simple and intuitive manner. However, review scores are also easily manipulable, hence they are often accompanied by explanations. A substantial amount of research has been devoted to ascertaining the quality of reviews, to identify the most useful and authentic scores through explanation analysis. In this paper, we advance the state of the art in review quality analysis. We introduce a rating system to identify review arguments and to define an appropriate weighted semantics through formal argumentation theory. We introduce an algorithm to construct a corresponding graph, based on a selection of weighted arguments, their semantic distance, and the supported ratings. We also provide an algorithm to identify the model of such an argumentation graph, maximizing the overall weight of the admitted nodes and edges. We evaluate these contributions on the Amazon review dataset by McAuley et al. (2015), by comparing the results of our argumentation assessment with the upvotes received by the reviews. Also, we deepen the evaluation by crowdsourcing a multidimensional assessment of reviews and comparing it to the argumentation assessment. Lastly, we perform a user study to evaluate the explainability of our method, i.e., to test whether the automated method we use to assess reviews is understandable by humans. Our method achieves two goals: (1) it identifies reviews that are considered useful, comprehensible, and complete by online users, and does so in an unsupervised manner, and (2) it provides an explanation of quality assessments.","tags":[],"title":"Transparent assessment of information quality of online reviews using formal argumentation theory","type":"publication"},{"authors":["Erik Brand","Kevin Roitero","Michael Soprano","Afshin Rahimi","Gianluca Demartini"],"categories":[],"content":"","date":1657324800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"c7ad261b2704356f61cb1b48f634be9a","permalink":"https://michaelsoprano.com/publication/journal-paper-jdiq-2022/","publishdate":"2021-09-15T15:00:00+01:00","relpermalink":"/publication/journal-paper-jdiq-2022/","section":"publication","summary":"Automated fact-checking (AFC) systems exist to combat disinformation, however their complexity usually makes them opaque to the end user, making it difficult to foster trust in the system. In this paper, we introduce the E-BART model with the hope of making progress on this front. E-BART is able to provide a veracity prediction for a claim, and jointly generate a human-readable explanation for this decision. We show that E-BART is competitive with the state-of-the-art on the e-FEVER and e-SNLI tasks. In addition, we validate the joint-prediction architecture by showing 1) that generating explanations does not significantly impede the model from performing well in its main task of veracity prediction, and 2) that predicted veracity and explanations are more internally coherent when generated jointly than separately. We also calibrate the E-BART model, allowing the output of the final model be correctly interpreted as the confidence of correctness. Finally, we also conduct and extensive human evaluation on the impact of generated explanations and observe that: explanations increase human ability to spot misinformation and make people more skeptical about claims, and explanations generated by E-BART are competitive with ground truth explanations.","tags":[],"title":"A Neural Model to Jointly Predict and Explain Truthfulness of Statements","type":"publication"},{"authors":["Tim Draws","David La Barbera","Michael Soprano","Kevin Roitero","Davide Ceolin","Alessandro Checco","Stefano Mizzaro"],"categories":[],"content":"","date":1655769600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627976,"objectID":"e9991656c12f02af2ce45a24adb3515d","permalink":"https://michaelsoprano.com/publication/conference-paper-facct-2022/","publishdate":"2021-03-15T14:36:35.536697Z","relpermalink":"/publication/conference-paper-facct-2022/","section":"publication","summary":"Due to the increasing amount of information shared online every day, the need for sound and reliable ways of distinguishing between trustworthy and non-trustworthy information is as present as ever. One technique for performing fact-checking at scale is to employ human intelligence in the form of crowd workers. Although earlier work has suggested that crowd workers can reliably identify misinformation, cognitive biases of crowd workers may reduce the quality of truthfulness judgments in this context. We performed a systematic exploratory analysis of publicly available crowdsourced data to identify a set of potential systematic biases that may occur when crowd workers perform fact-checking tasks. Following this exploratory study, we collected a novel data set of crowdsourced truthfulness judgments to validate our hypotheses. Our findings suggest that workers generally overestimate the truthfulness of statements and that different individual characteristics (i.e., their belief in science) and cognitive biases (i.e., the affect heuristic and overconfidence) can affect their annotations. Interestingly, we find that, depending on the general judgment tendencies of workers, their biases may sometimes lead to more accurate judgments.","tags":["bias","crowdsourcing","misinformation","explainability","truthfulness"],"title":"The Effects of Crowd Worker Biases in Fact-Checking Tasks","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1655251200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"9338404eca03ca4ce1ca36306994970f","permalink":"https://michaelsoprano.com/talk/longitudinal-loyalty-understanding-the-barriers-to-running-longitudinal-studies-on-crowdsourcing-platforms/","publishdate":"2022-06-15T00:00:00Z","relpermalink":"/talk/longitudinal-loyalty-understanding-the-barriers-to-running-longitudinal-studies-on-crowdsourcing-platforms/","section":"event","summary":"Internal Talk - Centrum Wiskunde \u0026 Informatica. June 14th, Amsterdam, The Netherlands.","tags":["crowdsourcing","longitudinal study","survey"],"title":"Longitudinal Loyalty: Understanding the Barriers to Running Longitudinal Studies on Crowdsourcing Platforms","type":"event"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1645574400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"4f0016a16cfec1e872ab8b5f250b1a75","permalink":"https://michaelsoprano.com/talk/crowd_frame-a-simple-and-complete-framework-to-deploy-complex-crowdsourcing-tasks-off-the-shelf/","publishdate":"2022-02-23T00:00:00Z","relpermalink":"/talk/crowd_frame-a-simple-and-complete-framework-to-deploy-complex-crowdsourcing-tasks-off-the-shelf/","section":"event","summary":"Conference Talk - The 15th ACM International Conference on Web Search and Data Mining. February 15th, 2022. Phoenix, U.S.A. Held remotely as a pre-recorded contribution due to the COVID-19 pandemic.","tags":["crowdsourcing","framework","user behavior"],"title":"Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelf","type":"event"},{"authors":["Michael Soprano","Kevin Roitero","Francesco Bombassei De Bona","Stefano Mizzaro"],"categories":[],"content":"","date":1644883200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"ecc886a321c43b35f809279214172822","permalink":"https://michaelsoprano.com/publication/conference-paper-wsdm-2022/","publishdate":"2022-02-15T14:36:35.998652Z","relpermalink":"/publication/conference-paper-wsdm-2022/","section":"publication","summary":"Due to their relatively low cost and ability to scale, crowdsourcing based approaches are widely used to collect a large amount of human annotated data. To this aim, multiple crowdsourcing platforms exist, where requesters can upload tasks and workers can carry them out and obtain payment in return. Such platforms share a task design and deploy workflow that is often counter-intuitive and cumbersome. To address this issue, we propose Crowd_Frame, a simple and complete framework which allows to develop and deploy diverse types of complex crowdsourcing tasks in an easy and customizable way. We show the abilities of the proposed framework and we make it available to researchers and practitioners.","tags":["framework","crowdsourcing","user behavior"],"title":"Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelf","type":"publication"},{"authors":["Michael Soprano","Kevin Roitero","David La Barbera","Davide Ceolin","Damiano Spina","Stefano Mizzaro"],"categories":[],"content":"","date":1635724800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"cdbd712d513d7625c4ef66c448f0c3ed","permalink":"https://michaelsoprano.com/publication/journal-paper-ipm-2021/","publishdate":"2021-09-15T15:00:00+01:00","relpermalink":"/publication/journal-paper-ipm-2021/","section":"publication","summary":"Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a balanced set of workers with different backgrounds and cognitive abilities; (2) using an adequate set of mechanisms to control the quality of the collected data; and (3) using a coarse grained assessment scale, the crowd can provide reliable identification of fake news. However, fake news are a subtle matter: statements can be just biased (“cherrypicked”), imprecise, wrong, etc. and the unidimensional truth scale used in existing work cannot account for such differences. In this paper we propose a multidimensional notion of truthfulness and we ask the crowd workers to assess seven different dimensions of truthfulness selected based on existing literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker’s Trustworthiness, and Informativeness. We deploy a set of quality control mechanisms to ensure that the thousands of assessments collected on 180 publicly available fact-checked statements distributed over two datasets are of adequate quality, including a custom search engine used by the crowd workers to find web pages supporting their truthfulness assessments. A comprehensive analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments are reliable when compared to an expert-provided gold standard; (2) the proposed dimensions of truthfulness capture independent pieces of information; (3) the crowdsourcing task can be easily learned by the workers; and (4) the resulting assessments provide a useful basis for a more complete estimation of statement truthfulness.","tags":["truthfulness","crowdsourcing","misinformation","explainability"],"title":"The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale","type":"publication"},{"authors":["Erik Brand","Michael Soprano","Kevin Roitero","Michael Soprano","Gianluca Demartini"],"categories":[],"content":"","date":1634342400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668602356,"objectID":"8806a23701d3f3ffc0c1fd5cbd03cdcb","permalink":"https://michaelsoprano.com/publication/conference-paper-tto-2021/","publishdate":"2021-10-06T14:36:35.998652Z","relpermalink":"/publication/conference-paper-tto-2021/","section":"publication","summary":"Automated fact-checking (AFC) systems exist to combat disinformation, however their complexity makes them opaque to the end user, making it difficult to foster trust. In this paper, we introduce the E-BART model with the hope of making progress on this front. E-BART is able to provide a veracity prediction for a claim, and jointly generate a human-readable explanation for this decision. We show that E-BART is competitive with the state-of-theart on the e-FEVER and e-SNLI tasks. In addition, we validate the joint-prediction architecture by showing 1) that generating explanations does not significantly impede the model from performing well in its main task of veracity prediction, and 2) that predicted veracity and explanations are more internally coherent when generated jointly than separately. Finally, we also conduct human evaluations on the impact of generated explanations and observe that explanations increase human ability to spot misinformation and make people more skeptical about claims.","tags":null,"title":"E-BART: Jointly Predicting and Explaining Truthfulness","type":"publication"},{"authors":["Michael Soprano"],"categories":["teaching"],"content":"Teachers Stefano Mizzaro - Theory Michael Soprano - Laboratory Aims The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, Twitter, etc.) and the crowdsourcing phenomenon.\nIn the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, theoretical foundations, and practical applications.\nTable of contents I contributed to the course by teaching the practical applications of the concepts outlined by Professor Stefano Mizzaro in a total of six lectures (12 hours). Additionally, I assigned course assignments to the students. The course is conducted in Italian.\n Lesson 1: Python / Rest: Richiami Lesson 2: Twitter API: Tweepy Lesson 3: Twitter API: Endpoint V2 Lesson 4: Grafi: NetworkX Lesson 5-6: Crowd_Frame ","date":1633863600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1728390520,"objectID":"de1d56014f1945e53d5be6adc54ac74a","permalink":"https://michaelsoprano.com/post/social-computing-2021_2022/","publishdate":"2021-10-10T11:00:00Z","relpermalink":"/post/social-computing-2021_2022/","section":"post","summary":"Bachelor's Degree in Internet of Things, Big Data \u0026 Web at the University of Udine. Academic Year 2021/2022. Lectures: 6. Hours: 12","tags":["Python","Jupyter","Twitter API","Tweepy","Postman","Graph","NetworkX","PyViz","Amazon Mechanical Turk","Angular","Crowdsourcing","Data Analysis"],"title":"Social Computing - 2021/2022","type":"post"},{"authors":["Kevin Roitero","Michael Soprano","Beatrice Portelli","Massimiliano De Luise","Vincenzo Della Mea","Giuseppe Serra","Stefano Mizzaro","Gianluca Demartini"],"categories":[],"content":"","date":1631750400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"d39e4e4a8fad306722a6298f1ae8170a","permalink":"https://michaelsoprano.com/publication/journal-paper-puc-2021/","publishdate":"2021-09-15T15:00:00+01:00","relpermalink":"/publication/journal-paper-puc-2021/","section":"publication","summary":"Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of non-expert is exploited. We study whether crowdsourcing is an effective and reliable method to assess truthfulness during a pandemic, targeting statements related to COVID-19, thus addressing (mis)information that is both related to a sensitive and personal issue and very recent as compared to when the judgment is done. In our experiments, crowd workers are asked to assess the truthfulness of statements, and to provide evidence for the assessments. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we report results on workers behavior, agreement among workers, effect of aggregation functions, of scales transformations, and of workers background and bias. We perform a longitudinal study by re-launching the task multiple times with both novice and experienced workers, deriving important insights on how the behavior and quality change over time. Our results show that workers are able to detect and objectively categorize online (mis)information related to COVID-19; both crowdsourced and expert judgments can be transformed and aggregated to improve quality; worker background and other signals (e.g., source of information, behavior) impact the quality of the data. The longitudinal study demonstrates that the time-span has a major effect on the quality of the judgments, for both novice and experienced workers. Finally, we provide an extensive failure analysis of the statements misjudged by the crowd-workers.","tags":["truthfulness","crowdsourcing","misinformation","covid"],"title":"Can The Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation About COVID-19","type":"publication"},{"authors":["Davide Ceolin","Giuseppe Primiero","Jan Wielemaker","Michael Soprano"],"categories":[],"content":"","date":1620691200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"2d5d3cbe6fbf6fa476bdcfa1ff8da2d0","permalink":"https://michaelsoprano.com/publication/conference-paper-icwe-2021/","publishdate":"2021-05-11T14:00:00Z","relpermalink":"/publication/conference-paper-icwe-2021/","section":"publication","summary":"Review scores collect users opinions in a simple and intuitive manner. However, review scores are also easily manipulable, hence they are often accompanied by explanations. A substantial amount of research has been devoted to ascertaining the quality of reviews, to identify the most useful and authentic scores through explanation analysis. In this paper, we advance the state of the art in review quality analysis. We introduce a rating system to identify review arguments and to define an appropriate weighted semantics through formal argumentation theory. We introduce an algorithm to construct a corresponding graph, based on a selection of weighted arguments, their semantic similarity, and the supported ratings. We provide an algorithm to identify the model of such an argumentation graph, maximizing the overall weight of the admitted nodes and edges. We evaluate these contributions on the Amazon review dataset by McAuley et al.  [15], by comparing the results of our argumentation assessment with the upvotes received by the reviews. Also, we deepen the evaluation by crowdsourcing a multidimensional assessment of reviews and comparing it to the argumentation assessment. Lastly, we perform a user study to evaluate the explainability of our method. Our method achieves two goals: (1) it identifies reviews that are considered useful, comprehensible, truthful by online users and does so in an unsupervised manner, and (2) it provides an explanation of quality assessments.","tags":["formal argumentation theory","online reviews","information quality"],"title":"Assessing the Quality of Online Reviews Using Formal Argumentation Theory","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1603065600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"e280df8a5bf2b13f4f946322a0d9f695","permalink":"https://michaelsoprano.com/talk/the-covid-19-infodemic-can-the-crowd-judge-recent-misinformation-objectively/","publishdate":"2020-10-19T00:00:00Z","relpermalink":"/talk/the-covid-19-infodemic-can-the-crowd-judge-recent-misinformation-objectively/","section":"event","summary":"Conference Talk - The 29th ACM International Conference on Information and Knowledge Management (CIKM 2020). October 19th, 2020. Galway, Ireland. Held remotely as a pre-recorded contribution due to the COVID-19 pandemic.","tags":["covid-19","crowdsourcing"],"title":"The COVID-19 Infodemic: Can The Crowd Judge Recent Misinformation Objectively?","type":"event"},{"authors":["Kevin Roitero","Michael Soprano","Beatrice Portelli","Damiano Spina","Vincenzo Della Mea","Giuseppe Serra","Stefano Mizzaro","Gianluca Demartini"],"categories":[],"content":"","date":1603065600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"b7c6cb68071cb5ffba17721e2afe0c55","permalink":"https://michaelsoprano.com/publication/conference-paper-cikm-2020/","publishdate":"2021-03-15T14:36:35.536697Z","relpermalink":"/publication/conference-paper-cikm-2020/","section":"publication","summary":"Misinformation is an ever increasing problem that is difficult to solve for the research community and has a negative impact on the society at large. Very recently, the problem has been addressed with a crowdsourcing-based approach to scale up labeling efforts: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of (non-expert) judges is exploited. We follow the same approach to study whether crowdsourcing is an effective and reliable method to assess statements truthfulness during a pandemic. We specifically target statements related to the COVID-19 health emergency, that is still ongoing at the time of the study and has arguably caused an increase of the amount of misinformation that is spreading online (a phenomenon for which the term \\\"infodemic\\\" has been used). By doing so, we are able to address (mis)information that is both related to a sensitive and personal issue like health and very recent as compared to when the judgment is done: two issues that have not been analyzed in related work.In our experiment, crowd workers are asked to assess the truthfulness of statements, as well as to provide evidence for the assessments as a URL and a text justification. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we also report results on many different aspects, including: agreement among workers, the effect of different aggregation functions, of scales transformations, and of workers background / bias. We also analyze workers behavior, in terms of queries submitted, URLs found / selected, text justifications, and other behavioral data like clicks and mouse actions collected by means of an ad hoc logger.","tags":["ordinal classification","infodemic","information behavior","covid-19","crowdsourcing"],"title":"The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?","type":"publication"},{"authors":["Michael Soprano"],"categories":["teaching"],"content":"Teachers Stefano Mizzaro - Theory Michael Soprano - Laboratory Aims The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, Twitter, etc.) and the crowdsourcing phenomenon.\nIn the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, theoretical foundations, and practical applications.\nTable of contents I contributed to the course by teaching the practical applications of the concepts outlined by Professor Stefano Mizzaro in a total of six lectures (12 hours). Additionally, I assigned course assignments to the students. The course is conducted in Italian.\n Lesson 1: Twitter API: Introduzione \u0026amp; Richiami Lesson 2-3: Twitter API: Tweepy \u0026amp; Requests Lesson 4: Grafi: NetworkX Lesson 4-5: Crowd_Frame: How To ","date":1602327600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1728390520,"objectID":"95c36c21ea72ea7890dd0b4aeab82a0b","permalink":"https://michaelsoprano.com/post/social-computing-2020_2021/","publishdate":"2020-10-10T11:00:00Z","relpermalink":"/post/social-computing-2020_2021/","section":"post","summary":"Bachelor's Degree in Internet of Things, Big Data \u0026 Web at the University of Udine. Academic Year 2020/2021. Lectures: 6. Hours: 12","tags":["Python","Jupyter","Twitter API","Tweepy","Postman","Graph","NetworkX","PyViz","Amazon Mechanical Turk","Angular","Crowdsourcing","Data Analysis"],"title":"Social Computing - 2020/2021","type":"post"},{"authors":["Kevin Roitero","Michael Soprano","Shaoyang Fan","Damiano Spina","Stefano Mizzaro","Gianluca Demartini"],"categories":[],"content":"","date":1595635200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1686216050,"objectID":"390c638489d322ff5d76f2352ebd6880","permalink":"https://michaelsoprano.com/publication/conference-paper-sigir-2020/","publishdate":"2021-03-15T14:36:35.998652Z","relpermalink":"/publication/conference-paper-sigir-2020/","section":"publication","summary":"Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true and false statements. Usually such judgments are made by experts, like journalists for political statements or medical doctors for medical statements. In this paper, we follow a different approach and rely on (non-expert) crowd workers. This of course leads to the following research question: Can crowdsourcing be reliably used to assess the truthfulness of information and to create large-scale labeled collections for information credibility systems? To address this issue, we present the results of an extensive study based on crowdsourcing: we collect thousands of truthfulness assessments over two datasets, and we compare expert judgments with crowd judgments, expressed on scales with various granularity levels. We also measure the political bias and the cognitive background of the workers, and quantify their effect on the reliability of the data provided by the crowd.","tags":["classification","crowdsourcing","information credibility"],"title":"Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1595635200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"44da9a7f22f7ab656a77d63742ac41fe","permalink":"https://michaelsoprano.com/talk/can-the-crowd-identify-misinformation-objectively-the-effects-of-judgment-scale-and-assessors-background./","publishdate":"2020-07-25T00:00:00Z","relpermalink":"/talk/can-the-crowd-identify-misinformation-objectively-the-effects-of-judgment-scale-and-assessors-background./","section":"event","summary":"Conference Talk - The 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. July 25th, 2020. Xi'an, China. Held remotely as a pre-recorded contribution due to the COVID-19 pandemic.","tags":["disinformation","misinformation","crowdsourcing"],"title":"Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor’s Background.","type":"event"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1568592e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"b98a09de373c0ca77ce37253cd54a086","permalink":"https://michaelsoprano.com/talk/bias-and-fairness-in-effectiveness-evaluation-by-means-of-network-analysis-and-mixture-model/","publishdate":"2019-09-16T00:00:00Z","relpermalink":"/talk/bias-and-fairness-in-effectiveness-evaluation-by-means-of-network-analysis-and-mixture-model/","section":"event","summary":"Workshop Talk - Italian Information Retrieval Workshop (IIR 2019). September 16th, 2019. Padova, Italy.","tags":["bias","fairness","evaluation"],"title":"Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Model","type":"event"},{"authors":["Michael Soprano","Kevin Roitero","Stefano Mizzaro"],"categories":[],"content":"","date":1568592e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668602356,"objectID":"a8d187a3d1f0e575d6b3d060bc6fc815","permalink":"https://michaelsoprano.com/publication/workshop-paper-iir-2019/","publishdate":"2021-03-15T14:36:36.438925Z","relpermalink":"/publication/workshop-paper-iir-2019/","section":"publication","summary":"Information retrieval effectiveness evaluation is often carried out by means of test collections. Many works investigated possible sources of bias in such an approach. We propose a systematic approach to identify bias and its causes, and to remove it, thus enforcing fairness in effectiveness evaluation by means of test collections.","tags":["test collection","bias","fairness","network analysis"],"title":"Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Models","type":"publication"},{"authors":["Michael Soprano","Kevin Roitero","Stefano Mizzaro"],"categories":[],"content":"","date":1564012800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668602356,"objectID":"174c9ca5c39cc4b0e4cb9a7b075950d1","permalink":"https://michaelsoprano.com/publication/workshop-paper-birnld-2019/","publishdate":"2021-03-15T14:36:36.88953Z","relpermalink":"/publication/workshop-paper-birnld-2019/","section":"publication","summary":"Peer review is a well known mechanism exploited within the scholarly publishing process to ensure the quality of scientific literature. Such a mechanism, despite being well established and reasonable, is not free from problems, and alternative approaches to peer review have been developed. Such approaches exploit the readers of scientific publications and their opinions, and thus outsource the peer review activity to the scholar community; an example of this approach has been formalized in the Readersourcing model [5]. Our contribution is two-fold:(i) we propose a stochastic validation of the Readersourcing model, and (ii) we employ network analysis techniques to study the bias of the model, and in particular the interactions between readers and papers and their goodness and effectiveness scores. Our results show that by using network analysis interesting model properties can be derived.","tags":["peer review","network analysis","hits"],"title":"HITS Hits Readersourcing: Validating Peer Review Alternatives Using Network Analysis","type":"publication"},{"authors":["Michael Soprano","Stefano Mizzaro"],"categories":[],"content":"","date":1548892800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668600751,"objectID":"257fb4e5e2639d89aabf652f90a7a6e2","permalink":"https://michaelsoprano.com/publication/conference-paper-ircdl-2019/","publishdate":"2021-03-15T14:36:37.331078Z","relpermalink":"/publication/conference-paper-ircdl-2019/","section":"publication","summary":"This paper describes Readersourcing 2.0, an ecosystem providing an implementation of the Readersourcing approach proposed by Mizzaro [10]. Readersourcing is proposed as an alternative to the standard peer review activity that aims to exploit the otherwise lost opinions of readers. Readersourcing 2.0 implements two different models based on the so-called codetermination algorithms. We describe the requirements, present the overall architecture, and show how the end-user can interact with the system. Readersourcing 2.0 will be used in the future to study also other topics, like the idea of shepherding the users to achieve a better quality of the reviews and the differences between a review activity carried out with a single-blind or a double-blind approach.","tags":["scholarly publishing","peer review","crowdsourcing"],"title":"Crowdsourcing Peer Review: As We May Do","type":"publication"},{"authors":["Michael Soprano","Stefano Mizzaro"],"categories":[],"content":"","date":1548201600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668600751,"objectID":"4418b58d9df2b2fbe71b648dfddc3b83","permalink":"https://michaelsoprano.com/publication/conference-paper-aiudc-2019/","publishdate":"2021-03-15T14:36:37.769985Z","relpermalink":"/publication/conference-paper-aiudc-2019/","section":"publication","summary":"We propose an alternative approach to the standard peer review activity that aims to exploit the otherwise lost opinions of readers of publications which is called Readersourcing, originally proposed by Mizzaro [1]. Such an approach can be formalized by means of different models which share the same general principles. These models should be able to define a way, to measure the overall quality of a publication as well the reputation of a reader as an assessor; moreover, from these measures it should be possible to derive the reputation of a scholar as an author. We describe an ecosystem called Readersourcing 2.0 which provides an implementation for two Readersourcing models [2, 3] by outlining its goals and requirements. Readersourcing 2.0 will be used in the future to gather fresh data to analyze and validate.","tags":["scholarly publishing","peer review","crowdsourcing"],"title":"Crowdsourcing Peer Review in the Digital Humanities?","type":"publication"},{"authors":["Kevin Roitero","Michael Soprano","Andrea Brunello","Stefano Mizzaro"],"categories":[],"content":"","date":153576e4,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1686216050,"objectID":"1c2648a6ba53773cf225e3ddddb6e0a7","permalink":"https://michaelsoprano.com/publication/journal-paper-jdiq-2018/","publishdate":"2021-03-15T14:36:38.653339Z","relpermalink":"/publication/journal-paper-jdiq-2018/","section":"publication","summary":"Effectiveness evaluation of information retrieval systems by means of a test collection is a widely used methodology. However, it is rather expensive in terms of resources, time, and money; therefore, many researchers have proposed methods for a cheaper evaluation. One particular approach, on which we focus in this article, is to use fewer topics: in TREC-like initiatives, usually system effectiveness is evaluated as the average effectiveness on a set of n topics (usually, n=50, but more than 1,000 have been also adopted); instead of using the full set, it has been proposed to find the best subsets of a few good topics that evaluate the systems in the most similar way to the full set. The computational complexity of the task has so far limited the analysis that has been performed. We develop a novel and efficient approach based on a multi-objective evolutionary algorithm. The higher efficiency of our new implementation allows us to reproduce some notable results on topic set reduction, as well as perform new experiments to generalize and improve such results. We show that our approach is able to both reproduce the main state-of-the-art results and to allow us to analyze the effect of the collection, metric, and pool depth used for the evaluation. Finally, differently from previous studies, which have been mainly theoretical, we are also able to discuss some practical topic selection strategies, integrating results of automatic evaluation approaches.","tags":["topic selection strategy","evolutionary algorithms","reproducibility","test collection","few topics","topic sets"],"title":"Reproduce and Improve: An Evolutionary Approach to Select a Few Good Topics for Information Retrieval Evaluation","type":"publication"},{"authors":["Kevin Roitero","Michael Soprano","Stefano Mizzaro"],"categories":[],"content":"","date":1531008e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1686216050,"objectID":"725560812dc18a4727af0d86074c5700","permalink":"https://michaelsoprano.com/publication/conference-paper-sigir-2018/","publishdate":"2021-03-15T14:36:38.202502Z","relpermalink":"/publication/conference-paper-sigir-2018/","section":"publication","summary":"Several researchers have proposed to reduce the number of topics used in TREC-like initiatives. One research direction that has been pursued is what is the optimal topic subset of a given cardinality that evaluates the systems/runs in the most accurate way. Such a research direction has been so far mainly theoretical, with almost no indication on how to select the few good topics in practice. We propose such a practical criterion for topic selection: we rely on the methods for automatic system evaluation without relevance judgments, and by running some experiments on several TREC collections we show that the topics selected on the basis of those evaluations are indeed more informative than random topics.","tags":["topic selection","trec","few topics","test collections"],"title":"Effectiveness Evaluation with a Subset of Topics: A Practical Approach","type":"publication"},{"authors":null,"categories":null,"content":"","date":-62135596800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1615470069,"objectID":"f26b5133c34eec1aa0a09390a36c2ade","permalink":"https://michaelsoprano.com/admin/config.yml","publishdate":"0001-01-01T00:00:00Z","relpermalink":"/admin/config.yml","section":"","summary":"","tags":null,"title":"","type":"wowchemycms"},{"authors":null,"categories":null,"content":"The site is completely static, no use of server-side cookies is made and the only information that might be stored is your choice to see the page in the night mode.\nTherefore there’s no harm to your privacy, unless you explicitly contact me by email or through a social network.\n","date":-62135596800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1615898929,"objectID":"18d05a63a1c8d7ed973cc51838494e41","permalink":"https://michaelsoprano.com/privacy/","publishdate":"0001-01-01T00:00:00Z","relpermalink":"/privacy/","section":"","summary":"The site is completely static, no use of server-side cookies is made and the only information that might be stored is your choice to see the page in the night mode.","tags":null,"title":"Privacy Policy","type":"page"}] \ No newline at end of file +[{"authors":null,"categories":null,"content":"Benvignût! This is my personal webpage. I am a PostDoc at the University of Udine, affiliated with the Social, Media, Data \u0026amp; Crowd Laboratory, and I hold a Ph.D. in Computer Science, Mathematics, and Physics.\nMy main research interests include Human Computation, Crowdsourcing, and Information Retrieval. Currently, I am focused on leveraging crowdsourcing to address the growing challenge of misinformation spread.\nIn my free time, I enjoy discovering new literature and work towards being a decent runner, hiker, and skier.\n Download my Curriculum Vitae in English or Italian. Feel free to explore my Master's, Bachelor's, and High School theses (in Italian).\n","date":-62135596800,"expirydate":-62135596800,"kind":"term","lang":"en","lastmod":1728386301,"objectID":"2525497d367e79493fd32b198b28f040","permalink":"","publishdate":"0001-01-01T00:00:00Z","relpermalink":"","section":"authors","summary":"Benvignût! This is my personal webpage. I am a PostDoc at the University of Udine, affiliated with the Social, Media, Data \u0026amp; Crowd Laboratory, and I hold a Ph.D. in Computer Science, Mathematics, and Physics.","tags":null,"title":"Michael Soprano","type":"authors"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1726572600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1726576025,"objectID":"c4cd58d6905d915159aabedf08ff501c","permalink":"https://michaelsoprano.com/talk/deploying-complex-crowdsourcing-tasks-without-fear-an-introduction-to-crowd_frame/","publishdate":"2024-09-17T11:30:00Z","relpermalink":"/talk/deploying-complex-crowdsourcing-tasks-without-fear-an-introduction-to-crowd_frame/","section":"event","summary":"Demo Presentation - The 3rd Italian Conference on Big Data and Data Science (ITADATA 2024). Pisa, Italy.","tags":["demo","human computation","crowdsourcing","crowd_frame"],"title":"Deploying Complex Crowdsourcing Tasks Without Fear: An Introduction to Crowd_Frame","type":"event"},{"authors":["Jaspreet Singh","Michael Soprano","Kevin Roitero","Davide Ceolin"],"categories":[],"content":"","date":1725148800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1725184728,"objectID":"37de9c683bbb318b815214077838e5d7","permalink":"https://michaelsoprano.com/publication/conference-paper-misdoom-2024/","publishdate":"2024-09-01T00:00:00Z","relpermalink":"/publication/conference-paper-misdoom-2024/","section":"publication","summary":"This paper explores the use of crowdsourcing to classify statement types in film reviews to assess their information quality. Employing the Argument Type Identification Procedure which uses the Periodic Table of Arguments to categorize arguments, the study aims to connect statement types to the overall argument strength and information reliability. Focusing on non-expert annotators in a crowdsourcing environment, the research assesses their reliability based on various factors including language proficiency and annotation experience. Results indicate the importance of careful annotator selection and training to achieve high inter-annotator agreement and highlight challenges in crowdsourcing statement classification for information quality assessment.","tags":["crowdsourcing annotation","information quality assessment","argument type identification"],"title":"Crowdsourcing Statement Classification to Enhance Information Quality Prediction","type":"publication"},{"authors":["Michael Soprano","Kevin Roitero","Ujwal Gadiraju","Eddy Maddalena","Gianluca Demartini"],"categories":[],"content":"","date":1720569600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1720705059,"objectID":"45b5e1627d200b8e3002b8d06787a02e","permalink":"https://michaelsoprano.com/publication/journal-paper-tsc-2024/","publishdate":"2024-07-10T15:00:00+01:00","relpermalink":"/publication/journal-paper-tsc-2024/","section":"publication","summary":"Crowdsourcing tasks have been widely used to collect a large number of human labels at scale. While some of these tasks are deployed by requesters and performed only once by crowd workers, others require the same worker to perform the same task or a variant of it more than once, thus participating in a so-called longitudinal study. Despite the prevalence of longitudinal studies in crowdsourcing, there is a limited understanding of factors that influence worker participation in them across different crowdsourcing marketplaces. We present results from a large-scale survey of 300 workers on 3 different micro-task crowdsourcing platforms: Amazon Mechanical Turk, Prolific and Toloka. The aim is to understand how longitudinal studies are performed using crowdsourcing. We collect answers about 547 experiences and we analyze them both quantitatively and qualitatively. We synthesize 17 take-home messages about longitudinal studies together with 8 recommendations for task requesters and 5 best practices for crowdsourcing platforms to adequately conduct and support such kinds of studies. We release the survey and the data at: https://osf.io/h4du9/.","tags":["longitudinal studies","crowdsourcing platforms","surveys","online sampling","amazon mechanical turk","prolific","toloka"],"title":"Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platforms","type":"publication"},{"authors":["David La Barbera","Eddy Maddalena","Michael Soprano","Kevin Roitero","Gianluca Demartini","Davide Ceolin","Damiano Spina","Stefano Mizzaro"],"categories":[],"content":"","date":1717113600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1717244786,"objectID":"a51d41f7a834b97be079ca417f213722","permalink":"https://michaelsoprano.com/publication/journal-paper-ipm-2024-2/","publishdate":"2024-05-31T15:00:00+01:00","relpermalink":"/publication/journal-paper-ipm-2024-2/","section":"publication","summary":"There is an important ongoing effort aimed to tackle misinformation and to perform reliable fact-checking by employing human assessors at scale, with a crowdsourcing-based approach. Previous studies on the feasibility of employing crowdsourcing for the task of misinformation detection have provided inconsistent results: some of them seem to confirm the effectiveness of crowdsourcing for assessing the truthfulness of statements and claims, whereas others fail to reach an effectiveness level higher than automatic machine learning approaches, which are still unsatisfactory. In this paper, we aim at addressing such inconsistency and understand if truthfulness assessment can indeed be crowdsourced effectively. To do so, we build on top of previous studies; we select some of those reporting low effectiveness levels, we highlight their potential limitations, and we then reproduce their work attempting to improve their setup to address those limitations. We employ various approaches, data quality levels, and agreement measures to assess the reliability of crowd workers when assessing the truthfulness of (mis)information. Furthermore, we explore different worker features and compare the results obtained with different crowds. According to our findings, crowdsourcing can be used as an effective methodology to tackle misinformation at scale. When compared to previous studies, our results indicate that a significantly higher agreement between crowd workers and experts can be obtained by using a different, higher-quality, crowdsourcing platform and by improving the design of the crowdsourcing task. Also, we find differences concerning task and worker features and how workers provide truthfulness assessments.","tags":["crowdsourcing","misinformation","fact-checking","truthfulness assessment"],"title":"Crowdsourced Fact-checking: Does It Actually Work?","type":"publication"},{"authors":["Michael Soprano","Kevin Roitero","David La Barbera","Davide Ceolin","Damiano Spina","Gianluca Demartini","Stefano Mizzaro"],"categories":[],"content":"","date":1706918400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1706992044,"objectID":"af6209263d25032cabcb8be029e4ac73","permalink":"https://michaelsoprano.com/publication/journal-paper-ipm-2024/","publishdate":"2024-02-03T15:00:00+01:00","relpermalink":"/publication/journal-paper-ipm-2024/","section":"publication","summary":"The increase of the amount of misinformation spread every day online is a huge threat to the society. Organizations and researchers are working to contrast this misinformation plague. In this setting, human assessors are indispensable to correctly identify, assess and/or revise the truthfulness of information items, i.e., to perform the fact-checking activity. Assessors, as humans, are subject to systematic errors that might interfere with their fact-checking activity. Among such errors, cognitive biases are those due to the limits of human cognition. Although biases help to minimize the cost of making mistakes, they skew assessments away from an objective perception of information. Cognitive biases, hence, are particularly frequent and critical, and can cause errors that have a huge potential impact as they propagate not only in the community, but also in the datasets used to train automatic and semi-automatic machine learning models to fight misinformation. In this work, we present a review of the cognitive biases which might occur during the fact-checking process. In more detail, inspired by PRISMA – a methodology used for systematic literature reviews – we manually derive a list of 221 cognitive biases that may affect human assessors. Then, we select the 39 biases that might manifest during the fact-checking process, we group them into categories, and we provide a description. Finally, we present a list of 11 countermeasures that can be adopted by researchers, practitioners, and organizations to limit the effect of the identified cognitive biases on the fact-checking activity.","tags":["cognitive bias","misinformation","fact-checking","truthfulness assessment"],"title":"Cognitive Biases in Fact-Checking and Their Countermeasures: A Review","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1698309e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1698394801,"objectID":"a4927be3f1353c53366ddd8350f71765","permalink":"https://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-healthcare/","publishdate":"2023-10-26T08:30:00Z","relpermalink":"/talk/towards-a-conversational-based-agent-for-healthcare/","section":"event","summary":"Workshop Talk - European Federation of Medical Informatics Special Topic Conference 2023 (EFMI STC 2023). Turin, Italy.","tags":["conversational agent","electronic health record","large language models","public administration"],"title":"Towards a Conversational-Based Agent for Healthcare","type":"event"},{"authors":["Michael Soprano","Kevin Roitero","Vincenzo Della Mea","Stefano Mizzaro"],"categories":[],"content":"","date":1695168e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1697143748,"objectID":"343710264000293cf3f4d7227bc8744b","permalink":"https://michaelsoprano.com/publication/workshop-paper-ital-ia-2023/","publishdate":"2023-09-20T17:40:00+01:00","relpermalink":"/publication/workshop-paper-ital-ia-2023/","section":"publication","summary":"Conversational agents provide new modalities to access and interact with services and applications. Recently, they saw a backfire in their popularity, due to the recent advancements in language models. Such agents have been adopted in various fields such as healthcare and education, yet they received little attention in public administration. We describe as a practical use case a service of the portal that provides citizens of the Italian region of Friuli-Venezia Giulia with services related to their own Electronic Health Records. The service considered allows them to search for the available doctors and pediatricians in the region's municipalities. We rely on the use case described to propose a model for a conversational agent-based access modality. The model proposed allows us to lay the foundation for more advanced chatbot-like implementations which will use also alternative input modalities, such as voice-based communication.","tags":["public administration","electronic health record","conversational agents"],"title":"Towards a Conversational-Based Agent for Health Services","type":"publication"},{"authors":["David La Barbera","Michael Soprano","Kevin Roitero","Eddy Maddalena","Stefano Mizzaro"],"categories":[],"content":"","date":1691193600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694093922,"objectID":"3834b58b74a20f1cd91a82784e2f3b63","permalink":"https://michaelsoprano.com/publication/workshop-paper-iir-2023/","publishdate":"2023-08-05T15:30:30+01:00","relpermalink":"/publication/workshop-paper-iir-2023/","section":"publication","summary":"In this paper, we present our journey in exploring the use of crowdsourcing for fact-checking. We discuss our early experiments aimed towards the identification of the best possible setting for misinformation assessment using crowdsourcing. Our results indicate that the crowd can effectively address misinformation at scale, showing some degree of correlation with experts. We also highlight the influence of worker background on the quality of truthfulness assessments.","tags":["crowdsourcing","human computation","fact-checking","misinformation","truthfulness assessment"],"title":"Fact-Checking at Scale with Crowdsourcing Experiments and Lessons Learned","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1685318400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1698394826,"objectID":"68aec2d0cd76ae46198120e98755b94a","permalink":"https://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-health-services/","publishdate":"2023-05-29T00:00:00Z","relpermalink":"/talk/towards-a-conversational-based-agent-for-health-services/","section":"event","summary":"Workshop Talk - Terzo Convegno Nazionale CINI sull'Intelligenza Artificiale (Ital-IA 2023). May 29th, 2023. Pisa, Italy.","tags":["conversational agent","public administration"],"title":"Towards a Conversational-Based Agent for Health Services","type":"event"},{"authors":["Kevin Roitero","David La Barbera","Michael Soprano","Gianluca Demartini","Stefano Mizzaro","Tetsuya Sakai"],"categories":[],"content":"","date":1684713600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1686216050,"objectID":"6486499d1482393e35a34e97f1c3935d","permalink":"https://michaelsoprano.com/publication/journal-paper-tois-2023/","publishdate":"2021-09-15T15:00:00+01:00","relpermalink":"/publication/journal-paper-tois-2023/","section":"publication","summary":"To scale the size of Information Retrieval collections, crowdsourcing has become a common way to collect relevance judgments at scale. Crowdsourcing experiments usually employ 100-10,000 workers, but such a number is often decided in a heuristic way. The downside is that the resulting dataset does not have any guarantee of meeting predefined statistical requirements as, for example, have enough statistical power to be able to distinguish in a statistically significant way between the relevance of two documents. We propose a methodology adapted from literature on sound topic set size design, based on t-test and ANOVA, which aims at guaranteeing the resulting dataset to meet a predefined set of statistical requirements. We validate our approach on several public datasets. Our results show that we can reliably estimate the recommended number of workers needed to achieve statistical power, and that such estimation is dependent on the topic, while the effect of the relevance scale is limited. Furthermore, we found that such estimation is dependent on worker features such as agreement. Finally, we describe a set of practical estimation strategies that can be used to estimate the worker set size, and we also provide results on the estimation of document set sizes.","tags":["relevance judgments","statistical analysis","crowdsourcing"],"title":"How Many Crowd Workers Do I Need? On Statistical Power When Crowdsourcing Relevance Judgments","type":"publication"},{"authors":["Davide Ceolin","Giuseppe Primiero","Michael Soprano","Jan Wielemaker"],"categories":[],"content":"","date":1659052800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"f5bea931bb7158f139afb104d61db308","permalink":"https://michaelsoprano.com/publication/journal-paper-is-2022/","publishdate":"2022-07-29T15:00:00+01:00","relpermalink":"/publication/journal-paper-is-2022/","section":"publication","summary":"Review scores collect users’ opinions in a simple and intuitive manner. However, review scores are also easily manipulable, hence they are often accompanied by explanations. A substantial amount of research has been devoted to ascertaining the quality of reviews, to identify the most useful and authentic scores through explanation analysis. In this paper, we advance the state of the art in review quality analysis. We introduce a rating system to identify review arguments and to define an appropriate weighted semantics through formal argumentation theory. We introduce an algorithm to construct a corresponding graph, based on a selection of weighted arguments, their semantic distance, and the supported ratings. We also provide an algorithm to identify the model of such an argumentation graph, maximizing the overall weight of the admitted nodes and edges. We evaluate these contributions on the Amazon review dataset by McAuley et al. (2015), by comparing the results of our argumentation assessment with the upvotes received by the reviews. Also, we deepen the evaluation by crowdsourcing a multidimensional assessment of reviews and comparing it to the argumentation assessment. Lastly, we perform a user study to evaluate the explainability of our method, i.e., to test whether the automated method we use to assess reviews is understandable by humans. Our method achieves two goals: (1) it identifies reviews that are considered useful, comprehensible, and complete by online users, and does so in an unsupervised manner, and (2) it provides an explanation of quality assessments.","tags":[],"title":"Transparent assessment of information quality of online reviews using formal argumentation theory","type":"publication"},{"authors":["Erik Brand","Kevin Roitero","Michael Soprano","Afshin Rahimi","Gianluca Demartini"],"categories":[],"content":"","date":1657324800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"c7ad261b2704356f61cb1b48f634be9a","permalink":"https://michaelsoprano.com/publication/journal-paper-jdiq-2022/","publishdate":"2021-09-15T15:00:00+01:00","relpermalink":"/publication/journal-paper-jdiq-2022/","section":"publication","summary":"Automated fact-checking (AFC) systems exist to combat disinformation, however their complexity usually makes them opaque to the end user, making it difficult to foster trust in the system. In this paper, we introduce the E-BART model with the hope of making progress on this front. E-BART is able to provide a veracity prediction for a claim, and jointly generate a human-readable explanation for this decision. We show that E-BART is competitive with the state-of-the-art on the e-FEVER and e-SNLI tasks. In addition, we validate the joint-prediction architecture by showing 1) that generating explanations does not significantly impede the model from performing well in its main task of veracity prediction, and 2) that predicted veracity and explanations are more internally coherent when generated jointly than separately. We also calibrate the E-BART model, allowing the output of the final model be correctly interpreted as the confidence of correctness. Finally, we also conduct and extensive human evaluation on the impact of generated explanations and observe that: explanations increase human ability to spot misinformation and make people more skeptical about claims, and explanations generated by E-BART are competitive with ground truth explanations.","tags":[],"title":"A Neural Model to Jointly Predict and Explain Truthfulness of Statements","type":"publication"},{"authors":["Tim Draws","David La Barbera","Michael Soprano","Kevin Roitero","Davide Ceolin","Alessandro Checco","Stefano Mizzaro"],"categories":[],"content":"","date":1655769600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627976,"objectID":"e9991656c12f02af2ce45a24adb3515d","permalink":"https://michaelsoprano.com/publication/conference-paper-facct-2022/","publishdate":"2021-03-15T14:36:35.536697Z","relpermalink":"/publication/conference-paper-facct-2022/","section":"publication","summary":"Due to the increasing amount of information shared online every day, the need for sound and reliable ways of distinguishing between trustworthy and non-trustworthy information is as present as ever. One technique for performing fact-checking at scale is to employ human intelligence in the form of crowd workers. Although earlier work has suggested that crowd workers can reliably identify misinformation, cognitive biases of crowd workers may reduce the quality of truthfulness judgments in this context. We performed a systematic exploratory analysis of publicly available crowdsourced data to identify a set of potential systematic biases that may occur when crowd workers perform fact-checking tasks. Following this exploratory study, we collected a novel data set of crowdsourced truthfulness judgments to validate our hypotheses. Our findings suggest that workers generally overestimate the truthfulness of statements and that different individual characteristics (i.e., their belief in science) and cognitive biases (i.e., the affect heuristic and overconfidence) can affect their annotations. Interestingly, we find that, depending on the general judgment tendencies of workers, their biases may sometimes lead to more accurate judgments.","tags":["bias","crowdsourcing","misinformation","explainability","truthfulness"],"title":"The Effects of Crowd Worker Biases in Fact-Checking Tasks","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1655251200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"9338404eca03ca4ce1ca36306994970f","permalink":"https://michaelsoprano.com/talk/longitudinal-loyalty-understanding-the-barriers-to-running-longitudinal-studies-on-crowdsourcing-platforms/","publishdate":"2022-06-15T00:00:00Z","relpermalink":"/talk/longitudinal-loyalty-understanding-the-barriers-to-running-longitudinal-studies-on-crowdsourcing-platforms/","section":"event","summary":"Internal Talk - Centrum Wiskunde \u0026 Informatica. June 14th, Amsterdam, The Netherlands.","tags":["crowdsourcing","longitudinal study","survey"],"title":"Longitudinal Loyalty: Understanding the Barriers to Running Longitudinal Studies on Crowdsourcing Platforms","type":"event"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1645574400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"4f0016a16cfec1e872ab8b5f250b1a75","permalink":"https://michaelsoprano.com/talk/crowd_frame-a-simple-and-complete-framework-to-deploy-complex-crowdsourcing-tasks-off-the-shelf/","publishdate":"2022-02-23T00:00:00Z","relpermalink":"/talk/crowd_frame-a-simple-and-complete-framework-to-deploy-complex-crowdsourcing-tasks-off-the-shelf/","section":"event","summary":"Conference Talk - The 15th ACM International Conference on Web Search and Data Mining. February 15th, 2022. Phoenix, U.S.A. Held remotely as a pre-recorded contribution due to the COVID-19 pandemic.","tags":["crowdsourcing","framework","user behavior"],"title":"Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelf","type":"event"},{"authors":["Michael Soprano","Kevin Roitero","Francesco Bombassei De Bona","Stefano Mizzaro"],"categories":[],"content":"","date":1644883200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"ecc886a321c43b35f809279214172822","permalink":"https://michaelsoprano.com/publication/conference-paper-wsdm-2022/","publishdate":"2022-02-15T14:36:35.998652Z","relpermalink":"/publication/conference-paper-wsdm-2022/","section":"publication","summary":"Due to their relatively low cost and ability to scale, crowdsourcing based approaches are widely used to collect a large amount of human annotated data. To this aim, multiple crowdsourcing platforms exist, where requesters can upload tasks and workers can carry them out and obtain payment in return. Such platforms share a task design and deploy workflow that is often counter-intuitive and cumbersome. To address this issue, we propose Crowd_Frame, a simple and complete framework which allows to develop and deploy diverse types of complex crowdsourcing tasks in an easy and customizable way. We show the abilities of the proposed framework and we make it available to researchers and practitioners.","tags":["framework","crowdsourcing","user behavior"],"title":"Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelf","type":"publication"},{"authors":["Michael Soprano","Kevin Roitero","David La Barbera","Davide Ceolin","Damiano Spina","Stefano Mizzaro"],"categories":[],"content":"","date":1635724800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"cdbd712d513d7625c4ef66c448f0c3ed","permalink":"https://michaelsoprano.com/publication/journal-paper-ipm-2021/","publishdate":"2021-09-15T15:00:00+01:00","relpermalink":"/publication/journal-paper-ipm-2021/","section":"publication","summary":"Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a balanced set of workers with different backgrounds and cognitive abilities; (2) using an adequate set of mechanisms to control the quality of the collected data; and (3) using a coarse grained assessment scale, the crowd can provide reliable identification of fake news. However, fake news are a subtle matter: statements can be just biased (“cherrypicked”), imprecise, wrong, etc. and the unidimensional truth scale used in existing work cannot account for such differences. In this paper we propose a multidimensional notion of truthfulness and we ask the crowd workers to assess seven different dimensions of truthfulness selected based on existing literature: Correctness, Neutrality, Comprehensibility, Precision, Completeness, Speaker’s Trustworthiness, and Informativeness. We deploy a set of quality control mechanisms to ensure that the thousands of assessments collected on 180 publicly available fact-checked statements distributed over two datasets are of adequate quality, including a custom search engine used by the crowd workers to find web pages supporting their truthfulness assessments. A comprehensive analysis of crowdsourced judgments shows that: (1) the crowdsourced assessments are reliable when compared to an expert-provided gold standard; (2) the proposed dimensions of truthfulness capture independent pieces of information; (3) the crowdsourcing task can be easily learned by the workers; and (4) the resulting assessments provide a useful basis for a more complete estimation of statement truthfulness.","tags":["truthfulness","crowdsourcing","misinformation","explainability"],"title":"The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale","type":"publication"},{"authors":["Erik Brand","Michael Soprano","Kevin Roitero","Michael Soprano","Gianluca Demartini"],"categories":[],"content":"","date":1634342400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668602356,"objectID":"8806a23701d3f3ffc0c1fd5cbd03cdcb","permalink":"https://michaelsoprano.com/publication/conference-paper-tto-2021/","publishdate":"2021-10-06T14:36:35.998652Z","relpermalink":"/publication/conference-paper-tto-2021/","section":"publication","summary":"Automated fact-checking (AFC) systems exist to combat disinformation, however their complexity makes them opaque to the end user, making it difficult to foster trust. In this paper, we introduce the E-BART model with the hope of making progress on this front. E-BART is able to provide a veracity prediction for a claim, and jointly generate a human-readable explanation for this decision. We show that E-BART is competitive with the state-of-theart on the e-FEVER and e-SNLI tasks. In addition, we validate the joint-prediction architecture by showing 1) that generating explanations does not significantly impede the model from performing well in its main task of veracity prediction, and 2) that predicted veracity and explanations are more internally coherent when generated jointly than separately. Finally, we also conduct human evaluations on the impact of generated explanations and observe that explanations increase human ability to spot misinformation and make people more skeptical about claims.","tags":null,"title":"E-BART: Jointly Predicting and Explaining Truthfulness","type":"publication"},{"authors":["Kevin Roitero","Michael Soprano","Beatrice Portelli","Massimiliano De Luise","Vincenzo Della Mea","Giuseppe Serra","Stefano Mizzaro","Gianluca Demartini"],"categories":[],"content":"","date":1631750400,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"d39e4e4a8fad306722a6298f1ae8170a","permalink":"https://michaelsoprano.com/publication/journal-paper-puc-2021/","publishdate":"2021-09-15T15:00:00+01:00","relpermalink":"/publication/journal-paper-puc-2021/","section":"publication","summary":"Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of non-expert is exploited. We study whether crowdsourcing is an effective and reliable method to assess truthfulness during a pandemic, targeting statements related to COVID-19, thus addressing (mis)information that is both related to a sensitive and personal issue and very recent as compared to when the judgment is done. In our experiments, crowd workers are asked to assess the truthfulness of statements, and to provide evidence for the assessments. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we report results on workers behavior, agreement among workers, effect of aggregation functions, of scales transformations, and of workers background and bias. We perform a longitudinal study by re-launching the task multiple times with both novice and experienced workers, deriving important insights on how the behavior and quality change over time. Our results show that workers are able to detect and objectively categorize online (mis)information related to COVID-19; both crowdsourced and expert judgments can be transformed and aggregated to improve quality; worker background and other signals (e.g., source of information, behavior) impact the quality of the data. The longitudinal study demonstrates that the time-span has a major effect on the quality of the judgments, for both novice and experienced workers. Finally, we provide an extensive failure analysis of the statements misjudged by the crowd-workers.","tags":["truthfulness","crowdsourcing","misinformation","covid"],"title":"Can The Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation About COVID-19","type":"publication"},{"authors":["Davide Ceolin","Giuseppe Primiero","Jan Wielemaker","Michael Soprano"],"categories":[],"content":"","date":1620691200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"2d5d3cbe6fbf6fa476bdcfa1ff8da2d0","permalink":"https://michaelsoprano.com/publication/conference-paper-icwe-2021/","publishdate":"2021-05-11T14:00:00Z","relpermalink":"/publication/conference-paper-icwe-2021/","section":"publication","summary":"Review scores collect users opinions in a simple and intuitive manner. However, review scores are also easily manipulable, hence they are often accompanied by explanations. A substantial amount of research has been devoted to ascertaining the quality of reviews, to identify the most useful and authentic scores through explanation analysis. In this paper, we advance the state of the art in review quality analysis. We introduce a rating system to identify review arguments and to define an appropriate weighted semantics through formal argumentation theory. We introduce an algorithm to construct a corresponding graph, based on a selection of weighted arguments, their semantic similarity, and the supported ratings. We provide an algorithm to identify the model of such an argumentation graph, maximizing the overall weight of the admitted nodes and edges. We evaluate these contributions on the Amazon review dataset by McAuley et al.  [15], by comparing the results of our argumentation assessment with the upvotes received by the reviews. Also, we deepen the evaluation by crowdsourcing a multidimensional assessment of reviews and comparing it to the argumentation assessment. Lastly, we perform a user study to evaluate the explainability of our method. Our method achieves two goals: (1) it identifies reviews that are considered useful, comprehensible, truthful by online users and does so in an unsupervised manner, and (2) it provides an explanation of quality assessments.","tags":["formal argumentation theory","online reviews","information quality"],"title":"Assessing the Quality of Online Reviews Using Formal Argumentation Theory","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1603065600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"e280df8a5bf2b13f4f946322a0d9f695","permalink":"https://michaelsoprano.com/talk/the-covid-19-infodemic-can-the-crowd-judge-recent-misinformation-objectively/","publishdate":"2020-10-19T00:00:00Z","relpermalink":"/talk/the-covid-19-infodemic-can-the-crowd-judge-recent-misinformation-objectively/","section":"event","summary":"Conference Talk - The 29th ACM International Conference on Information and Knowledge Management (CIKM 2020). October 19th, 2020. Galway, Ireland. Held remotely as a pre-recorded contribution due to the COVID-19 pandemic.","tags":["covid-19","crowdsourcing"],"title":"The COVID-19 Infodemic: Can The Crowd Judge Recent Misinformation Objectively?","type":"event"},{"authors":["Kevin Roitero","Michael Soprano","Beatrice Portelli","Damiano Spina","Vincenzo Della Mea","Giuseppe Serra","Stefano Mizzaro","Gianluca Demartini"],"categories":[],"content":"","date":1603065600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1685627806,"objectID":"b7c6cb68071cb5ffba17721e2afe0c55","permalink":"https://michaelsoprano.com/publication/conference-paper-cikm-2020/","publishdate":"2021-03-15T14:36:35.536697Z","relpermalink":"/publication/conference-paper-cikm-2020/","section":"publication","summary":"Misinformation is an ever increasing problem that is difficult to solve for the research community and has a negative impact on the society at large. Very recently, the problem has been addressed with a crowdsourcing-based approach to scale up labeling efforts: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd of (non-expert) judges is exploited. We follow the same approach to study whether crowdsourcing is an effective and reliable method to assess statements truthfulness during a pandemic. We specifically target statements related to the COVID-19 health emergency, that is still ongoing at the time of the study and has arguably caused an increase of the amount of misinformation that is spreading online (a phenomenon for which the term \\\"infodemic\\\" has been used). By doing so, we are able to address (mis)information that is both related to a sensitive and personal issue like health and very recent as compared to when the judgment is done: two issues that have not been analyzed in related work.In our experiment, crowd workers are asked to assess the truthfulness of statements, as well as to provide evidence for the assessments as a URL and a text justification. Besides showing that the crowd is able to accurately judge the truthfulness of the statements, we also report results on many different aspects, including: agreement among workers, the effect of different aggregation functions, of scales transformations, and of workers background / bias. We also analyze workers behavior, in terms of queries submitted, URLs found / selected, text justifications, and other behavioral data like clicks and mouse actions collected by means of an ad hoc logger.","tags":["ordinal classification","infodemic","information behavior","covid-19","crowdsourcing"],"title":"The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?","type":"publication"},{"authors":["Kevin Roitero","Michael Soprano","Shaoyang Fan","Damiano Spina","Stefano Mizzaro","Gianluca Demartini"],"categories":[],"content":"","date":1595635200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1686216050,"objectID":"390c638489d322ff5d76f2352ebd6880","permalink":"https://michaelsoprano.com/publication/conference-paper-sigir-2020/","publishdate":"2021-03-15T14:36:35.998652Z","relpermalink":"/publication/conference-paper-sigir-2020/","section":"publication","summary":"Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true and false statements. Usually such judgments are made by experts, like journalists for political statements or medical doctors for medical statements. In this paper, we follow a different approach and rely on (non-expert) crowd workers. This of course leads to the following research question: Can crowdsourcing be reliably used to assess the truthfulness of information and to create large-scale labeled collections for information credibility systems? To address this issue, we present the results of an extensive study based on crowdsourcing: we collect thousands of truthfulness assessments over two datasets, and we compare expert judgments with crowd judgments, expressed on scales with various granularity levels. We also measure the political bias and the cognitive background of the workers, and quantify their effect on the reliability of the data provided by the crowd.","tags":["classification","crowdsourcing","information credibility"],"title":"Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background","type":"publication"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1595635200,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"44da9a7f22f7ab656a77d63742ac41fe","permalink":"https://michaelsoprano.com/talk/can-the-crowd-identify-misinformation-objectively-the-effects-of-judgment-scale-and-assessors-background./","publishdate":"2020-07-25T00:00:00Z","relpermalink":"/talk/can-the-crowd-identify-misinformation-objectively-the-effects-of-judgment-scale-and-assessors-background./","section":"event","summary":"Conference Talk - The 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. July 25th, 2020. Xi'an, China. Held remotely as a pre-recorded contribution due to the COVID-19 pandemic.","tags":["disinformation","misinformation","crowdsourcing"],"title":"Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor’s Background.","type":"event"},{"authors":["Michael Soprano"],"categories":null,"content":"","date":1568592e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1694424132,"objectID":"b98a09de373c0ca77ce37253cd54a086","permalink":"https://michaelsoprano.com/talk/bias-and-fairness-in-effectiveness-evaluation-by-means-of-network-analysis-and-mixture-model/","publishdate":"2019-09-16T00:00:00Z","relpermalink":"/talk/bias-and-fairness-in-effectiveness-evaluation-by-means-of-network-analysis-and-mixture-model/","section":"event","summary":"Workshop Talk - Italian Information Retrieval Workshop (IIR 2019). September 16th, 2019. Padova, Italy.","tags":["bias","fairness","evaluation"],"title":"Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Model","type":"event"},{"authors":["Michael Soprano","Kevin Roitero","Stefano Mizzaro"],"categories":[],"content":"","date":1568592e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668602356,"objectID":"a8d187a3d1f0e575d6b3d060bc6fc815","permalink":"https://michaelsoprano.com/publication/workshop-paper-iir-2019/","publishdate":"2021-03-15T14:36:36.438925Z","relpermalink":"/publication/workshop-paper-iir-2019/","section":"publication","summary":"Information retrieval effectiveness evaluation is often carried out by means of test collections. Many works investigated possible sources of bias in such an approach. We propose a systematic approach to identify bias and its causes, and to remove it, thus enforcing fairness in effectiveness evaluation by means of test collections.","tags":["test collection","bias","fairness","network analysis"],"title":"Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Models","type":"publication"},{"authors":["Michael Soprano","Kevin Roitero","Stefano Mizzaro"],"categories":[],"content":"","date":1564012800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668602356,"objectID":"174c9ca5c39cc4b0e4cb9a7b075950d1","permalink":"https://michaelsoprano.com/publication/workshop-paper-birnld-2019/","publishdate":"2021-03-15T14:36:36.88953Z","relpermalink":"/publication/workshop-paper-birnld-2019/","section":"publication","summary":"Peer review is a well known mechanism exploited within the scholarly publishing process to ensure the quality of scientific literature. Such a mechanism, despite being well established and reasonable, is not free from problems, and alternative approaches to peer review have been developed. Such approaches exploit the readers of scientific publications and their opinions, and thus outsource the peer review activity to the scholar community; an example of this approach has been formalized in the Readersourcing model [5]. Our contribution is two-fold:(i) we propose a stochastic validation of the Readersourcing model, and (ii) we employ network analysis techniques to study the bias of the model, and in particular the interactions between readers and papers and their goodness and effectiveness scores. Our results show that by using network analysis interesting model properties can be derived.","tags":["peer review","network analysis","hits"],"title":"HITS Hits Readersourcing: Validating Peer Review Alternatives Using Network Analysis","type":"publication"},{"authors":["Michael Soprano","Stefano Mizzaro"],"categories":[],"content":"","date":1548892800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668600751,"objectID":"257fb4e5e2639d89aabf652f90a7a6e2","permalink":"https://michaelsoprano.com/publication/conference-paper-ircdl-2019/","publishdate":"2021-03-15T14:36:37.331078Z","relpermalink":"/publication/conference-paper-ircdl-2019/","section":"publication","summary":"This paper describes Readersourcing 2.0, an ecosystem providing an implementation of the Readersourcing approach proposed by Mizzaro [10]. Readersourcing is proposed as an alternative to the standard peer review activity that aims to exploit the otherwise lost opinions of readers. Readersourcing 2.0 implements two different models based on the so-called codetermination algorithms. We describe the requirements, present the overall architecture, and show how the end-user can interact with the system. Readersourcing 2.0 will be used in the future to study also other topics, like the idea of shepherding the users to achieve a better quality of the reviews and the differences between a review activity carried out with a single-blind or a double-blind approach.","tags":["scholarly publishing","peer review","crowdsourcing"],"title":"Crowdsourcing Peer Review: As We May Do","type":"publication"},{"authors":["Michael Soprano","Stefano Mizzaro"],"categories":[],"content":"","date":1548201600,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1668600751,"objectID":"4418b58d9df2b2fbe71b648dfddc3b83","permalink":"https://michaelsoprano.com/publication/conference-paper-aiudc-2019/","publishdate":"2021-03-15T14:36:37.769985Z","relpermalink":"/publication/conference-paper-aiudc-2019/","section":"publication","summary":"We propose an alternative approach to the standard peer review activity that aims to exploit the otherwise lost opinions of readers of publications which is called Readersourcing, originally proposed by Mizzaro [1]. Such an approach can be formalized by means of different models which share the same general principles. These models should be able to define a way, to measure the overall quality of a publication as well the reputation of a reader as an assessor; moreover, from these measures it should be possible to derive the reputation of a scholar as an author. We describe an ecosystem called Readersourcing 2.0 which provides an implementation for two Readersourcing models [2, 3] by outlining its goals and requirements. Readersourcing 2.0 will be used in the future to gather fresh data to analyze and validate.","tags":["scholarly publishing","peer review","crowdsourcing"],"title":"Crowdsourcing Peer Review in the Digital Humanities?","type":"publication"},{"authors":["Kevin Roitero","Michael Soprano","Andrea Brunello","Stefano Mizzaro"],"categories":[],"content":"","date":153576e4,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1686216050,"objectID":"1c2648a6ba53773cf225e3ddddb6e0a7","permalink":"https://michaelsoprano.com/publication/journal-paper-jdiq-2018/","publishdate":"2021-03-15T14:36:38.653339Z","relpermalink":"/publication/journal-paper-jdiq-2018/","section":"publication","summary":"Effectiveness evaluation of information retrieval systems by means of a test collection is a widely used methodology. However, it is rather expensive in terms of resources, time, and money; therefore, many researchers have proposed methods for a cheaper evaluation. One particular approach, on which we focus in this article, is to use fewer topics: in TREC-like initiatives, usually system effectiveness is evaluated as the average effectiveness on a set of n topics (usually, n=50, but more than 1,000 have been also adopted); instead of using the full set, it has been proposed to find the best subsets of a few good topics that evaluate the systems in the most similar way to the full set. The computational complexity of the task has so far limited the analysis that has been performed. We develop a novel and efficient approach based on a multi-objective evolutionary algorithm. The higher efficiency of our new implementation allows us to reproduce some notable results on topic set reduction, as well as perform new experiments to generalize and improve such results. We show that our approach is able to both reproduce the main state-of-the-art results and to allow us to analyze the effect of the collection, metric, and pool depth used for the evaluation. Finally, differently from previous studies, which have been mainly theoretical, we are also able to discuss some practical topic selection strategies, integrating results of automatic evaluation approaches.","tags":["topic selection strategy","evolutionary algorithms","reproducibility","test collection","few topics","topic sets"],"title":"Reproduce and Improve: An Evolutionary Approach to Select a Few Good Topics for Information Retrieval Evaluation","type":"publication"},{"authors":["Kevin Roitero","Michael Soprano","Stefano Mizzaro"],"categories":[],"content":"","date":1531008e3,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1686216050,"objectID":"725560812dc18a4727af0d86074c5700","permalink":"https://michaelsoprano.com/publication/conference-paper-sigir-2018/","publishdate":"2021-03-15T14:36:38.202502Z","relpermalink":"/publication/conference-paper-sigir-2018/","section":"publication","summary":"Several researchers have proposed to reduce the number of topics used in TREC-like initiatives. One research direction that has been pursued is what is the optimal topic subset of a given cardinality that evaluates the systems/runs in the most accurate way. Such a research direction has been so far mainly theoretical, with almost no indication on how to select the few good topics in practice. We propose such a practical criterion for topic selection: we rely on the methods for automatic system evaluation without relevance judgments, and by running some experiments on several TREC collections we show that the topics selected on the basis of those evaluations are indeed more informative than random topics.","tags":["topic selection","trec","few topics","test collections"],"title":"Effectiveness Evaluation with a Subset of Topics: A Practical Approach","type":"publication"},{"authors":null,"categories":null,"content":"","date":-62135596800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1615470069,"objectID":"f26b5133c34eec1aa0a09390a36c2ade","permalink":"https://michaelsoprano.com/admin/config.yml","publishdate":"0001-01-01T00:00:00Z","relpermalink":"/admin/config.yml","section":"","summary":"","tags":null,"title":"","type":"wowchemycms"},{"authors":null,"categories":null,"content":"The site is completely static, no use of server-side cookies is made and the only information that might be stored is your choice to see the page in the night mode.\nTherefore there’s no harm to your privacy, unless you explicitly contact me by email or through a social network.\n","date":-62135596800,"expirydate":-62135596800,"kind":"page","lang":"en","lastmod":1615898929,"objectID":"18d05a63a1c8d7ed973cc51838494e41","permalink":"https://michaelsoprano.com/privacy/","publishdate":"0001-01-01T00:00:00Z","relpermalink":"/privacy/","section":"","summary":"The site is completely static, no use of server-side cookies is made and the only information that might be stored is your choice to see the page in the night mode.","tags":null,"title":"Privacy Policy","type":"page"}] \ No newline at end of file diff --git a/index.xml b/index.xml index f7cc0887..b1fe1800 100644 --- a/index.xml +++ b/index.xml @@ -1,79 +1,2 @@ -Michael Sopranohttps://michaelsoprano.com/Michael SopranoWowchemy (https://wowchemy.com)en-usTue, 17 Sep 2024 11:30:00 +0000https://michaelsoprano.com/media/icon_hu4525676fb5a133c3f276d8a3d210c954_1943161_512x512_fill_lanczos_center_3.pngMichael Sopranohttps://michaelsoprano.com/Deploying Complex Crowdsourcing Tasks Without Fear: An Introduction to Crowd_Framehttps://michaelsoprano.com/talk/deploying-complex-crowdsourcing-tasks-without-fear-an-introduction-to-crowd_frame/Tue, 17 Sep 2024 11:30:00 +0000https://michaelsoprano.com/talk/deploying-complex-crowdsourcing-tasks-without-fear-an-introduction-to-crowd_frame/Crowdsourcing Statement Classification to Enhance Information Quality Predictionhttps://michaelsoprano.com/publication/conference-paper-misdoom-2024/Sun, 01 Sep 2024 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-misdoom-2024/Social Computing - 2024/2025https://michaelsoprano.com/post/social-computing-2024_2025/Wed, 10 Jul 2024 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2024_2025/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>Hafska Akebli</strong> - Laboratory (Part 1-2)</li> -<li><strong>Michele Lizzit</strong> - Laboratory (Part 1-2)</li> -<li><strong>Michael Soprano</strong> - Laboratory (Part 2)</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, -Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, -theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by providing a lecture on Crowd_Frame, a framework I developed for the design and deployment of crowdsourcing tasks (2 hours). Additionally, I provided the students with -their second course assignment. The course is conducted in Italian.</p>Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platformshttps://michaelsoprano.com/publication/journal-paper-tsc-2024/Wed, 10 Jul 2024 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-tsc-2024/Crowdsourced Fact-checking: Does It Actually Work?https://michaelsoprano.com/publication/journal-paper-ipm-2024-2/Fri, 31 May 2024 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-ipm-2024-2/Cognitive Biases in Fact-Checking and Their Countermeasures: A Reviewhttps://michaelsoprano.com/publication/journal-paper-ipm-2024/Sat, 03 Feb 2024 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-ipm-2024/Towards a Conversational-Based Agent for Healthcarehttps://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-healthcare/Thu, 26 Oct 2023 08:30:00 +0000https://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-healthcare/Social Computing - 2023/2024https://michaelsoprano.com/post/social-computing-2023_2024/Tue, 10 Oct 2023 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2023_2024/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>David La Barbera</strong> - Laboratory (Part 1)</li> -<li><strong>Michael Soprano</strong> - Laboratory (Part 2)</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, -Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, -theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by providing two lectures on Crowd_Frame, a framework I developed for the design and deployment of crowdsourcing tasks (4 hours). Additionally, I provided the students with -their second course assignment. The course is conducted in Italian.</p> -<ul> -<li><strong>Lesson 4-5</strong>: <a href="https://www.dropbox.com/scl/fi/4rmiryatmpzhyrcmejuyb/SC_MS_4_Crowd_Frame.pptx?rlkey=1cd5jhx1n2fi491qvkagau3un&amp;dl=0" target="_blank" rel="noopener">Crowd_Frame</a></li> -</ul>Towards a Conversational-Based Agent for Health Serviceshttps://michaelsoprano.com/publication/workshop-paper-ital-ia-2023/Wed, 20 Sep 2023 00:00:00 +0000https://michaelsoprano.com/publication/workshop-paper-ital-ia-2023/Fact-Checking at Scale with Crowdsourcing Experiments and Lessons Learnedhttps://michaelsoprano.com/publication/workshop-paper-iir-2023/Sat, 05 Aug 2023 00:00:00 +0000https://michaelsoprano.com/publication/workshop-paper-iir-2023/Towards a Conversational-Based Agent for Health Serviceshttps://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-health-services/Mon, 29 May 2023 00:00:00 +0000https://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-health-services/How Many Crowd Workers Do I Need? On Statistical Power When Crowdsourcing Relevance Judgmentshttps://michaelsoprano.com/publication/journal-paper-tois-2023/Mon, 22 May 2023 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-tois-2023/Social Computing - 2022/2023https://michaelsoprano.com/post/social-computing-2022_2023/Mon, 10 Oct 2022 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2022_2023/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>David La Barbera</strong> - Laboratory (Part 1)</li> -<li><strong>Michael Soprano</strong> - Laboratory (Part 2)</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, -Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, -theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by providing two lectures on Crowd_Frame, a framework I developed for the design and deployment of crowdsourcing tasks (4 hours). Additionally, I provided the students with -their second course assignment. The course is conducted in Italian.</p> -<ul> -<li><strong>Lesson 4-5</strong>: <a href="https://www.dropbox.com/scl/fi/ez7aanl9b9vn4m6zt6ckw/22-23.Crowd_Frame.pdf?rlkey=95kkc3nrf9ejy6r17s5i44y7w&amp;st=b24tubx6&amp;dl=0" target="_blank" rel="noopener">Crowd_Frame</a></li> -</ul>Transparent assessment of information quality of online reviews using formal argumentation theoryhttps://michaelsoprano.com/publication/journal-paper-is-2022/Fri, 29 Jul 2022 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-is-2022/A Neural Model to Jointly Predict and Explain Truthfulness of Statementshttps://michaelsoprano.com/publication/journal-paper-jdiq-2022/Sat, 09 Jul 2022 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-jdiq-2022/The Effects of Crowd Worker Biases in Fact-Checking Taskshttps://michaelsoprano.com/publication/conference-paper-facct-2022/Tue, 21 Jun 2022 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-facct-2022/Longitudinal Loyalty: Understanding the Barriers to Running Longitudinal Studies on Crowdsourcing Platformshttps://michaelsoprano.com/talk/longitudinal-loyalty-understanding-the-barriers-to-running-longitudinal-studies-on-crowdsourcing-platforms/Wed, 15 Jun 2022 00:00:00 +0000https://michaelsoprano.com/talk/longitudinal-loyalty-understanding-the-barriers-to-running-longitudinal-studies-on-crowdsourcing-platforms/Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelfhttps://michaelsoprano.com/talk/crowd_frame-a-simple-and-complete-framework-to-deploy-complex-crowdsourcing-tasks-off-the-shelf/Wed, 23 Feb 2022 00:00:00 +0000https://michaelsoprano.com/talk/crowd_frame-a-simple-and-complete-framework-to-deploy-complex-crowdsourcing-tasks-off-the-shelf/Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelfhttps://michaelsoprano.com/publication/conference-paper-wsdm-2022/Tue, 15 Feb 2022 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-wsdm-2022/The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scalehttps://michaelsoprano.com/publication/journal-paper-ipm-2021/Mon, 01 Nov 2021 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-ipm-2021/E-BART: Jointly Predicting and Explaining Truthfulnesshttps://michaelsoprano.com/publication/conference-paper-tto-2021/Sat, 16 Oct 2021 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-tto-2021/Social Computing - 2021/2022https://michaelsoprano.com/post/social-computing-2021_2022/Sun, 10 Oct 2021 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2021_2022/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>Michael Soprano</strong> - Laboratory</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by teaching the practical applications of the concepts outlined by Professor Stefano Mizzaro in a total of six lectures (12 hours). Additionally, I assigned course assignments to the students. The course is conducted in Italian.</p> -<ul> -<li><strong>Lesson 1</strong>: <a href="https://www.dropbox.com/scl/fi/az4z8x0zu3ysajpjsfs0h/SC_MS_1_Python_Rest_Richiami.pptx?rlkey=ozxl5bwwpp56wjgp3ty01v7tf&amp;dl=0" target="_blank" rel="noopener">Python / Rest: Richiami</a></li> -<li><strong>Lesson 2</strong>: <a href="https://www.dropbox.com/scl/fi/xrtha37m67zob5k7sresz/SC_MS_2-Twitter_API_Tweepy.pptx?rlkey=i54k7eha11f59ies21pyn9u16&amp;dl=0" target="_blank" rel="noopener">Twitter API: Tweepy</a></li> -<li><strong>Lesson 3</strong>: <a href="https://www.dropbox.com/scl/fi/93umif7xa71woszewr5zo/SC_MS_3-EndpointV2_Req.pptx?rlkey=zp1vux6vjp1x6xc7evip6l0d4&amp;dl=0" target="_blank" rel="noopener">Twitter API: Endpoint V2</a></li> -<li><strong>Lesson 4</strong>: <a href="https://www.dropbox.com/scl/fi/d22udpapqnitwuc32ban4/SC_MS_4_Grafi_NetworkX.pptx?rlkey=3r949h14dqxsnzwoffprdgaz7&amp;dl=0" target="_blank" rel="noopener">Grafi: NetworkX</a></li> -<li><strong>Lesson 5-6</strong>: <a href="https://www.dropbox.com/scl/fi/tmgtqxt3ihnvtgj5r0150/SC_MS_5_Crowd_Frame.pptx?rlkey=prdahfpu3u0y7k48dzi2mc1fk&amp;dl=0" target="_blank" rel="noopener">Crowd_Frame</a></li> -</ul>Can The Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation About COVID-19https://michaelsoprano.com/publication/journal-paper-puc-2021/Thu, 16 Sep 2021 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-puc-2021/Assessing the Quality of Online Reviews Using Formal Argumentation Theoryhttps://michaelsoprano.com/publication/conference-paper-icwe-2021/Tue, 11 May 2021 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-icwe-2021/The COVID-19 Infodemic: Can The Crowd Judge Recent Misinformation Objectively?https://michaelsoprano.com/talk/the-covid-19-infodemic-can-the-crowd-judge-recent-misinformation-objectively/Mon, 19 Oct 2020 00:00:00 +0000https://michaelsoprano.com/talk/the-covid-19-infodemic-can-the-crowd-judge-recent-misinformation-objectively/The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?https://michaelsoprano.com/publication/conference-paper-cikm-2020/Mon, 19 Oct 2020 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-cikm-2020/Social Computing - 2020/2021https://michaelsoprano.com/post/social-computing-2020_2021/Sat, 10 Oct 2020 11:00:00 +0000https://michaelsoprano.com/post/social-computing-2020_2021/<h2 id="teachers">Teachers</h2> -<ul> -<li><a href="https://users.dimi.uniud.it/~stefano.mizzaro/" title="Stefano Mizzaro" target="_blank" rel="noopener">Stefano Mizzaro</a> - Theory</li> -<li><strong>Michael Soprano</strong> - Laboratory</li> -</ul> -<h2 id="aims">Aims</h2> -<p>The aim of the course is to provide students with foundational knowledge and practical skills in the area of social informatics. The course discusses both the so-called social media (Facebook, Twitter, etc.) and the crowdsourcing phenomenon.</p> -<p>In the first case, social behavior is supported by computational systems; in the second case, computational systems are supported by social behavior. The course addresses conceptual topics, theoretical foundations, and practical applications.</p> -<h2 id="table-of-contents">Table of contents</h2> -<p>I contributed to the course by teaching the practical applications of the concepts outlined by Professor Stefano Mizzaro in a total of six lectures (12 hours). Additionally, I assigned course assignments to the students. The course is conducted in Italian.</p> -<ul> -<li><strong>Lesson 1</strong>: <a href="https://www.dropbox.com/scl/fi/xpq3b6zhzf3zpczqkwn9g/SC_MS_1_Twitter_API_Intro_Ric.pptx?rlkey=trdwm6yxgohyktj79c0bu3xsx&amp;dl=0" target="_blank" rel="noopener">Twitter API: Introduzione &amp; Richiami</a></li> -<li><strong>Lesson 2-3</strong>: <a href="https://www.dropbox.com/scl/fi/rtdvumim9mzuihlu3zlxu/SC_MS_2-Twitter_API_Tweepy_Req.pptx?rlkey=t7rms0spo58a612oniuda1vey&amp;dl=0" target="_blank" rel="noopener">Twitter API: Tweepy &amp; Requests</a></li> -<li><strong>Lesson 4</strong>: <a href="https://www.dropbox.com/scl/fi/0e8iz71e45mxix4cmxiam/SC_MS_3_Grafi_NetworkX.pptx?rlkey=ab76koii4j1we6zpwatihhh10&amp;dl=0" target="_blank" rel="noopener">Grafi: NetworkX</a></li> -<li><strong>Lesson 4-5</strong>: <a href="https://www.dropbox.com/scl/fi/56egim1ixzm1bm4tn988x/SC_MS_4_Crowd_Frame.pptx?rlkey=yl0z8mqxb6q6n4u1h2etfuz5e&amp;dl=0" target="_blank" rel="noopener">Crowd_Frame: How To</a></li> -</ul>Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Backgroundhttps://michaelsoprano.com/publication/conference-paper-sigir-2020/Sat, 25 Jul 2020 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-sigir-2020/Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor’s Background.https://michaelsoprano.com/talk/can-the-crowd-identify-misinformation-objectively-the-effects-of-judgment-scale-and-assessors-background./Sat, 25 Jul 2020 00:00:00 +0000https://michaelsoprano.com/talk/can-the-crowd-identify-misinformation-objectively-the-effects-of-judgment-scale-and-assessors-background./Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Modelhttps://michaelsoprano.com/talk/bias-and-fairness-in-effectiveness-evaluation-by-means-of-network-analysis-and-mixture-model/Mon, 16 Sep 2019 00:00:00 +0000https://michaelsoprano.com/talk/bias-and-fairness-in-effectiveness-evaluation-by-means-of-network-analysis-and-mixture-model/Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Modelshttps://michaelsoprano.com/publication/workshop-paper-iir-2019/Mon, 16 Sep 2019 00:00:00 +0000https://michaelsoprano.com/publication/workshop-paper-iir-2019/HITS Hits Readersourcing: Validating Peer Review Alternatives Using Network Analysishttps://michaelsoprano.com/publication/workshop-paper-birnld-2019/Thu, 25 Jul 2019 00:00:00 +0000https://michaelsoprano.com/publication/workshop-paper-birnld-2019/Crowdsourcing Peer Review: As We May Dohttps://michaelsoprano.com/publication/conference-paper-ircdl-2019/Thu, 31 Jan 2019 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-ircdl-2019/Crowdsourcing Peer Review in the Digital Humanities?https://michaelsoprano.com/publication/conference-paper-aiudc-2019/Wed, 23 Jan 2019 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-aiudc-2019/Reproduce and Improve: An Evolutionary Approach to Select a Few Good Topics for Information Retrieval Evaluationhttps://michaelsoprano.com/publication/journal-paper-jdiq-2018/Sat, 01 Sep 2018 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-jdiq-2018/Effectiveness Evaluation with a Subset of Topics: A Practical Approachhttps://michaelsoprano.com/publication/conference-paper-sigir-2018/Sun, 08 Jul 2018 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-sigir-2018/<link>https://michaelsoprano.com/admin/config.yml</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://michaelsoprano.com/admin/config.yml</guid><description/></item><item><title>Privacy Policyhttps://michaelsoprano.com/privacy/Mon, 01 Jan 0001 00:00:00 +0000https://michaelsoprano.com/privacy/<p>The site is completely static, no use of server-side cookies is made and the only information that might be stored is your choice to see the page in the night mode.</p> +Michael Sopranohttps://michaelsoprano.com/Michael SopranoWowchemy (https://wowchemy.com)en-usTue, 17 Sep 2024 11:30:00 +0000https://michaelsoprano.com/media/icon_hu4525676fb5a133c3f276d8a3d210c954_1943161_512x512_fill_lanczos_center_3.pngMichael Sopranohttps://michaelsoprano.com/Deploying Complex Crowdsourcing Tasks Without Fear: An Introduction to Crowd_Framehttps://michaelsoprano.com/talk/deploying-complex-crowdsourcing-tasks-without-fear-an-introduction-to-crowd_frame/Tue, 17 Sep 2024 11:30:00 +0000https://michaelsoprano.com/talk/deploying-complex-crowdsourcing-tasks-without-fear-an-introduction-to-crowd_frame/Crowdsourcing Statement Classification to Enhance Information Quality Predictionhttps://michaelsoprano.com/publication/conference-paper-misdoom-2024/Sun, 01 Sep 2024 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-misdoom-2024/Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platformshttps://michaelsoprano.com/publication/journal-paper-tsc-2024/Wed, 10 Jul 2024 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-tsc-2024/Crowdsourced Fact-checking: Does It Actually Work?https://michaelsoprano.com/publication/journal-paper-ipm-2024-2/Fri, 31 May 2024 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-ipm-2024-2/Cognitive Biases in Fact-Checking and Their Countermeasures: A Reviewhttps://michaelsoprano.com/publication/journal-paper-ipm-2024/Sat, 03 Feb 2024 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-ipm-2024/Towards a Conversational-Based Agent for Healthcarehttps://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-healthcare/Thu, 26 Oct 2023 08:30:00 +0000https://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-healthcare/Towards a Conversational-Based Agent for Health Serviceshttps://michaelsoprano.com/publication/workshop-paper-ital-ia-2023/Wed, 20 Sep 2023 00:00:00 +0000https://michaelsoprano.com/publication/workshop-paper-ital-ia-2023/Fact-Checking at Scale with Crowdsourcing Experiments and Lessons Learnedhttps://michaelsoprano.com/publication/workshop-paper-iir-2023/Sat, 05 Aug 2023 00:00:00 +0000https://michaelsoprano.com/publication/workshop-paper-iir-2023/Towards a Conversational-Based Agent for Health Serviceshttps://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-health-services/Mon, 29 May 2023 00:00:00 +0000https://michaelsoprano.com/talk/towards-a-conversational-based-agent-for-health-services/How Many Crowd Workers Do I Need? On Statistical Power When Crowdsourcing Relevance Judgmentshttps://michaelsoprano.com/publication/journal-paper-tois-2023/Mon, 22 May 2023 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-tois-2023/Transparent assessment of information quality of online reviews using formal argumentation theoryhttps://michaelsoprano.com/publication/journal-paper-is-2022/Fri, 29 Jul 2022 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-is-2022/A Neural Model to Jointly Predict and Explain Truthfulness of Statementshttps://michaelsoprano.com/publication/journal-paper-jdiq-2022/Sat, 09 Jul 2022 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-jdiq-2022/The Effects of Crowd Worker Biases in Fact-Checking Taskshttps://michaelsoprano.com/publication/conference-paper-facct-2022/Tue, 21 Jun 2022 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-facct-2022/Longitudinal Loyalty: Understanding the Barriers to Running Longitudinal Studies on Crowdsourcing Platformshttps://michaelsoprano.com/talk/longitudinal-loyalty-understanding-the-barriers-to-running-longitudinal-studies-on-crowdsourcing-platforms/Wed, 15 Jun 2022 00:00:00 +0000https://michaelsoprano.com/talk/longitudinal-loyalty-understanding-the-barriers-to-running-longitudinal-studies-on-crowdsourcing-platforms/Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelfhttps://michaelsoprano.com/talk/crowd_frame-a-simple-and-complete-framework-to-deploy-complex-crowdsourcing-tasks-off-the-shelf/Wed, 23 Feb 2022 00:00:00 +0000https://michaelsoprano.com/talk/crowd_frame-a-simple-and-complete-framework-to-deploy-complex-crowdsourcing-tasks-off-the-shelf/Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelfhttps://michaelsoprano.com/publication/conference-paper-wsdm-2022/Tue, 15 Feb 2022 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-wsdm-2022/The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scalehttps://michaelsoprano.com/publication/journal-paper-ipm-2021/Mon, 01 Nov 2021 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-ipm-2021/E-BART: Jointly Predicting and Explaining Truthfulnesshttps://michaelsoprano.com/publication/conference-paper-tto-2021/Sat, 16 Oct 2021 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-tto-2021/Can The Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation About COVID-19https://michaelsoprano.com/publication/journal-paper-puc-2021/Thu, 16 Sep 2021 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-puc-2021/Assessing the Quality of Online Reviews Using Formal Argumentation Theoryhttps://michaelsoprano.com/publication/conference-paper-icwe-2021/Tue, 11 May 2021 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-icwe-2021/The COVID-19 Infodemic: Can The Crowd Judge Recent Misinformation Objectively?https://michaelsoprano.com/talk/the-covid-19-infodemic-can-the-crowd-judge-recent-misinformation-objectively/Mon, 19 Oct 2020 00:00:00 +0000https://michaelsoprano.com/talk/the-covid-19-infodemic-can-the-crowd-judge-recent-misinformation-objectively/The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?https://michaelsoprano.com/publication/conference-paper-cikm-2020/Mon, 19 Oct 2020 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-cikm-2020/Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Backgroundhttps://michaelsoprano.com/publication/conference-paper-sigir-2020/Sat, 25 Jul 2020 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-sigir-2020/Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor’s Background.https://michaelsoprano.com/talk/can-the-crowd-identify-misinformation-objectively-the-effects-of-judgment-scale-and-assessors-background./Sat, 25 Jul 2020 00:00:00 +0000https://michaelsoprano.com/talk/can-the-crowd-identify-misinformation-objectively-the-effects-of-judgment-scale-and-assessors-background./Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Modelhttps://michaelsoprano.com/talk/bias-and-fairness-in-effectiveness-evaluation-by-means-of-network-analysis-and-mixture-model/Mon, 16 Sep 2019 00:00:00 +0000https://michaelsoprano.com/talk/bias-and-fairness-in-effectiveness-evaluation-by-means-of-network-analysis-and-mixture-model/Bias and Fairness in Effectiveness Evaluation by Means of Network Analysis and Mixture Modelshttps://michaelsoprano.com/publication/workshop-paper-iir-2019/Mon, 16 Sep 2019 00:00:00 +0000https://michaelsoprano.com/publication/workshop-paper-iir-2019/HITS Hits Readersourcing: Validating Peer Review Alternatives Using Network Analysishttps://michaelsoprano.com/publication/workshop-paper-birnld-2019/Thu, 25 Jul 2019 00:00:00 +0000https://michaelsoprano.com/publication/workshop-paper-birnld-2019/Crowdsourcing Peer Review: As We May Dohttps://michaelsoprano.com/publication/conference-paper-ircdl-2019/Thu, 31 Jan 2019 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-ircdl-2019/Crowdsourcing Peer Review in the Digital Humanities?https://michaelsoprano.com/publication/conference-paper-aiudc-2019/Wed, 23 Jan 2019 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-aiudc-2019/Reproduce and Improve: An Evolutionary Approach to Select a Few Good Topics for Information Retrieval Evaluationhttps://michaelsoprano.com/publication/journal-paper-jdiq-2018/Sat, 01 Sep 2018 00:00:00 +0000https://michaelsoprano.com/publication/journal-paper-jdiq-2018/Effectiveness Evaluation with a Subset of Topics: A Practical Approachhttps://michaelsoprano.com/publication/conference-paper-sigir-2018/Sun, 08 Jul 2018 00:00:00 +0000https://michaelsoprano.com/publication/conference-paper-sigir-2018/<link>https://michaelsoprano.com/admin/config.yml</link><pubDate>Mon, 01 Jan 0001 00:00:00 +0000</pubDate><guid>https://michaelsoprano.com/admin/config.yml</guid><description/></item><item><title>Privacy Policyhttps://michaelsoprano.com/privacy/Mon, 01 Jan 0001 00:00:00 +0000https://michaelsoprano.com/privacy/<p>The site is completely static, no use of server-side cookies is made and the only information that might be stored is your choice to see the page in the night mode.</p> <p>Therefore there’s no harm to your privacy, unless you explicitly contact me by email or through a social network.</p> \ No newline at end of file diff --git a/post/index.html b/post/index.html index dc6dd58b..d693b36a 100644 --- a/post/index.html +++ b/post/index.html @@ -5,7 +5,7 @@ - + @@ -31,16 +31,16 @@ - - + + -Social Computing - 2022/2023 | Michael Soprano +Posts | Michael Soprano - +