Lay Perceptions of Crowd-Scientific Findings: The Risks of Variability and Lack of Consensus

# Lay Perceptions of Crowd-Scientific Findings

**The Risks of Variability and Lack of Consensus**  
[Shilaan Alzahawi](http://shilaan.rbind.io) & [Benoît Monin](https://www.gsb.stanford.edu/faculty-research/faculty/benoit-monin)

.pull-left[
![](talk/title_plot.jpg)
]

.pull-right[
`\(~~\)`

[<svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <path d="M459.37 151.716c.325 4.548.325 9.097.325 13.645 0 138.72-105.583 298.558-298.558 298.558-59.452 0-114.68-17.219-161.137-47.106 8.447.974 16.568 1.299 25.34 1.299 49.055 0 94.213-16.568 130.274-44.832-46.132-.975-84.792-31.188-98.112-72.772 6.498.974 12.995 1.624 19.818 1.624 9.421 0 18.843-1.3 27.614-3.573-48.081-9.747-84.143-51.98-84.143-102.985v-1.299c13.969 7.797 30.214 12.67 47.431 13.319-28.264-18.843-46.781-51.005-46.781-87.391 0-19.492 5.197-37.36 14.294-52.954 51.655 63.675 129.3 105.258 216.365 109.807-1.624-7.797-2.599-15.918-2.599-24.04 0-57.828 46.782-104.934 104.934-104.934 30.213 0 57.502 12.67 76.67 33.137 23.715-4.548 46.456-13.32 66.599-25.34-7.798 24.366-24.366 44.833-46.132 57.827 21.117-2.273 41.584-8.122 60.426-16.243-14.292 20.791-32.161 39.308-52.628 54.253z"></path></svg> `\(~~\)` twitter.com/shilaan01](https://twitter.com/shilaan01)    
[<svg viewBox="0 0 496 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <path d="M336.5 160C322 70.7 287.8 8 248 8s-74 62.7-88.5 152h177zM152 256c0 22.2 1.2 43.5 3.3 64h185.3c2.1-20.5 3.3-41.8 3.3-64s-1.2-43.5-3.3-64H155.3c-2.1 20.5-3.3 41.8-3.3 64zm324.7-96c-28.6-67.9-86.5-120.4-158-141.6 24.4 33.8 41.2 84.7 50 141.6h108zM177.2 18.4C105.8 39.6 47.8 92.1 19.3 160h108c8.7-56.9 25.5-107.8 49.9-141.6zM487.4 192H372.7c2.1 21 3.3 42.5 3.3 64s-1.2 43-3.3 64h114.6c5.5-20.5 8.6-41.8 8.6-64s-3.1-43.5-8.5-64zM120 256c0-21.5 1.2-43 3.3-64H8.6C3.2 212.5 0 233.8 0 256s3.2 43.5 8.6 64h114.6c-2-21-3.2-42.5-3.2-64zm39.5 96c14.5 89.3 48.7 152 88.5 152s74-62.7 88.5-152h-177zm159.3 141.6c71.4-21.2 129.4-73.7 158-141.6h-108c-8.8 56.9-25.6 107.8-50 141.6zM19.3 352c28.6 67.9 86.5 120.4 158 141.6-24.4-33.8-41.2-84.7-50-141.6h-108z"></path></svg> `\(~~\)` shilaan.rbind.io](https://shilaan.rbind.io)     
[<svg viewBox="0 0 496 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"></path></svg> `\(~~\)` github.com/shilaan](https://github.com/shilaan)  
[<svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <path d="M440 6.5L24 246.4c-34.4 19.9-31.1 70.8 5.7 85.9L144 379.6V464c0 46.4 59.2 65.5 86.6 28.6l43.8-59.1 111.9 46.2c5.9 2.4 12.1 3.6 18.3 3.6 8.2 0 16.3-2.1 23.6-6.2 12.8-7.2 21.6-20 23.9-34.5l59.4-387.2c6.1-40.1-36.9-68.8-71.5-48.9zM192 464v-64.6l36.6 15.1L192 464zm212.6-28.7l-153.8-63.5L391 169.5c10.7-15.5-9.5-33.5-23.7-21.2L155.8 332.6 48 288 464 48l-59.4 387.3z"></path></svg> `\(~~\)` shilaan@stanford.edu](mailto:shilaan@stanford.edu)

]

???

Hi, thank you for being here. Today  I'll present some recent findings on crowdsourcing data analysis that I've been working on in collaboration with my co-author Professor Benoit Monin.

---
# Outline

???

First, I'll share a brief open science statement and links to all the available materials. Then, I'll define crowd science.

--
<svg viewBox="0 0 576 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <path d="M423.5 0C339.5.3 272 69.5 272 153.5V224H48c-26.5 0-48 21.5-48 48v192c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48V272c0-26.5-21.5-48-48-48h-48v-71.1c0-39.6 31.7-72.5 71.3-72.9 40-.4 72.7 32.1 72.7 72v80c0 13.3 10.7 24 24 24h32c13.3 0 24-10.7 24-24v-80C576 68 507.5-.3 423.5 0z"></path></svg> `\(~~\)` [**Open Science Statement**](#openscience)   
--

???

Then, I'll define crowd science and crowdsourced data analysis and introduce our question of interest, which is "does crowdsourcing data analysis improve lay perceptions of scientific findings?".
--

???

Afterwards we'll briefly talk about the methods and results of an experiment, and hopefully we'll have some time at the end for some discussion.

???

If you have any questions, thoughts, comments, please feel free to raise your hand or unmute yourself and jump in. We have quite some time today for this session, so I'd be happy to make this as interactive as you'd like it to be.

---
class: inverse, middle, center
name: openscience

# Open Science Statement

---
background-image: url("https://github.com/allisonhorst/stats-illustrations/raw/master/rstats-artwork/reproducibility_court.png")
background-position: 90% 50%
background-size: 50%

# Open Science Statement

--
[<svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <g groupmode="layer" id="layer6" label="icon">    <path id="path246" style="fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352777" d="m 255.96701,7.9999992 c -14.42056,0 -28.9134,3.1667198 -39.84197,9.5105508 L 69.402174,102.20596 c -21.954611,12.67476 -39.899208,43.75438 -39.899208,69.08441 v 169.41924 c 0,25.33653 17.944597,56.42568 39.899208,69.0939 L 216.12504,494.51788 C 227.05361,500.84232 241.54645,504 255.96701,504 c 14.48556,0 28.97013,-3.15768 39.89917,-9.48212 l 146.7224,-84.71437 c 21.92213,-12.66822 39.90845,-43.75737 39.90845,-69.0939 V 171.29037 c 0,-25.33003 -17.98632,-56.40965 -39.90845,-69.08441 L 295.86618,17.51055 C 284.93714,11.166719 270.45257,7.9999992 255.96701,7.9999992 Z M 355.1558,152.37339 c 4.22726,0 8.45019,1.60138 11.65118,4.81215 l 17.84669,17.82761 c 6.402,6.42793 6.402,16.93826 0,23.3597 L 228.22149,354.82393 c -6.46654,6.41499 -16.93871,6.41499 -23.34074,0 l -68.94211,-68.91386 c -6.40202,-6.41497 -5.044,-18.2773 0,-23.34073 l 17.85586,-17.82761 c 6.40202,-6.42793 16.9387,-6.42793 23.34073,0 l 27.74552,27.72655 c 6.40203,6.415 16.8742,6.415 23.34074,0 L 343.45702,157.18554 c 3.23307,-3.21072 7.47102,-4.81215 11.69878,-4.81215 z"></path>  </g></svg> `\(~~~\)` Preregistration](https://osf.io/rpu98)

[<svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <g groupmode="layer" id="layer6" label="icon">    <path id="path86" style="fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352777" d="m 255.99051,7.9999992 c -14.45957,0 -28.91558,3.1667158 -39.88958,9.5105468 L 69.397238,102.20596 C 47.455739,114.88072 29.50769,145.96034 29.50769,171.29036 v 169.41925 c 0,25.33653 17.948049,56.42568 39.889548,69.0939 l 146.703692,84.71437 c 10.974,6.32444 25.4365,9.48212 39.88958,9.48212 14.46596,0 28.92849,-3.15768 39.88953,-9.48212 l 146.72261,-84.71437 c 21.9352,-12.66822 39.88966,-43.75737 39.88966,-69.0939 V 171.29036 c 0,-25.33002 -17.95446,-56.40964 -39.88966,-69.0844 L 295.88952,17.510546 c -10.974,-6.343831 -25.43305,-9.5105468 -39.89901,-9.5105468 z m -58.05822,145.7374608 58.06766,22.86711 58.09612,-22.86711 100.07901,40.79892 -58.06755,22.85762 V 317.49208 L 255.99995,358.27204 155.90196,317.49208 V 217.394 L 97.815363,194.53638 Z M 155.90196,217.394 255.99995,258.16451 356.10753,217.394 255.99995,176.60457 Z"></path>  </g></svg> `\(~~~\)` Open materials](https://osf.io/6vze3)

[<svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <g groupmode="layer" id="layer6" label="icon">    <path id="path90" style="fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.0352777" d="M 255.99349,8.0446377 C 223.80996,6.8710755 198.88846,29.106812 171.90108,43.078975 135.19601,64.83668 97.398702,84.909644 61.424592,107.79531 c -23.529448,19.12486 -34.340913,49.89944 -31.869027,79.61823 0.136467,52.69231 -0.273926,105.39637 0.206898,158.08127 3.046554,31.90013 25.166618,58.97257 53.630746,72.31691 45.693451,26.22554 91.119371,52.93874 136.981251,78.85735 29.15402,13.29373 63.66608,7.7032 89.45702,-10.28262 45.55933,-26.46403 91.40188,-52.46098 136.78286,-79.21917 26.10444,-18.58601 38.49713,-51.27731 35.81722,-82.60095 -0.13644,-52.69193 0.27402,-105.39561 -0.20674,-158.08014 C 479.17554,134.58854 457.05921,107.51038 428.59249,94.170602 382.89744,67.950466 337.46312,41.255709 291.60404,15.334368 280.50454,10.177688 268.1747,8.0789301 255.99349,8.0446377 Z M 227.29819,140.73479 c 19.13338,0 38.26677,0 57.40015,0 0,76.83994 0,153.67988 0,230.51981 -19.13338,0 -38.26677,0 -57.40015,0 0,-76.83993 0,-153.67987 0,-230.51981 z m 72.23096,50.17436 c 19.12396,0 38.2479,0 57.37188,0 0,60.11515 0,120.2303 0,180.34545 -19.12398,0 -38.24792,0 -57.37188,0 0,-60.11515 0,-120.2303 0,-180.34545 z m -144.45251,94.07924 c 19.13025,0 38.26049,0 57.39074,0 0,28.7554 0,57.51081 0,86.26621 -19.13025,0 -38.26049,0 -57.39074,0 0,-28.7554 0,-57.51081 0,-86.26621 z"></path>  </g></svg> `\(~~~\)` Open data](https://osf.io/b527v/)

[<svg viewBox="0 0 581 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <path d="M581 226.6C581 119.1 450.9 32 290.5 32S0 119.1 0 226.6C0 322.4 103.3 402 239.4 418.1V480h99.1v-61.5c24.3-2.7 47.6-7.4 69.4-13.9L448 480h112l-67.4-113.7c54.5-35.4 88.4-84.9 88.4-139.7zm-466.8 14.5c0-73.5 98.9-133 220.8-133s211.9 40.7 211.9 133c0 50.1-26.5 85-70.3 106.4-2.4-1.6-4.7-2.9-6.4-3.7-10.2-5.2-27.8-10.5-27.8-10.5s86.6-6.4 86.6-92.7-90.6-87.9-90.6-87.9h-199V361c-74.1-21.5-125.2-67.1-125.2-119.9zm225.1 38.3v-55.6c57.8 0 87.8-6.8 87.8 27.3 0 36.5-38.2 28.3-87.8 28.3zm-.9 72.5H365c10.8 0 18.9 11.7 24 19.2-16.1 1.9-33 2.8-50.6 2.9v-22.1z"></path></svg> `\(~~\)` Reproducible manuscript](https://github.com/shilaan/Many-Analysts/tree/main/manuscript)

[<svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <g label="icon" id="layer6" groupmode="layer">    <path id="path2" d="m 255.9997,7.9999987 c -34.36057,0 -62.21509,27.8545563 -62.21509,62.2151643 0,20.303056 9.87066,38.160947 24.91769,49.517247 0.18814,-20.457899 16.79601,-36.993393 37.29685,-36.993393 20.50082,0 37.11091,16.535494 37.29909,36.993393 15.04533,-11.3563 24.9177,-29.212506 24.9177,-49.517247 C 318.21272,35.854555 290.35915,7.9999987 255.99915,7.9999987 Z M 293.29654,392.2676 c -0.18814,20.4601 -16.79601,36.99338 -37.29684,36.99338 -20.50082,0 -37.10922,-16.53551 -37.29684,-36.99338 -15.04759,11.35627 -24.91769,29.21246 -24.91769,49.51722 0,34.36059 27.85453,62.21518 62.2151,62.21518 34.36056,0 62.21508,-27.85459 62.21508,-62.21518 0,-20.30306 -9.87066,-38.16095 -24.91767,-49.51722 z M 441.78489,193.78484 c -20.30301,0 -38.16309,9.87068 -49.51717,24.91769 20.45786,0.18819 36.99333,16.79605 36.99333,37.29689 0,20.50085 -16.53547,37.11096 -36.9911,37.29916 11.35634,15.04533 29.21249,24.91769 49.51721,24.91769 C 476.14549,318.21327 504,290.35948 504,255.99942 504,221.6394 476.14549,193.78425 441.78489,193.78425 Z M 82.738898,255.99997 c 0,-20.50139 16.535509,-37.11096 36.993392,-37.29689 -11.35632,-15.04756 -29.214164,-24.91769 -49.517197,-24.91769 -34.36057,0 -62.2150945,27.85455 -62.2150945,62.21517 0,34.3606 27.8545245,62.21516 62.2150945,62.21516 20.303033,0 38.160877,-9.87068 49.517197,-24.91773 -20.457883,-0.18818 -36.993391,-16.796 -36.993391,-37.29688 z M 431.3627,80.636814 c -24.29549,-24.295544 -63.68834,-24.295544 -87.9844,0 -14.35704,14.357057 -20.00298,33.963346 -17.39331,52.633806 -0.0824,0.0809 -0.18198,0.13437 -0.26434,0.21491 -14.578,14.57799 -14.578,38.21689 0,52.79488 14.57797,14.57799 38.21681,14.57799 52.79484,0 0.0824,-0.0824 0.13455,-0.18198 0.21732,-0.26434 18.66819,2.60796 38.27445,-3.03799 52.63151,-17.39336 24.29378,-24.29778 24.29378,-63.68837 -0.003,-87.986153 z M 186.2806,378.51178 c 14.57798,-14.57799 14.57798,-38.21461 0,-52.79319 -14.57798,-14.57853 -38.21683,-14.57798 -52.79481,0 -0.0825,0.0824 -0.13448,0.18199 -0.21476,0.26215 -18.67046,-2.60795 -38.276723,3.03572 -52.63376,17.39505 -24.297753,24.29552 -24.297753,63.6884 0,87.98449 24.29551,24.29552 63.68833,24.29552 87.98439,0 14.35702,-14.35703 20.00297,-33.96333 17.39333,-52.63386 0.0848,-0.0786 0.18364,-0.13228 0.26672,-0.21505 z m 0,-245.02583 c -0.0826,-0.0824 -0.18198,-0.13436 -0.26445,-0.21494 2.60795,-18.66823 -3.038,-38.27452 -17.39332,-52.633811 -24.29777,-24.295544 -63.68832,-24.295544 -87.984405,0 -24.297747,24.297781 -24.297747,63.688381 0,87.986151 14.357042,14.35706 33.963315,20.00301 52.631515,17.39336 0.0808,0.0824 0.13447,0.18199 0.21475,0.26434 14.57799,14.57799 38.21684,14.57799 52.79482,0 14.57797,-14.57802 14.57797,-38.21689 0,-52.79488 z m 245.0821,209.89048 c -14.35703,-14.35703 -33.96329,-20.00301 -52.63378,-17.39505 -0.0809,-0.0824 -0.13228,-0.18199 -0.21506,-0.26215 -14.57797,-14.57799 -38.21685,-14.57799 -52.79482,0 -14.57797,14.57799 -14.57797,38.21461 0,52.79316 0.0827,0.0828 0.18198,0.13455 0.26434,0.21505 -2.60796,18.67053 3.03802,38.27683 17.39334,52.63386 24.29552,24.29552 63.68834,24.29552 87.98439,0 24.29775,-24.29552 24.29775,-63.68841 0.003,-87.98451 z" style="stroke-width:0.07717"></path>  </g></svg> `\(~~~\)` osf.io/vedb4](https://osf.io/vedb4)

[<svg viewBox="0 0 496 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:2em;" xmlns="http://www.w3.org/2000/svg">  <path d="M165.9 397.4c0 2-2.3 3.6-5.2 3.6-3.3.3-5.6-1.3-5.6-3.6 0-2 2.3-3.6 5.2-3.6 3-.3 5.6 1.3 5.6 3.6zm-31.1-4.5c-.7 2 1.3 4.3 4.3 4.9 2.6 1 5.6 0 6.2-2s-1.3-4.3-4.3-5.2c-2.6-.7-5.5.3-6.2 2.3zm44.2-1.7c-2.9.7-4.9 2.6-4.6 4.9.3 2 2.9 3.3 5.9 2.6 2.9-.7 4.9-2.6 4.6-4.6-.3-1.9-3-3.2-5.9-2.9zM244.8 8C106.1 8 0 113.3 0 252c0 110.9 69.8 205.8 169.5 239.2 12.8 2.3 17.3-5.6 17.3-12.1 0-6.2-.3-40.4-.3-61.4 0 0-70 15-84.7-29.8 0 0-11.4-29.1-27.8-36.6 0 0-22.9-15.7 1.6-15.4 0 0 24.9 2 38.6 25.8 21.9 38.6 58.6 27.5 72.9 20.9 2.3-16 8.8-27.1 16-33.7-55.9-6.2-112.3-14.3-112.3-110.5 0-27.5 7.6-41.3 23.6-58.9-2.6-6.5-11.1-33.3 2.6-67.9 20.9-6.5 69 27 69 27 20-5.6 41.5-8.5 62.8-8.5s42.8 2.9 62.8 8.5c0 0 48.1-33.6 69-27 13.7 34.7 5.2 61.4 2.6 67.9 16 17.7 25.8 31.5 25.8 58.9 0 96.5-58.9 104.2-114.8 110.5 9.2 7.9 17 22.9 17 46.4 0 33.7-.3 75.4-.3 83.6 0 6.5 4.6 14.4 17.3 12.1C428.2 457.8 496 362.9 496 252 496 113.3 383.5 8 244.8 8zM97.2 352.9c-1.3 1-1 3.3.7 5.2 1.6 1.6 3.9 2.3 5.2 1 1.3-1 1-3.3-.7-5.2-1.6-1.6-3.9-2.3-5.2-1zm-10.8-8.1c-.7 1.3.3 2.9 2.3 3.9 1.6 1 3.6.7 4.3-.7.7-1.3-.3-2.9-2.3-3.9-2-.6-3.6-.3-4.3.7zm32.4 35.6c-1.6 1.3-1 4.3 1.3 6.2 2.3 2.3 5.2 2.6 6.5 1 1.3-1.3.7-4.3-1.3-6.2-2.2-2.3-5.2-2.6-6.5-1zm-11.4-14.7c-1.6 1-1.6 3.6 0 5.9 1.6 2.3 4.3 3.3 5.6 2.3 1.6-1.3 1.6-3.9 0-6.2-1.4-2.3-4-3.3-5.6-2z"></path></svg> `\(~~~\)` github.com/shilaan/many-analysts](https://github.com/shilaan/many-analysts)

[<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.7em;" xmlns="http://www.w3.org/2000/svg">  <path d="M592 0H48A48 48 0 0 0 0 48v320a48 48 0 0 0 48 48h240v32H112a16 16 0 0 0-16 16v32a16 16 0 0 0 16 16h416a16 16 0 0 0 16-16v-32a16 16 0 0 0-16-16H352v-32h240a48 48 0 0 0 48-48V48a48 48 0 0 0-48-48zm-16 352H64V64h512z"></path></svg> `\(~~\)` Slides at bit.ly/many-analysts](https://bit.ly/many-analysts)

.right[
Artwork by [@allison_horst](https://github.com/allisonhorst/stats-illustrations)
]

???

In terms of the materials, these are all available online, including a preregistration, the survey, data and code, and a fully reproducible manuscript. You can find all these materials on the OSF and on Github, and you can also find these slides online if you want to have them in front of you at bit.ly/many-analysts.

---
class: inverse, center, middle
name: crowdscience

# Crowdsourcing Science

---
.center[
# Crowd Science
]

### The organization of scientific research in open and collaborative projects

From highly **inclusive** (sourcing citizens) to highly **selective** (sourcing specialists)

Used at different **stages** of the research process

???

You may have already heard of crowdfunding, which uses the crowd for funding, but you can also use a crowd of individuals for generating research ideas, designing studies, collecting data, and analyzing data, which is what I'll be talking about today.
--

.pull-left[

]

.pull-right[

]

.pull-right[

]

---
background-image: url("talk/meteor.jpeg")
background-position: 90% 73%
background-size: 42%

.center[
# Crowd Science
]

### The organization of scientific research in open and collaborative projects

From highly **inclusive** (sourcing citizens) to highly **selective** (sourcing specialists)

Used at different **stages** of the research process

.pull-left[

]

.pull-right[

]

.pull-right[

]

.right[

`\(~~\)`  
`\(~~\)`  
The great meteor storm of 1833: birth of *Citizen Science*

]

???

In 1833, Denison Olmsted used letter correspondence to recruit citizen scientists to help document a meteor shower. This led to a detailed documentation of the great meteor shower and the birth of the citizen-science movement (highly inclusive, used for data collection. In terms of the two dimensions we just discussed, this project can be categorized as highly inclusive - because it sourced citizens - and as used in the stage of data collection. 
---
class: inverse, center, middle
name: crowddata

# Crowdsourcing Data Analysis

---
.center[
# Crowdsourced Data Analysis

]

???

Involves giving different teams of scientists the same dataset, who independently analyze it to answer the same research question or estimate and report on a parameter of interest.

### Different teams of scientists independently analyze the same dataset

<div id="htmlwidget-f1273964fae68d796baa" style="width:100%;height:auto;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-f1273964fae68d796baa">{"x":{"filter":"none","vertical":false,"fillContainer":false,"data":[["1","2","3","4","5","6","7"],["Breznau et al.","Breznau et al.","Schweinsberg et al.","Botvinik-Nezer et al.","Huntington-Klein et al.","Bastiaansen et al.","Silberzahn et al."],[2021,2021,2021,2020,2020,2019,2018],["How many replicators does it take to achieve reliability? Investigating researcher variability in a crowdsourced replication","Observing many researchers using the same data and hypothesis\nreveals a hidden universe of uncertainty","Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis","Variability in the analysis of a single neuroimaging dataset by many teams","The influence of hidden researcher decisions in applied microeconomics","Time to get personal? The impact of researchers choices on the selection of\ntreatment targets using the experience sampling methodology","Many analysts, one data set: Making transparent how variations in analytic choices affect results"]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>Author<\/th>\n      <th>Year<\/th>\n      <th>Title<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"pageLength":2,"columnDefs":[{"className":"dt-right","targets":2},{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[2,10,25,50,100]}},"evals":[],"jsHooks":[]}</script>

???

I'm including an interactive table with 7 papers that have recently been published in the social and behavioral science that use the many-analyst approach. They include interesting topics such as variability in replications, opinions about different social policies, the effects of the gender and professional status of scientists on their verbosity during group meetings, variability in the analysis of fmri (so neuroimaging datasets), and we can scroll back all the way to 2018, to a quite well-known study that looked at discrimination against darker skin toned soccer players, and arguably started the recent crowd science movement in behavioral science.

---
# The Rationale of Crowdsourced Data Analysis

???

What is the rationale of crowdsourced data analysis - why should we be convinced by its findings? The phenomenon that underlies crowd science in general is known as the wisdom of the crowd.

### **The Wisdom of the Crowd**

---
background-image: url("talk/ox.png")
background-position: 99% 60%
background-size: 35%

# The Rationale of Crowdsourced Data Analysis

### **The Wisdom of the Crowd**

`\(~~\)`

In 1906, 787 people guessed the weight of an ox at a farmers' fair...

???

Perhaps the first documented case of the wisdom of the crowd was back in 1906, when... 1197 pounds, cattle experts, individual best guesser who took home the grand prize that day.

This may seem unsurprising now, but it wasn't back then. Back then, the fact that crowds were wise was a surprise. Crowds mostly had negative associations, were referred to as mobs, and there was heated intellectual debate about restricting the right to vote to intellectuals, rather than democratizing society.

To benefit from the wisdom of crowds several conditions must be in place. First each individual member of the crowd must have their own independent source of information. Second they must make individual decisions and not be swayed by the decisions of those around them. And third, there must be a mechanism in place that can collate these diverse opinions.

---
# Benefits of Crowdsourced Data Analysis

### **Crowdsourced data analysis can help to...**

???

I'll mention some benefits, some of which will overlap, and most of them are intimately related with each other.

--
<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M96 224c35.3 0 64-28.7 64-64s-28.7-64-64-64-64 28.7-64 64 28.7 64 64 64zm448 0c35.3 0 64-28.7 64-64s-28.7-64-64-64-64 28.7-64 64 28.7 64 64 64zm32 32h-64c-17.6 0-33.5 7.1-45.1 18.6 40.3 22.1 68.9 62 75.1 109.4h66c17.7 0 32-14.3 32-32v-32c0-35.3-28.7-64-64-64zm-256 0c61.9 0 112-50.1 112-112S381.9 32 320 32 208 82.1 208 144s50.1 112 112 112zm76.8 32h-8.3c-20.8 10-43.9 16-68.5 16s-47.6-6-68.5-16h-8.3C179.6 288 128 339.6 128 403.2V432c0 26.5 21.5 48 48 48h288c26.5 0 48-21.5 48-48v-28.8c0-63.6-51.6-115.2-115.2-115.2zm-223.7-13.4C161.5 263.1 145.6 256 128 256H64c-35.3 0-64 28.7-64 64v32c0 17.7 14.3 32 32 32h65.9c6.3-47.4 34.9-87.3 75.2-109.4z"></path></svg> `\(~~\)`   **Diversify** contributions to research

???

First, crowdsourced data analysis can help to diversify contributions to research. I will say that this benefit is not specific to crowdsourcing data analysis, but a property of crowd science and big team science in general. A goal of crowd science and big team science is to promote inclusivity and include researchers from underrepresented institutions. This comes with all the benefits of diversity. It allows us to leverage under-used resources and skills. We of course can't all be an expert on everything. If we recruit a diverse sample of data analysts, some may have expertise, for example, on a certain statistical method that is most appropriate for the data at hand, or they'll have some other expertise that may benefit the data analysis.

???

Second, the many-analyst approach helps to acknowledge that results in social science tend to be very variable, and can depend on idiosyncratic choices we have to make among multiple valid approaches, sometimes even choices we make unconsciously, such as the version of statistical software we rely on or the different defaults in for instance SPSS and R: it is important to note that the many-analyst approach does not *create* this variability in results, it simply *exposes* it. that should give us pause and epistemic humility when we assess conventional scientific research, which is often done by a single analyst or a single team: the particular outcome you're looking at is one from a distribution of outcomes that could have been possible, if the analyst had made different choices.

Like Nicholas said in his opening talk, approaches like this allow us to explore the size of the multiverse.

???

Third, we can assess the impact of individual analytic choices on the results. Here, we can ask if we can explain the strength of the evidence obtained by the specific individual decisions that researchers make. For instance, we can asses the impact of including or excluding certain control variables, the impact of running a certain statistical model such as a t-test or a mixed effects model, or the impact of excluding or including certain participants or outliers. We could ask, for example, if people who obtain stronger evidence are more likely to do one or the other, and vice versa.

???

Fourth, the many-analyst approach can help us assess the overall analytic robustness of a finding. We can assess to what extent a finding holds across many different analytic specification. We can ask: do the results come out consistently positive, or consistently negative, across different independent analyses? 
--

???

And when we do find that the result come out consistently positive or negative, then this can increase our confidence in the specific conclusion we're drawing.

???

And going back to the wisdom of the crowd phenomenon that we discussed, another benefit is that we can average across all these idiosyncratic analytic choices to obtain more accurate parameter estimates and to balance out individual biases and errors in the aggregate
--
<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M256 336h-.02c0-16.18 1.34-8.73-85.05-181.51-17.65-35.29-68.19-35.36-85.87 0C-2.06 328.75.02 320.33.02 336H0c0 44.18 57.31 80 128 80s128-35.82 128-80zM128 176l72 144H56l72-144zm511.98 160c0-16.18 1.34-8.73-85.05-181.51-17.65-35.29-68.19-35.36-85.87 0-87.12 174.26-85.04 165.84-85.04 181.51H384c0 44.18 57.31 80 128 80s128-35.82 128-80h-.02zM440 320l72-144 72 144H440zm88 128H352V153.25c23.51-10.29 41.16-31.48 46.39-57.25H528c8.84 0 16-7.16 16-16V48c0-8.84-7.16-16-16-16H383.64C369.04 12.68 346.09 0 320 0s-49.04 12.68-63.64 32H112c-8.84 0-16 7.16-16 16v32c0 8.84 7.16 16 16 16h129.61c5.23 25.76 22.87 46.96 46.39 57.25V448H112c-8.84 0-16 7.16-16 16v32c0 8.84 7.16 16 16 16h416c8.84 0 16-7.16 16-16v-32c0-8.84-7.16-16-16-16z"></path></svg> `\(~~\)`   ... and to **balance out individual biases and errors** in the aggregate

---
class: inverse, center, middle
name: lay

# Lay Perceptions

???

Now I'd like to move on to talking about lay perceptions. One of the reasons crowdsourced data analysis was proposed, was to increase the credibility of scientific research. Now, we know there's quite a great deal of science skepticism in some public circles, so we want scientific research to become more credible not just to ourselves as academics, but to lay people as well. 
---
# Lay Perceptions of Crowd-Scientific Findings

According to its proponents, crowdsourcing data analysis should **improve lay perceptions of scientific findings**  
--

<div id="htmlwidget-86927bafd82a99f986ee" style="width:100%;height:auto;" class="datatables html-widget"></div>
<script type="application/json" data-for="htmlwidget-86927bafd82a99f986ee">{"x":{"filter":"none","vertical":false,"fillContainer":false,"autoHideNavigation":true,"data":[["1","2","3"],["Arbon et al.","Breznau","Breznau et al."],[2019,2021,2021],["\"<b>For the public to have faith in the conclusions of scientists<\/b> it is important that the methods they employ are robust and transparent. This study will examine robustness by recruiting teams of independent data analysts and looking at how they answer a controversial research question using the same data, effectively ‘crowd-sourcing’ the data analysis.\"","\"crowdsourcing provides a new way to <b>increase credibility<\/b> for political and social research—in both sample populations and among the researchers themselves (…) It is hoped that these developments are tangible outcomes that <b>increase public, private, and government views of social science<\/b>.\"","\"Organized scientific knowledge production (…) should generate inter-researcher reliability, <b>offering consumers of scientific findings assurance<\/b> that they are not arbitrary flukes but that other researchers would generate similar findings given the same data.\""]],"container":"<table class=\"display\">\n  <thead>\n    <tr>\n      <th> <\/th>\n      <th>Author<\/th>\n      <th>Year<\/th>\n      <th>Quote<\/th>\n    <\/tr>\n  <\/thead>\n<\/table>","options":{"pageLength":3,"columnDefs":[{"className":"dt-right","targets":2},{"orderable":false,"targets":0}],"order":[],"autoWidth":false,"orderClasses":false,"lengthMenu":[3,10,25,50,100]}},"evals":[],"jsHooks":[]}</script>

???

I'm including here another table with three quotes. The first one is from a preregistration, in which the authors justify their use of the many-analyst approach by referring to the public needing to have faith in the conclusions of scientists. The second one is from a commentary on the many-analyst approach, in which the author provides the hopeful view that crowdsourcing provides a new way to increase credibility for social research, not just among researchers themselves, but in sample populations as well; and that it's hoped that the many-analyst approach will increase public views of social science. The third one is from a preprint relying on the many-analyst approach, in which the authors argue that many-analyst studies that generate consistent results across independent analysis should offer consumers of scientific findings assurance that they are not arbitrary flukes.

---
class: inverse, center, middle

# Does Crowdsourcing Data Analysis Improve the Credibility of Scientific Findings <svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:white;height:1em;" xmlns="http://www.w3.org/2000/svg">  <path d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 448c-110.532 0-200-89.431-200-200 0-110.495 89.472-200 200-200 110.491 0 200 89.471 200 200 0 110.53-89.431 200-200 200zm107.244-255.2c0 67.052-72.421 68.084-72.421 92.863V300c0 6.627-5.373 12-12 12h-45.647c-6.627 0-12-5.373-12-12v-8.659c0-35.745 27.1-50.034 47.579-61.516 17.561-9.845 28.324-16.541 28.324-29.579 0-17.246-21.999-28.693-39.784-28.693-23.189 0-33.894 10.977-48.942 29.969-4.057 5.12-11.46 6.071-16.666 2.124l-27.824-21.098c-5.107-3.872-6.251-11.066-2.644-16.363C184.846 131.491 214.94 112 261.794 112c49.071 0 101.45 38.304 101.45 88.8zM298 368c0 23.159-18.841 42-42 42s-42-18.841-42-42 18.841-42 42-42 42 18.841 42 42z"></path></svg>

???

Now, these quotes describe a worthy goal: improving the credibility of scientific findings through crowdsourcing data analysis. However, this is also an empirical claim, and as far as we know there is no evidence yet that crowdsourcing data analysis can indeed achieve this goal. So that's what we'll be looking at today: does crowdsourcing data analysis improve lay perceptions of scientific findings?
---
class: inverse, center, middle
name: method

# Method

---

# Method

We run an experiment with **three conditions**

???

Three scenarios. The first condition is a conventional, single-analyst condition where a single team of six scientists analyzes the same dataset and reports a single, aggregrate parameter estimate of 5% for the relationship between two variables that I'll introduce shortly.

???

We contrast this condition with two multi-analyst conditions. In the multi-consistent conditon, six independent teams of scientists report six independent estimates that show low variance and high consensus: they all agree in their conclusion, and the spread in their effect size estimate is small.
--

???

In the multi-consistent condition, six independent teams of scientists report six independent estimates that show high variance and low consensus. There is a large spread between the estimates, and the qualitative conclusions differ: two of the scientists find a negative relationship, and four of the scientists find a positive relationship.

`\(~~\)`  
In all three conditions, the given estimates average to 5%. We study the **effects on**

.pull-left[
<svg viewBox="0 0 256 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"></path></svg> **Posterior beliefs** about the reported phenomenon  
<svg viewBox="0 0 256 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"></path></svg> **Credibility** of the results  
<svg viewBox="0 0 256 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"></path></svg> **Confidence** in the aggregate estimate of 5%  
]
.pull-right[
<svg viewBox="0 0 256 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"></path></svg> Perceived impact of **bias** on the 5% estimate  
<svg viewBox="0 0 256 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"></path></svg> Perceived impact of **error** on the 5% estimate  
<svg viewBox="0 0 256 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M224.3 273l-136 136c-9.4 9.4-24.6 9.4-33.9 0l-22.6-22.6c-9.4-9.4-9.4-24.6 0-33.9l96.4-96.4-96.4-96.4c-9.4-9.4-9.4-24.6 0-33.9L54.3 103c9.4-9.4 24.6-9.4 33.9 0l136 136c9.5 9.4 9.5 24.6.1 34z"></path></svg> Perceived impact of **discretion** on the 5% estimate   
]

???

In all three conditions the estimates average to 5%

---
class: left

# Experimental design

???

In terms of the experimental design, participants are first introduced to a research question - do religious people report higher well-being - and then report their prior beliefs about this research question. They are then allocated to one of the three conditions, and afterwards report the outcomes that I just described. 
---

.center[

# Hypotheses

]

### **Social norms theory**
Observing consensus among a crowd increases **conformity** in opinion

???

We based our hypotheses on three lines of thought. First, in line with social norms theory, we expect that observing consensus among a crowd (the consistent crowd condition) will -- compared to the conclusion of a single scientist (the single estimate condition) -- increase conformity in opinion.

### **Intuitive statistics **
Observers intuitively accord to the logic of **the wisdom of crowds**:  
The ability of an aggregate of estimates to reduce noise stemming from individual bias or error

???

Second, Drawing from work on intuitive statistics, we also expect laypeople to intuitively accord to the logic of "the wisdom of crowds": the ability of an aggregate of estimates (rather than a single estimate) to reduce noise stemming from individual bias or error.

In contrast, when crowd estimates show low consensus and high variance (the inconsistent crowd condition), we predict that observers will be less swayed and more likely to attribute the findings to bias and error.

### **Aversion to variability**
Observing **variable estimates** decreases confidence in the overall estimate

???

In sum, our preregistered hypotheses are as follows: when laypeople observe multiple consistent estimates from a crowd of independent scientists, we expect -- compared to a single estimate and controlling for prior beliefs -- higher posterior beliefs and credibility of the results, lower confidence in the precise average parameter estimate, and lower ratings of bias and error. When laypeople observe multiple inconsistent estimates from a crowd of independent scientists, we expect -- compared to a single estimate and controlling for prior beliefs -- lower posterior beliefs and credibility of the results, lower confidence in the precise average parameter estimate, and greater ratings of bias and error.

---

.center[

# Hypotheses

]

`\(~~\)`

.pull-left[

]

.pull-right[

???

*Note*. Table indicates the predicted direction of the effect for all dependent variables, compared to the single-analyst condition and controlling for prior beliefs. For example, we hypothesized that, compared to a single-analyst study and controlling for prior beliefs,  ratings of credibility would be greater in the multi-analyst: consistent condition and lower in the multi-analyst: inconsistent condition.

???

Third, due to the difficulty of lay reasoning about variation, we predict an aversion to variability: i.e., we expect that observing variable estimates will decrease lay confidence in the precise average parameter estimate in both crowd conditions. In sum, our preregistered hypotheses are as follows: when laypeople observe multiple consistent estimates from a crowd of independent scientists, we expect -- compared to a single estimate and controlling for prior beliefs -- higher posterior beliefs and credibility of the results, lower confidence in the precise average parameter estimate, and lower ratings of bias and error. When laypeople observe multiple inconsistent estimates from a crowd of independent scientists, we expect -- compared to a single estimate and controlling for prior beliefs -- lower posterior beliefs and credibility of the results, lower confidence in the precise average parameter estimate, and greater ratings of bias and error.

---
.center[

# Hypotheses

]

`\(~~\)`

.pull-left[

]

.pull-right[

???

Our pre-registered hypotheses can be found in the table: we hypothesized that in the multi-consistent condition (compared to the single-analyst condition), lay consumers would have higher posterior beliefs, would find the results more credible, and would be less likely to believe the results stem from bias or error. For the multi-inconsistent condition, we hypothesized that lay consumers would have lower posterior beliefs, would find the results less credible, and would be more likely to believe the results stem from bias or error. In addition, we expected that the act of providing multiple (slightly to widely varying) parameter estimates would decrease confidence in the aggregate parameter estimate in both multi-analyst conditions.

---
.center[

# Hypotheses

]

`\(~~\)`

.pull-left[

]

.pull-right[

???

---
.center[

# Hypotheses

]

`\(~~\)`

.pull-left[

]

.pull-right[

| Measure | Multi-consistent | Multi-inconsistent |
| ------------- |  ------------- | ------------- | 
| Posterior beliefs | .center[<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:green;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M624 208h-64v-64c0-8.8-7.2-16-16-16h-32c-8.8 0-16 7.2-16 16v64h-64c-8.8 0-16 7.2-16 16v32c0 8.8 7.2 16 16 16h64v64c0 8.8 7.2 16 16 16h32c8.8 0 16-7.2 16-16v-64h64c8.8 0 16-7.2 16-16v-32c0-8.8-7.2-16-16-16zm-400 48c70.7 0 128-57.3 128-128S294.7 0 224 0 96 57.3 96 128s57.3 128 128 128zm89.6 32h-16.7c-22.2 10.2-46.9 16-72.9 16s-50.6-5.8-72.9-16h-16.7C60.2 288 0 348.2 0 422.4V464c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48v-41.6c0-74.2-60.2-134.4-134.4-134.4z"></path></svg>]  | .center[<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:red;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M624 208H432c-8.8 0-16 7.2-16 16v32c0 8.8 7.2 16 16 16h192c8.8 0 16-7.2 16-16v-32c0-8.8-7.2-16-16-16zm-400 48c70.7 0 128-57.3 128-128S294.7 0 224 0 96 57.3 96 128s57.3 128 128 128zm89.6 32h-16.7c-22.2 10.2-46.9 16-72.9 16s-50.6-5.8-72.9-16h-16.7C60.2 288 0 348.2 0 422.4V464c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48v-41.6c0-74.2-60.2-134.4-134.4-134.4z"></path></svg>] |
|Credibility | .center[<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:green;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M624 208h-64v-64c0-8.8-7.2-16-16-16h-32c-8.8 0-16 7.2-16 16v64h-64c-8.8 0-16 7.2-16 16v32c0 8.8 7.2 16 16 16h64v64c0 8.8 7.2 16 16 16h32c8.8 0 16-7.2 16-16v-64h64c8.8 0 16-7.2 16-16v-32c0-8.8-7.2-16-16-16zm-400 48c70.7 0 128-57.3 128-128S294.7 0 224 0 96 57.3 96 128s57.3 128 128 128zm89.6 32h-16.7c-22.2 10.2-46.9 16-72.9 16s-50.6-5.8-72.9-16h-16.7C60.2 288 0 348.2 0 422.4V464c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48v-41.6c0-74.2-60.2-134.4-134.4-134.4z"></path></svg>] | .center[<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:red;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M624 208H432c-8.8 0-16 7.2-16 16v32c0 8.8 7.2 16 16 16h192c8.8 0 16-7.2 16-16v-32c0-8.8-7.2-16-16-16zm-400 48c70.7 0 128-57.3 128-128S294.7 0 224 0 96 57.3 96 128s57.3 128 128 128zm89.6 32h-16.7c-22.2 10.2-46.9 16-72.9 16s-50.6-5.8-72.9-16h-16.7C60.2 288 0 348.2 0 422.4V464c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48v-41.6c0-74.2-60.2-134.4-134.4-134.4z"></path></svg>] |
| Confidence  | .center[<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:red;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M624 208H432c-8.8 0-16 7.2-16 16v32c0 8.8 7.2 16 16 16h192c8.8 0 16-7.2 16-16v-32c0-8.8-7.2-16-16-16zm-400 48c70.7 0 128-57.3 128-128S294.7 0 224 0 96 57.3 96 128s57.3 128 128 128zm89.6 32h-16.7c-22.2 10.2-46.9 16-72.9 16s-50.6-5.8-72.9-16h-16.7C60.2 288 0 348.2 0 422.4V464c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48v-41.6c0-74.2-60.2-134.4-134.4-134.4z"></path></svg>]  | .center[<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:red;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M624 208H432c-8.8 0-16 7.2-16 16v32c0 8.8 7.2 16 16 16h192c8.8 0 16-7.2 16-16v-32c0-8.8-7.2-16-16-16zm-400 48c70.7 0 128-57.3 128-128S294.7 0 224 0 96 57.3 96 128s57.3 128 128 128zm89.6 32h-16.7c-22.2 10.2-46.9 16-72.9 16s-50.6-5.8-72.9-16h-16.7C60.2 288 0 348.2 0 422.4V464c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48v-41.6c0-74.2-60.2-134.4-134.4-134.4z"></path></svg>] | 
| Bias | .center[<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:red;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M624 208H432c-8.8 0-16 7.2-16 16v32c0 8.8 7.2 16 16 16h192c8.8 0 16-7.2 16-16v-32c0-8.8-7.2-16-16-16zm-400 48c70.7 0 128-57.3 128-128S294.7 0 224 0 96 57.3 96 128s57.3 128 128 128zm89.6 32h-16.7c-22.2 10.2-46.9 16-72.9 16s-50.6-5.8-72.9-16h-16.7C60.2 288 0 348.2 0 422.4V464c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48v-41.6c0-74.2-60.2-134.4-134.4-134.4z"></path></svg>] | .center[<svg viewBox="0 0 640 512" style="position:relative;display:inline-block;top:.1em;fill:green;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M624 208h-64v-64c0-8.8-7.2-16-16-16h-32c-8.8 0-16 7.2-16 16v64h-64c-8.8 0-16 7.2-16 16v32c0 8.8 7.2 16 16 16h64v64c0 8.8 7.2 16 16 16h32c8.8 0 16-7.2 16-16v-64h64c8.8 0 16-7.2 16-16v-32c0-8.8-7.2-16-16-16zm-400 48c70.7 0 128-57.3 128-128S294.7 0 224 0 96 57.3 96 128s57.3 128 128 128zm89.6 32h-16.7c-22.2 10.2-46.9 16-72.9 16s-50.6-5.8-72.9-16h-16.7C60.2 288 0 348.2 0 422.4V464c0 26.5 21.5 48 48 48h352c26.5 0 48-21.5 48-48v-41.6c0-74.2-60.2-134.4-134.4-134.4z"></path></svg>] |
]

???

---
.center[

# Hypotheses

]

`\(~~\)`

.pull-left[

]

.pull-right[

???

---
.center[

# Hypotheses

]

`\(~~\)`

.pull-left[

]

.pull-right[

???

---
class: inverse, center, middle

# Survey Materials

---

.panelset.sideways[
.panel[.panel-name[Introduction]
<img src="talk/introduction.png" width="90%" />
]
.panel[.panel-name[Prior Beliefs]

![](talk/prior_beliefs.png)
]

.panel[.panel-name[Thanks!]

![](talk/thanks.png)
]
]

---
.panelset.sideways[
.panel[.panel-name[Study design & research approach (Single-analyst)]
<img src="talk/design_single.png" width="93%" />
]

.panel[.panel-name[Study design & research approach (Multi-analyst)]
<img src="talk/design_multi.png" width="91%" />
]
]

???

Participants are first introduced to the research question and research approach. In the single-analyst condition, a single team of six researchers analyzes the same dataset, and in the two multi-analyst conditions, six independent researchers analyze the same dataset. Importantly, in all conditions we emphasize in the last paragraph that there are a lot of choices to be made. Participants had to wait 30 seconds before proceeding to the next slide.

---

.panelset[
.panel[.panel-name[Results (Single-analyst)]
<img src="talk/results_single.png" width="80%" />
]

.panel[.panel-name[Results (Multi-consistent)]
<img src="talk/results_multi-consistent.png" width="80%" />
]

.panel[.panel-name[Results (Multi-inconsistent)]
<img src="talk/results_multi-inconsistent.png" width="80%" />
]
]

???

Here are the results we show participants. In the single-analyst condition, the team of six researchers concludes that religious people do indeed report higher well-being, and they find a 5% increase in well-being for every one point increase in religiosity. In the multi-consistent condition, the six independent researchers show consensus in that they all report a positive relationship, and the spread in their estimates is quite small; they range from 2% to 8%. In the multi-inconsistent condition, the six independent researchers do not show consensus, in that two of the scientists conclude that the relationship is negative, and four of the scientists conclude that the relationship is positive, and we can see that the spread in their estimates is quite large, ranging from -6% to 16%.

---

.panelset.sideways[
.panel[.panel-name[Posterior beliefs]
<img src="talk/posterior_beliefs.png" width="100%" />
]

.panel[.panel-name[Credibility]
<img src="talk/credibility.png" width="100%" />
]

.panel[.panel-name[Confidence]
<img src="talk/confidence.png" width="100%" />
]

.panel[.panel-name[Bias]
<img src="talk/bias.png" width="100%" />
]
.panel[.panel-name[Error]
<img src="talk/bias.png" width="100%" />
]
.panel[.panel-name[Discretion]
<img src="talk/bias.png" width="100%" />
]

.panel[.panel-name[Attention check]
<img src="talk/attention.png" width="100%" />
]

]

---
class: inverse, center, middle
name: results

# Results

---
class: left

# Participants

???

500 in the multi-consistent condition, 499 in the single-analyst and the multi-inconsistent condition

---
class: inverse, center, middle

# Main results

---
background-image: url("talk/Figure1b.jpg")
background-position: 50% 50%
background-size: 78%

???
For the results, we regressed all outcomes on prior beliefs and condition, with the single-analyst condition as the reference category. So all results that I will show you control for prior beliefs and show the results for the two multi-analyst conditions compared to the single-analyst condition. I will now walk you through each individual aspect of this Figure.

---
background-image: url("talk/Figure1.jpg")
background-position: 95% 57%
background-size: 40%

.center[

# Results: Multi-inconsistent

]

**In line with our hypotheses**, lay consumers of multi-analyst studies with **inconsistent** results  
--

`\(~~\)`  
`\(~~\)`

For the additional **exploratory measure**, lay consumers of multi-analyst studies with **inconsistent** results
<svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:green;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M8 256C8 119 119 8 256 8s248 111 248 248-111 248-248 248S8 393 8 256zm143.6 28.9l72.4-75.5V392c0 13.3 10.7 24 24 24h16c13.3 0 24-10.7 24-24V209.4l72.4 75.5c9.3 9.7 24.8 9.9 34.3.4l10.9-11c9.4-9.4 9.4-24.6 0-33.9L273 107.7c-9.4-9.4-24.6-9.4-33.9 0L106.3 240.4c-9.4 9.4-9.4 24.6 0 33.9l10.9 11c9.6 9.5 25.1 9.3 34.4-.4z"></path></svg> `\(~~~\)` Believe the 5% estimate is more likely to stem from idiosyncratic choices

???

In line with out hypotheses, we find that lay consumers of multi-analyst studies with inconsistent results have lower posterior beliefs (you can see this estimate here), find the results less credible, have less confidence in the aggregate estimate of 5%, and believe this estimate is more likely to stem from bias, error, and discretion (which we define as idiosyncratic choices among multiple valid approaches, or what we researchers often refer to as researcher degrees of freedom).

---
background-image: url("talk/Figure1.jpg")
background-position: 95% 57%
background-size: 40%

.center[

# Results: Multi-consistent

]

**Contrary to our hypotheses**, lay consumers of multi-analyst studies with **consistent** results  
--

`\(~~\)`  
We found **no significant effects** on  
--

`\(~~\)`  
For the additional **exploratory measure**, lay consumers of multi-analyst studies with **consistent** results
<svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:green;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M8 256C8 119 119 8 256 8s248 111 248 248-111 248-248 248S8 393 8 256zm143.6 28.9l72.4-75.5V392c0 13.3 10.7 24 24 24h16c13.3 0 24-10.7 24-24V209.4l72.4 75.5c9.3 9.7 24.8 9.9 34.3.4l10.9-11c9.4-9.4 9.4-24.6 0-33.9L273 107.7c-9.4-9.4-24.6-9.4-33.9 0L106.3 240.4c-9.4 9.4-9.4 24.6 0 33.9l10.9 11c9.6 9.5 25.1 9.3 34.4-.4z"></path></svg> `\(~~~\)` Believe the 5% estimate is more likely to stem from idiosyncratic choices

???

Contrary to our hypotheses, we find that lay consumers of multi-analyst studies with consistent results also have lower beliefs and believe the 5% estimate is more likely to stem from error or from idiosyncratic choices. We found no significant effects on the credibility of the results, confidence in the aggregate parameter estimate of 5%,  and the perceived impact of bias on the estimate.

---
background-image: url("talk/Figure2.jpg")
background-position: 50% 50%
background-size: 78%

???

Note.  Prior beliefs are displayed in blue; posterior beliefs are displayed in orange. The respective boxes display the lower quartiles, medians, and upper quartiles of prior and posterior beliefs by condition.

It is worth noting on the basis of Figure 2 and a post-hoc, paired t-test that, while multi-analyst studies with consistent results perform worse or no better than single-analyst studies on all measures, there is a significant, positive effect of the findings on posterior beliefs within the multi-consistent condition: i.e., beliefs in the research hypothesis are greater after reading the multi-consistent study results, M_d=4.75, p<.001. This finding clarifies that multi-analyst studies are not necessarily bad in absolute terms — however, when comparing to conventional, single-analyst scientific research, crowdsourced data analysis does not seem to provide an improvement in the sway and credibility of scientific research to lay consumers.

---

class: inverse, center, middle
name: discussion

# Discussion

---

`\(~~\)`

.center[
### **Compared to a single estimate, we find no evidence that crowd estimates improve lay perceptions of scientific findings**
]

`\(~~\)`

???

We conclude that, compared to a single estimate, we do not find evidence that crowd estimates improve lay perceptions of scientific findings. In the future, we are curious to explore whether it might be aversion to variability that explains these findings, and what the perceptions of scientists are. We also believe that in the future it is important for crowd scientists to consider how -- for example, by drawing from research on science communication -- to tackle science skepticism and effectively communicate variable crowd estimates to lay consumers.

From the proliferation of big team science and large-scale replication initiatives to preregistration and registered reports, several scientific fields have undergone significant reform with the well-intended goal of improving the reliability of scientific research. The multi-analyst approach has many worthy uses, from demonstrating the arbitrariness and impact of individual analytic choices to acknowledging the inherent variability of results and averaging across idiosyncratic analytic choices to obtain more accurate parameter estimates. However, as with any real-world intervention, scientific reform can have unintended consequences. Here, we focus on the effects of crowdsourcing data analysis, and find that the multi-analyst approach may have an unintended consequence.  
While partly instituted with the goal of improving the credibility of scientific research, lay consumers appear to resist the inherent variability and lack of consensus that often reflect the reality of multi-analyst research. To our surprise, even when results generated by independent analysts are largely consistent, lay consumers are less likely to believe in the reported phenomenon and more likely to think that the findings stem from error or idiosyncratic choices. 
--

## Future directions
<svg viewBox="0 0 512 512" style="position:relative;display:inline-block;top:.1em;fill:#035AA6;height:1.5em;" xmlns="http://www.w3.org/2000/svg">  <path d="M256 8C119.043 8 8 119.083 8 256c0 136.997 111.043 248 248 248s248-111.003 248-248C504 119.083 392.957 8 256 8zm0 448c-110.532 0-200-89.431-200-200 0-110.495 89.472-200 200-200 110.491 0 200 89.471 200 200 0 110.53-89.431 200-200 200zm107.244-255.2c0 67.052-72.421 68.084-72.421 92.863V300c0 6.627-5.373 12-12 12h-45.647c-6.627 0-12-5.373-12-12v-8.659c0-35.745 27.1-50.034 47.579-61.516 17.561-9.845 28.324-16.541 28.324-29.579 0-17.246-21.999-28.693-39.784-28.693-23.189 0-33.894 10.977-48.942 29.969-4.057 5.12-11.46 6.071-16.666 2.124l-27.824-21.098c-5.107-3.872-6.251-11.066-2.644-16.363C184.846 131.491 214.94 112 261.794 112c49.071 0 101.45 38.304 101.45 88.8zM298 368c0 23.159-18.841 42-42 42s-42-18.841-42-42 18.841-42 42-42 42 18.841 42 42z"></path></svg> `\(~~\)` Does **variability aversion** explain the findings?

???

---
class: inverse, center, middle

# Thank you!