We've been taking a look at the data from our first test run and there's some pretty interesting stuff!
Good morning! Preliminary and very exciting results are in from our ‘private trial’ of the platform, conducted at the NIH Intramural Campus and Brown University. At time of writing, 128 scientists have now joined the ‘Engine, and have provided 498 unique ratings. Great credit goes to ‘Engineer Dmitrijs Celinskis for mapping DOIs from the rated papers to Altmetric scores for corresponding papers, and starting to play with the results.
The upshot: Our initial analyses indicate Discovery Value can identify unique subsets of papers. The figure shows the quartile distribution of Discovery Value ratings for our first 400 ratings given, and their mean Altmetric scores. For the first 3 quartiles, increasing Discovery Value predicts increasing Altmetric scores, with a distinct peak in the 3rd quartile.
This 3rd quartile is, by definition, those papers judged as making substantial but not elite Discovery contributions. This makes sense: Papers that are changing minds about a topic should get more attention, a quantity that is very well measured by our colleagues at Altmetric.
In contrast, though, the group of papers with highest Discovery Value (the 4th quartile) showed lower Altmetric scores. These data suggest that public attention is most prominently awarded to findings that change opinions, but do so in understandable ways, potentially living within current paradigmatic structures. The most transformative papers, those that potentially threaten paradigms (and, by extension, the research programs of those reading the papers) received less public attention. This lack of attention could reflect, of course, not even knowing how to mention the more revolutionary work, it could also reflect another signature of the most transformative findings—a lack of Confidence in them until they are confirmed by replication in other labs. Whatever the reasons, DV is, in fact, doing what we hoped—finding groups of papers that go beyond what public opinion (expressed in citations or Altmetrics) can show.
While this is all early days—have I used the words preliminary and initial enough?!—we are further encouraged that a different relationship was observed between Altmetric scores and ratings of Actionability—the rated utility of findings in a paper. In this case, the utility of a paper had a mostly flat relationship with Altmetric scores, with one notable exception—the least useful papers are those that received the least attention. Also, no quartile of Actionability data has a very profound relationship to Altmetric scores: Basically, there isn’t a peak, nothing that comes close to the high scored seen in the 3rd quartile of DV.
So, in sum: DV and Actionability have different relationships to Altmetric numbers, underscoring the independent of these two categories we are having people rate. And, maybe (just maybe?) we are really able to find those papers that are making a change ahead of the curve of broader appreciation. Lets see how the next 1000 ratings go as we march onward through the public beta, but its exciting times in any event.