Will changing how science is measured change what science produces?


Author unknown, image from Altmetrics: a manifesto.
Click on the image to see more information about this topic.

I have written previously that the definition of  success will deeply influence the outcome of the project. When looking at innovation projects as a system, I believe that the design criteria are the greatest place to intervene and inform the overall flow of creative thinking. When you look at a project through the lens of evolution, the design criteria are the selective pressures that shape how organisms adapt to their contextual conditions. It’s all biomimicry… of course.

In Nature, there was an incredible Comment article by Heather Piwowar titled “Value all research products” which discussed the emergence of different measures of success in scientific research. These “altmetrics” are simply:

New ways to measure engagement with research output.

Basically, the National Science Foundation (NSF) is now measuring researchers on content beyond papers. This means that data sets, or engagement in online research forums will now be taken into account when determining the success of a given researcher and whether they should receive further funding. This new selective pressure could deeply influence the shift towards open information, alternative research models and increased collaboration.

There is a “manifesto” outlining the details behind altmetrics which discusses the bottle necks currently occurring in the status quo of peer reviewed journals and how new metrics will not just alter the content being produced, but how it gets validated and shared. One of their interesting statements is around the opportunity of speeding up information exchange:

The speed of altmetrics presents the opportunity to create real-time recommendation and collaborative filtering systems: instead of subscribing to dozens of tables-of-contents, a researcher could get a feed of this week’s most significant work in her field. This becomes especially powerful when combined with quick “alt-publications” like blogs or preprint servers, shrinking the communication cycle from years to weeks or days. Faster, broader impact metrics could also play a role in funding and promotion decisions.

I don’t know enough about this space to comment deeply, but Heather Piwowar runs a fantastic blog (of course), that outlines her deep thinking in this topic. Most amazingly, she even shared the process for writing a Comment article for Nature. She reflects on the involvement of the editors, suggesting that perhaps they should be co-authors due to their contribution, and even shares the first draft to highlight what was lost in the refinement process.

This level of transparency is very intriguing. Design thinking is still fairly opaque, especially if you’re looking for a specific answer around how a particular decision was made. The pressure to own and protect ideas leads to very closed interactions. Open innovation methodologies are developing mechanisms to encourage sharing and engagement, which suggests that the selective pressures of design thinking might be shifting along a parallel stream to scientific research.

Fun logo for their second conference/workshop.

Fun logo for their second conference/workshop.

As part of my follow up research around this topic I stumbled upon a group of researchers with the project titled; “Beyond the PDF“. It appears to be an intriguing discussion about the future of the “white paper”. The current model as we know is very static, with a one way flow of information while the shifts in how we generate content suggests a more interactive and flexible platform for communicating and evolving research.

I’d like to go the next step and have “beyond the magazine”. Blogs are really only informal, unedited variations of a magazine. The comments offer a certain level of interactivity, but the content is still a one-way stream. We are still a ways away from a “dialogue” format…



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s