Replicability for research means that the same workflow is used by different people for a research project using a new dataset yielding different but expected results. What happens to research that can’t be replicated?
For this activity, you will read the case study of Liliana Mayes, a third-year doctoral student in developmental psychology who is dealing with an issue of replicability with her research.
As you are reading, take on the role of a fifth-year graduate student who is friendly with Liliana but uninvolved in her research program. She has described her situation to you and asks for your advice.
Case Study
Complete this Activity
In the comments below, write a reflection on the following:
- Should Liliana follow her advisor’s recommendation and publish the paper as soon as possible, or should she heed Dr. Hobson’s caution?
- If she does not choose to publish the research, what should her next steps be?
Image Credit: Image used on featured image: Multicoloured brain illustration by Pxfuel free for commercial and personal use
I would advise Liliana to heed Dr. Hobson’s caution over her advisor, Dr. Blackburn. In any case, racing to publish results is not an ethical practice especially if the motivation behind it is to just gain novelty and recognition rather than publishing accurate results. Furthermore, the replication study would disprove her published results which may cause study retraction, making it a public affair and worse overall for the team. Dr. Hobson has a good point, and it doesn’t hurt to take a step back to consider their options.
If Liliana does not choose to publish the research, she can submit it as a pre-print, document it on her personal website, or participate in knowledge translation such as presenting her findings at conferences or research panels. As a doctoral student, her responsibility is to her own professional and research integrity and she should be empowered to raise her concerns to her advisor, regardless of his status as a full tenured professor. I don’t believe students should feel intimidated by their professors or advisors, they have a right to their own research practice and control over its publication.
I agree with everything in the comment above and would add that she could acknowledge the other studies findings in her paper and point to areas of concern in her own. Her research doesn’t have to be 100% correct, but what is does need to do is be in conversation with other researchers in her field. It shouldn’t be all or nothing.
I also think Liliana should wait and take all possible cautions to ensure the validity of her work. It would also be good to get a more detailed idea of the specific study performed by another group. It should be possible to compare the differences in the two studies and the factors that can be affecting the change in results. More detailed discussions with both professors should be done for good quality research work publication.
I agree with Dr. Blackburn. The research should be published, it may be the case that other research groups have the same idea out there. Liliana’s work is a good set of data helpful for others to improve the work. As for now, in this case, there was only one replication that didn’t show any effect in the condition, opening room for adapting by publishing articles and data together and building the infrastructure to collaboration opens the room to preserve our work, accelerate pace for new findings.
Perhaps Liliana’s research could be published in collaboration with the folks who got different results. Attaching those findings together could be an interesting way to present the nuances of the research.
I think she should proceed, but with caution. Her research is not invaluable. I don’t know what other factors may have been involved (ie. the children), but there’s no reason she can’t note those down, note the different results, and publish. While the positive feedback from the media and social media is not a gauge of good results, I think the fact that the research has been presented and spoken about in public view is a nudge to publish. She could also try to find out if others had replicated her studies. Perhaps, she can also speak with Drs. Blackburn and Hobson together.
If she chooses not to publish, there are options available to her whether she decides to keep researching further and then publish, or publish on her own personal site, collaborate with others, present at panels, and so on.
This is a tricky scenario! My perspective on it is totally coloured by the opening paragraph that describes the status of the two faculty members involved: Dr. Blackburn as a tenured professor and Dr. Hobson as an assistant professor, who, if he’s on the typical academic timeline, would be going for tenure very soon. In this scenario, Dr. Hobson would be “hungry” for publications to add to his tenure and promotion dossier. So the fact that he is the one expressing additional caution makes me pause even more.
Others have made excellent points above. I think it would be really helpful for Lilianna to start collaborating more closely with the external research team that was unable to replicate her results and try to suss out some of the variables that could have been contributing to the different result. It could be a great opportunity to publish an even more interesting study together. I think that she should eventually publish her research, but that she should take a bit of a pause to dig deeper into the differences between the two studies first.
I agree with many in the comments above, and agree with the comment directly above that Dr.Hobson would most likely be hoping for more publications to add to their dossier. I think that Lilianna should take a pause to acknowledge the other studies findings in her paper and point to areas of concern in her own and it should work to see why her work was unable to be replicated.
I’m thinking about both Christie’s comments above and the 15-minute Ted Talk we watched during the modules for this topic. I agree with Christie’s comments, that she thinks Dr. Hobson has more to lose if the paper was not published. I also think about Lillianna, and her own academic integrity: if she publishes the paper, and if the other research group also publishes something, disproving Lillianna’s results, then it seems more likely that she would be blacklisted. There may be discrepancies with the other research groups results, as there may be with Lillianna’s but if I were her peer, I would advise taking the constructive feedback and re-assessing her own results, to avoid becoming another name on the the Retraction Watch website.
As her peer, as this question asks, I would suggest she re-evaluate her analysis and conclusions before publishing. Reproducibility, I think, should become a condition for tenure, etc, much like the previous module in this POSE course, about Open Publishing.
I believe that some caution might be needed before Lilianna can proceed. I think that she needs to ensure that she has a full picture of the other study to see if there is anything, however small, that could have resulted in the change of results. Then if she does decide to publish she could acknowledge the other study and explain why the differences happened. Also, would it be possible for her to publish the work as either a working paper or early version of her work? That way it will still be published, but she can acknowledge that things might change.
Also, we should not forget that she/they ran the experiment twice with similar results. Why were they able to replicate the results while the other groups wasn’t? Perhaps she needs to ensure that all of the relevant data was published on OSF for her original study. Maybe the issue lies there.
No matter what, I think caution is the key to any move that she makes.
I believe Liliana should heed Dr. Hobson’s caution, and not publish it as soon as possible. Since another study with a larger sample of children has come to light, it might be in Liliana’s best interest to either acknowledge the external study in her own findings, or run her experiment again to replicate her results with different sample sizes. Though her initial results may be enlightening, erring on the side of caution couldn’t do any harm, so she should perhaps experiment more (than just twice) to see the variations in her results. In the end, caution should be exercised here.
In order to answer this question, I think it is important to refer back to the definition of replicability.
Repllicability is defined as the production of a consistent outcome when the same workload is used using a new dataset. While a small margin of error is possible, in this case this margin of error is possible, in this case the results produced an opposite outcome.
This makes me wonder about the transferability of the workflow. It could be that Liliana did not report her workflow/methods of her original research in a fully transparent manner. This, thus led to such an outcome. So, before we can compare apples to apples re the two studies and their outcomes, I feel like we have some unknowns to investigate first. Liliana should definitely take the time to establish a collaborative partnership with the other researchers to investigate the process and the methods they used to generate their results to see if there is as much harmony as possible between their approaches. It may be that her study is valid afterall.
In terms of publishing, I do not see an issue with her publishing her study if she followed methodological processes, and ensured rigour was maintained in her study, but she may want to include a section, based on her conversations with the other researchers, outlining any potential issues with replicability that they discovered through their conversation to allow full transparency for the readers. This process will also ensure she learns from any omissions or mistakes she made during the creation of her study’s workload to avoid any issues with future studies.
I agree with the others who have encouraged Liliana to consider Dr. Hobson’s caution and take time to review and reconsider her paper before publishing. As a doctoral candidate, producing work that is both reproducible and replicable should be important for Liliana, who is developing their own professional standing while also building up Drs. Blackburn & Hobson’s academic standing. Rather than pushing the publication forward as-is, I would encourage Liliana to use the second research group’s results as a learning opportunity and take time to re-run the original experiment with a different sample size, as well as considering any unintended biases that may have been introduced into her study or the replication study. Reviewing the other research group’s study can also be an opportunity for Liliana to revise and clarify her dissemination of the experiment design and analysis plan, as the way it’s been written up may have unintentionally affected the replication of the study.
I agree with many of the commenters before me that Liliana would be right to heed Dr. Hobson’s caution. As a graduate student and non-tenured faculty member, respectively, Liliana and Dr. Hobson have more to lose than Dr. Blackburn if their findings are published prematurely, and so he should be able to respect their decision.
In terms of what Liliana should do next, I would advise her to find out as much as she can about the outside, larger study, and find out if there were any changes to her study design that may have caused a different result. If there are no clear deviations, then she would ideally conduct her own larger study, or if this is not possible, address the divergent study in the results she hopes to publish.