The BioChain Advisory · Engagement

Testing Whether a Scientific Result Was Genuinely Reproducible

One example of the kind of work The BioChain Advisory does: helping organisations work out whether their systems can actually prove what they claim, before a regulator, investor, customer or incident forces the question. Here is how that played out.

A research organisation believed its environment was well controlled and its scientific outputs reproducible. One particular result, however, had passed through several stages before reaching the final report.

The source material was generated in one system. Raw data was exported for analysis. A researcher processed the files using a specialist pipeline. Intermediate outputs were stored elsewhere. Additional reference data was incorporated. A later version of the software changed part of the analysis. The final interpretation then appeared in a presentation, report and publication.

Each individual step looked reasonable. The question was whether the entire evidence chain could still be reconstructed.

We selected a representative result and worked backwards from conclusion to source: whether the original source records remained available, whether the correct software version could be identified, whether parameters and reference datasets were recorded, whether intermediate transformations were preserved, whether manual changes were documented, and whether responsibility was clear at each stage. We also tested whether two people, working only from the available records, would reach the same understanding of how the result was produced.

This revealed gaps that a conventional policy review would not have found. The organisation had excellent SOPs while still relying on undocumented file renaming. It had an approved analytical process while not retaining the configuration used for a specific run. It preserved the final output but not the intermediate transformations needed to explain it.

Did reproducibility exist in practice, not merely in principle?

The organisation received a prioritised view of which weaknesses materially affected evidential confidence and which were lower-value improvements. That distinction mattered: not every process needed a sophisticated technical solution. In this case, a missing identifier convention and one poorly retained class of record created more risk than any expensive systems problem would have.

Questions & Answers

Do you need access to our raw data to do this?

Yes, for the specific result or results being tested. We work backwards from a real conclusion through the actual records, not a description of your process.

What if the audit finds a real problem with a published result?

That is exactly what this is designed to surface early, while it can still be addressed quietly, rather than after a challenge or retraction forces the question.

How many results do you test in one engagement?

Usually a small number of representative results, chosen for how consequential or how typical they are, rather than an exhaustive review of everything the organisation has published.

Can this be done without involving the original researchers?

Some involvement is usually necessary, since they often hold context the records alone do not capture. Their time commitment is normally limited to a small number of focused conversations.

About The BioChain Advisory.

The BioChain Advisory helps organisations understand how evidence actually moves through their existing systems, independent of whether they ever deploy The BioChain platform itself. Engagements typically take one of two forms: a Data Integrity Audit, tracing representative evidence chains from source to conclusion to establish what can genuinely be demonstrated, or a Regulatory & Cross-Border Readiness Review, examining whether an organisation's systems and records can support the claims a new jurisdiction or regulation requires.

See the full Advisory offering →