The evidence-synthesis stack is being rebuilt, all at once

Ten Journal of Clinical Epidemiology papers and one arXiv stat.ME preprint landed in the same window, and read together they describe a single, uncomfortable fact: the methodological substrate that HTA bodies and regulators treat as settled — GRADE, PRISMA-P, guideline appraisal, gap-to-priority frameworks — is being renegotiated in public, while AI is quietly being inserted into the meta-analytic pipeline itself.

The frontier: LLMs inside the meta-analysis, not around it

The arXiv preprint is the provocation. The authors feed study-level clinical and methodological characteristics to an LLM, ask it to generate triplets (anchor, most-similar, most-dissimilar), then train an embedding model with triplet loss so that “similar” studies cluster in a learned space. Within-cluster meta-analyses replace pooled analysis. Applied to 58 observational studies of cognitive outcomes in preterm- versus term-born children, three clusters emerged; two retained substantial residual heterogeneity, and the most homogeneous cluster produced a more extreme pooled effect with a narrower prediction interval than the overall analysis.

That last sentence is the one to dwell on. A method that reliably produces tighter, larger effects in its “cleanest” stratum is exactly the method a sponsor would want to deploy — and exactly the one a reviewer should interrogate hardest. The LLM’s triplet assignments are not validated against blinded expert consensus, the embedding architecture is underspecified, and stability of three clusters across 58 studies is not assessed. The triplet-loss framing is borrowed cleanly from computer vision; the assumption that similarity in a learned embedding corresponds to exchangeability in a random-effects sense is doing all the inferential work, and is not defended. Elegant frame, unaudited interior.

The rule-makers are catching up to themselves

While the frontier moves, the standards are admitting their seams. A JCE systematic survey finds that guidelines claiming GRADE compliance often don’t actually adhere — and that the criteria used to judge adherence themselves vary across studies. A companion methods paper takes on the awkward case GRADE has long underspecified: how to rate certainty when the target threshold is value-based (“little or no difference,” “moderate effect”) and the point estimate sits on top of it. This is the case where imprecision judgments and direction-of-effect calls become discretionary, and it is also, not coincidentally, the case that drives label language disputes at HTA. A cross-sectional study of PRISMA-P 2015 adherence finds the protocol-stage checklist is patchily followed by the people who claim to follow it.

On top of those diagnoses, three constructive efforts. TRUSTGUIDES is a new appraisal tool aimed at individual recommendations rather than whole guidelines, with explicit scope for living guidelines, GRADE EtD, adaptation, COI, and — notably — AI-assisted guideline development; the authors argue AGREE II and RIGHT do not cover this terrain. STREAM is a protocol for methodology-about-methodology: how should research methods guidance itself be developed, across design, statistics, and qualitative work. A scoping review maps frameworks for translating evidence syntheses into research gaps and priorities, organised around GRADE CoE domains and EtD criteria.

Why biometrics teams should care now, not later

Most readers here do not write guidelines. They cite them. Indirect treatment comparisons, NMAs supporting external control arms, RWE packages for FDA and EMA, and HTA dossiers for NICE, ICER, and G-BA all sit downstream of GRADE certainty ratings, PRISMA-P-compliant protocols, and guideline-backed comparator choices. The new Bayesian NMA in advanced HCC (10 RCTs, n=7,301; OS HR 0.79, 95% CI 0.74–0.84 for ICI-based regimens vs TKI; HBV-positive PFS HR 0.54) is a clean illustration: rigorous network, etiology-stratified subgroups, and a top-ranked OS regimen — sintilimab plus a bevacizumab biosimilar — that is not approved by FDA or EMA. The ranking is statistically defensible and operationally inert for a Western submission. That gap between methodologically valid and regulatorily usable is precisely what the JCE cluster is trying to systematise.

The renegotiations are not binding yet. TRUSTGUIDES is in development, STREAM is a protocol, the LLM-clustering paper is a preprint, and the GRADE-near-threshold guidance is a methods article rather than a GRADE Working Group update. But the direction of travel — AI inside synthesis, tighter appraisal of recommendations, formal frameworks for gap-to-priority translation — will reach HTA reviewer checklists inside 12–24 months. Submissions drafted on today’s assumptions will be appraised on tomorrow’s.

Protocol read: The methodological substrate under your evidence packages is moving in several directions at once; none of it is binding this quarter, but the teams tracking which renegotiation lands first — and resisting the urge to adopt TRUSTGUIDES or STREAM prospectively before either is validated — will write fewer surprised response-to-questions in 2027.

What to do now:

• Flag any submission-supporting NMA or ITC that relies on LLM-assisted study selection or clustering — pre-specify human adjudication and document it.
• For HTA-facing GRADE ratings near value-based thresholds, pre-specify the target of certainty rating in the SAP rather than leaving it to the synthesis author.
• Audit cited CPGs in dossier comparator justifications for actual GRADE adherence, not just claimed adherence; the JCE survey gives the criteria landscape.