Signal Detection: From First Principles to a Validated Signal

A signal is information suggesting a new potentially causal association between a medicine and an event, serious enough to warrant verification — it is a hypothesis, never a proven fact. An SDR (Signal of Disproportionate Reporting) is the statistical alert a disproportionality method raises (e.g. PRR ≥ 2, chi-square ≥ 4, a ≥ 3). An SDR only becomes a validated signal after it passes validation (is it real?) and assessment (what does it mean, causally?).

Quantitative disproportionality methods count Drug × MedDRA Preferred Term pairs, which requires coded, de-duplicated, locked data. Without MedDRA coding there is no pair to count; without de-duplication a case counted twice inflates the count into a false signal; and running on half-finished cases produces a result that can't be defended. The one exception is a single, well-documented Designated Medical Event case, which can be a qualitative signal at n = 1 — but it still has to be processed and coded first.

All four measures are computed from the same 2×2 table (a = this drug with this event, b/c/d fill out the rest of the database). PRR and ROR are frequentist ratios, typically screened with a ≥ 3, PRR ≥ 2 and chi-square ≥ 4. IC (Information Component / BCPNN) and EBGM (MGPS) are Bayesian methods that shrink weak, low-count estimates toward the null, which makes them more stable for rare events where a is small.

The same fetched content plays two roles depending on the question being asked: in intake it answers is there a reportable case here?; in the signal system it answers what evidence supports or detects this drug–event signal? The design rule that prevents duplication is fetch once, route twice — a single ingestion worker fetches and de-duplicates once, and a router emits two independent events (case.candidate to intake, evidence.linked to signal). The signal system never creates cases itself.

Test it the way computer-system validation (OQ/PQ) tests any tool: you are not testing whether the drug is dangerous, you are testing whether the tool computes and records correctly. Six angles cover it — math correctness against a hand calculation, a positive control (a known true pair must fire), a negative control (a random pair must stay quiet), edge cases (a<3, zero cells, single-case DME), data integrity (duplicates, PT roll-up), and reproducibility (the same data-lock re-run must give identical, audit-trailed output).

Screening tens of thousands of pairs at a nominal significance threshold guarantees a large number of false positives by chance alone, so raw p-values or an uncorrected chi-square cutoff are not enough at scale. Two established mitigations: run a Bayesian shrinkage method (IC/BCPNN or EBGM/MGPS) instead of a pure frequentist ratio, since shrinkage pulls noisy, low-count estimates toward the null before they can fire; and, if you keep a frequentist screen, apply a multiplicity adjustment (e.g. a false-discovery-rate control) rather than a flat PRR ≥ 2 cutoff across the whole pair space. Either way, treat the review queue's false-positive rate as a metric you monitor and tune, not a one-time design decision.

The standard 2×2 (drug × event) doesn't capture interaction effects — a case with two co-administered drugs and an event can look identical to two separate single-drug reports in a naive count. Interaction signal detection instead compares the observed rate of the drug pair + event combination against the rate expected if the two drugs' individual effects were simply additive (an Omega shrinkage measure, or an interaction term added to a logistic/Bayesian model). This is a materially different computation from single-drug disproportionality, not a variant of it — plan for a separate pair-level aggregate table (drug A × drug B × PT) if interaction detection is in scope, rather than trying to bolt it onto the drug × PT schema.

The mechanics are similar but the interpretation differs enough that vaccine signal detection is treated as its own discipline. Vaccines are typically given once or a few times to large, otherwise-healthy cohorts (often children), so background incidence of the event in the unexposed population matters more than it does for a chronic-use drug — observed-versus-expected (using an independent background incidence rate) is used alongside or instead of PRR/ROR precisely because the comparator arm inside spontaneous-report data ("other vaccines") is a weaker denominator than an epidemiological background rate. If a platform supports vaccine PV, the disproportionality engine needs a background-rate input path, not just the drug × PT database comparator.

Validation confirms the SDR is real (not duplicates, coding error, or already labelled); assessment is the separate step that weighs causality once validity is established, and Bradford Hill's criteria are the structured checklist a reviewer works through there — strength of association (how large is the effect versus background), consistency (seen across multiple, independent case series or databases), temporality (exposure clearly precedes the event), biological plausibility (a mechanism fits), dose-response, and coherence with related information (e.g. class-effect precedent). No single criterion is decisive and Bradford Hill is a framework for documented judgement, not a scoring formula — which is exactly why the assessment step's reasoning has to be recorded in the audit trail alongside the SDR's statistics, not just the final action taken.

At high volume, a fixed PRR ≥ 2 / a ≥ 3 screen becomes noisy in the opposite direction from the small-n problem: with tens of thousands of reports, statistically significant but clinically trivial ratios (PRR = 2.1 driven by a huge denominator) start firing constantly, and masking from the product's own dominant, well-known reactions gets worse as those reactions accumulate more reports. The usual response is to move from a single flat threshold to a stratified and trend-aware review — track a pair's ratio over successive data-locks rather than reacting to one snapshot, stratify by market/region/time period to unmask pairs hidden behind a dominant reaction, and set review priority by clinical seriousness of the event, not purely by the size of the ratio.