DEVIDENCE leverages large language models (LLMs), deterministic code, and human-in-the-loop validation to construct and maintain our relational databases. The integrated pipeline is designed to convert published studies and raw survey data into structured, comparable evidence, modeled on how researchers have traditionally done manual evidence aggregation. The pipeline is reproducible, documented, and designed to be run continuously as literature grows:
Step 1: Metadata Collection and Verification
Publication metadata is collected through a combination of LLMs and API connections to external databases, then manually reviewed for accuracy.
Step 2: Structured Data Extraction
Treatment effects and other study-level fields are extracted from the full text of each paper using a combination of LLMs and deterministic code. No extracted value is trusted on first pass; coding scripts re-read each figure at its source, and a second model checks every field against the paper's text. An AI agent orchestrator approves each stage of the extraction process before the pipeline proceeds to the next. Only once this multi-stage verification is complete does a record move on to human validation.
Step 3: Validation with DEVIDENCE Fellows
A cohort of scholars from low- and middle-income countries, convened through the DEVIDENCE Fellows Program, reviews and validates the extracted study-level information. Fellows work asynchronously through an application developed and maintained by the DEVIDENCE team.
Step 4: Cataloguing Microdata Variables
When underlying datasets are available, including household surveys, administrative records, and other individual-level datasets, the variables used in each analysis are systematically catalogued. This includes recording each variable's original name, definition, units, and role within the study.
Step 5: Harmonizing Like Variables
Because studies often measure similar concepts using different definitions, scales, or naming conventions, all related variables are harmonized into a common framework, using economic methods. This allows outcomes and covariates that are substantively comparable but reported under different labels or units to be aligned for cross-study comparison and synthesis.
Step 6: Quality review
Every decision and judgment made by the LLMs is recorded, along with supporting evidence, creating a transparent audit trail for each piece of data. Before any information enters the DEVIDENCE database, humans review this audit trail and approve the AI’s work at key checkpoints, verifying the accuracy and quality of the aggregated data. The end result is a single database researchers can use, along with clear documentation of how each variable of interest was constructed.