cordis-pic-ror dataset → github →

Manual curation workflow

The pipeline applies only high and medium confidence matches automatically. Everything weaker is held back for a human to verify. This page walks through the loop from "pipeline finished a run" to "new Zenodo version published".

What needs human review

review  Pipeline made a guess

The resolver had some evidence (a fuzzy name match, a token-sort name hit, an ambiguous direct OpenAIRE link) but not enough to call it automatic. The PIC is in pic_ror_resolutions with confidence = "review". The reviewer either promotes the guess to an override or leaves the PIC unresolved with an empty ror.

missing  No pipeline candidate at all

The resolver couldn't link the PIC to any ROR. Usually OpenAIRE doesn't carry that PIC, the institution lacks a public website, or the legal name doesn't match a ROR record's aliases closely enough. The reviewer looks the institution up by hand (legal name, country, website) and writes a fresh override entry.

Why frequency matters

CORDIS participation is heavy-tailed. A handful of major institutions (CNRS, Max Planck, Wageningen, …) account for thousands of unresolved appearances; a long tail of one-off SMEs accounts for the rest. review_queue.py joins cordis_participants with pic_ror_resolutions and emits one row per not-auto-resolved PIC, sorted by total occurrence count across HE + H2020 + FP7. The Numbers page has a slider that lets you eyeball the diminishing-returns curve directly.

Run the loop

# 1. Re-run the pipeline against the latest source dumps.
just pipeline                  # produces tmp/org-pipeline/organizations.sqlite + exports

# 2. Refresh the consumer-ready outputs.
just export                      # output/pic_mapping.csv (+ .jsonl + summary.json)

# 3. Generate the manual-review queue.
just review-queue              # writes tmp/.../review_queue.csv

# 4. Open review_queue.csv in your spreadsheet of choice. Walk top-down.
#    Each row carries: pic, count, country, short_name, legal_name,
#    website_url, openaire_alt_names, openaire_external_ids,
#    pipeline_ror, pipeline_confidence, pipeline_source.
#
# 5. For each kept decision, append to overrides/pic_ror_overrides.yaml.
#    The schema is below.

# 6. Commit overrides/pic_ror_overrides.yaml. Re-run `just export` so the
#    new override is merged into output/pic_mapping.csv.
# 7. Upload output/pic_mapping.csv as a new version of the Zenodo record.

The override schema

- pic: "999996549"
  ror: "01ggx4157"                # bare ROR id; "" = leave unresolved
  name: "European Organization for Nuclear Research (CERN)"   # editor affordance
  note: "Promoted from `review`; OpenAIRE was missing the ROR pid."

- pic: "888777666"
  ror: ""
  name: "Some defunct subsidiary"
  note: "Pipeline matched the parent group; we want exact matches only."

The consumer-side loader reads only pic and ror. name and note exist for the human editor; they're ignored downstream. A row with empty ror explicitly leaves the PIC unresolved; it removes the pipeline's guess from the auto-resolved set, even if the pipeline had picked something for it.

AI-assisted bootstrap

The first 3000 rows of the queue have already been run through an AI-assisted review pass. The decisions live at overrides/ai_review/top_3000.yaml (see the README in that directory). Each entry carries source: ai-assisted and a reasoning string explaining what was checked. They are not auto-merged into pic_ror_overrides.yaml; the human reviewer triages them first.

The agents ran a strict pass first (require a confident, dedicated ROR for the exact subsidiary), then a relaxed pass over the strict-pass unresolved entries (accept the parent group's ROR when the subsidiary itself isn't in ROR). Generalized entries are flagged in their reasoning with the literal string "Generalized to parent: …", so they can be filtered or re-reviewed.

What stays unresolved

Four patterns account for most unresolved entries after both passes: