SourceBench: A Benchmark for Cited Source Quality in Generative Engines

SourceBench focuses on the quality of cited sources along dimensions such as:

• semantic relevance
• factual accuracy
• freshness
• objectivity / tone
• layout / ad density
• accountability
• transparency
• authority

This repository is the benchmark codebase. It contains:

• the 100 benchmark queries, collected sources, and source scores used in the paper
• source collection scripts
• source judging scripts
• metric computation scripts
• official submission validation and runner scripts
• split policy and official submission contract

The public leaderboard site should be hosted separately from this repository.

data/queries/
  sourcebench_queries.csv
data/collected-sources/
data/content-scores/
leaderboard/
  QUERY_SPLIT_POLICY.md
  OFFICIAL_SUBMISSION_CONTRACT.md
  README.md
  examples/
src/source-collection/
src/content-scoring/scripts/
src/evaluation/
requirements.txt

Install dependencies:

pip install -r requirements.txt

Run the public evaluation pipeline:

python src/source-collection/get_urls.py \
  --input path/to/public_queries.jsonl \
  --output output/public_urls.json \
  --model YOUR_MODEL_NAME \
  --openai-base-url YOUR_OPENAI_COMPATIBLE_ENDPOINT \
  --openai-api-key YOUR_GE_API_KEY \
  --ai-model-name YOUR_MODEL_NAME

python src/source-collection/collect_sources_from_urls.py \
  --input output/public_urls.json \
  --output output/public_sources.json \
  --rejected-output output/public_rejected.jsonl \
  --ai-model-name YOUR_MODEL_NAME

export QWEN_API_KEY=YOUR_QWEN_API_KEY
python src/content-scoring/scripts/scoring.py \
  --input-file output/public_sources.json \
  --out-dir output/scored \
  --run-name YOUR_MODEL_NAME

python src/evaluation/compute_metrics.py \
  --run YOUR_MODEL_NAME=output/scored/YOUR_MODEL_NAME.enriched.json \
  --query-metadata data/queries/sourcebench_queries.csv \
  --out-dir output/metrics

The evaluation-side scripts are documented in:

• src/evaluation/README.md

In short:

• validate_official_submission.py: validate official submission schema
• official_submission_backend.py: intake and queue a submission
• official_run.py: run hidden official evaluation
• compute_metrics.py: aggregate final metrics

Open evaluation can be run locally.

Official leaderboard evaluation is intended to be run server-side by the SourceBench team using:

• a hidden holdout split
• the fixed judge model and prompts
• the fixed metrics code
• a standardized submission contract

Relevant files:

• leaderboard/QUERY_SPLIT_POLICY.md
• leaderboard/OFFICIAL_SUBMISSION_CONTRACT.md
• leaderboard/README.md
• leaderboard/examples/

The hidden holdout queries should not be committed here.

Internal official submission artifacts are also excluded. The ignored directory:

• leaderboard/.official_submissions/

is only for server-side submission intake and official evaluation runs.

If you use SourceBench in your research, please cite:

@article{sourcebench2026,
  title={SourceBench: Can AI Answers Reference Quality Web Sources?},
  author={Hexi Jin and Stephen Liu and Yuheng Li and Simran Malik and Yiying Zhang},
  journal={arXiv preprint arXiv:2602.16942},
  year={2026}
}