# Research & benchmarks

The tool-use and federated-learning research lineage.

## OKF Authoring Research

- [Open data for the Explore OKF pilot](explore-okf-open-data-test-candidates.md)
  selects a bounded Coventry everyday-services journey and authoritative UK
  sources for testing readable graph labels, useful semantic linking,
  CPSV-AP/SKOS mappings and multi-resolution geography before reviewing
  `okf-uk-living`.

## AI And Tool-Use Research

- [FedAvg — McMahan et al. (2017)](fedavg.md) — Communication-efficient decentralised training by model averaging.
- [Google keyboard FL — Hard et al. (2018)](google-keyboard-fl.md) — On-device federated training for next-word prediction at production scale.
- [Kairouz et al. — Advances and open problems (2019)](kairouz-open-problems.md) — The field's open problems: heterogeneity, privacy, fairness, evaluation.
- [REALM (2020)](realm.md) — Retrieval-augmented pre-training; knowledge can be retrieved, not only stored.
- [MELLODDY (2019–2022)](melloddy.md) — Ten pharma companies trained a shared model without sharing raw data.
- [MRKL (2022)](mrkl.md) — Neuro-symbolic modular routing to expert/knowledge modules.
- [ReAct (2022)](react.md) — Interleaving reasoning traces with actions improves task success.
- [Toolformer (2023)](toolformer.md) — Self-supervised learning of when and how to call tools.
- [API-Bank (2023)](api-bank.md) — Runnable evaluation exposing planning, retrieval and calling gaps.
- [Gorilla & APIBench (2023)](gorilla.md) — Fine-tuned LLM + document retriever; adapts to doc change, cuts hallucination.
- [ToolLLM / ToolBench (2023)](toolllm-toolbench.md) — 16,000+ APIs; ToolLLaMA with a neural API retriever; DFSDT.
- [AgentBench (2023)](agentbench.md) — Multi-environment agent evaluation across 8 interactive environments.
- [Berkeley Function Calling Leaderboard (BFCL)](bfcl.md) — Function-call accuracy: relevance, parallel and sequential calls.
- [APIGen (2024)](apigen.md) — Pipeline for verifiable function-calling data; 3,673 executable APIs.
- [τ-bench (2024)](tau-bench.md) — Tool-Agent-User evaluation with domain policies.
- [ToolSandbox (2024)](toolsandbox.md) — Stateful, conversational evaluation with a user simulator.
- [ToolACE (2024)](toolace.md) — Self-evolution synthesis; 26,507 APIs; dual verification.