Round-Trip Factual Consistency

A Round-Trip Factual Consistency is a natural language generation evaluation metric that measures whether factual information remains accurate and unchanged after transformations (e.g., summarization, translation, paraphrasing) and reconstruction back to its original form.

• AKA: Bidirectional Factual Consistency, Round-Trip Correctness (RTC), Factual Round-Trip Verification.
• Context:
  • It can be used to evaluate automated writing systems (that supports natural language generation).
  • It can assess the factual alignment between input data and generated content.
  • It can evaluate the consistency of information when content is transformed back to its original form.
  • It can measure the reliability of automated writing systems in maintaining factual accuracy.
  • It can determine the robustness of content generation models against factual distortions.
  • It can evaluate text summarization systems by comparing reconstructed summaries to source documents.
  • It can validate knowledge graph updates by ensuring facts remain consistent after entity resolution or relation extraction.
  • It can test machine translation systems for semantic preservation across language pairs.
  • It can detect hallucinations in LLM-generated content through iterative encoding-decoding cycles.
  • It can use cross-document alignment to verify factual coherence in multi-step workflows.
  • It can range from being a simple consistency check to being a comprehensive evaluation metric, depending on the complexity of the writing task.
  • ...
• Example(s):
  • AlignScore (Zao et al., 2023), which uses a unified alignment function to evaluate factual consistency.
  • QAFactEval (Durmus et al., 2022), which assesses factual consistency in text summarization via question generation and answering.
  • TRUE (Honovich et al., 2022), which re-evaluates factual consistency evaluation metrics for text generation.
  • Summarization Consistency Check: Reconstructing a summary back to its original article to verify retained facts.
  • Translation Round-Trip Test: Translating text to another language and back to assess fidelity (e.g., EN→FR→EN).
  • Paraphrase Validation: Comparing paraphrased text to original content for factual equivalence.
  • Knowledge Graph Repair: Detecting inconsistencies after merging datasets from disparate sources.
  • ...
• Counter-Example(s):
  • Basic Grammar Checks, which ignore semantic/content preservation.
  • Surface-Level Consistency Metrics, which lack deep semantic analysis.
  • Surface-Level Similarity Metrics (e.g., BLEU, ROUGE), which measure lexical overlap but not factual accuracy.
  • Lexical Overlap Measures, which serve different purposes.
  • Syntactic Consistency Checks, which use different approaches.
  • Single-Pass Fact Checking, which lacks iterative reconstruction.
  • ...
• See: Contextual Relevance Score, Cycle Consistency Check, Information Preservation Metric, Natural Language Generation Evaluation Metric, Automated Writing Evaluation, Factual Verification, Hallucination Detection, Semantic Preservation, Knowledge Graph Consistency, Text Reconstruction Task.

---

---

References

2025

• (Malaviya et al., 2025) ⇒ Malaviya, Agrawal, et al. (2025). "Dolomites: Domain-Specific Long-Form Methodical Tasks". In: Transactions of the Association for Computational Linguistics.
  • QUOTE: Round-Trip Factual Consistency measures the extent to which statements in the model output are consistent with statements in the reference output. We compute 1) forward entailment considering a reference section as the premise and the corresponding model section as the hypothesis and 2) reverse entailment considering a model output section as the premise and the corresponding reference section as the hypothesis. Scores are aggregated over all sections and examples. These metrics loosely capture the notions of precision and recall, we also report the harmonic mean of the two.

2024

• (Gu et al., 2024) ⇒ Gu, Choi, et al. (2024). "Unsupervised Evaluation of Code LLMs with Round-Trip Correctness". In: arXiv.
  • QUOTE: Round-trip correctness (RTC) allows us to measure an LLM's performance over a wide-range of real-life software domains — without human-provided annotations — and complements existing narrow-domain benchmarks. Intuitively, for a "good" forward and backward model we expect ˆx = M −1(M(x)) to be semantically equivalent to x. For example, we can describe code with natural language in the forward pass and then generate back the code from the sampled natural language descriptions in the backward pass.

2023

• (Zao et al., 2023) ⇒ Zhao, W., Liu, J., Wei, Z., and Huang, X. (2023). "AlignScore: Evaluating Factual Consistency with a Unified Alignment Function". In: arXiv.
  • QUOTE: AlignScore applies a unified alignment function to assess factual consistency across various text generation tasks.

    It achieves robust performance by aligning generated text with reference data while mitigating hallucination in automated writing systems.

2022a

• (Durmus et al., 2022) ⇒ Durmus, E., He, H., and Liang, P. (2022). "QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization". In: ACL Anthology.
  • QUOTE: QAFactEval is a factual consistency evaluation metric that utilizes question generation and question answering to assess the faithfulness of text summarization.

    Unlike prior lexical overlap approaches, QAFactEval emphasizes semantic consistency and fact verification.

2022b

• (Honovich et al., 2022) ⇒ Honovich, O., Scialom, T., Guu, K., Shtok, A., and Sen, P. (2022). "TRUE: Re-evaluating Factual Consistency Evaluation". In: ACL Anthology.
  • QUOTE: TRUE is a framework designed to analyze existing factual consistency metrics and their ability to detect factual errors in text generation.

    It highlights critical shortcomings in conventional evaluation methods, urging for more reliable factual verification systems.

2021

• (Honovich et al., 2021) ⇒ Honovich, O., Levy, O., Ravfogel, S., and Goldberg, Y. (2021). "Q²: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering". In: arXiv.
  • QUOTE: Q² evaluates factual consistency in knowledge-grounded dialogues by utilizing an automated question generation and answering framework.

    It effectively detects hallucinations in conversational AI responses, ensuring factual accuracy.

2021b

• (Dolomites, 2021) ⇒ Mitchell, M., Zellers, R., He, D., and Chang, K. (2021). "Dolomites: Domain-Specific Long-Form Methodical Tasks". In: Transactions of the Association for Computational Linguistics.
  • QUOTE: Dolomites Benchmark provides an evaluation framework for domain-specific writing tasks, emphasizing the importance of factual consistency metrics in long-form writing.

    It introduces datasets for measuring the accuracy of automated content creation systems in preserving domain-specific knowledge.

2020

• (FactEval, 2020) ⇒ Gabriel, S., Cohen, S., Yu, S., Cattan, A., and Eban, E. (2020). "FactEval: Evaluating the Factual Consistency of Abstractive Text Summarization". In: arXiv.
  • QUOTE: FactEval is an evaluation framework designed to analyze the factual consistency of abstractive summarization models.

    It introduces multi-layer evaluation, capturing semantic similarity and factual entailment in generated summaries.

---