A Systematic Evaluation of Retrieval-Augmented Generation and Language Models for Space Operations

The rapid expansion of space activities has led to an unprecedented accumulation of technical documentation, operational guidelines, and scientific literature, creating challenges for timely decision-making in space operations. Effective management in space operations requires tools capable of efficiently processing vast and heterogeneous information sources. This paper systematically evaluates the performance of Retrieval-Augmented Generation (RAG) pipelines, combining Large Language Models (LLMs) with information retrieval techniques for extracting and synthesizing actionable knowledge from domain-specific documents. We compare various retrieval strategies, embedding models, and LLM answers to assess their impact on information accuracy, relevance, and reliability. Our results demonstrate that RAG pipelines can significantly enhance knowledge access, reduce uncertainty, and support decision-making in complex space operations.

Access paper here

Bibtex:

@inproceedings{Belo2026a-systematic-evaluation,
    author = "Belo, Ruben Catarino and Guimaraes, Marta and Soares, Claudia",
    title = "A Systematic Evaluation of Retrieval-Augmented Generation and Language Models for Space Operations",
    booktitle = "AI4Space",
    url = "https://openreview.net/forum?id=hLv173CNNk",
    year = "2026",
    eprint = "https://openreview.net/pdf?id=belo|a\\_systematic\\_evaluation\\_of\\_retrievalaugmented\\_generation\\_and\\_language\\_models\\_for\\_space\\_operations",
    organization = "thecvf.com/CVPR/2026/Workshop/AI4Space",
    abstract = "The rapid expansion of space activities has led to an unprecedented accumulation of technical documentation, operational guidelines, and scientific literature, creating challenges for timely decision-making in space operations. Effective management in space operations requires tools capable of efficiently processing vast and heterogeneous information sources. This paper systematically evaluates the performance of Retrieval-Augmented Generation (RAG) pipelines, combining Large Language Models (LLMs) with information retrieval techniques for extracting and synthesizing actionable knowledge from domain-specific documents. We compare various retrieval strategies, embedding models, and LLM answers to assess their impact on information accuracy, relevance, and reliability. Our results demonstrate that RAG pipelines can significantly enhance knowledge access, reduce uncertainty, and support decision-making in complex space operations."
}