An example of three documents and relationships among different document conditions. Models need to reason with unmentioned conditions and condition relationships for optimization.
What is conditional reasoning?
The same real-life questions posed to different individuals may lead to different answers based on their unique situations. For instance, whether a student is eligible for a scholarship depends on eligibility conditions, such as major or degree required.
ConditionalQA (Sun et al. 2021) was proposed to evaluate models' capability of reading a document and answering eligibility questions, considering unmentioned conditions.
What is multi-document conditional reasoning?
ConditionalQA is limited to questions on single documents, neglecting harder cases that may require cross-document reasoning and optimization, for example, "What is the maximum number of scholarships attainable?" Such questions over multiple documents are not only more challenging due to more context to understand, but also because the model has to
@article{chen2024mdcr,
title={MDCR: A Dataset for Multi-Document Conditional Reasoning},
author={Chen, Peter Baile and Zhang, Yi and Liu, Chunwei and Gupta, Sejal and Kim, Yoon and Cafarella, Michael},
journal={arXiv preprint arXiv:2406.11784},
year={2024}
}