ConditionalQA

What is ConditionalQA?

ConditionalQA is a question answering dataset featuring complex questions with conditional answers, i.e. answers are only applicable if certain conditions apply. Questions require compositional logical reasoning and are extremely challenging. This dataset is collected by the team at Carnegie Mellon University.

For more details about ConditionalQA, please refer to our paper:

ConditionalQA (Sun et al. 2021)

Example

Getting started

ConditionalQA is distributed under a CC BY-SA 4.0 License. The training, development, and test (no answer) sets can be downloaded below. Questions are associated with provided documents. The URL is the unique identifier for a document.

Training set

Dev set

Test set (no answer)

Documents

You can evaluate your results with the official evaluation script:

python evaluate.py --pred_file=PRED_FILE --ref_file=REF_FILE

Evaluation script

Sample output

Submission

To evaluate your predictions on the official test sets, please submit your predictions to the Codalab competition.

Codalab competition

Please email your Codalab results to us if you would like your model added to the leaderboard. Please include your model name, your organization, a link to your paper, and your open-sourced codes if applicable. The email address is [username]@cs.cmu.edu where the username is "haitians".

Stay connected!

If you have any questions or comments on ConditionalQA, please email the first author of the paper. If you use ConditionalQA in your research, please cite our paper with the following BibTeX entry.

@misc{sun2021conditionalqa,
  title={ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers}, 
  author={Haitian Sun and William W. Cohen and Ruslan Salakhutdinov},
  year={2021},
  eprint={2110.06884},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

Thank you and have fun!