Uncertain Natural Language Inference

UNLI

UNLI (Uncertain natural language inference) is a refinement of Natural Language Inference (NLI) that shifts away from categorical labels, targeting instead the direct prediction of subjective probability assessments to model subtle distinctions on the likelihood of a hypothesis conditioned on a premise.

Read our paper here.

Data

A subset of the popular SNLI dataset is labeled under a probabilistic scale to create our proof-of-concept UNLI dataset, u-SNLI, which can be downloaded here. The dataset comprises of 3 files, {train|dev|test}.csv, where each file contains the following columns: ID, Premise, Hypothesis, NLI label, UNLI label.

Example annotations:

• Premise: A man is singing into a microphone.
• Hypotheses:
  • [0.946] A man performs a song.
  • [0.840] A man is performing on stage.
  • [0.152] A male performer is singing a special and meaningful song.
  • [0.144] A man performing in a bar.
  • [0.062] A man is singing the national anthem at a crowded stadium.

The code to reproduce the results reported in the paper can be found here.

Citation

@inproceedings{UNLI,
  title={Uncertain Natural Language Inference},
  author={Tongfei Chen and
           Zhengping Jiang and
           Adam Poliak and
           Keisuke Sakaguchi and
           Benjamin {Van Durme}},
  booktitle={Proceedings of ACL},
  year={2020}
}