A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation: Paper accepted at ACL 2020
Our paper “A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation” was accepted at ACL 2020 (Annual Conference of the Association for Computational Linguistics), which is the major international conference for computational linguistics (ranked tier A*).
The paper is a collaboration between the EU projects Lihlith and INODE .
Title: A Methodology for Creating Question Answering Corpora Using Inverse Data Annotation
Authors: Jan Deriu, Katsiaryna Mlynchyk, Philippe Schläpfer, Alvaro Rodrigo, Dirk von Grünigen, Nicolas Kaiser, Kurt Stockinger, Eneko Agirre and Mark Cieliebak
Abstract: In this paper, we introduce a novel methodology to efficiently construct a corpus for question answering over structured data. For this, we introduce an intermediate representation that is based on the logical query plan in a database, called Operation Trees (OT). This representation allows us to invert the annotation process without loosing flexibility in the types of queries that we generate. Furthermore, it allows for fine-grained alignment of the tokens to the operations.
In our method, we randomly generate OTs from a context free grammar, and annotators just have to write the appropriate question and assign the tokens. We apply the method to create a new corpus OTTA (Operation Trees and Token Assignment), a large semantic parsing corpus for evaluating natural language interfaces to databases. We compare OTTA to Spider and LC-QuaD 2.0 and show that our methodology more than triples the annotation speed while maintaining the complexity of the queries. Finally, we train a state-of-the-art semantic parsing model on our data and show that our corpus is a challenging dataset and that the token alignment can be leveraged to significantly increase the performance.