Call for Papers

We are pleased to announce the first call for papers for the First Workshop on Dynamic Adversarial Data Collection (DADC), taking place on the 14th July, 2022 and co-located with NAACL 2022 in Seattle, Washington.

Dynamic Adversarial Data Collection (DADC) has been gaining traction in the NLP research community as a promising approach to improving data collection practices, model evaluation and task performance (Yang et al., 2017; Dua et al., 2019; Dinan et al., 2019; Nie et al., 2020; Bartolo et al., 2020; Kiela et al. 2021). DADC allows the dynamic collection of human-written data with models in the loop. Human annotators can be tasked with finding adversarial examples that fool current state-of-the-art models (SOTA) or they can cooperate with assistive models-in-the-loop to find interesting examples.

Recently, various efforts have shown that the DADC process can yield richer training datasets for tasks such as Question Answering (Bartolo et al., 2020; Kaushik et al., 2021), Natural Language Inference (Nie et al., 2020; Sheng et al., 2021), Sentiment Analysis (Potts et al., 2021), and Hate Speech Detection (Vidgen et al., 2021; Kirk et al., 2021). Research on DADC-based approaches has also found that it provides a more realistic evaluation setting (Ma et al., 2021), that the benefits of DADC can be scaled using synthetic data generation (Bartolo et al., 2021), that it yields more diverse training data with fewer artifacts (Wallace et al., 2021), and that generative assistants can improve both collection efficiency and effectiveness (Bartolo et al., 2021).

Topics

Building on this interest in the community, we would like to invite researchers to share their latest work in designing and understanding methods around dynamic adversarial data collection methods. We welcome work on (but not limited to) the following topics:

• Dynamic Benchmarking: Improving or understanding dynamic benchmarking and evaluation, along with investigation into the roles of humans and models for better evaluation, more reliable metrics, etc.
• Data Collection: Dynamic and/or adversarial data collection for the purposes of collecting model training data including the design of creative interfaces, system design, improving annotator engagement, reducing annotator artifacts or bias, investigating the benefits of approaches such as expert annotation or crowdsourcing, and improving data quality and efficiency.
• Active Learning for DADC: Investigating ways to incorporate existing or new active learning algorithms into the DADC pipeline.
• Model Design, Interpretability and Bias: Developing models that are more robust in dynamic and/or adversarial settings, as well as understanding the roles of human and model competition or collaboration for improved model robustness, interpretability, bias mitigation and algorithmic fairness.
• Applications to New Tasks: Extending the ideas behind dynamic adversarial data collection to new tasks, such as generative, multi-lingual and multimodal tasks.
• Analysis and Limitations of Current Approaches: Critiques or discussions of the limitations of traditional and/or dynamic adversarial data collection approaches and model training and evaluation methodologies.

Format and Submissions

We will have an archival track as well as a non-archival track. Archival track submissions either go through a standard double-blind review process, or can be submitted with ARR reviews. The non-archival track seeks recently published work -- it does not need to be anonymized and will not go through the review process. The submission should clearly indicate the original venue and will be accepted if the committee thinks the work will benefit from exposure to the workshop audience. Non-archival papers will not be included in the workshop proceedings. For archival and non-archival papers, we accept short papers (4 pages of content + references) and long papers (8 pages of content + references).

The OpenReview DADC workshop page is now open and accepting paper submissions. Papers should follow the official *ACL Paper Styles and paper formatting guidelines (also see Overleaf template).

To submit your paper, please see the OpenReview DADC workshop page.

Submitting through OpenReview

• Papers requiring peer review: please use the "NAACL 2022 Workshop DADC Submission" button
• Papers with ARR reviews: please use the "NAACL 2022 Workshop DADC Commitment Submission" button
  

DADC Shared Task

We will also be running a Shared Task competition focused on better annotation, better training data and better models. For more information, see our Call for Participation.

Important Dates

All deadlines are 11:59pm AOE (anywhere on earth).

Contact

Please contact the conference organizers at dadc-workshop@googlegroups.com with any questions.