The First Workshop on Dynamic Adversarial Data Collection (DADC)

Latest News:

• May 2, 2022: The Official Example Creation Window for Track 1 is now live on the Dynabench platform. To participate, register your details here.
• April 25, 2022: The DADC Shared Task registration deadline has been extended to May 15, 2022. Sign up today!
• April 22, 2022: We are delighted to announce that the awesome folks at MLCommons will be sponsoring the DADC workshop! Their support will allow us to award prizes for the shared task winners as well as a best paper award 🏆!
• April 7, 2022: Following extensions by other NAACL workshops, we have also decided to extend the submission deadline for papers from April 8 to April 15. We look forward to your submissions! Submit at https://openreview.net/group?id=aclweb.org/NAACL/2022/Workshop/DADC.
• March 29, 2022: The OpenReview DADC workshop page is now open and accepting paper submissions!
• March 23, 2022: The DADC Shared Task Call for Participation is now live! To register your team's interest, fill out the registration form.
• February 14, 2022: See our Call for Papers here!

The DADC Shared Task

The DADC Shared Task this year will focus on the Extractive Question Answering (QA) task. We have three (3) tracks focusing on better annotators, better training data and better models.

Specific details and a call for participation can be found here. Here's a quick overview:

Track 1: Better Annotators

Participants will submit 100 "official" question answering (QA) examples through the Dynabench platform. The collected dataset will form parts of the evaluation set for Tracks 2 and 3. The objective is to find as many model-fooling examples as possible -- the winning team will be the one with the highest validated model error rate (vMER).

If you think you have what it takes to fool our best AI model, sign up today!

Track 2: Better Training Data

In this data-centric track, participants will submit 10,000 training examples (in SQuAD v1.1 JSON format, see https://huggingface.co/datasets/adversarial_qa#dataset-structure). These examples can be selected from existing datasets, expert-annotated, crowdsourced, or synthetically-generated. The workshop organisers will then train ELECTRA-Large models and evaluate them on the data collected in Track 1. The team with the highest word-overlap F1 score on the test set will be considered the winner.

Track 3: Better Models

The workshop organisers have pre-specified a set of evaluation weights for the Dynabench QA leaderboard. Participants can train any models on any data and submit their models directly to Dynabench. The team with the highest dynascore will be considered the winner.

Overall

Teams can choose to participate on individual tracks only. To further encourage the formation of diverse teams working on a range of challenges, we will also have an overall DADC Shared Task Winning Team based on performance across all 3 tracks.

Submission Information

The OpenReview DADC workshop page is now open and accepting paper submissions. The paper commitment page for work already having ARR reviews will be coming shortly. Papers should follow the official *ACL Paper Styles and paper formatting guidelines (also see Overleaf template).

For more information, refer to the Call for Papers page.

Important Dates (Paper Submission)

Important Dates (Shared Task)

Organizers

Core Team

• Max Bartolo
• Hannah Rose Kirk
• Pedro Rodriguez
• Katerina Margatina
• Tristan Thrush
• Mimansa Jaiswal

Advisory Committee

• Pontus Stenetorp
• Robin Jia
• Adina Williams
• Douwe Kiela

Contact

Please join our mailing list to get important updates. If you have any questions, please contact dadc-workshop@googlegroups.com.

Sponsors

We would like to thank our generous sponsors for their help and support.

Platinum Sponsors

Gold Sponsors