Logo

Dynamic Adversarial Data Collection (DADC) Workshop at NAACL 2022

The First Workshop on Dynamic Adversarial Data Collection (DADC) at NAACL 2022 in Seattle, Washington.

Home
Invited Speakers
Shared Task
Call for Papers
Submit Paper
Sign up for notifications

The First Workshop on Dynamic Adversarial Data Collection (DADC)


Latest News:

The DADC Shared Task

The DADC Shared Task this year will focus on the Extractive Question Answering (QA) task. We have three (3) tracks focusing on better annotators, better training data and better models.

Specific details and a call for participation can be found here. Here's a quick overview:

Track 1: Better Annotators

Participants will submit 100 "official" question answering (QA) examples through the Dynabench platform. The collected dataset will form parts of the evaluation set for Tracks 2 and 3. The objective is to find as many model-fooling examples as possible -- the winning team will be the one with the highest validated model error rate (vMER).

If you think you have what it takes to fool our best AI model, sign up today!

Track 2: Better Training Data

In this data-centric track, participants will submit 10,000 training examples (in SQuAD v1.1 JSON format, see https://huggingface.co/datasets/adversarial_qa#dataset-structure). These examples can be selected from existing datasets, expert-annotated, crowdsourced, or synthetically-generated. The workshop organisers will then train ELECTRA-Large models and evaluate them on the data collected in Track 1. The team with the highest word-overlap F1 score on the test set will be considered the winner.

Track 3: Better Models

The workshop organisers have pre-specified a set of evaluation weights for the Dynabench QA leaderboard. Participants can train any models on any data and submit their models directly to Dynabench. The team with the highest dynascore will be considered the winner.

Overall

Teams can choose to participate on individual tracks only. To further encourage the formation of diverse teams working on a range of challenges, we will also have an overall DADC Shared Task Winning Team based on performance across all 3 tracks.


Submission Information

The OpenReview DADC workshop page is now open and accepting paper submissions. The paper commitment page for work already having ARR reviews will be coming shortly. Papers should follow the official *ACL Paper Styles and paper formatting guidelines (also see Overleaf template).

For more information, refer to the Call for Papers page.

Important Dates (Paper Submission)

February 14, 2022First Call for Workshop Papers
April 8 15, 2022Submission deadline (papers requiring peer review)
May 1 6, 2022Submission deadline (papers with ARR reviews)
May 1 6, 2022Submission deadline (non-archival papers)
May 6 16, 2022Notification of Acceptance
May 20 27, 2022Camera-ready Papers Due
July 14, 2022Workshop Date at NAACL 2022

Important Dates (Shared Task)

April 25 May 22, 2022Team Registration Deadline
May 2 - 15 22, 2022Official Example Creation Window for Track 1
June 3, 2022Track 2 Submission Deadline
June 3, 2022Track 3 Submission Deadline
June 10, 2022System Description Paper (Optional) Submission Deadline
June 17, 2022System Description Paper Notification of Acceptance
June 17, 2022Results Announced
June 24, 2022System Description Paper Camera-Ready Deadline
July 14, 2022Workshop Dates & Overall Winning Team Announcement 🏆

Organizers

Core Team

Advisory Committee

Contact

Please join our mailing list to get important updates. If you have any questions, please contact dadc-workshop@googlegroups.com.


Sponsors

We would like to thank our generous sponsors for their help and support.

Platinum Sponsors

Gold Sponsors