The First Workshop on Dynamic Adversarial Data Collection (DADC) at NAACL 2022 in Seattle, Washington.
HomeThe DADC Shared Task this year will focus on the Extractive Question Answering (QA) task. We have three (3) tracks focusing on better annotators, better training data and better models.
Specific details and a call for participation can be found here. Here's a quick overview:
Participants will submit 100 "official" question answering (QA) examples through the Dynabench platform. The collected dataset will form parts of the evaluation set for Tracks 2 and 3. The objective is to find as many model-fooling examples as possible -- the winning team will be the one with the highest validated model error rate (vMER).
If you think you have what it takes to fool our best AI model, sign up today!
In this data-centric track, participants will submit 10,000 training examples (in SQuAD v1.1 JSON format, see https://huggingface.co/datasets/adversarial_qa#dataset-structure). These examples can be selected from existing datasets, expert-annotated, crowdsourced, or synthetically-generated. The workshop organisers will then train ELECTRA-Large models and evaluate them on the data collected in Track 1. The team with the highest word-overlap F1 score on the test set will be considered the winner.
The workshop organisers have pre-specified a set of evaluation weights for the Dynabench QA leaderboard. Participants can train any models on any data and submit their models directly to Dynabench. The team with the highest dynascore will be considered the winner.
Teams can choose to participate on individual tracks only. To further encourage the formation of diverse teams working on a range of challenges, we will also have an overall DADC Shared Task Winning Team based on performance across all 3 tracks.
The OpenReview DADC workshop page is now open and accepting paper submissions. The paper commitment page for work already having ARR reviews will be coming shortly. Papers should follow the official *ACL Paper Styles and paper formatting guidelines (also see Overleaf template).
For more information, refer to the Call for Papers page.
February 14, 2022 | First Call for Workshop Papers |
April | Submission deadline (papers requiring peer review) |
May | Submission deadline (papers with ARR reviews) |
May | Submission deadline (non-archival papers) |
May | Notification of Acceptance |
May | Camera-ready Papers Due |
July 14, 2022 | Workshop Date at NAACL 2022 |
Team Registration Deadline | |
May 2 - | Official Example Creation Window for Track 1 |
June 3, 2022 | Track 2 Submission Deadline |
June 3, 2022 | Track 3 Submission Deadline |
June 10, 2022 | System Description Paper (Optional) Submission Deadline |
June 17, 2022 | System Description Paper Notification of Acceptance |
June 17, 2022 | Results Announced |
June 24, 2022 | System Description Paper Camera-Ready Deadline |
July 14, 2022 | Workshop Dates & Overall Winning Team Announcement 🏆 |
Please join our mailing list to get important updates. If you have any questions, please contact dadc-workshop@googlegroups.com
.
We would like to thank our generous sponsors for their help and support.