1. Registration
- Register your team through the registration form. One registration per team, listing the team name, all members, affiliations and a contact e-mail.
- Registration stays open until the test submissions close. Registering early means you receive the data-release announcement, the Slack invitation and any schedule changes.
- Registration is also required to create your Codabench account for the competition.
2. Teams
- Anyone can participate, individually or as a team: students, academics, industry and independent researchers.
- Team names appear on the public leaderboard and in the overview paper. Please keep them appropriate.
- Organizers and their close collaborators may submit runs for reference, but these are marked as such and excluded from the official ranking.
3. Data and terms of use
- The training and development sets (with labels) and the test set (without labels) are released from the official GitHub repository on the dates of the timeline.
- Data licence: TBA. By downloading the data you agree to use it for research purposes within the terms of that licence and to cite the task overview paper in any publication that uses it.
- Requesting the data commits your team to submit at least one official run. Teams that download the data and silently withdraw may be excluded from future editions.
- The test set must not be annotated manually, used for training or model selection, or redistributed with labels. Gold labels for the test set are released after the evaluation phase.
4. Track and allowed resources
| Single track · Open | |
|---|---|
| Open-source or open-weight pre-trained models | Allowed |
| Proprietary models or APIs | Not allowed |
| Fine-tuning on the official training data | Allowed |
| Additional training data (synthetic or collected) | Allowed, must be documented |
| Retrieval over external documents at inference time (legal texts, decrees, web) | Allowed |
| Tool use, agents, code execution at inference time | Allowed |
| Ensembles, self-consistency, chain-of-thought | Allowed |
| Human intervention on test predictions | Not allowed |
External documents and tools may be used at inference time, but human intervention on test predictions is not allowed. Document models, data, retrieval sources and tools in your system description paper.
5. Phases and runs
| Phase | Data | Runs | Leaderboard |
|---|---|---|---|
| Development | Development set (labels public) | Unlimited | Public, for sanity checks |
| Evaluation | Test set (labels withheld) | Up to 3 per team | Hidden until results are announced |
| Post-evaluation | Test set | Unlimited | Public, marked as unofficial |
- Your latest valid run submitted during the evaluation phase, before the deadline, will count as your official result.
- Runs uploaded outside the evaluation window are not official and cannot be reported as such in papers.
- All deadlines are 23:59 anywhere on Earth (UTC-12).
6. Submission format
Upload a zip archive to Codabench. The archive must contain a single file named predictions.csv at its root, encoded in UTF-8, with a header row and one line per test question:
id,answer
TQA-00001,A
TQA-00002,C
TQA-00003,Bidis the question identifier from the test file; every test id must appear exactly once.answeris exactly one ofA,BorC. Missing, empty or invalid values are scored as wrong.- Validate the file locally with the script in the repository before uploading.
7. Leaderboard and ranking
- Systems are ranked by accuracy on the test set. Ties are broken by F1 score over the three labels.
- Per-variety accuracy (MSA and Darija) and the split between driving-knowledge and article-grounded questions are reported for every official run.
- Results are announced on the timeline date and summarised in the task overview paper, which also includes a short description of every system that submitted a paper.
8. Papers
- Every team with an official run is invited to submit a system description paper (4 pages plus unlimited references and appendices) through OpenReview. Details are on the paper submission page.
- Papers must cite the TariQA overview paper and report the official run, clearly separated from any post-evaluation experiments.
9. Frequently asked questions
Do I need to register before downloading the data?
Yes. Registration takes two minutes. It lets us keep teams informed and keep the leaderboard tidy.
Can I use a proprietary model or API?
No. Only open-source or open-weight models may be used. Document the model and version in your paper.
May I translate the questions or generate extra training data?
Yes. Translation, paraphrasing and synthetic data generation are allowed and must be described in the paper. Generating data from the test questions is not allowed.
Is the test set labelled by language variety?
The test file contains only the question and its options. Variety labels are released after the evaluation phase, together with the gold answers, so that teams can analyse their results.
How many official runs do I get?
Up to three per team during the evaluation phase. Development-phase runs are unlimited.
Do I need an OpenReview account?
Yes, every author needs one to submit and to review. System and overview papers are not anonymous, and teams may post their papers on arXiv.
Should I release my code?
It is strongly encouraged. Add the URL to your paper; replicability is one of the review criteria.
Are there ethical considerations I should discuss?
Yes. Describe potential misuse and limitations of your system, in particular the risk of giving wrong legal advice to drivers, in a short paragraph of the paper.
10. Contact
- Task questions: the Slack workspace, the mailing list, or .
- Workshop logistics: the AbjadNLP Workshop organizers.
- OpenReview technical problems: OpenReview support.
TariQA follows the ACL Anti-Harassment Policy.