How Deep is your Learning: The DL-HARD Annotated Deep Learning Dataset

Mackie, Iain; Dalton, Jeffery; Yates, Andrew

Datensatz

DATENSATZ AKTIONENEXPORT

Zur Ablage hinzufügen

Lokale TagsFreigabegeschichteDetailsÜbersicht

Freigegeben

Forschungspapier

How Deep is your Learning: The DL-HARD Annotated Deep Learning Dataset

MPG-Autoren

/persons/resource/persons206666

Yates, Andrew
Databases and Information Systems, MPI for Informatics, Max Planck Society;

Externe Ressourcen

Es sind keine externen Ressourcen hinterlegt

Volltexte (beschränkter Zugriff)

Für Ihren IP-Bereich sind aktuell keine Volltexte freigegeben.

Volltexte (frei zugänglich)

arXiv:2105.07975.pdf
(Preprint), 905KB

Ergänzendes Material (frei zugänglich)

Es sind keine frei zugänglichen Ergänzenden Materialien verfügbar

Zitation

Mackie, I., Dalton, J., & Yates, A. (2021). How Deep is your Learning: The DL-HARD Annotated Deep Learning Dataset. Retrieved from https://arxiv.org/abs/2105.07975.

Zitierlink: https://hdl.handle.net/21.11116/0000-0009-67AB-3

Zusammenfassung

Deep Learning Hard (DL-HARD) is a new annotated dataset designed to more
effectively evaluate neural ranking models on complex topics. It builds on TREC
Deep Learning (DL) topics by extensively annotating them with question intent
categories, answer types, wikified entities, topic categories, and result type
metadata from a commercial web search engine. Based on this data, we introduce
a framework for identifying challenging queries. DL-HARD contains fifty topics
from the official DL 2019/2020 evaluation benchmark, half of which are newly
and independently assessed. We perform experiments using the official submitted
runs to DL on DL-HARD and find substantial differences in metrics and the
ranking of participating systems. Overall, DL-HARD is a new resource that
promotes research on neural ranking methods by focusing on challenging and
complex topics.