|
WebMall - A multi-shop benchmark for evaluating web agents
Peeters, Ralph
;
Steiner, Aaron
;
Schwarz, Luca
;
Caspary, Julian Yuya
;
Bizer, Christian
|
DOI:
|
https://doi.org/10.1145/3805712.3808592
|
|
URL:
|
https://dl.acm.org/doi/10.1145/3805712.3808592
|
|
URN:
|
urn:nbn:de:bsz:180-madoc-727626
|
|
Document Type:
|
Conference or workshop publication
|
|
Year of publication:
|
2026
|
|
Book title:
|
Proceedings of the 49th International ACM SIGIR Conference on Research and Development in Information Retrieval : SIGIR '26, July 20-24, 2026, Melbourne, Australia
|
|
Page range:
|
3342-3348
|
|
Conference title:
|
SIGIR 2026, 49th International ACM SIGIR Conference on Research and Development in Information Retrieval
|
|
Location of the conference venue:
|
Melbourne, Australia
|
|
Date of the conference:
|
20-24.07.2026
|
|
Publisher:
|
Moffat, Alistair
;
Scholer, Falk
|
|
Place of publication:
|
New York, NY
|
|
Publishing house:
|
Association for Computing Machinery
|
|
ISBN:
|
979-8-4007-2599-9
|
|
Related URLs:
|
|
|
Publication language:
|
English
|
|
Institution:
|
School of Business Informatics and Mathematics > Information Systems V: Web-based Systems (Bizer 2012-)
|
|
Pre-existing license:
|
 Creative Commons Attribution 4.0 International (CC BY 4.0)
|
|
Subject:
|
004 Computer science, internet
|
|
Keywords (English):
|
web agents , product search , e-commerce , benchmarking
|
|
Abstract:
|
LLM-based web agents have the potential to automate long-running web tasks, such as searching for products in multiple e-shops and subsequently ordering the cheapest products that meet the user's needs. Benchmarks for evaluating web agents either require agents to perform tasks online using the live Web or offline using simulated environments, the latter allowing for the exact reproduction of the experimental setup. While DeepShop and ShoppingComp provide online benchmarks that require agents to perform challenging shopping tasks, existing offline benchmarks such as WebShop, WebArena, and Mind2Web cover only comparatively simple e-commerce tasks performed against a single shop containing product data from a single source. What is missing is an e-commerce benchmark that simulates multiple shops containing heterogeneous product data and requires agents to perform complex retrieval tasks. We fill this gap by introducing WebMall, the first offline multi-shop benchmark for evaluating web agents on challenging comparison shopping tasks. WebMall consists of four simulated shops populated with product data extracted from the Common Crawl. The WebMall tasks range from specific product searches and price comparisons to advanced searches for complementary or substitute products, as well as checkout processes. We validate WebMall using eight agents that differ in observation space, availability of short-term memory, and the employed LLM. The validation highlights the difficulty of the benchmark, with the best-performing agents achieving task completion rates below 65% in the task categories cheapest product search and vague product search.
|
 | Dieser Eintrag ist Teil der Universitätsbibliographie. |
 | Das Dokument wird vom Publikationsserver der Universitätsbibliothek Mannheim bereitgestellt. |
 Search Authors in
BASE:
Peeters, Ralph
;
Steiner, Aaron
;
Schwarz, Luca
;
Caspary, Julian Yuya
;
Bizer, Christian
Google Scholar:
Peeters, Ralph
;
Steiner, Aaron
;
Schwarz, Luca
;
Caspary, Julian Yuya
;
Bizer, Christian
ORCID:
Peeters, Ralph  ORCID: 0000-0003-3174-2616 ; Steiner, Aaron ORCID: 0009-0006-6946-7057 ; Schwarz, Luca ; Caspary, Julian Yuya ; Bizer, Christian ORCID: 0000-0003-2367-0237
You have found an error? Please let us know about your desired correction here: E-Mail
Actions (login required)
 |
Show item |
|
