Data Challenge Track
The International Conference on Performance Engineering (ICPE) is hosting its sixth edition of the Data Challenge track. We call upon everyone interested to apply approaches and analyses to a common selection of performance datasets. The Data Challenge track is open-ended: participants can choose the research questions they find most interesting. The proposed approaches/analyses and their findings are discussed in short papers and presented at the main conference.
In this track, we will provide four different performance datasets. Participants are invited to come up with new research questions and approaches for performance analysis. For their papers, participants must choose one or more datasets from a predefined list derived from prior academic/industry research. Participants are expected to use this year’s datasets to answer their research questions, and report their findings in a four-page challenge paper. If the paper is accepted, participants will be invited to present the results at ICPE 2027 in Gothenburg, Sweden. Details on the datasets are provided below.
Datasets
This year’s ICPE data challenge is based on four datasets from both academic and industry studies. Each dataset includes performance measurements gathered from either industrial or open-source systems, with each dataset having its own unique data format and content as described in their respective repositories.
The first dataset is provided by the 2026 arxiv.org paper “Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale”. This paper provides a dataset consisting of GitHub Copilot Production-Scale Workload Traces that provide a production-scale characterization of multi-step agentic tool execution interleaved with LLM inferencing requests. Agentic coding sessions consist of sparse user-initiated turns resulting in an autonomous agent loop of LLM calls coupled with tool execution. This data set characterizes a subset of the data from the paper. The dataset characterizes 301K agentic sessions from 1.2M user turns, with 9.3M LLM calls and 8.7M Tool calls. By capturing the temporal, computational, and resource utilization characteristics of production agentic workloads, this dataset provides researchers with empirical metadata to develop realistic workload models for evaluating the performance, scalability, and resource efficiency of agentic AI infrastructure.
- Paper: https://arxiv.org/pdf/2608.00101
- Repository: https://github.com/Azure/AzurePublicDataset/blob/master/GitHubCopilotCodingAgentDataset2026.md
The second dataset is introduced in the 2026 arxiv paper “Rethinking AI Cloud Infrastructure for Agentic Serving Systems with the Aries Experimentation Framework”. Hyscale-lab provides a 24-hour workload trace for agentic LLM workflows where an agent was used as a wrapper/harness around an LLM to instrument LLM workflows. The trace provides observation of the LLM/agent task trajectory which shows repeated LLM calls, harness (thin-client) decisions, stateful tool execution, task outcomes, and resource telemetry. The trace has been extracted from an anonymized commercial platform. This dataset provides a second example of dynamic LLM (GPU) and agentic (CPU) workloads characteristic of those that have begun to dominate data centers and cloud platforms. The dataset captures LLM/agentic interactions which would allow someone to reproduce the agent side (CPU-only), the LLM side (GPU only), or the whole stack (GPU + CPUs) for performance/profiling experiments.
- Paper (accepted to SoCC 2026): https://arxiv.org/pdf/2607.29069
- Repository: https://github.com/hyscale-lab/ARIES/blob/main/docs/ant-group-agent-LLM-trace.md
The third dataset accompanies the arxiv paper “Exploring Statistical Change Point Detection Techniques for Performance Anomaly Detection at Mozilla”. It contains 174 software performance time series manually annotated by 11 Mozilla performance engineers to identify performance change points. These expert annotations provide ground-truth reference data for benchmarking statistical change-point detection techniques and evaluating the accuracy of automated performance anomaly detection systems.
- Paper: https://arxiv.org/abs/2606.18377
- Repository: https://zenodo.org/records/20382322
The fourth dataset is the New Kids dataset, resulting from the ACM TOIT paper “New Kids: An Architecture and Performance Investigation of Second-Generation Serverless Platforms” and the follow-up arxiv preprint “Tremulant: Latency in Serverless Platforms beyond the 99th Percentile”. The dataset includes extensive benchmarking results with serverless platforms, including AWS Lambda and Google Cloud Run Functions as well as a number of underexplored second generation FaaS platforms. See the repository link for a more detailed dataset description.
- Papers: https://arxiv.org/abs/2604.15916 and https://arxiv.org/abs/2610.18532
- Repository: https://github.com/3s-rg-codes/faas-bench-explore
Challenge
High-level possible ideas for participants include but are not limited to (these are only examples):
- Create and/or apply visualization techniques to navigate the extensive data generated by systems.
- Beschastnikh et al., 2020: https://doi.org/10.1145/3375633
- Silva et al., 2021: https://doi.org/10.1109/IV53921.2021.00028
- Anand et al., 2020: https://doi.org/10.48550/arXiv.2010.13681
- Develop automated techniques to identify patterns associated with performance degradations.
- Wang et al., 2022: https://dl.acm.org/doi/10.1145/3545008.3545026
- Traini and Cortellessa, 2023: https://doi.org/10.1109/TSE.2023.3266041
- Bansal et al., 2020: https://doi.org/10.1145/3377813.3381353
- Evaluation of previous/novel root cause analysis techniques.
- Noferesti et al., 2024: https://doi.org/10.1016/j.jss.2024.112117
- Mariani et al., 2018: https://doi.org/10.1109/ICST.2018.00034
- Ma et al., 2020: https://doi.org/10.1145/3366423.3380111
- Model performance of systems using machine learning algorithms.
- Xiong et al., 2013: https://doi.org/10.1145/2479871.2479909
- Liao et al., 2020: https://doi.org/10.1007/s10664-020-09866-z
- Replicate prior study/approach on a selected dataset or correlate one of the datasets with other open datasets.
Submission
A data challenge paper should outline the findings of your research. Starting with an introduction to the problem tackled and its relevance in the field. Detail the datasets utilized, the methods and tools applied, and the results achieved. Discuss the implications of the study findings, and highlight the paper contributions and their importance.
To maintain clarity and consistency in research submissions, authors are required to specify the dataset (or portion thereof) utilized when detailing methodologies or presenting findings. Additionally, authors are reminded to be precise in their references to datasets.
We highly encourage the solution’s source code to be included with the submission (e.g., in a permanent repository such as GitHub or similar), but this is not mandatory for acceptance of a data challenge paper.
The page limit for challenge papers is 4 pages (including all figures and tables) + 1 page for references. Data challenge papers will be published in the companion to the ICPE 2027 proceedings. All Data Challenge papers will be reviewed by the program committee members and potentially additional reviewers. Note that submissions to this track are double-anonymous: for details, see the Double Anonymized FAQ page. The best Data Challenge paper will be awarded by the track chairs and the program committee members.
Submissions are made via HotCRP .
The submission deadline can be found here .
Data Challenge Chairs
- David Bermbach, TU Berlin, Germany
- Wes Lloyd, University of Washington Tacoma, USA