AI agent trained on its own mistakes: AutoSynthData boosts performance by 35%

Learning from failures. AutoSynthData turns the target model's errors and a stronger model's successful solutions into skill cards. A generator uses those cards to create new training tasks. Each contains:

- An instruction describing the working environment. - A user request. - A check of the result.

The team selected tasks that the target model solved in at most 1 out of 3 attempts and the stronger model solved in at least 2 out of 3. The generator received only skill cards, without the original evaluation prompts or solutions. The new tasks used different requests, environment states and solution paths.

For the ITSM dataset, the team generated 1,994 examples in 66 hours with DeepSeek-V4.1-Flash as the teacher. These were used to fine-tune Gemma-4-26B-A4B-it.

For the Hybrid dataset, AutoSynthData created 2,000 examples in about 18 hours with Qwen3.8-27B as the teacher. After fine-tuning the same Gemma model, its first-attempt success rate rose by 7.2 percentage points, or 35% relative to the baseline. The team achieved its best result after the fifth pass through the training data.

The team also tested the generated result checks themselves. The share of successful checks rose from 63.01% to 68.55%.

Environment for replication. EnterpriseOps-Gym is available on GitHub in the ServiceNow/EnterpriseOps-Gym repository. Running it requires Python 3.11+, uv and Docker. The README describes how to set up the environment:

1. After cloning, run `uv sync --extra all`. 2. Copy `conf.example/` to `conf/` and configure the model's API key. 3. Unpack `gym_dbs.zip` and start the Docker MCP servers for the required domains.

The README includes commands for starting the servers and a separate evaluation example for Hybrid.

The original EnterpriseOps-Gym tasks are available via `load_dataset("ServiceNow-AI/EnterpriseOps-Gym", "oracle", split="teams")`.