Fighting Game · Guides & deep dives

V2 Bot Authoring Guide

BotAuthoringGuide_V2_Contract.json

10sections6 minread1table
Markdown source
On this page

This guide covers TODOS.phase-72.72.19.1.15: authoring custom test scenarios for the V2 playtesting bot framework. It defines the scenario contract, review workflow, validation commands, and reporting requirements used by the headless V2BotHarness and the nightly v2-nightly-bot-suite.

Source Of Truth#

Scenario Contract#

Every custom test scenario is a deterministic manifest row reviewed by QA and run through one existing bot profile. The manifest is not free-form prose. It is the source used to build FV2BotHarnessAction scripts, bind FV2BotHarnessPolicyBinding controllers, select automation filters, configure reporting, and route CI failures.

Use stable ids with the bot.scenario.* prefix. Example ids:

  • bot.scenario.market_vendor_loop
  • bot.scenario.combat_arena_smoke
  • bot.scenario.quest_marker_regression
  • bot.scenario.performance_downtown_path

Each scenario must declare its target map, deterministic seed, frame budget, selected bot profile, automation filter, setup preconditions, action or policy inputs, observation schema, assertions, artifacts, and alert policy. The same scenario id must be used in bot result reports and screenshot artifact paths so failures can be compared across runs.

Custom Scenario Fields#

Field Authoring rule
ScenarioId Stable bot.scenario.* id. Do not reuse for a different gameplay purpose.
Owner QA, gameplay, level, economy, or performance owner responsible for triage.
Map Cooked or editor map path. Use one map per scenario unless testing travel.
Seed Deterministic seed. Changing it requires review because baseline output moves.
FrameBudget Maximum automation frames, plus per-frame action budget when applicable.
AutomationFilter Existing filter such as V2.Playtesting.BotHarness.
BotProfile One of the existing bot profiles: exploration, navigation, combat, puzzle, quest, economy, soak, load, RL, performance, or reporting.
Preconditions Spawn state, inventory, quest state, economy balance, data fixtures, and server mode.
ActionScript Ordered FV2BotHarnessAction sequence or policy input fixture.
ObservationSchema Required observation fields, hashes, coverage tags, and telemetry counters.
Assertions Expected pass criteria, disallowed warnings, crash detection, stuck spot detection, and performance budget checks.
Artifacts Transcript, report summary, coverage rows, frame-time table, and screenshot artifacts.
AlertPolicy Whether failures page qa-bot-alerts, performance owners, or release owners.

Keep scenario data compact. Large fixture content should live in data assets or test fixture files referenced by stable path. Scenario manifests should remain reviewable in code review and deterministic in CI.

Bot Selection#

Choose the smallest existing bot profile that can prove the scenario's risk:

  • Use V2.Playtesting.BotHarness for exact scripted player actions or CustomCommand payloads.
  • Use V2.Playtesting.RandomExplorationBot when the goal is crash detection, stuck spot detection, or noisy item and door interaction pressure.
  • Use V2.Playtesting.BotCoverageMapping for area, transition, and nav-linked coverage requirements.
  • Use V2.Playtesting.NavigationStressTestBot for pathfinding, fallback movement, unreachable target, and stuck-recovery regressions.
  • Use V2.Playtesting.CombatBot for target selection, ability rotation, cooldown use, health/resource checks, and combat outcome assertions.
  • Use V2.Playtesting.PuzzleSolvingBot for interactable sequences, puzzle states, and completion assertions.
  • Use V2.Playtesting.QuestCompletionBot for objective marker flow, quest progress events, and completable quest validation.
  • Use V2.Playtesting.EconomyBot for vendor loops, trades, inflation probes, item duplication probes, and economy exploit detection.
  • Use V2.Playtesting.PerformanceProfilingBot for standardized paths, performance-intensive areas, frame-time capture, and regression comparison.
  • Use V2.Playtesting.BotResultReporting only when validating report assembly, screenshot references, or dashboard-ready output.

Do not create a new bot profile for a one-off content case until the existing profiles cannot express the risk with FV2BotHarnessAction, CustomCommand, or policy input fixtures.

Action And Observation Authoring#

Scripted scenarios use ordered FV2BotHarnessAction rows. Each row should state frame, action kind, axis values, target id, payload JSON, duration, and whether the action is valid for headless execution. Use RunBotHarnessScript to verify that the harness accepts the action surface and records a transcript.

Policy scenarios use FV2BotHarnessPolicyBinding. The binding must name the controller kind, observation schema, action schema, and whether the controller can emit any player action. Keep ML or scripted policy fixtures pinned by path and version so reruns produce the same observation hash.

Observations must include the minimum data needed for a deterministic assertion: location or area id, health/resource state, quest objective id, economy balance, inventory state, interactable id, coverage tag, frame-time sample, crash signature, stuck spot id, and final observation hash as applicable.

Assertions And Failure Signals#

Every scenario must contain at least one pass/fail assertion. Use explicit checks instead of manual review notes. Common assertion categories are:

  • No crash signatures and no harness rejections.
  • No stuck spot id after recovery retries.
  • Required areas visited and required transitions crossed.
  • Expected interactable, combat, puzzle, quest, or economy event observed.
  • Economy balance stays within approved source/sink bounds.
  • Performance budget holds for p95 frame time, hitch count, and regression threshold.
  • Final observation hash matches the approved baseline when the scenario is a regression fixture.

When a scenario is expected to produce warnings, declare the allowed warning ids in the manifest. Unexpected warnings must fail CI validation so new regressions are visible.

Artifacts And Reporting#

Custom scenarios must produce enough artifacts for triage without rerunning the editor locally. Required artifacts are the action transcript, result summary, assertion table, final observation hash, and report row. Add coverage rows, frame-time samples, economy flow tables, or quest progress rows when the bot profile produces them.

Screenshot artifacts are required for crash, stuck spot, coverage gap, performance anomaly, puzzle failure, and quest failure reports. Route those references through GenerateBotResultReport so BotResultReporting can aggregate areas visited, crashes found, stuck spots, performance anomalies, and screenshots in the same report contract used by nightly CI.

Authoring Workflow#

  1. Pick the risk being tested and choose the smallest existing bot profile that can prove it.
  2. Create a stable ScenarioId, owner, map, deterministic seed, frame budget, automation filter, and alert policy.
  3. Author preconditions for spawn state, data fixtures, server mode, inventory, quest state, economy state, or performance baseline.
  4. Write the ActionScript as FV2BotHarnessAction rows, or bind a scripted or ML controller through FV2BotHarnessPolicyBinding.
  5. Define the ObservationSchema and baseline hashes required for deterministic assertions.
  6. Add explicit assertions for crash detection, stuck spot detection, gameplay outcome, coverage, economy, and performance budget expectations.
  7. Define artifacts, including transcript, report summary, screenshot artifacts, coverage rows, and frame-time data when relevant.
  8. Run the scenario with RunBotHarnessScript or the selected bot helper and generate a report with GenerateBotResultReport.
  9. Add the scenario to the nightly bot suite only after the targeted checker, result reporting checker, bot-run CI checker, and doc validation pass.
  10. Attach command output and artifact paths to review.

Review Checklist#

Before approving a custom test scenario, confirm:

  • ScenarioId is stable and uses the bot.scenario.* namespace.
  • Owner, map, deterministic seed, frame budget, automation filter, and bot profile are explicit.
  • Preconditions do not depend on editor-only state unless the scenario is editor-only by design.
  • Action rows are valid FV2BotHarnessAction data and do not exceed per-frame budgets.
  • Policy bindings declare their observation schema, action schema, controller kind, and versioned fixture paths.
  • Assertions fail on new crashes, harness rejections, stuck spots, missing coverage, gameplay outcome regressions, economy exploits, or performance budget regressions.
  • Allowed warnings are named explicitly; all other warnings fail validation.
  • Screenshot artifacts are present for crash, stuck, coverage, performance, puzzle, and quest findings.
  • Bot result reporting includes the scenario id in report rows and artifact paths.
  • The scenario is covered by CI validation and the nightly v2-nightly-bot-suite before it is treated as a release signal.

Validation Commands#

Run these commands after changing bot authoring docs, scenario manifests, bot contracts, result reporting, CI wiring, or Horde gates:

bash
python3 V2/ue/Tools/check-v2-bot-authoring-guide.py
python3 V2/ue/Tools/check-v2-bot-run-ci.py
python3 V2/ue/Tools/check-v2-bot-result-reporting.py
python3 V2/ue/Tools/check-v2-bot-harness.py
python3 V2/ue/Tools/check-v2-random-exploration-bot.py
python3 V2/ue/Tools/check-v2-bot-coverage-mapping.py
python3 V2/ue/Tools/check-v2-navigation-stress-test-bot.py
python3 V2/ue/Tools/check-v2-combat-bot.py
python3 V2/ue/Tools/check-v2-puzzle-solving-bot.py
python3 V2/ue/Tools/check-v2-soak-test-bot.py
python3 V2/ue/Tools/check-v2-load-test-bot-army.py
python3 V2/ue/Tools/check-v2-rl-trained-exploration-agent.py
python3 V2/ue/Tools/check-v2-quest-completion-bot.py
python3 V2/ue/Tools/check-v2-economy-bot.py
python3 V2/ue/Tools/check-v2-performance-profiling-bot.py
python3 V2/ue/Tools/check-v2-tests-module.py
python3 V2/ue/Tools/check-v2-ci-workflow.py
python3 -m json.tool V2/ue/Content/V2/QA/BotAuthoringGuide_V2_Contract.json
python3 -m json.tool V2/ue/Build/Horde/v2-buildgraph-job.json
python3 V2/tools/validate-v2-docs.py
pnpm exec prettier --check V2/docs/qa/bot-authoring-guide.md V2/ue/Content/V2/QA/BotAuthoringGuide_V2_Contract.json V2/ue/Build/Horde/v2-buildgraph-job.json