Eve library

Isis User Telemetry and Qualitative Feedback Report Runbook

Runbook

Revision 1

On this page

Last Updated: 2026-02-24
Scope: WS18-008 user telemetry + qualitative feedback collection for ergonomic improvements.

Purpose#

Generate a repeatable evidence artifact that combines UI telemetry, benchmark signals, on-call incident patterns, and qualitative user feedback into a prioritized ergonomic improvement plan.

Artifacts#

  • docs/domains/isis/remediation-v2/ISIS_V2_USER_TELEMETRY_FEEDBACK_REPORT.json
  • docs/domains/isis/remediation-v2/ISIS_V2_USER_TELEMETRY_FEEDBACK_REPORT.md

Inputs#

  • UX benchmark baseline:
    • docs/domains/isis/remediation-v2/ISIS_V2_UX_INTERACTION_BENCHMARK.json
  • benchmark dashboard telemetry:
    • docs/domains/isis/remediation-v2/ISIS_V2_BENCHMARK_DASHBOARD.json
  • creative on-call triage reports:
    • tmp/isis-oncall/**/creative-pipeline-oncall-triage-*.json
  • UI telemetry event stream (NDJSON, configurable):
    • default: scripts/isis/fixtures/ws18-008-ui-telemetry.ndjson
  • qualitative feedback stream (NDJSON, configurable):
    • default: scripts/isis/fixtures/ws18-008-qualitative-feedback.ndjson

Generator#

bash
node scripts/isis/generate-v2-user-telemetry-feedback-report.mjs

Options:

  • --window-start=YYYY-MM-DD
  • --window-end=YYYY-MM-DD
  • --window-days=<int> (default 14)
  • --telemetry-file=<path-to-ndjson>
  • --feedback-file=<path-to-ndjson>
  • --strict-sources=true|false (fail if source files are missing)

Validator#

bash
node scripts/isis/validate-v2-user-telemetry-feedback-report.mjs

NDJSON Input Contract#

UI telemetry sample line:

json
{
  "category": "ui.interaction",
  "interaction": "click",
  "route": "/jobs",
  "target": { "label": "Submit Job" },
  "timestamp": "2026-02-24T18:03:01Z"
}

Qualitative feedback sample line:

json
{
  "feedbackId": "FB-001",
  "persona": "creator",
  "sentiment": "negative",
  "ergonomicsScore": 2.4,
  "themes": ["form-complexity"],
  "comment": "Parameter forms need preset grouping.",
  "submittedAt": "2026-02-24T16:15:00Z"
}

Operational Cadence#

  1. Generate report weekly during stabilization and pre-quarter planning.
  2. Validate artifact integrity before governance review.
  3. Review top P0/P1 improvement items with @oshun/isis-team and @oshun/platform.
  4. Convert selected improvements to tracked delivery tickets.
  5. Attach JSON+Markdown report to release evidence package.
  • docs/domains/isis/remediation/user-journey-certification/UX-010-ui-interaction-api-telemetry-parity.md
  • docs/domains/isis/remediation-v2/ISIS_V2_UX_BENCHMARK_STUDIES_AND_ERGONOMICS_TARGETS.md
  • docs/domains/isis/runbooks/post-launch-stabilization-sprint.md