Testing Playbooks

Test suites let you define expected inputs and assertions for a playbook and run them on demand. Catch regressions before they reach production.

---

What is a test suite?

A test suite is a named collection of test cases. Each case provides:

• Input: the form data or inputs to pass to the playbook
• Assertion: a condition the output must satisfy

When you run a suite, Studio executes each case against the playbook and reports pass/fail with the actual output.

---

Create a test suite in the Studio UI

1. Open the playbook you want to test
2. Go to the Tests tab
3. Click New Test Suite
4. Give it a name: Weather Briefing Tests

Add a test case

1. Click Add case
2. Name: Returns non-empty briefing
3. Inputs: (leave empty — this playbook has no inputs)
4. Assertion type: string_contains
5. Expected value: Amsterdam (the AI should mention the city)
6. Click Save

Assertion types

Type	Checks
string_contains	Output contains a substring
string_not_empty	Output is not empty or blank
regex_match	Output matches a regular expression
json_path	A JSON path expression evaluates to a value
length_min	Output has at least N characters

---

Run the suite from the UI

1. Open the test suite
2. Click Run suite
3. Wait for results — each case shows ✅ pass or ❌ fail with the actual output

---

Run via provision CLI

The provision tool can run test suites as part of a deploy pipeline:

cd /path/to/your-customer-repo

# Run test suites
go run ./cmd/provision \
  --dir studio/ \
  --org-slug your-org \
  --url https://studio.taufinity.io \
  --run-test-suites

Exit code 0 = all suites passed. Exit code 1 = one or more failures.

Output:

Suite "Weather Briefing Tests" — 2 cases
  ✅ Returns non-empty briefing (0.8s)
  ✅ Mentions Amsterdam (1.1s)
Suite PASSED (2/2)

---

CI integration (CloudBuild example)

Add a test step after provisioning:

# cloudbuild.yaml
steps:
  - name: gcr.io/cloud-builders/go
    args:
      - run
      - ./cmd/provision
      - --dir=studio/
      - --org-slug=my-org
      - --url=https://studio.taufinity.io
    env:
      - TAUFINITY_ADMIN_TOKEN=$_ADMIN_TOKEN

  - name: gcr.io/cloud-builders/go
    args:
      - run
      - ./cmd/provision
      - --dir=studio/
      - --org-slug=my-org
      - --url=https://studio.taufinity.io
      - --run-test-suites
    env:
      - TAUFINITY_ADMIN_TOKEN=$_ADMIN_TOKEN

---

assert step vs test suite

	assert step	Test suite
When it runs	Every time the playbook runs	Only when you explicitly run the suite
Good for	Runtime guardrails, catching bad AI output	Regression testing during development
Blocks the run?	Yes — fails the run if condition is false	No — run history is separate

Use assert steps to enforce invariants in production. Use test suites to safely test changes during development.

# Example: assert step that blocks if output is too short
- name: Assert output quality
  step_type: assert
  output_key: quality_check
  config:
    condition: 'len(step_outputs["summary"]) > 100'
    message: "Summary is too short — AI may have returned an error"

---

Test suite YAML (for provision)

Define test suites in studio/test-suites/. Provision auto-discovers all files in that directory and deploys them alongside playbooks:

# studio/test-suites/weather-briefing.yaml
name: Weather Briefing Tests
target_slug: weather-briefing    # matches playbook slug

cases:
  - name: Returns non-empty briefing
    inputs: {}
    assertion:
      type: string_not_empty
      output_key: briefing_html

  - name: Mentions Amsterdam
    inputs: {}
    assertion:
      type: string_contains
      output_key: briefing_text
      value: Amsterdam

---

Writing good assertions

• Test behaviour, not exact wording — AI output varies. Prefer string_contains "Amsterdam" over exact match on a full sentence.
• One assertion per case — keep cases focused so failures are easy to diagnose.
• Test the failure case — add a case with intentionally bad inputs to verify the playbook handles it gracefully.
• Name cases descriptively — "Returns non-empty briefing" is better than "Test 1".