AI Test Harness

Getting Started

Run AI Test Harness locally, register agents, and execute your first autonomous test workflow.

Quick Start Guide

Get AI Test Harness running in under 10 minutes with this step-by-step guide.

Option 1: Cloud (Fastest)

Sign up for a free Starter account and start testing immediately:

  1. Visit app.aitestharness.com/signup
  2. Create account and verify email
  3. Generate API key from Settings → API Keys
  4. Install CLI or SDK to start testing

Option 2: Local Deployment

Deploy the platform locally using Docker Compose or Kubernetes:

Docker Compose (Recommended for Local Dev)

bash
# Clone the repository
git clone https://github.com/ai-test-harness/platform.git
cd platform

# Start all services
docker-compose up -d

# Wait for services to be healthy
docker-compose ps

# Access dashboard at http://localhost:3000

Kubernetes (Production-Ready)

Install the platform control plane and runtime services in your local Kubernetes cluster:

bash
# Add Helm repository
helm repo add ai-test-harness https://charts.aitestharness.com
helm repo update

# Install with default configuration
helm install ai-test-harness ai-test-harness/platform \
  --namespace ai-test-harness \
  --create-namespace \
  --wait

# Verify installation
kubectl get pods -n ai-test-harness

Expose the control plane API to your workstation:

bash
kubectl port-forward -n ai-test-harness service/control-plane 8080:80

# API now available at http://localhost:8080

Install CLI

Install the AI Test Harness CLI for command-line access:

bash
# macOS/Linux
curl -fsSL https://get.aitestharness.com | sh

# Or via Homebrew
brew install ai-test-harness/tap/ath

# Windows (PowerShell)
iwr https://get.aitestharness.com/windows | iex

# Verify installation
ath version

Configure Authentication

Set up your API key for CLI and SDK access:

bash
# For cloud deployments
ath auth login

# For local deployments
export AI_TEST_HARNESS_API_URL=http://localhost:8080
ath auth create-key --name "local-dev"

Connect Your Application

Register your application and environment:

bash
# Create application
ath apps create \
  --name "My SaaS App" \
  --url "https://app.example.com"

# Add environment
ath envs create \
  --app "My SaaS App" \
  --name "staging" \
  --url "https://staging.example.com"

Create Your First Test Plan

Generate a test plan from code changes:

bash
# Analyze current git diff
ath plan create \
  --name "Release v1.0.0" \
  --auto-detect \
  --environment staging

# Or specify files manually
ath plan create \
  --name "Payment flow update" \
  --files "src/checkout/payment.ts,src/api/orders.ts" \
  --environment staging

Run Tests

Execute the generated test plan:

bash
# Execute and wait for completion
ath execute run \
  --plan "Release v1.0.0" \
  --wait

# Or run in background
ath execute run \
  --plan "Release v1.0.0" \
  --async

# Check status
ath execute status <execution-id>

View Results

View test results and analytics:

bash
# View execution summary
ath execute results <execution-id>

# Open dashboard for detailed analysis
ath dashboard open

# Get release readiness score
ath analytics readiness --version "v1.0.0"

Using the SDK

Integrate AI Test Harness into your application code:

bash
# Install SDK
npm install @ai-test-harness/sdk

# Or for Python
pip install ai-test-harness

Example TypeScript usage:

bash
import { TestHarness } from '@ai-test-harness/sdk';

const client = new TestHarness({
  apiKey: process.env.AI_TEST_HARNESS_API_KEY,
});

async function runTests() {
  // Create test plan
  const plan = await client.testPlans.create({
    name: 'Release v1.0.0',
    changeSet: [
      { file: 'src/checkout/payment.ts', impact: 'high' }
    ],
    environment: 'staging',
  });

  console.log(`Created plan: ${plan.id}`);

  // Execute tests
  const execution = await client.executions.create({
    testPlanId: plan.id,
  });

  // Wait for completion
  await execution.wait();

  // Get results
  const results = await execution.getResults();
  console.log(`Pass rate: ${results.passRate}%`);

  if (results.passRate < 90) {
    throw new Error('Quality gate failed');
  }
}

runTests().catch(console.error);

Next Steps

Now that you have AI Test Harness running, explore these topics:

Getting Help

If you run into issues: