Testing Overview

Testing allows users to execute prompt variants against structured test data, compare results, and evaluate outputs using expected outputs and metrics. The Testing tab becomes available after the first release has been published.

​

Testing Structure

Testing is built around three primary entities: Each test run is executed against a selected release and a selected Test Suite.

​

Selecting a Release

The release selector is located at the top of the page. Testing always operates on a specific release. When a different release is selected, the testing table is automatically updated using the configuration stored in that release.

​

Test Suites

A Test Suite defines a collection of test data, expected outputs, and evaluation metrics. Multiple Test Suites can be created for the same release. This makes it possible to evaluate the same prompt against different datasets and testing scenarios. New Test Suites can be created directly from the Test Suite selector.

​

Testing Table Structure

The testing table is the primary workspace within the Testing tab. In the table:

• Rows represent variables
• Columns represent scenarios
• Variants are grouped within each scenario

Each scenario contains its own set of variable values used for testing. The initial variable structure is imported from the selected release and can then be managed within the Test Suite. Model outputs and expected outputs are displayed below the input data.

​

Editing Test Data

To modify test data, select any editable cell in the testing table. A side drawer will open. The drawer allows editing all variable values associated with the selected scenario. Changes are saved automatically.

​

Test Suite Settings

The Settings action allows users to manage the current Test Suite. Available actions include:

• Updating the Label
• Updating the Name
• Managing Expected Outputs
• Deleting the Test Suite

​

Managing Test Data

The Manage action provides access to additional testing configuration. Through Manage, users can configure:

• Test Cases
• Inputs
• Expected Outputs
• Metrics

All changes are applied to the currently selected Test Suite.

​

Adding Metrics

Metrics are attached through Manage → Metrics. The metrics list displays all metrics created in the Metrics section. The same metric can be reused across multiple Test Suites. Some metrics may require additional configuration before they can be used.

​

Running Tests

Tests can be executed in two ways. Results are automatically displayed in the table after execution completes.

​

Release Synchronization

If the selected release has been updated in the Editor after a Test Suite was created, Testing automatically detects the changes. When this occurs, a synchronization notification is displayed. Users can:

• Apply the updates
• Ignore the changes

For more information about release synchronization, see Release Sync.