Skip to main content
Metrics allow prompt outputs to be evaluated automatically during testing. Metrics are created independently and can be reused across multiple Test Suites. Once a metric is attached to a Test Suite, it is executed automatically as part of the testing workflow.

​
Metrics List

The Metrics section contains all metrics available within the workspace. For each metric, the list displays basic information such as:
  • Name
  • Metric Type
  • Creation Date
Metrics can be reused across multiple Test Suites. Selecting a metric opens the metric editor dialog.

​
Creating and Editing Metrics

To create a new metric, click New Metric. Metrics are created and edited directly within a modal dialog. The same dialog is used for both creating new metrics and modifying existing custom metrics. Available settings depend on the selected metric type and may include:
  • Label Name
  • Evaluation Type
  • Test Variables
  • Test Data
  • Evaluation Configuration
  • Test Results
Custom metrics can be updated at any time. This includes changing the metric type when a different evaluation approach is required. After saving, the metric becomes available in Testing through: 
Manage → Metrics

​
Metric Types

Intelleap supports two metric types:

Code Metric

Runs custom Python evaluation logic during Testing.

JSON Schema Metric

Validates model outputs against a structured schema.
The selected type determines how the metric evaluates prompt outputs. Detailed configuration for each metric type is covered in the following sections.

​
Default Metrics

Some metrics are available by default. Examples include:

Latency

Measures execution timing.

Exact Match

Checks whether an output matches the expected value exactly.

Word Count Adherence

Checks whether an output follows a word count constraint.
Default metrics are maintained by the platform and are available for immediate use in Test Suites.

​
Using Metrics

Metrics do not run independently. To use a metric, attach it to a Test Suite. Once attached, the metric is executed automatically whenever tests are run. The same metric can be reused across multiple Test Suites.

​
Metrics and Testing

All metrics are executed through the Testing workflow. During test execution, metrics evaluate model outputs and produce structured results that can be reviewed alongside execution results. This allows prompt quality to be measured consistently across different datasets and releases.