Evaluation Criteria

Overview

Evaluation Criteria (also called Prompt Profiles) define what the AI looks for when analyzing a conversation. They control which aspects are evaluated and what the output includes. Different projects can use different evaluation profiles.

Evaluation Profiles

Client Admin

Aelo provides base categories that serve as starting points:

Category	Best For
Sales Call	Outbound/inbound sales phone conversations
Support Call	Customer service and technical support calls
Sales Chat	Text-based sales conversations (web chat, messaging)
Complaint	Complaint handling and escalation calls

Each category comes with pre-configured modules optimized for that conversation type.

Category-Bound Profiles (Automatic Selection)

Client Admin

Aelo automatically detects the category of every call (Purchase, Support, Complaint, etc.). Each organization gets a set of category-bound profiles — one per category — so every conversation is analyzed with criteria tuned for its type:

10 call categories: Product Inquiry, Purchase, Scheduling, Support, Billing, Onboarding, Complaint, Cancellation, Follow-up, Other.
Chats: detected into the same categories as calls, and analyzed with their category’s profile. The channel only affects the shape of the evaluation: criteria that apply only to speech (filler words, interruptions, diction, timestamped key moments, the recording disclaimer) don’t apply to chats.
Missed and short calls are not analyzed, so they have no profiles.

When a record is analyzed, the profile is selected in this order:

Category-bound profile — the org profile bound to the detected call category.
Project profile — the profile linked to the project (fallback when no bound profile exists).
Default profile — used when neither of the above is configured.

Managing Bindings

Assigning, changing, or removing a category binding happens only in the matrix: Analysis Setup → Categories and Criteria → Categories tab. Each row of the matrix is a category with its current bound profile (or “Base profile” if unbound):

Assign — pick a profile for the category.
Restore default — recreates the pre-configured profile for a category after a binding is removed.
A binding already taken by another profile is rejected (“This binding is already taken by another profile”).

The profile editor itself (Profiles tab, formerly the Prompt Builder) shows the binding read-only, with a link back to the matrix (“Binding is changed in “Categories and Criteria"").

Each category can have at most one bound profile per organization.

Pinned Analysis Config

Client Admin

Category-bound profiles are convenient, but some projects need to be locked to one profile no matter what category the AI detects — for example, a project dedicated entirely to a single sales motion. Pinning does that: it opts a project out of the org’s category auto-selection.

Open Projects → select a project → Settings → Quality tab → Pinned analysis config:

Pin prompt profile — always use the project’s selected prompt profile for every analysis in this project, regardless of the detected call category.
Pin script — always use the project’s default script for script adherence scoring, regardless of the detected call category.

Both toggles require the project to already have the corresponding item selected — each toggle stays disabled with a hint until you pick a prompt profile (or assign a default script) above it. Pinning without a selection is rejected.

Full Resolution Order

With pinning in the picture, a record’s prompt profile is chosen in this order:

Explicit override — a manual choice made when re-analyzing a specific record; always wins.
Pinned project profile — used when Pin prompt profile is on.
Deal-stage mapping — for orgs with stage-based scorecard selection turned on and connected to Bitrix24: an exact mapping for the linked deal’s current stage, falling back to a mapping for the stage’s semantic phase when no exact mapping exists. Skipped entirely when the call isn’t linked to a Bitrix24 deal.
Category-bound profile — the org profile bound to the detected call category.
Project profile — the profile linked to the project (fallback when nothing above applies).
Default profile — used when none of the above is configured.

Script selection for adherence scoring follows the same shape minus the deal-stage and override steps (re-analysis doesn’t let you choose a script, and script adherence is not stage-aware): pinned project script → category-bound script → project default script → no script (adherence scoring is skipped for that conversation).

Things To Know

Deactivating a pinned profile does not reliably unpin it. If you deactivate the profile that’s currently pinned, the pinned step is skipped on the next analysis — but if no category binding matches that call, the project-profile fallback step can still pick up that same (inactive) profile. If you want a project to stop using a profile, unpin it explicitly rather than deactivating the profile.
A pin with nothing left to pin is a safe no-op, not an error. If a pinned default script is later unassigned from the project (from the Scripts management screens) while Pin script stays on, the pin simply becomes inert — the resolver skips it and falls through to the category binding or project default. We recommend turning the toggle off explicitly once you notice it has nothing attached, rather than relying on this fallback.
Deleting a pinned profile also falls through cleanly: category binding → project profile → default.

Stage-Based Scorecard Selection

Client Admin

For organizations connected to Bitrix24, Aelo can select the evaluation scorecard based on the linked deal’s current CRM pipeline stage, on top of the detected call category — useful when, say, a discovery call and a closing call in the same category deserve different evaluation criteria. This applies to calls linked to a Bitrix24 deal; leads are not covered in this version.

Deal-stage mapping sits between the pinned project profile and the category-bound profile in the resolution order above: an exact stage mapping wins first, falling back to a mapping for the stage’s semantic phase when no exact mapping exists for that stage (in practice the automatically-derived phase reflects the deal’s won/lost status — see Deal Phases below). It is skipped entirely — falling straight through to the category-bound profile — for chats, calls not linked to a Bitrix24 deal, and orgs that haven’t connected Bitrix24 or turned this on.

Deal Stages Tab

Client Admin

Open Analysis Setup → Categories and Criteria → Deal Stages tab to configure the mapping:

Deal Phases — set a default evaluation profile for each of the 7 coarse semantic phases (Discovery, Qualification, Proposal, Negotiation, Closing, Post-sale, Other). These defaults feed Accept suggested below, which turns them into exact per-stage mappings — that’s how Discovery, Qualification, Proposal, Negotiation, and Closing actually take effect. Post-sale and Other additionally apply automatically at analysis time, as a fallback derived straight from the deal’s won/lost status when a stage has no exact mapping.
Pipelines & Stages — map an individual CRM pipeline stage to a profile; each stage shows a suggested phase (a keyword-based hint, not an automatic binding) to speed up mapping. An exact mapping always takes priority over the phase default.
Accept suggested — for every unmapped stage with a usable suggested phase, creates an exact binding to that phase’s default profile in one action. Suggestions are never applied automatically — you always choose to accept them.

The Pipelines & Stages list requires an active Bitrix24 connection; without one, the tab prompts you to connect a CRM source first.

On The Analysis Panel

When a call is scored using the deal-stage mapping, the analysis panel shows the CRM stage and a “Scored using [profile] scorecard” caption, so you can see which mapping applied. This is visible in the standalone Aelo record view; it is not yet shown inside the embedded Bitrix24 panel.

Analysis Modules

Each profile consists of toggleable analysis modules. Enable or disable modules to control what the AI evaluates:

Summary Module

Produces high-level metrics:

Quality Score (0–100)
Sale Probability (0–100%)
Attention Flag — marks conversations needing manual review

Content Module

Extracts conversation content:

Topics — main subjects discussed
Keywords — important terms and phrases
Key Moments — critical events (objection raised, pricing discussed, agreement reached)
Summary — natural language overview of the conversation

Speech Module

Evaluates speaking patterns:

Filler Words — “um”, “uh”, “like” frequency
Agent Uncertainty — the agent hedging on their own price, deadline, or promise (not retelling the customer’s words, instructions, or estimates). The signal list is built into the analysis and is not configurable
Forbidden Words — words from the project’s own list that must not be said. Set the list in project settings → Quality; inflected forms count, and every hit shows the speaker and the quote. An empty list means the metric is not computed
Mandatory Phrases — required phrases (e.g., compliance statements)
Profanity — detection of inappropriate language
Speech Clarity — overall speaking clarity score

Dialogue Module

Evaluates conversation structure:

Greeting — proper opening and introduction
BANT Qualification — Budget, Authority, Need, Timeline assessment
Objection Handling — how the agent responds to customer objections
Closing — effective conversation conclusion and next steps

Emotions Module

Analyzes emotional aspects:

Sentiment — positive, neutral, negative classification
Politeness — agent courtesy and professionalism
Interruptions — frequency and handling of speaker interruptions

Compliance Module

Checks regulatory and policy adherence:

Speech Clarity — meets minimum clarity standards
Recording Disclaimer — required disclosure at conversation start
CRM Completeness — only assessed when there is a connected CRM and collecting the customer’s identifying details was actually this call’s job (a new customer not yet linked to any record); calls tied to an existing deal or known contact are skipped rather than scored low, and so are calls from organizations with no CRM integration, where there is no CRM to measure hygiene against. When applicable, it checks that the customer/contact name and contact info (phone or email) were captured — plus the company name for business customers only, since a private individual has none. This is analyzed and shown as its own compliance indicator and does not contribute to the quality score.

Recommendations Module

Generates coaching output:

Strengths — what the agent did well
Improvements — specific areas to improve with actionable suggestions

Configuring a Profile

Open Analysis Setup → Categories and Criteria → Profiles tab in the main menu (formerly the Prompt Builder).
Select a base category or create a custom profile.
Toggle modules on or off based on your needs.
Review the Live Preview panel on the right:
- Estimated token count (affects processing cost)
- Sample output — what the AI will return for a typical conversation
Save the profile.

Live Preview

The right-side preview panel updates in real-time as you configure modules:

Token Count — estimated tokens consumed per analysis (directly affects cost)
Cost Estimate — approximate cost per analysis based on your LLM provider
Sample Output — example of what the analysis result looks like

Use the preview to find the right balance between analysis depth and cost.

Linking Profiles to Projects

To assign an evaluation profile to a project:

Go to Projects > select a project > Settings.
Find the Prompt Profile section.
Select the desired evaluation profile from the dropdown.
Save.

All new records in that project will use the selected profile. Existing records are not affected unless you manually re-analyze them.

The project profile acts as a fallback: it applies when no category-bound profile exists for the conversation’s detected category (see Category-Bound Profiles) — unless the project has pinned its own profile, which always takes priority over category bindings.

Access by Role

Feature	Client	Admin	Supervisor	Agent
View profiles	✅	✅	❌	❌
Create/edit profiles	✅	✅	❌	❌
Delete profiles	✅	✅	❌	❌
Link to projects	✅	✅	❌	❌
View live preview	✅	✅	❌	❌

Tips & Best Practices

Start with a base category and customize — don’t build from scratch.
Enable only the modules you need. More modules = higher cost per analysis and longer processing.
Use the Live Preview to estimate costs before deploying a new profile.
Create different profiles for different conversation types (e.g., one for sales calls, another for support).
After changing a profile, re-analyze a few recent records to verify the results meet expectations.
The Compliance module is recommended for regulated industries — it ensures recording disclaimers and mandatory phrases are consistently checked.