Evaluation Profiles

Overview

Evaluation profiles (formerly Prompt Profiles) define what the AI looks for when analyzing a conversation. They control which aspects are evaluated and what the output includes. Different projects can use different evaluation profiles.

Not the same as Criteria. A profile is the scorecard Aelo applies to a whole category of conversations. Questions asked of every conversation — yes/no, a score or a choice — are a separate feature — see Criteria.

Evaluation Profiles

Client Admin

Aelo provides base categories that serve as starting points:

Category	Best For
Sales Call	Outbound/inbound sales phone conversations
Support Call	Customer service and technical support calls
Sales Chat	Text-based sales conversations (web chat, messaging)
Complaint	Complaint handling and escalation calls

Each category comes with pre-configured modules optimized for that conversation type.

Category-Bound Profiles (Automatic Selection)

Client Admin

Aelo automatically detects the category of every call (Purchase, Support, Complaint, etc.). Each organization gets a set of category-bound profiles — one per category — so every conversation is analyzed with metrics tuned for its type:

10 call categories: Product Inquiry, Purchase, Scheduling, Support, Billing, Onboarding, Complaint, Cancellation, Follow-up, Other.
Chats: detected into the same categories as calls, and analyzed with their category’s profile. The channel only affects the shape of the evaluation: metrics that apply only to speech (filler words, interruptions, diction, timestamped key moments, the recording disclaimer) don’t apply to chats.
Missed and short calls are not analyzed, so they have no profiles. The same applies to a chat with no operator reply, or with fewer real messages than the project’s Min Chat Messages setting (4 by default).

When a record is analyzed, the profile is selected in this order:

Category-bound profile — the org profile bound to the detected call category.
Project profile — the profile linked to the project (fallback when no bound profile exists).
Default profile — used when neither of the above is configured.

Managing Bindings

Assigning, changing, or removing a category binding happens only in the matrix: Analysis Setup → Categories and Profiles → Categories tab. Each row of the matrix is a category with its current bound profile (or “Base profile” if unbound):

Assign — pick a profile for the category.
Restore default — recreates the pre-configured profile for a category after a binding is removed.
A binding already taken by another profile is rejected (“This binding is already taken by another profile”).

The profile editor itself (Profiles tab, formerly the Prompt Builder) shows the binding read-only, with a link back to the matrix (“Binding is changed on the “Categories” tab”).

Each category can have at most one bound profile per organization.

Pinned Analysis Config

Client Admin

Category-bound profiles are convenient, but some projects need to be locked to one profile no matter what category the AI detects — for example, a project dedicated entirely to a single sales motion. Pinning does that: it opts a project out of the org’s category auto-selection.

Open Projects → select a project → Settings → Quality tab → Pinned analysis config:

Pin prompt profile — always use the project’s selected prompt profile for every analysis in this project, regardless of the detected call category. There is no pin for scripts, and none is needed: a script you assign to a project already beats the category binding on its own (see the selection order below).

The toggle requires the project to already have a prompt profile selected — it stays disabled with a hint until you pick one above it. Pinning without a selection is rejected.

Full Resolution Order

With pinning in the picture, a record’s prompt profile is chosen in this order:

Explicit override — a manual choice made when re-analyzing a specific record; always wins.
Pinned project profile — used when Pin prompt profile is on.
Deal-stage mapping — for orgs with stage-based scorecard selection turned on and connected to Bitrix24: an exact mapping for the linked deal’s current stage, falling back to a mapping for the stage’s semantic phase when no exact mapping exists. Skipped entirely when the call isn’t linked to a Bitrix24 deal.
Category-bound profile — the org profile bound to the detected call category.
Project profile — the profile linked to the project (fallback when nothing above applies).
Default profile — used when none of the above is configured.

Script selection for adherence scoring follows a similar shape minus the deal-stage and override steps (re-analysis doesn’t let you choose a script, and script adherence is not stage-aware) — but with the project ahead of the category binding, unlike profiles: project script → category-bound script → no script (adherence scoring is skipped for that conversation). A script you assign to a project is a deliberate choice, so it wins; category bindings apply to projects that have no script of their own.

Things To Know

Deactivating a pinned profile does not reliably unpin it. If you deactivate the profile that’s currently pinned, the pinned step is skipped on the next analysis — but if no category binding matches that call, the project-profile fallback step can still pick up that same (inactive) profile. If you want a project to stop using a profile, unpin it explicitly rather than deactivating the profile.
Only one control stops adherence scoring for a project: Enable rule-compliance scoring in its Quality tab. Switch it off and the step is skipped, whatever any binding says.
Clearing a project’s script, or deactivating it, changes WHICH script is used — not whether scoring happens. Both make resolution fall through to the category-bound script, so a conversation whose category has a binding is still scored, just against that script. Deactivating also reaches beyond this project: a script belongs to the organization, so it stops being used everywhere it was assigned, and in its category binding too.
Deleting a pinned profile also falls through cleanly: category binding → project profile → default.
A script assigned to nothing is never applied — and the list says so. A script that is bound to no call category and is not the default script of any active project matches no step of the cascade, so script adherence is skipped for every conversation. In Script rules, such a script carries a Not assigned marker; the others show the category they are bound to and the projects that use them. An attachment that cannot work — the project uses another script as its default, or the project is not active — is dimmed and explains itself on hover.

Stage-Based Scorecard Selection

Client Admin

For organizations connected to Bitrix24, Aelo can select the evaluation scorecard based on the linked deal’s current CRM pipeline stage, on top of the detected call category — useful when, say, a discovery call and a closing call in the same category deserve different evaluation profiles. This applies to calls linked to a Bitrix24 deal; leads are not covered in this version.

Deal-stage mapping sits between the pinned project profile and the category-bound profile in the resolution order above: an exact stage mapping wins first, falling back to a mapping for the stage’s semantic phase when no exact mapping exists for that stage (the phase comes from Bitrix24’s own won/lost marker, else from the phase you chose for the stage, else is guessed from the stage name — see Deal Phases below). It is skipped entirely — falling straight through to the category-bound profile — for chats, calls not linked to a Bitrix24 deal, and orgs that haven’t connected Bitrix24 or turned this on.

Deal Stages Tab

Client Admin

Open Analysis Setup → Categories and Profiles → Deal Stages tab to configure the mapping:

Deal Phases — set a default evaluation profile for each of the 7 coarse semantic phases (Discovery, Qualification, Proposal, Negotiation, Closing, Post-sale, Other). They apply automatically at analysis time to a stage with no exact mapping: Post-sale and Other from the deal’s won/lost status in Bitrix24, the rest from the phase you chose for the stage under Pipelines & Stages, or — until you choose — the phase guessed from the stage’s name (for example, a stage named “Negotiations” counts as Negotiation). The phase is stored on the call; a guessed one is marked as a guess on the Objections screen.
Pipelines & Stages — each stage has two settings. Phase: choose the stage’s phase, or leave it on the guess from the name (a won or lost stage shows the phase Bitrix24 sets, with no choice). Changing a phase also moves the calls already recorded at that stage to it — on the Objections screen; their scores are not recalculated. Profile: map the stage to a profile directly; an exact mapping always takes priority over the phase default.
Keep guessed phases — stores every phase still guessed from a stage name as your own choice in one action, so renaming a stage in Bitrix24 no longer changes its phase.
Accept suggested — for every unmapped stage with a usable guessed phase, creates an exact binding to that phase’s default profile in one action, so the stage keeps that profile even if its name changes later.

The Pipelines & Stages list requires an active Bitrix24 connection; without one, the tab prompts you to connect a CRM source first.

On The Analysis Panel

When a call is scored using the deal-stage mapping, the analysis panel shows the CRM stage and a “Scored using [profile] scorecard” caption, so you can see which mapping applied. This is visible in the standalone Aelo record view; it is not yet shown inside the embedded Bitrix24 panel.

Analysis Modules

Each profile consists of toggleable analysis modules. Enable or disable modules to control what the AI evaluates:

Summary Module

Produces high-level metrics:

Quality Score (0–100)
Sale Probability (0–100%)
Attention Flag — marks conversations needing manual review

Content Module

Extracts conversation content:

Topics — main subjects discussed
Keywords — important terms and phrases
Key Moments — critical events (objection raised, pricing discussed, agreement reached)
Summary — natural language overview of the conversation

Speech Module

Evaluates speaking patterns:

Filler Words — how many hesitation sounds the agent made (“um”, “uh”, “hmm”), counted in the agent’s own speech only. Words are not counted, even common fillers like “like” or “well”: they often carry meaning, and a count without context would blame the agent for them. The score depends on the rate per minute: each sound a minute takes 10 points off 100, so ten or more a minute score 0. The call transcript hides these sounds to stay readable; the count is made on the verbatim text. If the agent can’t be identified in the call, the metric is not shown
Agent Uncertainty — the agent hedging on their own price, deadline, or promise (not retelling the customer’s words, instructions, or estimates). The signal list is built into the analysis and is not configurable
Forbidden Words — words from the project’s own list that must not be said. Set the list in project settings → Quality; inflected forms count, and every hit shows the speaker and the quote. An empty list means the metric is not computed
Mandatory Phrases — required phrases (e.g., compliance statements)
Profanity — detection of inappropriate language
Speech Clarity — overall speaking clarity score

Dialogue Module

Evaluates conversation structure:

Greeting — proper opening and introduction
BANT Qualification — Budget, Authority, Need, Timeline assessment
Objection Handling — how the agent responds to customer objections
Closing — effective conversation conclusion and next steps

Emotions Module

Analyzes emotional aspects:

Sentiment — positive, neutral, negative classification
Politeness — agent courtesy and professionalism
Interruptions — frequency and handling of speaker interruptions

Compliance Module

Checks regulatory and policy adherence:

Speech Clarity — meets minimum clarity standards
Recording Disclaimer — required disclosure at conversation start
CRM Completeness — only assessed when there is a connected CRM and collecting the customer’s identifying details was actually this call’s job (a new customer not yet linked to any record); calls tied to an existing deal or known contact are skipped rather than scored low, and so are calls from organizations with no CRM integration, where there is no CRM to measure hygiene against. When applicable, it checks that the customer/contact name and contact info (phone or email) were captured — plus the company name for business customers only, since a private individual has none. This is analyzed and shown as its own compliance indicator and does not contribute to the quality score.

Recommendations Module

Generates coaching output:

Strengths — what the agent did well
Improvements — specific areas to improve with actionable suggestions

Configuring a Profile

Open Analysis Setup → Categories and Profiles → Profiles tab in the main menu (formerly the Prompt Builder).
Select a base category or create a custom profile.
Toggle modules on or off based on your needs.
Review the Live Preview panel on the right:
- Estimated token count (affects processing cost)
- Sample output — what the AI will return for a typical conversation
Save the profile.

Live Preview

The right-side preview panel updates in real-time as you configure modules:

Token Count — estimated tokens consumed per analysis (directly affects cost)
Cost Estimate — approximate cost per analysis based on your LLM provider
Sample Output — example of what the analysis result looks like

Use the preview to find the right balance between analysis depth and cost.

Preview Scope

By default the preview is computed at organization level, and project rules are not part of it. The toolbar lets you pick a project — the preview then applies that project’s settings: the forbidden-word list, the report language, and the project’s name and description. Those are the prompt inputs configured on the project; data a real analysis pulls in as it runs (CRM fields for autofill, a tailored analysis configuration) is not part of the preview.

Linking Profiles to Projects

To assign an evaluation profile to a project:

Go to Projects > select a project > Settings.
Find the Prompt Profile section.
Select the desired evaluation profile from the dropdown.
Save.

All new records in that project will use the selected profile. Existing records are not affected unless you manually re-analyze them.

The project profile acts as a fallback: it applies when no category-bound profile exists for the conversation’s detected category (see Category-Bound Profiles) — unless the project has pinned its own profile, which always takes priority over category bindings.

Access by Role

Feature	Client	Admin	Supervisor	Agent
View profiles	✅	✅	❌	❌
Create/edit profiles	✅	✅	❌	❌
Delete profiles	✅	✅	❌	❌
Link to projects	✅	✅	❌	❌
View live preview	✅	✅	❌	❌

Tips & Best Practices

Start with a base category and customize — don’t build from scratch.
Enable only the modules you need. More modules = higher cost per analysis and longer processing.
Use the Live Preview to estimate costs before deploying a new profile.
Create different profiles for different conversation types (e.g., one for sales calls, another for support).
After changing a profile, re-analyze a few recent records to verify the results meet expectations.
The Compliance module is recommended for regulated industries — it ensures recording disclaimers and mandatory phrases are consistently checked.