WDYM? Business Intelligence agents that understand you.

Overview

Humans communicate in different ways about the same concepts. They bring their own terminology, phrasing, and context when interacting with an agent.

In this post, we explore how LLM-based workflows can improve an agent's ability to identify, clarify, and interpret user intent in the context of a Business Intelligence Agent.

Building on the State-Driven Agent Architecture we introduced in our previous post, we extend the agent's context beyond business intelligence objects. While understanding those objects is essential, the agent must also understand how people refer to them, relate them to one another, and discuss them within their own business context.

The Problem

Through understanding user interactions with agents, we identify terminology drift and referential ambiguity as key challenges when building business intelligence agents. The following scenarios show how a simple misalignment in terminology or context can break the reporting pipeline for a Business Intelligence Agent.

The Solution

We use two key mechanisms to address these problems: Entity Extraction & Resolution and Semantic Search. These are key system components required to resolve these two core problems of terminology drift and referential ambiguity. As an indirect but beneficial impact, having both of these mechanisms also reduces conversational back-and-forth as the entity resolutions build out the ontology.

Entity Resolution

An essential part of understanding how different users communicate is to build an entity extraction layer on the conversational data, by User, and by Organization. This allows for identified entities to store in the user's or organization's context, while unresolved entities can be escalated for clarification.

Part 1: Entity Extraction & Resolution

As an example, we have the message:

User: Show me the top 10 investments by amount for John Doe

Running the message through a Small Language Model (SLM) or even some more tailored Entity Extraction models, you can classify and maintain an entity relationship structure that can be conversation-specific as well as generalized by organization.

{
  "operation": "READ",
  "target_objects": [],
  "request_terms": [
    {
      "text": "investments",
      "role": "requested_record_type",
      "resolved_object": null,
      "confidence": 0.6
    }
  ],
  "entities": {
    "person_name": "John Doe"
  }
}

Escalate ontology matches

As per the JSON output, the entity extraction identified "Investments" and has not surfaced any resolved objects by default.

{
    "domain_ontology_matches": [],
    "entity_lenses": [],
    "ambiguities": [
      {
        "id": "entity_lens",
        "kind": "entity_lens",
        "status": "unresolved",
        "blocking": false,
        "default_policy": "default_overview",
        "reason": "No explicit Salesforce object or domain term was detected."
      }
    ],
    "clarification": {
      "needed_now": true,
      "ask_if_lookup_fails": false,
      "question": null
    },
    "resolved_context": {
      "objects": [],
      "entities": {},
      "entity_lenses": [],
      "response_language": "en"
    }
  }

The first action taken is that, the first time a user makes a request like this, the agent cannot identify but can suggest objects that are classified as investments. Therefore the agent workflow triggers a clarification path, in which it asks the user which object were they referring to, with some potential matches. As an example, the Agent would say:

Agent: Which Salesforce object do you mean by investments?

A user would naturally suggest "Opportunities", and the Entity Extraction "resolves" the entity ambiguity for subsequent requests.

  {
    "domain_ontology_matches": [
      {
        "canonical": "investment",
        "phrase": "investments",
        "primary_object": "Opportunity",
        "also_consider": ["OpportunityLineItem"],
        "default_lens": "investment_pipeline"
      }
    ],
    "resolved_context": {
      "objects": ["Opportunity"],
      "entity_lenses": ["investment_pipeline"]
    }
  }

In subsequent requests, variants of the term "investment" can be mapped to the stored ontology entry without requiring another clarification. These mappings may be conversation, user, and organizational level depending on your architecture. In the case of our system, we apply these mapping autonomously after mention-thresholds are met at the user and organizational level.

Part 2: Semantic Search

Humans embed understanding of who, what, where, when, why to conversations, whereas LLMs and Agents do not. This understanding has to be developed either by encoding a memory, or into other systemic processes that occur in the agentic architecture. In our case as part of the entity resolution, subsequently after entity extraction, we attempt to semantically find any ambiguous nouns such as "John Doe" (an Accoun, User, or Contact) using vector search.

To do this, we store light versions of the Salesforce objects in pgvector, and store their core attributes such as the object nice name, API name, object record id, and object record name, which gives us enough insight to either resolve an ambiguity or request a clarification. Continuing with the earlier example:

User: Show me the top 10 investments by amount for John Doe

{
  "operation": "READ",
  "target_objects": [],
  "request_terms": [
    {
      "text": "investments",
      "role": "requested_record_type",
      "resolved_object": null,
      "confidence": 0.6
    }
  ],
  "entities": {
    "person_name": "John Doe"
  }
}

The agent has identified "John Doe" as a person, but it has not yet linked the name to a specific Salesforce record. In all cases where we have entities defined we apply a semantic search to any entity identified, and return the best object match(es) over a certain semantic similarity.

Search Results

Below is a payload of the semantic search results for "John Doe".

 [
    {
      "object_type": "Account",
      "record_id": "007Hu00000A1b2CIAG",
      "score": 0.93,
      "vector_score": 0.88,
      "name_score": 1.0,
      "text": "Account: Name: John Doe | Email: john.doe@example.com | Username: john.doe@example.com | IsActive: true",
      "metadata": {
        "field_name": "John Doe",
        "field_email": "john.doe@example.com",
        "field_username": "john.doe@example.com",
        "field_isactive": true,
        "field_profileid": "00fHu0000012345IAB"
      }
    }
  ]

If there is more than one result returned over the threshold (i.e. A cosine similarity score over 0.90), then a clarification request can be sent to the User to resolve the ambiguity.

The Outcome

Through the painstaking process of entity extraction, ontology search and generation, semantic search, and entity resolution, we've constructed a deterministic reporting schema that can then be transformed into a proper Salesforce SOQL query (or any SQL-like query language).

We've learned through clarification, and subsequently ontological matches, that the "investments" refers to "Opportunity". We also learned that "John Doe" is an "Account" through semantic search. From there, we construct the full report specification that can then be sent through to our SOQL constructor to create a properly formatted report, with the resolved entities and objects.

"report_spec": {
      "target_object": "Opportunity",
      "columns": [
        {"field": "Name", "label": "Opportunity"},
        {"field": "Account.Name", "label": "Account"},
        {"field": "StageName", "label": "Stage"},
        {"field": "Amount", "label": "Amount"},
        {"field": "CloseDate", "label": "Close Date"},
      ],
      "filters": [
        {
          "field": "AccountId",
          "operator": "in",
          "value": ["005USER"],
          "removable": true
        }
      ],
      "sort": [
        {
          "field": "Amount",
          "direction": "DESC"
        }
      ],
      "include_all_matching_rows": true
}

The resulting reports all correctly resolve in any of the scenarios identified as a result of correctly resolving ambiguities and clarifying uncertainties with the user.

In our experiment, our simple assessment mechanism was determining whether or not an agent loop could resolve a complete chain of entities with the entity resolution and semantic search mechanisms added. Across 10 questions which required resolving one or more entity ambiguities, 100% of the initial assessment questions led to failed agent responses as a result of incorrect report structuring, hallucinations, and simply "Could not find {entity}." scenarios. When seeding the ontology with known terminology and introducing this workflow: Extraction entity mentions → map business terms to canonical concepts → resolve named records → clarify uncertainty → construct and validate the report specification – we were able to resolve all 10 report generation requests.

Example Successful Agent Responses