Large Language Models are excellent at generating text.
But generation alone isn't enough to build truly useful AI applications.
Imagine asking an AI assistant:
What's the status of my order?A normal LLM can explain how order tracking works.
But it cannot magically access your order database.
Or suppose you ask:
Cancel my order #ORD-10291.The model can tell you how to cancel an order.
But it cannot actually cancel anything unless your application gives it the ability to perform that action.
This is where tool calling and AI agents come in.
Instead of simply generating an answer, an AI application can:
Understand the request
↓
Decide what action is required
↓
Select a tool
↓
Execute the tool
↓
Observe the result
↓
Continue reasoning
↓
Generate the final responseIn this article, we'll explore how to build this architecture using Spring AI.
An AI agent is an application where an LLM can decide what actions need to be performed and use available tools to accomplish a goal.
A traditional LLM application looks like:
User
↓
Prompt
↓
LLM
↓
ResponseAn agent-based application looks more like:
User
↓
LLM
↓
Decision
↓
Tool
↓
Result
↓
LLM
↓
Decision
↓
Another Tool
↓
Result
↓
Final AnswerThe important difference is:
The LLM is no longer limited to generating text.
It can interact with the application through controlled capabilities.
Tool calling allows an LLM to request the execution of a function exposed by your application.
For example, imagine our application provides:
getOrderStatus()
cancelOrder()
getCustomer()
createSupportTicket()The user asks:
Where is my order?The model might determine that it needs:
getOrderStatus()The application executes the function and returns:
Order #10291
Status: Shipped
Expected delivery: September 10The LLM can then generate:
Your order has been shipped and is expected to arrive on September 10.The LLM didn't directly access the database.
Instead:
LLM
↓
Tool Request
↓
Application
↓
Database
↓
Tool Result
↓
LLM
↓
AnswerThis distinction is extremely important for enterprise applications.
Without tools:
LLM
↓
TextWith tools:
LLM
↓
Tools
├── Database
├── REST APIs
├── Search
├── Payment systems
├── CRM
├── Internal services
└── Business workflowsThis turns the LLM from a text-generation component into an interface for interacting with your application.
For example, an AI sales assistant could have:
getCustomer()
getCustomerOrders()
createLead()
updateLead()
sendEmail()
scheduleMeeting()A support agent could have:
searchKnowledgeBase()
getCustomerAccount()
getOrder()
createTicket()
updateTicket()An internal developer assistant could have:
searchDocumentation()
searchGitRepository()
getBuildStatus()
createIssue()The possibilities are much broader than simple question answering.
At this point, it is useful to distinguish RAG from tool calling.
RAG is primarily about retrieving information.
Tool calling is about performing actions or retrieving live data through application capabilities.
For example:
RAG
↓
Retrieve company documentation
↓
Answer questionTool calling:
LLM
↓
Call order API
↓
Get live order status
↓
AnswerThey can also be combined.
For example:
User
↓
AI Agent
├── RAG → Search company policies
│
├── Tool → Get customer account
│
└── Tool → Check order status
↓
LLM
↓
AnswerThis combination is extremely powerful.
Spring AI provides abstractions that make it easier to expose application capabilities to language models.
A simplified architecture looks like:
Spring Boot
│
├── ChatModel
│
├── Tools
│
├── Advisors
│
├── Chat Memory
│
└── Vector StoreThe application controls which tools are available.
The model decides whether a tool is needed.
This separation is important.
The model should not have unrestricted access to your application.
Instead, the application exposes specific capabilities.
Imagine we have an order service.
@Service
public class OrderService {
public OrderStatus getOrderStatus(String orderId) {
// Fetch order from database
return orderRepository.findStatus(orderId);
}
}We can expose a controlled method as an AI tool.
Conceptually:
@Tool(
description = "Get the current status of an order"
)
public OrderStatus getOrderStatus(String orderId) {
return orderService.getOrderStatus(orderId);
}The description is important.
The model uses the tool description to understand:
What does this tool do?
When should I use it?
What parameters does it require?For example:
Tool:
getOrderStatus
Description:
Returns the current shipping and delivery status
for a customer order.
Input:
orderIdThe model can then determine whether this tool is appropriate.
A tool can be thought of as:
Tool Name
+
Description
+
Input Schema
+
Execution LogicFor example:
getOrderStatus(
orderId: String
)The model might produce a tool request conceptually like:
{
"name": "getOrderStatus",
"arguments": {
"orderId": "ORD-10291"
}
}The application receives the request and executes the corresponding Java method.
A typical interaction looks like this:
User
│
│ "What's the status of ORD-10291?"
↓
LLM
│
│ Tool Request
↓
getOrderStatus("ORD-10291")
│
↓
Order Service
│
↓
Database
│
↓
Tool Result
│
↓
LLM
│
↓
Final AnswerNotice something important.
The LLM doesn't execute Java code itself.
The application remains responsible for execution.
The model only requests the action.
Spring AI's ChatClient provides a convenient API for interacting with chat models.
Conceptually:
ChatClient chatClient;
String response = chatClient.prompt()
.user("What's the status of order ORD-10291?")
.tools(orderTools)
.call()
.content();The exact APIs may vary depending on the Spring AI version you're using, but the architecture remains the same:
ChatClient
↓
ChatModel
↓
Tool Selection
↓
Tool Execution
↓
Tool Result
↓
Final ResponseA real agent usually has more than one tool.
For example:
CustomerAgentTools
├── getCustomer()
├── getCustomerOrders()
├── getOrderStatus()
├── createSupportTicket()
└── updateCustomer()Now consider this question:
My order is late. Please check the status
and create a support ticket if necessary.The model might determine:
1. getOrderStatus()
2. Analyze result
3. createSupportTicket()
4. Return final responseThe application executes each requested operation.
This is where the concept of an agent starts becoming much more interesting.
A simple agent loop can be represented as:
┌───────────────┐
│ User │
└───────┬───────┘
↓
┌───────────┐
│ LLM │
└─────┬─────┘
↓
Need a Tool?
/ \
No Yes
↓ ↓
Final Answer Tool Call
↓
Tool Execution
↓
Tool Result
↓
LLM
↓
Need Another Tool?The model can repeatedly interact with tools until it has enough information to produce the final response.
This distinction is important.
People often hear:
AI Agent
and immediately think:
Give AI access to everything
↓
Let AI do whatever it wantsThat is not how production systems should be designed.
A production agent should operate inside clear boundaries.
For example:
Allowed Tools
↓
Authorization
↓
Validation
↓
Execution
↓
AuditThe application remains in control.
The model should not be trusted with unrestricted capabilities.
Imagine we expose:
@Tool
public void deleteCustomer(String customerId) {
...
}This is potentially dangerous.
An LLM should not automatically receive unrestricted permission to perform destructive operations.
Instead, sensitive tools should have additional controls.
For example:
User
↓
Authentication
↓
Authorization
↓
Agent
↓
Tool Request
↓
Permission Check
↓
Confirmation
↓
ExecutionFor destructive operations, you may require explicit user confirmation.
Example:
AI:
I found customer account C-19291.
Deleting this account is irreversible.
Do you want me to continue?
User:
Yes.
AI:
Executing deletion...The AI should assist with the decision process, not bypass your security model.
A useful production architecture is to classify tools.
READ
getCustomer()
getOrder()
searchDocuments()
getInvoice()Then:
WRITE
createTicket()
updateCustomer()
createLead()And:
DESTRUCTIVE
deleteCustomer()
cancelSubscription()
refundPayment()Different permission levels can then be applied.
For example:
READ
→ Automatically allowed
WRITE
→ Role-based authorization
DESTRUCTIVE
→ Authorization + confirmationThis makes agent behavior much safer.
Tool calling solves one problem.
But another problem appears quickly:
What does the agent remember?
Consider this conversation:
User:
My order is late.
AI:
What's your order number?
User:
ORD-10291.
AI:
Let me check it.Now the next message is:
Can you create a support ticket for it?The AI needs to understand that:
"it"refers to:
ORD-10291This requires conversational context.
That's where chat memory becomes important.
A simple conversation can be represented as:
User:
My order is late.
Assistant:
What's your order number?
User:
ORD-10291.
Assistant:
Let me check that order.The application maintains the conversation history.
Conceptually:
Conversation ID
↓
Chat Memory
↓
Previous Messages
↓
Current Prompt
↓
LLMSpring AI provides abstractions for managing chat memory.
It is useful to distinguish two concepts.
Conversation context.
User:
My order is late.
User:
It's order 10291.
User:
Can you check it?The system remembers the current conversation.
Persistent information about the user.
For example:
Customer:
Ayush
Preferences:
Preferred language = English
Preferred notification = EmailLong-term memory usually requires persistence in a database or another storage system.
A production architecture might look like:
Conversation
↓
Chat Memory Store
↓
PostgreSQL / RedisThe exact storage mechanism depends on the application.
Now we can combine:
User
↓
Agent
↓
Memory
↓
LLM
↓
Tools
↓
Tool Results
↓
Memory
↓
LLM
↓
AnswerThis enables more natural multi-turn interactions.
Another important Spring AI concept is the Advisor.
Advisors can intercept and influence the interaction between the application and the model.
They can be used for concerns such as:
Conversation memory
RAG
Logging
Security
Prompt modification
Context injection
ObservabilityConceptually:
User
↓
ChatClient
↓
Advisor
↓
ChatModel
↓
Advisor
↓
ResponseThis allows cross-cutting AI behavior to be separated from business logic.
Now things become much more powerful.
Imagine an enterprise support agent.
It has:
RAG
↓
Company documentationTools:
getCustomer()
getOrder()
createTicket()Memory:
Conversation historyThe architecture becomes:
User
↓
AI Agent
↓
┌───────┼────────┐
↓ ↓ ↓
RAG Tools Memory
↓ ↓ ↓
Knowledge APIs Conversation
│ │ │
└───────┼────────┘
↓
LLM
↓
ResponseThis is much closer to a production AI application.
Consider the request:
My payment failed for order ORD-19291.
Can you check what happened and tell me
what I should do?The agent could perform:
1. getOrder("ORD-19291")
2. getPaymentStatus("ORD-19291")
3. searchKnowledgeBase("payment failure")
4. Generate explanationThe final answer could be:
Your payment attempt failed because the transaction
was declined by the payment provider.
According to the payment policy, you can retry the
payment using another payment method.
Would you like me to create a support ticket?The model combined:
Live application data
+
Knowledge base
+
Conversation contextThis is significantly more useful than a standalone chatbot.
Agents can also perform multi-step workflows.
For example:
User:
Find my overdue invoices and send reminders.The agent could reason through:
getCustomer()
↓
getInvoices()
↓
Filter overdue invoices
↓
sendReminder()
↓
Return summaryThe workflow becomes:
Goal
↓
Plan
↓
Tool
↓
Observe
↓
Next Decision
↓
Tool
↓
Observe
↓
Final ResultThis pattern is often called an agent loop.
A simplified conceptual implementation looks like:
while (!completed) {
AgentDecision decision =
llm.decide(context);
if (decision.requiresTool()) {
ToolResult result =
toolExecutor.execute(
decision.toolCall()
);
context.add(result);
} else {
return decision.finalAnswer();
}
}In real applications, frameworks handle much of this interaction.
But understanding the underlying loop is important.
An important engineering lesson:
Not every AI feature needs an agent.
If your workflow is deterministic:
Validate request
↓
Call API
↓
Save result
↓
Return responseyou probably don't need an autonomous agent.
A normal service workflow may be better.
Agents become more useful when:
The next step depends on the current result.For example:
Check order
↓
If delayed
↓
Check refund policy
↓
If eligible
↓
Ask for confirmation
↓
Create refund requestThe dynamic decision-making is where agents become valuable.
A → B → C → DEverything is predetermined.
A
↓
LLM decides
├── B
├── C
└── D
↓
Observe result
↓
Decide againAgents provide flexibility.
Traditional workflows provide predictability.
Production systems often use both.
A practical Spring Boot architecture might look like:
┌───────────────┐
│ Frontend │
└───────┬───────┘
↓
┌───────────────┐
│ Spring Boot │
│ API │
└───────┬───────┘
↓
┌────────────┐
│ ChatClient │
└─────┬──────┘
↓
┌──────────────┐
│ Agent │
└──────┬───────┘
↓
┌────────────┼────────────┐
↓ ↓ ↓
Memory RAG Tools
↓ ↓ ↓
PostgreSQL pgvector APIs
↓
MicroservicesThis architecture fits naturally into existing Spring Boot applications.
Agent systems can become difficult to debug.
Imagine an agent performs:
Tool 1
Tool 2
Tool 3
Tool 4and the final response is incorrect.
You need to know:
What did the model decide?
Which tools were selected?
What arguments were sent?
How long did each tool take?
What did each tool return?
How many model calls happened?
How many tokens were consumed?Therefore, observability is critical.
Track:
LLM latency
Tool latency
Retrieval latency
Token usage
Tool calls
Tool failures
Model responses
Agent iterations
ErrorsAn agent can potentially continue calling tools indefinitely.
For example:
LLM
↓
Tool
↓
LLM
↓
Tool
↓
LLM
↓
Tool
↓
...Production systems should enforce limits.
For example:
Maximum iterations = 10
Maximum tool calls = 20
Maximum execution time = 30 secondsYou should also define clear failure behavior.
Agent limit reached
↓
Stop execution
↓
Return safe response
↓
Log failureNever blindly trust model-generated tool arguments.
Suppose the model requests:
{
"orderId": "ORD-999999999"
}Your application should still validate:
Does the order exist?
Does the user own the order?
Is the user authorized?
Is the order accessible to this tenant?The architecture should be:
LLM
↓
Tool Request
↓
Schema Validation
↓
Authorization
↓
Business Validation
↓
Tool ExecutionThe LLM is not your security boundary.
Your application is.
This becomes especially important in SaaS applications.
Imagine:
Tenant A
├── Customers
├── Orders
└── Documents
Tenant B
├── Customers
├── Orders
└── DocumentsAn AI agent must never retrieve Tenant B's information while processing a Tenant A request.
Every tool and retrieval operation should carry tenant context.
For example:
tenant_id
user_id
roles
permissionsThen:
User
↓
Authentication
↓
Tenant Context
↓
Agent
↓
Tool
↓
Authorization
↓
Tenant-scoped DataThe same principle applies to RAG.
Vector Search
+
tenant_id filtershould ensure that retrieved documents belong to the correct tenant.
Production agents should have explicit guardrails.
Examples:
Input validation
Output validation
Tool authorization
Rate limiting
Token limits
Iteration limits
PII protection
Audit logging
Human approvalFor high-risk actions:
Agent
↓
Tool Request
↓
Risk Evaluation
↓
Human Approval
↓
ExecutionThis creates a human-in-the-loop architecture.
Not every decision should be fully automated.
For example:
Refund amount < $50
↓
Automatic
Refund amount > $50
↓
Human approvalOr:
Create support ticket
↓
Automatic
Delete account
↓
Confirmation requiredThis gives us a practical balance:
AI Automation
+
Business Rules
+
Human OversightAt this point, we can combine everything we've discussed.
User
↓
Spring Boot
↓
ChatClient
↓
AI Agent
↓
┌───────────┼───────────┐
↓ ↓ ↓
Memory RAG Tools
↓ ↓ ↓
PostgreSQL pgvector REST APIs
↓
Business Services
↓
DatabaseThis is a strong foundation for enterprise AI applications.
Imagine a sales assistant.
The user asks:
Show me the latest opportunities for Acme
and tell me which ones are likely to close this month.The agent could:
1. getCustomer("Acme")
2. getOpportunities("Acme")
3. retrieve sales documentation
4. analyze opportunity information
5. generate summaryNow the user says:
Create a follow-up task for the highest priority opportunity.The agent can:
1. Identify opportunity
2. createFollowUpTask()
3. Return task detailsThis is where AI starts becoming an application interface rather than simply a chatbot.
Think about the responsibilities this way:
LLM
=
Reasoning + Language
RAG
=
Knowledge Retrieval
Tools
=
Actions + Live Data
Memory
=
Conversation Context
Spring Boot
=
Application + Security + Business LogicTogether:
LLM
+
RAG
+
Tools
+
Memory
+
Business Logic
=
AI ApplicationSpring AI provides abstractions that allow Java developers to work with AI capabilities using familiar Spring patterns.
Important building blocks include:
ChatClient
ChatModel
EmbeddingModel
VectorStore
Document
Advisors
Chat Memory
ToolsThis means an enterprise Java team can integrate AI into an existing Spring Boot architecture instead of creating an entirely separate AI stack.
For example:
Existing Spring Boot Application
↓
Spring AI Layer
↓
Model + RAG + Tools
↓
Existing MicroservicesThis makes AI integration much more practical for Java teams.
A more complete production system might eventually look like:
┌───────────────┐
│ User │
└───────┬───────┘
↓
API Gateway
↓
Authentication
↓
Spring Boot API
↓
AI Agent
↓
┌──────────────────────┼──────────────────────┐
↓ ↓ ↓
Memory RAG Tools
↓ ↓ ↓
PostgreSQL pgvector Microservices
↓
Business Database
↓
LLM Provider
↓
ResponseAnd around the entire system:
Security
Observability
Rate Limiting
Audit Logging
Guardrails
EvaluationThese are not optional concerns in serious enterprise deployments.
It is useful to understand the difference.
User
↓
LLM
↓
AnswerUser
↓
Retrieve Knowledge
↓
LLM
↓
AnswerUser
↓
LLM
↓
Tool
↓
Result
↓
AnswerUser
↓
Agent
↓
Reason
↓
Tool
↓
Observe
↓
Reason
↓
Tool
↓
Observe
↓
Final AnswerThe complexity increases at every stage.
We can now think about the evolution of an AI application:
Level 1
LLM
↓
Text Generation
Level 2
LLM + RAG
↓
Knowledge Retrieval
Level 3
LLM + Tools
↓
Actions
Level 4
LLM + Tools + Memory
↓
Contextual Assistant
Level 5
LLM + RAG + Tools + Memory
↓
Agent
Level 6
Multiple Agents + Workflows
↓
Agentic SystemThis progression is useful when deciding how much complexity your application actually needs.
The evolution from traditional AI applications to agentic applications can be summarized as:
LLM
↓
Generate Text
RAG
↓
Retrieve Knowledge
Tool Calling
↓
Take Actions
Memory
↓
Remember Context
Agents
↓
Make Decisions
Workflows
↓
Coordinate Multiple StepsSpring AI provides Java developers with abstractions for building many of these capabilities inside the Spring ecosystem.
The most important engineering principle is:
Let the model decide, but let your application control.
The LLM can decide which tool may be useful.
Your application should decide whether that tool is actually allowed to execute.
That separation gives us a much safer architecture for enterprise AI.
We've now covered three major capabilities:
LLM
↓
Generate
RAG
↓
Retrieve
Tools
↓
ActBut there is another challenge.
What happens when a system has:
Multiple agents
↓
Multiple tools
↓
Multiple services
↓
Multiple AI modelsHow do these agents communicate?
How do we standardize tool discovery?
How can an AI agent securely interact with external tools and services?
This leads us toward another important concept in modern AI engineering:
Model Context Protocol — MCP.
In the next article, we'll explore:
Building MCP Clients and Tool-Based AI Applications with Spring AI.
- Tool calling allows LLMs to interact with application capabilities.
- The LLM requests tools; your application executes them.
- Tools can expose APIs, databases, business operations, and services.
- Chat memory provides conversational context.
- RAG provides external knowledge.
- Agents can combine RAG, memory, and tools.
- Not every workflow requires an agent.
- Deterministic workflows are often better for predictable business processes.
- Tool permissions and authorization are critical.
- Destructive operations should require stronger controls.
- Multi-tenant applications must enforce tenant isolation at the tool and retrieval layers.
- Agent loops should have iteration, timeout, and tool-call limits.
- Observability is essential for debugging agent behavior.
- Human-in-the-loop approval is useful for high-risk actions.
- Spring AI provides abstractions that make these patterns accessible to Java and Spring Boot developers.
The future of enterprise AI isn't just:
LLM → AnswerIt's increasingly:
LLM
↓
Reason
↓
Retrieve
↓
Call Tools
↓
Observe
↓
Act
↓
Remember
↓
Complete the GoalAnd that's where AI agents with Spring AI become truly interesting.