{"article":{"slug":"creating-custom-agents-inside-brontos-alert-investigation-agent","title":"Creating Custom Agents: Inside Bronto's Alert Investigation Agent","subtitle":null,"summary":"Bronto details how its open-source alert investigation agent was built with LangGraph and LangChain, and how custom agents pair with observability tools for faster incident response.","content_type":"blog_post","language":"en","canonical_url":"https://bronto.io/blog/creating-custom-agents-inside-brontos-alert-investigation-agent","author":{"name":null,"url":null,"person_slug":null,"person_url":null},"authored_by":"human","publisher":{"name":"Bronto","url":null,"listing_slug":null,"listing":null},"topics":[{"name":"AI Agents","slug":"ai-agents","url":"https://listedarticles.com/topics/ai-agents"},{"name":"Infrastructure","slug":"infrastructure","url":"https://listedarticles.com/topics/infrastructure"},{"name":"Software Engineering","slug":"software-engineering","url":"https://listedarticles.com/topics/software-engineering"},{"name":"Open Source","slug":"open-source","url":"https://listedarticles.com/topics/open-source"}],"about_listings":[],"cover_image_url":null,"license":"all-rights-reserved","word_count":2252,"reading_minutes":10,"published_at":"2026-11-06T05:13:47.027Z","added_at":"2026-11-06T05:13:47.027Z","updated_at":"2026-11-06T05:13:47.027Z","added_via":"api","contributor":{"type":"agent","name":"ListedStartups Using Bot","registered":true},"profile_url":"https://listedarticles.com/articles/creating-custom-agents-inside-brontos-alert-investigation-agent","markdown_url":"https://listedarticles.com/articles/creating-custom-agents-inside-brontos-alert-investigation-agent.md","example":false,"citation":"Bronto. \"Creating Custom Agents: Inside Bronto's Alert Investigation Agent.\" 6 Nov 2026. https://bronto.io/blog/creating-custom-agents-inside-brontos-alert-investigation-agent (all-rights-reserved)","access":{"human_view":"preview","full_text_available":true,"source_url":"https://bronto.io/blog/creating-custom-agents-inside-brontos-alert-investigation-agent"},"body_markdown":"# Creating Custom Agents: Inside Bronto’s Alert Investigation Agent\n\nNikolai Gladychev\n\nSenior Software Engineer\n\nUsing generalist agents (e.g., Claude Code) with Bronto’s MCP server is powerful and effective, but for well-defined and repeated tasks, specialised *custom* agents can offer significant advantages including improved reliability, faster execution, and better token efficiency. It is becoming increasingly common for teams to write specialised agents for SRE workloads, and we want to make it *effortless* for people to write their own custom agents with Bronto. \n\nWe [__previously showed__](https://bronto.io/blog/building-a-log-analysis-agent-with-langgraph-and-brontos-mcp-server) how a basic LangGraph agent could leverage our MCP server, and now we’ve released [__an open source alert investigation agent__](https://github.com/bronto-community/bronto-alert-investigation-agent) that replicates our in-platform [__Alert Investigation feature__](https://bronto.io/blog/investigation-reports-when-monitors-get-smarter). It is a small reference project that you (or your agent) can use to see the best patterns for integrating Bronto into your LangGraph agents.\n\nBelow, I’ll demonstrate how *easy* it is to write custom agents with Bronto’s powerful set of MCP tools, and then dive into the architecture and code that make our open source alert investigation agent work. If you want to learn more about what our agent does and how to use it, check out our [announcement post](https://bronto.io/blog/introducing-brontos-open-source-alert-investigation-agent). Don't worry if you're not a LangGraph expert, we'll show you in detail how we used it to build our agent. We will also show how easy it is to send telemetry data from the agent to Bronto.\n\n### Building Custom Agents with LangGraph and Bronto MCP\n\nLangChain and LangGraph are two powerful and complementary libraries for wiring LLM calls into structured state machines. To make that process even smoother, we’ve released the [__Bronto SDK for Python__](https://github.com/bronto-community/bronto-sdk-python): a small, typed client for the Bronto REST API, plus helpers that encode Bronto-specific details. Combine these tools, and you can build a remarkably effective agent connected to Bronto's MCP in about 20 lines of code:\n\n```\nimport asyncio\nimport os\nimport sys\nfrom bronto_sdk import mcp as bronto_mcp\nfrom langchain_mcp_adapters.sessions import create_session\nfrom langchain_mcp_adapters.tools import load_mcp_tools\nfrom langchain.agents import create_agent\nfrom langchain.chat_models import init_chat_model\nasync def investigate(user_prompt: str):\n    api_key = os.environ.get(\"BRONTO_API_KEY\")\n    bronto_region = os.environ.get(\"BRONTO_REGION\")\n    async with create_session({\n        \"transport\": \"streamable_http\",\n        \"url\": bronto_mcp.url(bronto_region),\n        \"headers\": bronto_mcp.auth_headers(api_key=api_key)\n    }) as session:\n        await session.initialize()\n        mcp_tools = await load_mcp_tools(session)\n        llm = init_chat_model(\"claude-sonnet-5\", model_provider=\"anthropic\")\n        agent = create_agent(llm, tools=mcp_tools)\n        result = await agent.ainvoke({\"messages\": [(\"user\", user_prompt)]})\n        print(result[\"messages\"][-1].content)\nasyncio.run(investigate(sys.argv[1]))\n```\nWith just this snippet, you have an agent that can handle complex, natural-language questions like, *\"What are the p99 response times on my API gateway, broken down by resource?\"*\n\n| To run the sample code in this blog, ensure you have Python and `uv` installed on your machine, then create a temporary Python venv that has the required dependencies by running this command (on mac or linux): `tmp=$(mktemp -d) && uv venv \"$tmp\" && source \"$tmp/bin/activate\" && uv pip install langgraph langchain langchain-core langchain-aws langchain-anthropic langchain-openai langchain-mcp-adapters pydantic opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-httpx opentelemetry-instrumentation-genai-langchain httpx https://github.com/bronto-community/bronto-sdk-python/releases/download/v0.1.2/bronto_sdk-0.1.2-py3-none-any.whl && cd $tmp` | \n\nThe magic in the python snippet above lies in LangGraph's `create_agent` method, which gives your LLM a built-in **ReAct (Reason + Act)** loop. Historically, you had to write a lot of boilerplate to give an agent a set of tools, then make it look at its context, execute a tool, re-evaluate, execute a tool, etc.., and loop until the task was done. Now, `create_agent` handles that entire lifecycle.\n\nTo connect that agent to Bronto, you can use the `langchain_mcp_adapters` library. Looking at the code above, it takes just three lines to connect:\n\n1. Open a connection to Bronto’s remote MCP using  `create_session` .\n2. Complete the required MCP handshake using `session.initialize()` .\n3. Load Bronto's MCP as LangChain-native tools using `load_mcp_tools(session)` .\n\n### Controlling Tool Use\n\nA very typical thing you might want to do, is to restrict the tools used by an agent. By restricting a ReAct loop to a specific subset of MCP tools, you give the LLM only what it needs for a particular step. This reduces the chance of the agent making an error by picking the wrong tool and it also reduces the context window size:\n\n```\nALLOWED_TOOLS = {\"get_datasets\", \"get_keys\", \"search_logs\", \"timeseries\"}\n...\n        all_mcp_tools = await load_mcp_tools(session)\n        mcp_tools = [t for t in all_mcp_tools if t.name in ALLOWED_TOOLS]\n        agent = create_agent(llm, tools=mcp_tools)\n```\n### Using Langraph Middleware\n\nYou can strictly control the internal state of a LangChain ReAct agent using its Middleware system. This allows you to intercept the agent mid-loop to modify LLM requests, insert human-in-the-loop approvals, or handle errors automatically. LangChain ships with several built-in middlewares for common use cases:\n\n```\nfrom langchain.agents.middleware import ToolRetryMiddleware, ModelCallLimitMiddleware\n...\n    agent = create_agent(\n                llm,\n                tools=mcp_tools,\n                middleware=[\n                    ToolRetryMiddleware(max_retries=2, on_failure=\"continue\"),\n                    ModelCallLimitMiddleware(run_limit=5, exit_behavior=\"end\")\n                ]\n            )\n```\nIn this snippet, there are two standard middlewares:\n\n- `ToolRetryMiddleware` : Automatically retries failed tool calls. If it continues to fail, it feeds the error details back to the LLM so the agent can decide if it should try a different approach.\n- `ModelCallLimitMiddleware` : Prevents excessive agent loops by hard-terminating the agent if it exceeds a set number of LLM calls.\n\nThe Bronto open-source investigation agent also uses some custom middleware such as:\n\n- `InvestigationWindowMiddleware` : Automatically injects the correct time-range parameters into tool calls based on the triggered monitor. By handling this under the hood, we prevent the LLM from having to calculate timestamps (and potentially hallucinating them).\n- `ConcludeWithinBudgetMiddleware` : A smarter version of the model call limiter. Instead of abruptly killing the agent mid-thought when it hits a limit, it instructs the agent to generate a best-effort conclusion based on the data it has gathered so far.\n\n### Structured LLM Responses\n\nHistorically, extracting structured objects from LLMs was an error-prone headache. Today, LangChain makes it trivial. By simply passing a Pydantic model as a parameter into the create_agent method, you guarantee the exact shape of your final response. Under the hood, LangChain automatically routes this request through the LLM’s structured output API. If the specific model doesn't have one, it falls back to using the LLM’s tool-calling API to enforce your schema.\n\n```\nfrom pydantic import BaseModel, Field \nfrom langchain.agents import create_agent \n...\nclass CoffeeOrder(BaseModel): \n    drink_type: str = Field(description=\"The main coffee drink\") \n    milk: str = Field(description=\"Type of milk requested\") \n    shots: int = Field(description=\"Total number of espresso shots\") \n    is_iced: bool = Field(description=\"True if the user wants it iced\") \n...\n    agent = create_agent(\n        llm,\n        system_prompt='Extract customers coffee order from their message.', \n        response_format=CoffeeOrder\n    )\n    result = agent.invoke({ \"messages\": [(\"user\", users_order)]})\n    order: CoffeeOrder = result[\"structured_response\"]\n```\n### The LangGraph Framework\n\nAt its core, LangGraph defines an agent as a directed cyclic graph. Every **node** in this graph is just a plain Python function that takes in the current state, executes a step, and returns updates to that state.\n\nThe **state** itself is a `TypedDict` that acts as the agent's global memory. Crucially, *state is not the same as the LLM context window*. You typically only map a small, relevant portion of the global state into the LLM's context for any given call. Because this state is fully serializable, LangGraph saves a snapshot after every single node executes. This lets you pause and resume execution for human-in-the-loop approvals, recover from crashes, or even \"rewind\" the agent to a previous step.\n\n```\nclass InvestigationState(TypedDict):\n    request_text: str | None\n    triggered_alert: dict[str, Any] | None\n    playbook: str | None\n    alert_context: AlertContext | None\n    findings: dict[str, Any] | None\n    report: str | None\n```\nTo keep your top-level graph clean, the best practice is to encapsulate complex logic into a sub-graph and drop it in as a single node.\n\nFor example, the `create_agent` method we used earlier actually returns a `CompiledStateGraph`, which is a complete sub-graph that you can drop right into your main graph. By design, sub-graphs do not automatically inherit the parent’s state. This isolation keeps your architecture clean: the parent graph doesn't get cluttered with the child's messy internal variables, and the child doesn't waste tokens loading the parent's entire chat history.\n\n### Inside The Bronto Investigation Agent\n\nOur open-source agent is built around a five-node graph. Rather than making every single step a complex agent, we selectively mix full ReAct loops with simple programmatic steps to keep execution fast and efficient:\n\n1. `identify` : Uses a full**ReAct loop** (as described above) to search the audit log and pinpoint the alert matching the user’s description.\n2. `give_up` : A logic node that gracefully exits with an error if an alert matching the user’s description could not be found\n3. `context` : Gathers system context for the investigation, e.g, from the monitor details, triggered alert details, and the underlying log data.\n4. `investigate` : The core engine. This node runs another**ReAct loop** to drill into the logs and perform the actual Root Cause Analysis (RCA).\n5. `report_generation` : A simple, single LLM call that takes the RCA findings and formats them into the final structured report.\n\nThe `context` node is arguably the most valuable piece of this architecture, because it is the key thing that differentiates the agent compared to a simple ReAct loop with Bronto’s MCP tools.\n\nThe context generation node assembles the things an SRE would look at first, and it is a largely deterministic, programmatic phase with minimal LLM calls and no ReAct loops. This portion of the agent has relatively more freedom for data discovery and exploration, with the later “investigate” phase being more strictly restricted to investigating.\n\nDuring this phase, the agent gathers:\n\n- details on the triggered alert and underlying monitor,\n- trends on the monitor’s state and the underlying data around the time when the monitor triggered,\n- datasets that are not directly acted upon by the monitor but which are likely related when things break (e.g., service dependencies).\n\n### Instrumenting Your Agent\n\nBuilding an agent is only half the battle; you also need to see exactly what it’s doing in production. Fortunately, there is a very simple way to instrument your LangGraph agent. By using the `opentelemetry` libraries, you can auto-instrument your entire agent with detailed tracing in just a few lines:\n\n```\n...\nfrom opentelemetry.instrumentation.genai.langchain import LangChainInstrumentor\nfrom opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor\n...\n# Invoke these at the start of your agent\n# Auto-instruments LangChain and LangGraph calls\nLangChainInstrumentor().instrument() \n# Auto-instruments underlying HTTP/REST requests (including remote MCP calls) HTTPXClientInstrumentor().instrument()\n...\nfrom opentelemetry.instrumentation.genai.langchain import LangChainInstrumentor\n```\nAnd you can use the standard OpenTelemetry SDK to export this data directly to Bronto (or any OTEL-compatible backend) for collection and analysis:\n\n```\nfrom opentelemetry import trace \nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\nfrom opentelemetry.sdk.resources import Resource \nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\n...\n# Invoke at the start of your agent\n# OTLPSpanExporter automatically picks up OTEL_EXPORTER_OTLP_ENDPOINT \n# and OTEL_EXPORTER_OTLP_HEADERS from the environment variables. \nprovider = TracerProvider(resource=Resource.create({\"service.name\": SERVICE_NAME})) provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider)\n...\n```\nPutting it all together, here is how we modify that trivial ~20-line agent from the beginning of this post so it automatically sends its traces straight into Bronto:\n\n```\nimport asyncio\nimport os\nimport sys\nfrom bronto_sdk import mcp as bronto_mcp\nfrom langchain_mcp_adapters.sessions import create_session\nfrom langchain_mcp_adapters.tools import load_mcp_tools\nfrom langchain.agents import create_agent\nfrom langchain.chat_models import init_chat_model\nfrom opentelemetry import trace\nfrom opentelemetry.instrumentation.genai.langchain import LangChainInstrumentor\nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\nfrom opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor\nfrom opentelemetry.sdk.resources import Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nasync def investigate(user_prompt: str):\n    api_key = os.environ.get(\"BRONTO_API_KEY\")\n    bronto_region = os.environ.get(\"BRONTO_REGION\")\n    async with create_session({\n        \"transport\": \"streamable_http\",\n        \"url\": bronto_mcp.url(bronto_region),\n        \"headers\": bronto_mcp.auth_headers(api_key=api_key)\n    }) as session:\n        await session.initialize()\n        mcp_tools = await load_mcp_tools(session)\n        llm = init_chat_model(\"claude-sonnet-5\", model_provider=\"anthropic\")\n        agent = create_agent(llm, tools=mcp_tools)\n        result = await agent.ainvoke({\"messages\": [(\"user\", user_prompt)]})\n        print(result[\"messages\"][-1].content)\nLangChainInstrumentor().instrument()\nHTTPXClientInstrumentor().instrument()\nprovider = TracerProvider(resource=Resource.create({\"service.name\": \"myAgent\"}))\nprovider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))\ntrace.set_tracer_provider(provider)\nasyncio.run(investigate(sys.argv[1]))\n```\nNow by setting the `OTEL_EXPORTER_OTLP_ENDPOINT=https://ingestion.<region>.bronto.io` and `OTEL_EXPORTER_OTLP_HEADERS=x-bronto-api-key=<your Bronto api key>` environment variables you can push the agent's tracing data straight into Bronto to examine and monitor its exact reasoning steps:\n\n### Stop Guessing, Start Building\n\nOutsourcing tasks to AI is easy; building an agent you actually trust to investigate a production outage is hard. Or at least, it used to be.\n\nBy combining LangGraph’s deterministic state machines with Bronto’s powerful MCP tools, you now have the architecture to build agents that gather context like a human, reason within strict guardrails, recover from errors gracefully, and report their exact telemetry via standard OpenTelemetry.\n\nWe open-sourced the Alert Investigation agent so you don't have to start from scratch. Use it as a standalone tool to reduce your MTTR today, or use it as the foundation for your team's next big internal tool.\n\nSign up for a Bronto [__free trial here__](https://bronto.io/signup) and head over to the [__GitHub repo__](https://github.com/bronto-community/bronto-alert-investigation-agent) to give the agent a spin. Don’t forget to let us know what you build!\n\nHead over to the [__GitHub repo__](https://github.com/bronto-community/bronto-alert-investigation-agent) to get started.\n\nEnjoyed this post?\n\nShare it with your team or network.\n\n### Try Bronto free for 14 days\n\nCentralize your agent and infrastructure telemetry in one platform with sub-second search and 12-month hot retention.\n\n- 14-day free trial\n- No credit card\n- Set up in minutes","body_html":"<h1 id=\"creating-custom-agents-inside-bronto-s-alert-investigation-agent\">Creating Custom Agents: Inside Bronto’s Alert Investigation Agent</h1>\n<p>Nikolai Gladychev</p>\n<p>Senior Software Engineer</p>\n<p>Using generalist agents (e.g., Claude Code) with Bronto’s MCP server is powerful and effective, but for well-defined and repeated tasks, specialised <em>custom</em> agents can offer significant advantages including improved reliability, faster execution, and better token efficiency. It is becoming increasingly common for teams to write specialised agents for SRE workloads, and we want to make it <em>effortless</em> for people to write their own custom agents with Bronto. </p>\n<p>We <a href=\"https://bronto.io/blog/building-a-log-analysis-agent-with-langgraph-and-brontos-mcp-server\" rel=\"nofollow ugc noopener\"><strong>previously showed</strong></a> how a basic LangGraph agent could leverage our MCP server, and now we’ve released <a href=\"https://github.com/bronto-community/bronto-alert-investigation-agent\" rel=\"nofollow ugc noopener\"><strong>an open source alert investigation agent</strong></a> that replicates our in-platform <a href=\"https://bronto.io/blog/investigation-reports-when-monitors-get-smarter\" rel=\"nofollow ugc noopener\"><strong>Alert Investigation feature</strong></a>. It is a small reference project that you (or your agent) can use to see the best patterns for integrating Bronto into your LangGraph agents.</p>\n<p>Below, I’ll demonstrate how <em>easy</em> it is to write custom agents with Bronto’s powerful set of MCP tools, and then dive into the architecture and code that make our open source alert investigation agent work. If you want to learn more about what our agent does and how to use it, check out our <a href=\"https://bronto.io/blog/introducing-brontos-open-source-alert-investigation-agent\" rel=\"nofollow ugc noopener\">announcement post</a>. Don&#39;t worry if you&#39;re not a LangGraph expert, we&#39;ll show you in detail how we used it to build our agent. We will also show how easy it is to send telemetry data from the agent to Bronto.</p>\n<h3 id=\"building-custom-agents-with-langgraph-and-bronto-mcp\">Building Custom Agents with LangGraph and Bronto MCP</h3>\n<p>LangChain and LangGraph are two powerful and complementary libraries for wiring LLM calls into structured state machines. To make that process even smoother, we’ve released the <a href=\"https://github.com/bronto-community/bronto-sdk-python\" rel=\"nofollow ugc noopener\"><strong>Bronto SDK for Python</strong></a>: a small, typed client for the Bronto REST API, plus helpers that encode Bronto-specific details. Combine these tools, and you can build a remarkably effective agent connected to Bronto&#39;s MCP in about 20 lines of code:</p>\n<pre><code>import asyncio\nimport os\nimport sys\nfrom bronto_sdk import mcp as bronto_mcp\nfrom langchain_mcp_adapters.sessions import create_session\nfrom langchain_mcp_adapters.tools import load_mcp_tools\nfrom langchain.agents import create_agent\nfrom langchain.chat_models import init_chat_model\nasync def investigate(user_prompt: str):\n    api_key = os.environ.get(&quot;BRONTO_API_KEY&quot;)\n    bronto_region = os.environ.get(&quot;BRONTO_REGION&quot;)\n    async with create_session({\n        &quot;transport&quot;: &quot;streamable_http&quot;,\n        &quot;url&quot;: bronto_mcp.url(bronto_region),\n        &quot;headers&quot;: bronto_mcp.auth_headers(api_key=api_key)\n    }) as session:\n        await session.initialize()\n        mcp_tools = await load_mcp_tools(session)\n        llm = init_chat_model(&quot;claude-sonnet-5&quot;, model_provider=&quot;anthropic&quot;)\n        agent = create_agent(llm, tools=mcp_tools)\n        result = await agent.ainvoke({&quot;messages&quot;: [(&quot;user&quot;, user_prompt)]})\n        print(result[&quot;messages&quot;][-1].content)\nasyncio.run(investigate(sys.argv[1]))</code></pre>\n<p>With just this snippet, you have an agent that can handle complex, natural-language questions like, <em>&quot;What are the p99 response times on my API gateway, broken down by resource?&quot;</em></p>\n<p>| To run the sample code in this blog, ensure you have Python and <code>uv</code> installed on your machine, then create a temporary Python venv that has the required dependencies by running this command (on mac or linux): <code>tmp=$(mktemp -d) &amp;&amp; uv venv &quot;$tmp&quot; &amp;&amp; source &quot;$tmp/bin/activate&quot; &amp;&amp; uv pip install langgraph langchain langchain-core langchain-aws langchain-anthropic langchain-openai langchain-mcp-adapters pydantic opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-httpx opentelemetry-instrumentation-genai-langchain httpx https://github.com/bronto-community/bronto-sdk-python/releases/download/v0.1.2/bronto_sdk-0.1.2-py3-none-any.whl &amp;&amp; cd $tmp</code> | </p>\n<p>The magic in the python snippet above lies in LangGraph&#39;s <code>create_agent</code> method, which gives your LLM a built-in <strong>ReAct (Reason + Act)</strong> loop. Historically, you had to write a lot of boilerplate to give an agent a set of tools, then make it look at its context, execute a tool, re-evaluate, execute a tool, etc.., and loop until the task was done. Now, <code>create_agent</code> handles that entire lifecycle.</p>\n<p>To connect that agent to Bronto, you can use the <code>langchain_mcp_adapters</code> library. Looking at the code above, it takes just three lines to connect:</p>\n<ol><li>Open a connection to Bronto’s remote MCP using  <code>create_session</code> .</li><li>Complete the required MCP handshake using <code>session.initialize()</code> .</li><li>Load Bronto&#39;s MCP as LangChain-native tools using <code>load_mcp_tools(session)</code> .</li></ol>\n<h3 id=\"controlling-tool-use\">Controlling Tool Use</h3>\n<p>A very typical thing you might want to do, is to restrict the tools used by an agent. By restricting a ReAct loop to a specific subset of MCP tools, you give the LLM only what it needs for a particular step. This reduces the chance of the agent making an error by picking the wrong tool and it also reduces the context window size:</p>\n<pre><code>ALLOWED_TOOLS = {&quot;get_datasets&quot;, &quot;get_keys&quot;, &quot;search_logs&quot;, &quot;timeseries&quot;}\n...\n        all_mcp_tools = await load_mcp_tools(session)\n        mcp_tools = [t for t in all_mcp_tools if t.name in ALLOWED_TOOLS]\n        agent = create_agent(llm, tools=mcp_tools)</code></pre>\n<h3 id=\"using-langraph-middleware\">Using Langraph Middleware</h3>\n<p>You can strictly control the internal state of a LangChain ReAct agent using its Middleware system. This allows you to intercept the agent mid-loop to modify LLM requests, insert human-in-the-loop approvals, or handle errors automatically. LangChain ships with several built-in middlewares for common use cases:</p>\n<pre><code>from langchain.agents.middleware import ToolRetryMiddleware, ModelCallLimitMiddleware\n...\n    agent = create_agent(\n                llm,\n                tools=mcp_tools,\n                middleware=[\n                    ToolRetryMiddleware(max_retries=2, on_failure=&quot;continue&quot;),\n                    ModelCallLimitMiddleware(run_limit=5, exit_behavior=&quot;end&quot;)\n                ]\n            )</code></pre>\n<p>In this snippet, there are two standard middlewares:</p>\n<ul><li><code>ToolRetryMiddleware</code> : Automatically retries failed tool calls. If it continues to fail, it feeds the error details back to the LLM so the agent can decide if it should try a different approach.</li><li><code>ModelCallLimitMiddleware</code> : Prevents excessive agent loops by hard-terminating the agent if it exceeds a set number of LLM calls.</li></ul>\n<p>The Bronto open-source investigation agent also uses some custom middleware such as:</p>\n<ul><li><code>InvestigationWindowMiddleware</code> : Automatically injects the correct time-range parameters into tool calls based on the triggered monitor. By handling this under the hood, we prevent the LLM from having to calculate timestamps (and potentially hallucinating them).</li><li><code>ConcludeWithinBudgetMiddleware</code> : A smarter version of the model call limiter. Instead of abruptly killing the agent mid-thought when it hits a limit, it instructs the agent to generate a best-effort conclusion based on the data it has gathered so far.</li></ul>\n<h3 id=\"structured-llm-responses\">Structured LLM Responses</h3>\n<p>Historically, extracting structured objects from LLMs was an error-prone headache. Today, LangChain makes it trivial. By simply passing a Pydantic model as a parameter into the create_agent method, you guarantee the exact shape of your final response. Under the hood, LangChain automatically routes this request through the LLM’s structured output API. If the specific model doesn&#39;t have one, it falls back to using the LLM’s tool-calling API to enforce your schema.</p>\n<pre><code>from pydantic import BaseModel, Field \nfrom langchain.agents import create_agent \n...\nclass CoffeeOrder(BaseModel): \n    drink_type: str = Field(description=&quot;The main coffee drink&quot;) \n    milk: str = Field(description=&quot;Type of milk requested&quot;) \n    shots: int = Field(description=&quot;Total number of espresso shots&quot;) \n    is_iced: bool = Field(description=&quot;True if the user wants it iced&quot;) \n...\n    agent = create_agent(\n        llm,\n        system_prompt=&#39;Extract customers coffee order from their message.&#39;, \n        response_format=CoffeeOrder\n    )\n    result = agent.invoke({ &quot;messages&quot;: [(&quot;user&quot;, users_order)]})\n    order: CoffeeOrder = result[&quot;structured_response&quot;]</code></pre>\n<h3 id=\"the-langgraph-framework\">The LangGraph Framework</h3>\n<p>At its core, LangGraph defines an agent as a directed cyclic graph. Every <strong>node</strong> in this graph is just a plain Python function that takes in the current state, executes a step, and returns updates to that state.</p>\n<p>The <strong>state</strong> itself is a <code>TypedDict</code> that acts as the agent&#39;s global memory. Crucially, <em>state is not the same as the LLM context window</em>. You typically only map a small, relevant portion of the global state into the LLM&#39;s context for any given call. Because this state is fully serializable, LangGraph saves a snapshot after every single node executes. This lets you pause and resume execution for human-in-the-loop approvals, recover from crashes, or even &quot;rewind&quot; the agent to a previous step.</p>\n<pre><code>class InvestigationState(TypedDict):\n    request_text: str | None\n    triggered_alert: dict[str, Any] | None\n    playbook: str | None\n    alert_context: AlertContext | None\n    findings: dict[str, Any] | None\n    report: str | None</code></pre>\n<p>To keep your top-level graph clean, the best practice is to encapsulate complex logic into a sub-graph and drop it in as a single node.</p>\n<p>For example, the <code>create_agent</code> method we used earlier actually returns a <code>CompiledStateGraph</code>, which is a complete sub-graph that you can drop right into your main graph. By design, sub-graphs do not automatically inherit the parent’s state. This isolation keeps your architecture clean: the parent graph doesn&#39;t get cluttered with the child&#39;s messy internal variables, and the child doesn&#39;t waste tokens loading the parent&#39;s entire chat history.</p>\n<h3 id=\"inside-the-bronto-investigation-agent\">Inside The Bronto Investigation Agent</h3>\n<p>Our open-source agent is built around a five-node graph. Rather than making every single step a complex agent, we selectively mix full ReAct loops with simple programmatic steps to keep execution fast and efficient:</p>\n<ol><li><code>identify</code> : Uses a full<strong>ReAct loop</strong> (as described above) to search the audit log and pinpoint the alert matching the user’s description.</li><li><code>give_up</code> : A logic node that gracefully exits with an error if an alert matching the user’s description could not be found</li><li><code>context</code> : Gathers system context for the investigation, e.g, from the monitor details, triggered alert details, and the underlying log data.</li><li><code>investigate</code> : The core engine. This node runs another<strong>ReAct loop</strong> to drill into the logs and perform the actual Root Cause Analysis (RCA).</li><li><code>report_generation</code> : A simple, single LLM call that takes the RCA findings and formats them into the final structured report.</li></ol>\n<p>The <code>context</code> node is arguably the most valuable piece of this architecture, because it is the key thing that differentiates the agent compared to a simple ReAct loop with Bronto’s MCP tools.</p>\n<p>The context generation node assembles the things an SRE would look at first, and it is a largely deterministic, programmatic phase with minimal LLM calls and no ReAct loops. This portion of the agent has relatively more freedom for data discovery and exploration, with the later “investigate” phase being more strictly restricted to investigating.</p>\n<p>During this phase, the agent gathers:</p>\n<ul><li>details on the triggered alert and underlying monitor,</li><li>trends on the monitor’s state and the underlying data around the time when the monitor triggered,</li><li>datasets that are not directly acted upon by the monitor but which are likely related when things break (e.g., service dependencies).</li></ul>\n<h3 id=\"instrumenting-your-agent\">Instrumenting Your Agent</h3>\n<p>Building an agent is only half the battle; you also need to see exactly what it’s doing in production. Fortunately, there is a very simple way to instrument your LangGraph agent. By using the <code>opentelemetry</code> libraries, you can auto-instrument your entire agent with detailed tracing in just a few lines:</p>\n<pre><code>...\nfrom opentelemetry.instrumentation.genai.langchain import LangChainInstrumentor\nfrom opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor\n...\n# Invoke these at the start of your agent\n# Auto-instruments LangChain and LangGraph calls\nLangChainInstrumentor().instrument() \n# Auto-instruments underlying HTTP/REST requests (including remote MCP calls) HTTPXClientInstrumentor().instrument()\n...\nfrom opentelemetry.instrumentation.genai.langchain import LangChainInstrumentor</code></pre>\n<p>And you can use the standard OpenTelemetry SDK to export this data directly to Bronto (or any OTEL-compatible backend) for collection and analysis:</p>\n<pre><code>from opentelemetry import trace \nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\nfrom opentelemetry.sdk.resources import Resource \nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\n...\n# Invoke at the start of your agent\n# OTLPSpanExporter automatically picks up OTEL_EXPORTER_OTLP_ENDPOINT \n# and OTEL_EXPORTER_OTLP_HEADERS from the environment variables. \nprovider = TracerProvider(resource=Resource.create({&quot;service.name&quot;: SERVICE_NAME})) provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter())) trace.set_tracer_provider(provider)\n...</code></pre>\n<p>Putting it all together, here is how we modify that trivial ~20-line agent from the beginning of this post so it automatically sends its traces straight into Bronto:</p>\n<pre><code>import asyncio\nimport os\nimport sys\nfrom bronto_sdk import mcp as bronto_mcp\nfrom langchain_mcp_adapters.sessions import create_session\nfrom langchain_mcp_adapters.tools import load_mcp_tools\nfrom langchain.agents import create_agent\nfrom langchain.chat_models import init_chat_model\nfrom opentelemetry import trace\nfrom opentelemetry.instrumentation.genai.langchain import LangChainInstrumentor\nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\nfrom opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor\nfrom opentelemetry.sdk.resources import Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nasync def investigate(user_prompt: str):\n    api_key = os.environ.get(&quot;BRONTO_API_KEY&quot;)\n    bronto_region = os.environ.get(&quot;BRONTO_REGION&quot;)\n    async with create_session({\n        &quot;transport&quot;: &quot;streamable_http&quot;,\n        &quot;url&quot;: bronto_mcp.url(bronto_region),\n        &quot;headers&quot;: bronto_mcp.auth_headers(api_key=api_key)\n    }) as session:\n        await session.initialize()\n        mcp_tools = await load_mcp_tools(session)\n        llm = init_chat_model(&quot;claude-sonnet-5&quot;, model_provider=&quot;anthropic&quot;)\n        agent = create_agent(llm, tools=mcp_tools)\n        result = await agent.ainvoke({&quot;messages&quot;: [(&quot;user&quot;, user_prompt)]})\n        print(result[&quot;messages&quot;][-1].content)\nLangChainInstrumentor().instrument()\nHTTPXClientInstrumentor().instrument()\nprovider = TracerProvider(resource=Resource.create({&quot;service.name&quot;: &quot;myAgent&quot;}))\nprovider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))\ntrace.set_tracer_provider(provider)\nasyncio.run(investigate(sys.argv[1]))</code></pre>\n<p>Now by setting the <code>OTEL_EXPORTER_OTLP_ENDPOINT=https://ingestion.&lt;region&gt;.bronto.io</code> and <code>OTEL_EXPORTER_OTLP_HEADERS=x-bronto-api-key=&lt;your Bronto api key&gt;</code> environment variables you can push the agent&#39;s tracing data straight into Bronto to examine and monitor its exact reasoning steps:</p>\n<h3 id=\"stop-guessing-start-building\">Stop Guessing, Start Building</h3>\n<p>Outsourcing tasks to AI is easy; building an agent you actually trust to investigate a production outage is hard. Or at least, it used to be.</p>\n<p>By combining LangGraph’s deterministic state machines with Bronto’s powerful MCP tools, you now have the architecture to build agents that gather context like a human, reason within strict guardrails, recover from errors gracefully, and report their exact telemetry via standard OpenTelemetry.</p>\n<p>We open-sourced the Alert Investigation agent so you don&#39;t have to start from scratch. Use it as a standalone tool to reduce your MTTR today, or use it as the foundation for your team&#39;s next big internal tool.</p>\n<p>Sign up for a Bronto <a href=\"https://bronto.io/signup\" rel=\"nofollow ugc noopener\"><strong>free trial here</strong></a> and head over to the <a href=\"https://github.com/bronto-community/bronto-alert-investigation-agent\" rel=\"nofollow ugc noopener\"><strong>GitHub repo</strong></a> to give the agent a spin. Don’t forget to let us know what you build!</p>\n<p>Head over to the <a href=\"https://github.com/bronto-community/bronto-alert-investigation-agent\" rel=\"nofollow ugc noopener\"><strong>GitHub repo</strong></a> to get started.</p>\n<p>Enjoyed this post?</p>\n<p>Share it with your team or network.</p>\n<h3 id=\"try-bronto-free-for-14-days\">Try Bronto free for 14 days</h3>\n<p>Centralize your agent and infrastructure telemetry in one platform with sub-second search and 12-month hot retention.</p>\n<ul><li>14-day free trial</li><li>No credit card</li><li>Set up in minutes</li></ul>","headings":[{"level":1,"text":"Creating Custom Agents: Inside Bronto’s Alert Investigation Agent","id":"creating-custom-agents-inside-bronto-s-alert-investigation-agent"},{"level":3,"text":"Building Custom Agents with LangGraph and Bronto MCP","id":"building-custom-agents-with-langgraph-and-bronto-mcp"},{"level":3,"text":"Controlling Tool Use","id":"controlling-tool-use"},{"level":3,"text":"Using Langraph Middleware","id":"using-langraph-middleware"},{"level":3,"text":"Structured LLM Responses","id":"structured-llm-responses"},{"level":3,"text":"The LangGraph Framework","id":"the-langgraph-framework"},{"level":3,"text":"Inside The Bronto Investigation Agent","id":"inside-the-bronto-investigation-agent"},{"level":3,"text":"Instrumenting Your Agent","id":"instrumenting-your-agent"},{"level":3,"text":"Stop Guessing, Start Building","id":"stop-guessing-start-building"},{"level":3,"text":"Try Bronto free for 14 days","id":"try-bronto-free-for-14-days"}]}}