Agents Observability: Smolagent and Open Telemetry

With the growing flexibility and use of AI agents, it is critical to understand the intermediate steps that lead to the final outcome in addition to the outcome itself.

Smolagents already provides the option of printing the intermediate steps on console by using verbose=True.

The task of printing and analyzing the output on the console is a little complicated and time-consuming. It is very difficult to inspect agent runs from console logs because, given the unpredictability of agents, the console quickly fills with random logs. By combining Smolagents with Open Telemetry, we can track, examine, and troubleshoot agent runs on the user interface.

What is OpenTelemetry?

OpenTelemetry, also known as OTel, is a vendor-neutral open source Observability framework for instrumenting, generating, collecting, and exporting telemetry data such as traces, metrics, and logs.

OpenTelemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior. It is a framework that is independent of both vendors and tools that adds the capability of comprehending a system's internal state by looking at its outputs.

Read more about open telemetry What is OpenTelemetry? | OpenTelemetry

Phoenix by ArizeAI

Phoenix is an OpenTelemetry compatible, open-source AI observability platform designed for experimentation, evaluation, and troubleshooting. It is built on top of OpenTelemetry and is vendor, language, and framework agnostic.

SetUp

Let us now go over how to set up Smolagents, Phoenix, and open telemetry for observability.

Start Phoenix Server

# install phoenix
pip install arize-phoenix

# start phoenix server locally on port 6006
phoenix serve

As an alternative, we can use Docker to launch the Phoenix server.

# pull the latest docker image
docker pull arizephoenix/phoenix

# start the phoenix server 6006 is UI port where 4317 is grpc port for OTLP trace collector
docker run -p 6006:6006 -p 4317:4317 -i -t arizephoenix/phoenix:latest