Arize is one of the more popular platforms for understanding and improving how AI agents work. What follows is my hands-on with their open-source toolkit, as led by their handy self-paced tutorial (from which you see the code below), and executed locally in VS Code.

Key learnings include:

• Where in the agentic workflow one intervene to iterates (e.g. deep diving into specific steps vs re-running the whole thing)
• Importance of scoring evals in binary way vs continuous scores
• How to avoid hitting rate limits by being more intentional about concurrency guidelines and which models to choose (appreciating that this can negatively impact eval quality/scoring)

📋 Table of Contents

Agent Structure & Execution

• First, I brought in and executed their agentic workflow, a financial research analyst and writer agent, respectively.

"""Phoenix tracing tutorial: CrewAI financial research crew."""

from dotenv import load_dotenv
load_dotenv()

from phoenix.otel import register

tracer_provider = register(
    project_name="crewai-tracing-quickstart",
    auto_instrument=True,
)
from crewai import Agent, Crew, Process, Task
from crewai_tools import SerperDevTool

search_tool = SerperDevTool()

researcher = Agent(
    role="Financial Research Analyst",
    goal="Gather up-to-date financial data, trends, and news for the target companies or markets",
    backstory="""
        You are a Senior Financial Research Analyst.
    """,
    verbose=True,
    allow_delegation=False,
    max_iter=2,
    tools=[search_tool],
)

writer = Agent(
    role="Financial Report Writer",
    goal="Compile and summarize financial research into clear, actionable insights",
    backstory="""
        You are an experienced financial content writer.
    """,
    verbose=True,
    allow_delegation=True,
    max_iter=1
)
task1 = Task(
    description="""
        Research: {tickers}
        Focus on: {focus}
        Today's date is May 2026. Prioritize information from the last 6 months.
        Cite source URLs and dates for every claim.
    """,
    expected_output="Detailed financial research summary with web search findings",
    agent=researcher,
)

task2 = Task(
    description="Write a report based on the research above.",
    expected_output="A polished financial analysis report",
    agent=writer,
)
crew = Crew(
    agents=[researcher, writer],
    tasks=[task1, task2],
    verbose=True,
    process=Process.sequential,
)

user_inputs = {
    "tickers": "DDOG",
    "focus": "financial analysis and market outlook"
}

result = crew.kickoff(inputs=user_inputs)

• Search API Serper showed usage triggered for Google searches, incl. credits used

Trace Capture

• Arize Phoenix’s dash showed how many traces were captured (1), time it took (about 48 seconds)

• Clicking into a trace reveals further context on the workflow:
  • Overall cost (this was v cheap, less than 1 cent)
  • The two agents (analyst, writer) and their respective workflows
  • The number of tokens used by the agents (e.g. 238 to trigger the Serper run; 2,803 to analyze the results; 1,720 to write the report)

Evals - Generating a Richer Dataset

• Evals: How is our agent firing? How will it handle different tickers, financial analysis tasks? This calls for evals. First, we need a richer dataset to evaluate against. So, we add some code to our script:

test_queries = [
    {"tickers": "AAPL", "focus": "financial analysis and market outlook"},
    {"tickers": "AMZN", "focus": "profitability and market share"},
    {"tickers": "AAPL, MSFT", "focus": "comparative financial analysis"},
    {"tickers": "META, SNAP, PINS", "focus": "social media sector trends"},
    {"tickers": "RIVN", "focus": "financial health and viability"},
    {"tickers": "SNOW", "focus": "revenue growth trajectory"},
    {"tickers": "META", "focus": "latest developments and stock performance"},
    {"tickers": "AAPL, MSFT, GOOGL, AMZN, META", "focus": "big tech comparison and market outlook"},
    {"tickers": "AMC", "focus": "financial analysis and market sentiment"},
]