Neuro SAN: Seeing What Agents Are Up To

A quick look at the tools Neuro SAN offers to help developers understand, debug, and improve multi-agent AI behavior in real time.

So you’ve got all these agents hooked up and talking to one another, but as a whole, what are they actually doing?

Among the Neuro SAN suite of tools are a few clients to help out.

• For the developer perspective, there is the nsflow tool which shows a visualization of the agent network graph with some observability as to the chatter. This is a python tool suitable for single developers.
• Another tool is the agent_cli which comes with the neuro-san toolkit. This is a nuts-and-bolts command line tool, again for an individual developer.
• Our team is also working on a multi-user React/NodeJS web interface to visualize agent networks in action. We hope to make this publicly available when its ready (see this article’s headline image).

But maybe your agents aren’t doing quite what you want. A fancy lightshow is impressive, but how do you figure out where things are going wrong (or not-wrong, aka “right”)?

In a single-agent system, one resorts to tweaking prompts to get better results, and Neuro SAN multi-agent networks (alas!) are no exception here. And now, the task is seemingly compounded as there are now multiple prompts to manage.

In previous articles we mentioned that each agent in a Neuro SAN network has its own private chat stream/history via its own connection to an LLM. The Neuro SAN system allows you to look at the inter-agent chatter by sending streams of chat messages back to a client to sort them out by their agent of origin as they happen, by asking for a MAXIMAL chat filter in a request.

Within each agent’s chat stream you can see:

• What its system prompt was
• What the values for its calling arguments were
• What other agents/tools it decided to invoke
• What those other agents returned to contribute to the specific agent’s task
• Some insights into its decision making (when available)

In the end you’ll see a round-up of what each agent decides to report upstream as well as these stats for that agent’s work:

• The time taken
• Token counts
• Cost estimation

Similar information is reported for Neuro SAN Coded Tools, giving you the opportunity to see (for instance) just how a web interface was invoked.

Armed with this information, you can begin the tradtional(!?) task of prompt engineering to get your agents behave better. It’s perhaps the New Drudgery, but in future articles I’ll talk about tools that are included with neuro-san that will allow you to unit test your agents and give you statistical sense for how well they are doing at the tasks you set them to.

Neuro SAN links:

• Home page
• GitHub library infra
• GitHub demo repo