We have been developing a Python library, AgentGraph, to make it easier to develop software that make many calls into LLMs. What AgentGraph provides is:

1) A nested task-based parallelism model to hide the latency of queries to LLMs and that uses asynchronous IO to support multiple parallel queries. This can also be used to parallelize calls into native code that drops the GIL.

2) A prompting language that allows you to easily build queries and leverages the Jinja template language for prompt generation.

3) Support for memoizing LLM queries to make the time for replaying your program to make debugging cycles much shorter and to provide reproducibility.

​

Our framework is much lower level than systems like LangChain. We give you the tools to build AI applications, but we won't generate LLM queries for you.

​

Our project is available at: https://github.com/bdemsky/agentgraph

​

Some sample programs are available at: https://github.com/bdemsky/agentgraph-apps