Conduit is a local proxy that sits between your code and LLM providers
recording billable usage.
Point SDK's base URL at Conduit, and get detailed
tracking with zero code changes.
Single binary, written in Rust.
What Conduit allows you to do
- run a proxy locally to intercept API calls
- set up one or more LLM API providers to forward the API calls to
- view the token consumption of each API call in a terminal UI
- send information to Valohai LLM for further cost analysis
Conduit is available through:
- installer script or prebuilt binary from GitHub Releases
cargo install valohai-conduitfrom crates.io (requires Rust)
You can confirm that Conduit is installed with:
conduit --help
# Usage: conduit [OPTIONS] [COMMAND]
#
# Commands:
# proxy Run the proxy server [default command]
# dashboard Open the terminal-based dashboard
# help Print this message or the help of the given subcommand(s)Conduit uses a conduit.toml file as its configuration file.
By default Conduit reads conduit.toml from the working directory. To keep the config somewhere else, pass --config <path> (-c) or set the CONDUIT_CONFIG environment variable. Pass it to every conduit command, or export CONDUIT_CONFIG, so conduit proxy and conduit dashboard read the same config:
conduit --config ~/.config/conduit/conduit.toml proxyconduit proxy won't start without a config that defines at least one [providers.<name>] section.
You can copy the example from here and adjust where Conduit will be listening to LLM calls and setup the providers you are using.
Conduit records the LLM calls it proxies to a SQLite database, conduit.db in the working directory by default. Set database in conduit.toml or the CONDUIT_DATABASE environment variable to keep it elsewhere, like a state directory when running Conduit as a service. Conduit creates the file but not its directory. conduit dashboard reads the same setting, so point both at the same file:
database = "/var/lib/conduit/conduit.db"Conduit can capture cost values from responses formatted in the common Anthropic (Messages API) and OpenAI (Responses & Chat Completions APIs) or similar responses from other models.
Costs are estimated from a built-in price table of OpenAI and Anthropic models. For any other model (or your own rates), add a [pricing] table to conduit.toml; each key is a model name prefix, and these prices are checked first:
[pricing."jev-"]
input = 0.042 # USD per million tokens
output = 0.0input and output are required. The optional cached_input, cache_write_5m and cache_write_1h price prompt cache reads and 5-minute and 1-hour cache writes, and default to 0.1x, 1.25x and 2x of input; set them for models whose cache rates differ. cached_input also prices OpenAI cached tokens, where leaving it out means no discount instead of 0.1x. For models that charge more for long prompts, an optional [pricing."<prefix>".long_prompt] table takes tokens, the prompt length above which it applies, and the same rate keys; a request over it pays those rates for all of its tokens, cache reads and writes included.
Requests to providers have no total time limit, so long streamed responses are never cut. Instead, an optional [upstream] table sets how long Conduit waits to connect and how long a response may go without sending any data, in seconds; 0 disables either one:
[upstream]
connect_timeout_s = 10 # default
idle_timeout_s = 600 # defaultOnce a configuration file exists, you can run Conduit in your working directory with the command:
conduit proxyOnce Conduit is running, point your SDK's base URL at it. The URL is http://<proxy-address>/<provider_key> where <provider_key> matches a [providers.<key>] section in your conduit.toml.
Set the base URL environment variable for your provider SDK:
# NB: OpenAI-like providers also need the `/v1` path suffix:
export OPENAI_BASE_URL=http://localhost:8080/openai/v1
export ANTHROPIC_BASE_URL=http://localhost:8080/anthropicMost SDKs automatically pick up the above env vars with no code changes needed.
Alternatively, pass the base API url directly, e.g. with common Python SDKs:
client = OpenAI(base_url="http://localhost:8080/openai/v1")
client = Anthropic(base_url="http://localhost:8080/anthropic")See examples/ for complete Python usage examples.
With conduit proxy running, send a request through it. You don't need an API key for this check:
curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:8080/anthropic/v1/modelsReplace anthropic with the name of one of your [providers.<name>] sections.
401: the request went through Conduit and reached the provider, which rejected it for having no key404: usually the provider name in the URL doesn't match a[providers.<name>]section502: Conduit is running but couldn't reach the provider; check itsupstreamURL and your network000with aFailed to connecterror: Conduit isn't running or is listening on another address
Run your code with real keys, then open conduit dashboard to see tokens and cost per call.
See the contributing guide to get started.