Skip to content
valohaiPublic

About

⚡️ local proxy for LLM token usage across providers, written in Rust

Topics

Resources

Contributing

Stars

13 stars

Watchers

0 watching

Forks

Repository files navigation

Valohai Conduit

License: MIT

Conduit is a local proxy that sits between your code and LLM providers recording billable usage.
Point SDK's base URL at Conduit, and get detailed tracking with zero code changes.

Single binary, written in Rust.

What Conduit allows you to do

  • run a proxy locally to intercept API calls
  • set up one or more LLM API providers to forward the API calls to
  • view the token consumption of each API call in a terminal UI
  • send information to Valohai LLM for further cost analysis

Install

Conduit is available through:

You can confirm that Conduit is installed with:

conduit --help

# Usage: conduit [OPTIONS] [COMMAND]
#
# Commands:
#   proxy      Run the proxy server [default command]
#   dashboard  Open the terminal-based dashboard
#   help       Print this message or the help of the given subcommand(s)

Configuration

Conduit uses a conduit.toml file as its configuration file.

By default Conduit reads conduit.toml from the working directory. To keep the config somewhere else, pass --config <path> (-c) or set the CONDUIT_CONFIG environment variable. Pass it to every conduit command, or export CONDUIT_CONFIG, so conduit proxy and conduit dashboard read the same config:

conduit --config ~/.config/conduit/conduit.toml proxy

conduit proxy won't start without a config that defines at least one [providers.<name>] section.

You can copy the example from here and adjust where Conduit will be listening to LLM calls and setup the providers you are using.

Conduit records the LLM calls it proxies to a SQLite database, conduit.db in the working directory by default. Set database in conduit.toml or the CONDUIT_DATABASE environment variable to keep it elsewhere, like a state directory when running Conduit as a service. Conduit creates the file but not its directory. conduit dashboard reads the same setting, so point both at the same file:

database = "/var/lib/conduit/conduit.db"

Conduit can capture cost values from responses formatted in the common Anthropic (Messages API) and OpenAI (Responses & Chat Completions APIs) or similar responses from other models.

Costs are estimated from a built-in price table of OpenAI and Anthropic models. For any other model (or your own rates), add a [pricing] table to conduit.toml; each key is a model name prefix, and these prices are checked first:

[pricing."jev-"]
input = 0.042    # USD per million tokens
output = 0.0

input and output are required. The optional cached_input, cache_write_5m and cache_write_1h price prompt cache reads and 5-minute and 1-hour cache writes, and default to 0.1x, 1.25x and 2x of input; set them for models whose cache rates differ. cached_input also prices OpenAI cached tokens, where leaving it out means no discount instead of 0.1x. For models that charge more for long prompts, an optional [pricing."<prefix>".long_prompt] table takes tokens, the prompt length above which it applies, and the same rate keys; a request over it pays those rates for all of its tokens, cache reads and writes included.

Requests to providers have no total time limit, so long streamed responses are never cut. Instead, an optional [upstream] table sets how long Conduit waits to connect and how long a response may go without sending any data, in seconds; 0 disables either one:

[upstream]
connect_timeout_s = 10   # default
idle_timeout_s = 600     # default

Once a configuration file exists, you can run Conduit in your working directory with the command:

conduit proxy

Usage

Once Conduit is running, point your SDK's base URL at it. The URL is http://<proxy-address>/<provider_key> where <provider_key> matches a [providers.<key>] section in your conduit.toml.

Set the base URL environment variable for your provider SDK:

# NB: OpenAI-like providers also need the `/v1` path suffix:
export OPENAI_BASE_URL=http://localhost:8080/openai/v1
export ANTHROPIC_BASE_URL=http://localhost:8080/anthropic

Most SDKs automatically pick up the above env vars with no code changes needed.

Alternatively, pass the base API url directly, e.g. with common Python SDKs:

client = OpenAI(base_url="http://localhost:8080/openai/v1")
client = Anthropic(base_url="http://localhost:8080/anthropic")

See examples/ for complete Python usage examples.

Check that it works

With conduit proxy running, send a request through it. You don't need an API key for this check:

curl -sS -o /dev/null -w "%{http_code}\n" http://localhost:8080/anthropic/v1/models

Replace anthropic with the name of one of your [providers.<name>] sections.

  • 401: the request went through Conduit and reached the provider, which rejected it for having no key
  • 404: usually the provider name in the URL doesn't match a [providers.<name>] section
  • 502: Conduit is running but couldn't reach the provider; check its upstream URL and your network
  • 000 with a Failed to connect error: Conduit isn't running or is listening on another address

Run your code with real keys, then open conduit dashboard to see tokens and cost per call.

Development

See the contributing guide to get started.

About

⚡️ local proxy for LLM token usage across providers, written in Rust

Topics

Resources

Contributing

Stars

13 stars

Watchers

0 watching

Forks

Releases

Used by

Contributors

Languages