Skip to content

🐛 inference: use custom fetch for queue polling and result downloads - #2868

Open
abhiramshaji-cyber wants to merge 4 commits into
huggingface:mainfrom
abhiramshaji-cyber:fix/inference-custom-fetch-polling
Open

abhiramshaji-cyber wants to merge 4 commits into
huggingface:mainfrom
abhiramshaji-cyber:fix/inference-custom-fetch-polling

Conversation

@abhiramshaji-cyber

@abhiramshaji-cyber abhiramshaji-cyber commented Nov 15, 2026 •

Copy link
Copy Markdown

Hey @Wauplin, hope you're doing well, and thanks again for looking at #2820! Following up on that thread, here's the small PR I offered: it passes fetch through getResponse the same way signal was threaded in #2112.

A custom fetch passed in the options was only used for the first request of a call, and now queue polling and output downloads go through it too.

What this code is for

Each task in packages/inference/src/tasks sends the first request with innerRequest, which already uses options.fetch. It then hands the provider's reply to providerHelper.getResponse(...). For queue based providers (fal-ai, together, novita, wavespeed, zai-org) that method polls a status URL, fetches the result, and often downloads the output file. replicate and hf-inference download output files there too.

Before this PR

Someone calls client.textToImage({ provider: "fal-ai", ... }, { fetch: proxyFetch }).

  1. The provider mapping lookup and the submit go through proxyFetch.
  2. textToImage calls providerHelper.getResponse(res, url, headers, outputType, signal). No fetch is passed.
  3. getResponseFromQueueApi polls with the global fetch(statusUrl, { headers, signal }), with the auth header attached.
  4. The result request and fetch(result.images[0].url) also use the global fetch, so they skip proxyFetch entirely.

After this PR

Same call, same steps 1 and 2, except getResponse now also gets options?.fetch.

  1. getResponseFromQueueApi polls with fetchFn(statusUrl, { headers, signal }), where fetchFn is proxyFetch.
  2. The result request and the image download go through proxyFetch as well.
  3. Without a custom fetch, fetchFn defaults to the global fetch, so nothing changes for everyone else.

The diagram

flowchart LR

classDef box fill:#eef1ff,stroke:#4c5bd4,color:#111827
classDef bad fill:#ffd6d6,stroke:#cc0000,color:#111827,stroke-width:3px
classDef good fill:#c8f0c8,stroke:#008800,color:#111827,stroke-width:3px

subgraph BEFORE["BEFORE: 2 of 5 requests use the custom fetch"]
direction TB
  B1["textToImage with options.fetch"] --> B2["innerRequest submit via options.fetch"]
  B2 --> B3["getResponse(res, url, headers, outputType, signal)"]
  B3 --> B4{"fetch(statusUrl)"}
  B4 -->|"global fetch, auth header attached"| B5["polling, result, download skip the custom fetch"]
end

subgraph AFTER["AFTER: 5 of 5 requests use the custom fetch"]
direction TB
  A1["textToImage with options.fetch"] --> A2["innerRequest submit via options.fetch"]
  A2 --> A3["THE CHANGE: getResponse(..., signal, options.fetch)"]
  A3 --> A4{"fetchFn(statusUrl)"}
  A4 -->|"options.fetch given"| A5["polling, result, download use the custom fetch"]
  A4 -->|"no options.fetch"| A6["fetchFn defaults to global fetch, same as today"]
end

B1 ~~~ A1

class B1,B2,B3 box
class A1,A2,A4,A6 box
class B4,B5 bad
class A3,A5 good

linkStyle 3 stroke:#cc0000,stroke-width:5px
linkStyle 7 stroke:#008800,stroke-width:5px

style BEFORE fill:#fff5f5,stroke:#cc0000,color:#111827
style AFTER fill:#f3fff3,stroke:#008800,color:#111827
Loading

Provider files in packages/inference/src/providers with fal-ai.ts selected

The providers folder with fal-ai.ts and its queue methods, Code view in Architect

The change

  • providerHelper.ts: getResponse on TaskProviderHelper and the task helper interfaces gets an optional fetchFn?: typeof fetch after signal. ImageToTextTaskHelper.preparePayloadAsync gets the same.
  • Providers: each getResponse that makes a request takes fetchFn: typeof fetch = fetch and uses it for every status poll, result request and output download, in fal-ai (including getResponseFromQueueApi), together, novita, wavespeed, zai-org, replicate and hf-inference. zai-org's preparePayloadAsync uses it for the image URL download.
  • Tasks: every call that already forwarded options?.signal now also forwards options?.fetch.
  • Decoding of data: URLs (fetch(\data:image/jpeg;base64,...`)) stays on the global fetch, since it never leaves the process and a proxy fetch` may not handle that scheme.
  • New test/provider-custom-fetch.spec.ts, with the global fetch stubbed in each case:
    • textToImage on fal-ai end to end: status, result and image download all go through the custom fetch.
    • ZaiImageToTextTask.preparePayloadAsync called directly: the image URL download goes through the custom fetch.
    • textToImage on hf-inference returning an output URL: the download goes through the custom fetch.
    • textToImage on hf-inference returning b64_json: the data: decode stays on the global fetch, and the custom fetch never sees a data: URL.
    • imageToText on zai-org with a URL input: the image download goes through the custom fetch, which covers the wiring in tasks/cv/imageToText.ts.
    • The other four cases fail on main and pass here. The b64_json case passes on both, since it pins behavior that should not change.

What changes for a user

  • With a custom fetch (proxy, extra headers, logging), every request in a queue or download path now goes through it.
  • Without one, behavior is unchanged: fetchFn defaults to the global fetch.
  • The new parameter is optional and comes last, so existing direct callers of getResponse keep working.
  • loadTaskInfo in makeRequestOptions.ts still uses the global fetch; I left it alone since 🐛 Fix custom fetch for default inference model discovery #2759 covers that spot.

I went with an extra positional argument to match how signal was added. Would you rather have an options object there instead? Happy to switch, and any tips are very welcome.

Fixes #2820


Note

Low Risk
Mechanical fetch plumbing with backward-compatible defaults; limited behavior change except when callers pass a custom fetch, where networking becomes more consistent and predictable.

Overview
Custom options.fetch now applies to the full inference call path, not only the initial submit via innerRequest. Provider helpers accept an optional fetchFn (defaulting to global fetch) on getResponse and, for ZAI image-to-text, preparePayloadAsync; task entrypoints forward options?.fetch the same way they already forward options?.signal.

Queue and async providers (fal-ai, novita, together, wavespeed, zai-org) use fetchFn for status polling, result retrieval, and media downloads. Replicate and hf-inference use it for output URL fetches. In-process data: URL decoding (e.g. hf-inference b64_json) still uses global fetch so proxies are not required to handle data: schemes.

New provider-custom-fetch.spec.ts tests end-to-end propagation (fal-ai queue flow, hf-inference URL vs base64, ZAI image URL inputs) and asserts the global fetch stub stays unused when a custom fetch is supplied.

Reviewed by Cursor Bugbot for commit c19c58e. Bugbot is set up for automated code reviews on this repo. Configure here.

Pass Options.fetch to getResponse (and to the zai image input download in preparePayloadAsync), so status polling, result requests and output downloads in fal-ai, together, novita, wavespeed, zai-org, replicate and hf-inference go through the caller's fetch instead of the global one. Decoding of data URLs keeps using the global fetch.

Fixes huggingface#2820

@geraldaddey geraldaddey left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this up, and for the careful write-up on #2820. I walked through the issue and ran the change locally.

The bug is real. I stubbed global fetch, passed a custom one that counts calls, and ran textToImage against fal-ai with a queued response:

On main:

  • custom fetch (2): model mapping + the initial submit
  • global fetch (3): status poll, result request, and the image download

On this branch, all five of those go through the custom fetch and global fetch is unused.

The approach looks right to me: thread options.fetch into getResponse the same way signal was in #2112, default fetchFn to the global fetch, and leave in-process data: URL decoding on the global one. I also checked the remaining fetch( call sites under packages/inference/src. The only leftover network call is loadTaskInfo in makeRequestOptions.ts (around line 214), which is rightly left for #2759. The data: decodes in hf-inference, together, deepinfra, and nscale are unchanged, which is what we want.

On the options-object question: I'd keep the extra positional argument, to stay consistent with how signal was added.

What I ran in packages/inference (GitHub Actions Test/Lint are waiting for approval, so this is from a local checkout of 2c7b304):

  • pnpm test: 28 passed, 123 skipped (the live InferenceClient suite)
  • pnpm lint:check, pnpm check, and pnpm format:check all clean
  • the existing provider-abort-signal.spec.ts cases still pass, so defaulting fetchFn to global fetch doesn't break direct getResponse callers, and the abort signal is still passed through to polling

Nits, all on tests rather than the fix itself:

  • test/provider-custom-fetch.spec.ts is a good regression for the fal-ai queue path, plus a direct ZaiImageToTextTask.preparePayloadAsync call. A lot of the other call sites aren't locked in, though, and those would be easy to miss later. A couple of extra cases would help, for example:
    • a download-only provider, such as replicate textToVideo or hf-inference textToImage with an output URL
    • hf-inference textToImage returning b64_json, to pin that data: decoding stays on global fetch (packages/inference/src/providers/hf-inference.ts, around line 162)
    • the zai image-input path through imageToText(...) rather than preparePayloadAsync directly, so the wiring in packages/inference/src/tasks/cv/imageToText.ts (line 15) is covered too

None of that blocks the change. LGTM, with those test nits.

…decoding and zai imageToText

Adds three cases to provider-custom-fetch.spec.ts: hf-inference textToImage with an output URL downloads through the custom fetch, hf-inference b64_json data URL decoding stays on the global fetch, and imageToText on zai-org downloads the image input through the custom fetch.
@abhiramshaji-cyber

Copy link
Copy Markdown
Author

Hey @geraldaddey, hope you're doing well! Thank you so much for running this locally and for the call count breakdown on main vs the branch, it made it really clear what each test needed to pin down. I added all three in a090377:

  • hf-inference textToImage with an output URL: the download goes through the custom fetch and global fetch is never called
  • hf-inference textToImage returning b64_json: the data: decode stays on global fetch, and the custom fetch never sees a data: URL
  • imageToText on zai-org with a URL input: the image download goes through the custom fetch, so the wiring in tasks/cv/imageToText.ts is covered end to end

I picked hf-inference for the download case since it shares setup with the b64_json one. Happy to add a replicate textToVideo case too if you think it's worth it. I also merged main into the branch. Locally pnpm test is 31 passed, 123 skipped, and lint, check and format are clean. With the src changes reverted, four of the five cases fail; the b64_json one passes on both since it pins existing behavior.

@geraldaddey

Copy link
Copy Markdown

Thanks, these look great. I don't think the replicate case is needed since hf-inference already covers the download-only path. Still LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[inference] custom fetch isn't used for queue polling and result downloads

2 participants