Repository navigation
🐛 inference: use custom fetch for queue polling and result downloads - #2868
abhiramshaji-cyber wants to merge 4 commits into
Conversation
Pass Options.fetch to getResponse (and to the zai image input download in preparePayloadAsync), so status polling, result requests and output downloads in fal-ai, together, novita, wavespeed, zai-org, replicate and hf-inference go through the caller's fetch instead of the global one. Decoding of data URLs keeps using the global fetch. Fixes huggingface#2820
geraldaddey
left a comment
There was a problem hiding this comment.
Thanks for putting this up, and for the careful write-up on #2820. I walked through the issue and ran the change locally.
The bug is real. I stubbed global fetch, passed a custom one that counts calls, and ran textToImage against fal-ai with a queued response:
On main:
- custom fetch (2): model mapping + the initial submit
- global fetch (3): status poll, result request, and the image download
On this branch, all five of those go through the custom fetch and global fetch is unused.
The approach looks right to me: thread options.fetch into getResponse the same way signal was in #2112, default fetchFn to the global fetch, and leave in-process data: URL decoding on the global one. I also checked the remaining fetch( call sites under packages/inference/src. The only leftover network call is loadTaskInfo in makeRequestOptions.ts (around line 214), which is rightly left for #2759. The data: decodes in hf-inference, together, deepinfra, and nscale are unchanged, which is what we want.
On the options-object question: I'd keep the extra positional argument, to stay consistent with how signal was added.
What I ran in packages/inference (GitHub Actions Test/Lint are waiting for approval, so this is from a local checkout of 2c7b304):
pnpm test: 28 passed, 123 skipped (the liveInferenceClientsuite)pnpm lint:check,pnpm check, andpnpm format:checkall clean- the existing
provider-abort-signal.spec.tscases still pass, so defaultingfetchFnto globalfetchdoesn't break directgetResponsecallers, and the abort signal is still passed through to polling
Nits, all on tests rather than the fix itself:
test/provider-custom-fetch.spec.tsis a good regression for the fal-ai queue path, plus a directZaiImageToTextTask.preparePayloadAsynccall. A lot of the other call sites aren't locked in, though, and those would be easy to miss later. A couple of extra cases would help, for example:- a download-only provider, such as replicate
textToVideoor hf-inferencetextToImagewith anoutputURL - hf-inference
textToImagereturningb64_json, to pin thatdata:decoding stays on globalfetch(packages/inference/src/providers/hf-inference.ts, around line 162) - the zai image-input path through
imageToText(...)rather thanpreparePayloadAsyncdirectly, so the wiring inpackages/inference/src/tasks/cv/imageToText.ts(line 15) is covered too
- a download-only provider, such as replicate
None of that blocks the change. LGTM, with those test nits.
…decoding and zai imageToText Adds three cases to provider-custom-fetch.spec.ts: hf-inference textToImage with an output URL downloads through the custom fetch, hf-inference b64_json data URL decoding stays on the global fetch, and imageToText on zai-org downloads the image input through the custom fetch.
|
Hey @geraldaddey, hope you're doing well! Thank you so much for running this locally and for the call count breakdown on
I picked hf-inference for the download case since it shares setup with the |
|
Thanks, these look great. I don't think the replicate case is needed since hf-inference already covers the download-only path. Still LGTM! |
Hey @Wauplin, hope you're doing well, and thanks again for looking at #2820! Following up on that thread, here's the small PR I offered: it passes
fetchthroughgetResponsethe same waysignalwas threaded in #2112.A custom
fetchpassed in the options was only used for the first request of a call, and now queue polling and output downloads go through it too.What this code is for
Each task in
packages/inference/src/taskssends the first request withinnerRequest, which already usesoptions.fetch. It then hands the provider's reply toproviderHelper.getResponse(...). For queue based providers (fal-ai, together, novita, wavespeed, zai-org) that method polls a status URL, fetches the result, and often downloads the output file. replicate and hf-inference download output files there too.Before this PR
Someone calls
client.textToImage({ provider: "fal-ai", ... }, { fetch: proxyFetch }).proxyFetch.textToImagecallsproviderHelper.getResponse(res, url, headers, outputType, signal). Nofetchis passed.getResponseFromQueueApipolls with the globalfetch(statusUrl, { headers, signal }), with the auth header attached.fetch(result.images[0].url)also use the globalfetch, so they skipproxyFetchentirely.After this PR
Same call, same steps 1 and 2, except
getResponsenow also getsoptions?.fetch.getResponseFromQueueApipolls withfetchFn(statusUrl, { headers, signal }), wherefetchFnisproxyFetch.proxyFetchas well.fetch,fetchFndefaults to the globalfetch, so nothing changes for everyone else.The diagram
flowchart LR classDef box fill:#eef1ff,stroke:#4c5bd4,color:#111827 classDef bad fill:#ffd6d6,stroke:#cc0000,color:#111827,stroke-width:3px classDef good fill:#c8f0c8,stroke:#008800,color:#111827,stroke-width:3px subgraph BEFORE["BEFORE: 2 of 5 requests use the custom fetch"] direction TB B1["textToImage with options.fetch"] --> B2["innerRequest submit via options.fetch"] B2 --> B3["getResponse(res, url, headers, outputType, signal)"] B3 --> B4{"fetch(statusUrl)"} B4 -->|"global fetch, auth header attached"| B5["polling, result, download skip the custom fetch"] end subgraph AFTER["AFTER: 5 of 5 requests use the custom fetch"] direction TB A1["textToImage with options.fetch"] --> A2["innerRequest submit via options.fetch"] A2 --> A3["THE CHANGE: getResponse(..., signal, options.fetch)"] A3 --> A4{"fetchFn(statusUrl)"} A4 -->|"options.fetch given"| A5["polling, result, download use the custom fetch"] A4 -->|"no options.fetch"| A6["fetchFn defaults to global fetch, same as today"] end B1 ~~~ A1 class B1,B2,B3 box class A1,A2,A4,A6 box class B4,B5 bad class A3,A5 good linkStyle 3 stroke:#cc0000,stroke-width:5px linkStyle 7 stroke:#008800,stroke-width:5px style BEFORE fill:#fff5f5,stroke:#cc0000,color:#111827 style AFTER fill:#f3fff3,stroke:#008800,color:#111827The providers folder with
fal-ai.tsand its queue methods, Code view in ArchitectThe change
providerHelper.ts:getResponseonTaskProviderHelperand the task helper interfaces gets an optionalfetchFn?: typeof fetchaftersignal.ImageToTextTaskHelper.preparePayloadAsyncgets the same.getResponsethat makes a request takesfetchFn: typeof fetch = fetchand uses it for every status poll, result request and output download, in fal-ai (includinggetResponseFromQueueApi), together, novita, wavespeed, zai-org, replicate and hf-inference. zai-org'spreparePayloadAsyncuses it for the image URL download.options?.signalnow also forwardsoptions?.fetch.data:URLs (fetch(\data:image/jpeg;base64,...`)) stays on the globalfetch, since it never leaves the process and a proxyfetch` may not handle that scheme.test/provider-custom-fetch.spec.ts, with the globalfetchstubbed in each case:textToImageon fal-ai end to end: status, result and image download all go through the customfetch.ZaiImageToTextTask.preparePayloadAsynccalled directly: the image URL download goes through the customfetch.textToImageon hf-inference returning anoutputURL: the download goes through the customfetch.textToImageon hf-inference returningb64_json: thedata:decode stays on the globalfetch, and the customfetchnever sees adata:URL.imageToTexton zai-org with a URL input: the image download goes through the customfetch, which covers the wiring intasks/cv/imageToText.ts.mainand pass here. Theb64_jsoncase passes on both, since it pins behavior that should not change.What changes for a user
fetch(proxy, extra headers, logging), every request in a queue or download path now goes through it.fetchFndefaults to the globalfetch.getResponsekeep working.loadTaskInfoinmakeRequestOptions.tsstill uses the globalfetch; I left it alone since 🐛 Fix custom fetch for default inference model discovery #2759 covers that spot.I went with an extra positional argument to match how
signalwas added. Would you rather have an options object there instead? Happy to switch, and any tips are very welcome.Fixes #2820
Note
Low Risk
Mechanical fetch plumbing with backward-compatible defaults; limited behavior change except when callers pass a custom
fetch, where networking becomes more consistent and predictable.Overview
Custom
options.fetchnow applies to the full inference call path, not only the initial submit viainnerRequest. Provider helpers accept an optionalfetchFn(defaulting to globalfetch) ongetResponseand, for ZAI image-to-text,preparePayloadAsync; task entrypoints forwardoptions?.fetchthe same way they already forwardoptions?.signal.Queue and async providers (fal-ai, novita, together, wavespeed, zai-org) use
fetchFnfor status polling, result retrieval, and media downloads. Replicate and hf-inference use it for output URL fetches. In-processdata:URL decoding (e.g. hf-inferenceb64_json) still uses globalfetchso proxies are not required to handledata:schemes.New
provider-custom-fetch.spec.tstests end-to-end propagation (fal-ai queue flow, hf-inference URL vs base64, ZAI image URL inputs) and asserts the globalfetchstub stays unused when a custom fetch is supplied.Reviewed by Cursor Bugbot for commit c19c58e. Bugbot is set up for automated code reviews on this repo. Configure here.