Repository navigation
fix: Correct retry logic - #3703
stevehipwell wants to merge 1 commit into
Conversation
|
👋 Hi, and thank you for this contribution! This repo is maintained by GitHub and community members on a best-effort basis. We'll get to this as soon as we can. You can help us prioritize by joining the discussion on open issues and PRs, sharing details on the changes you need, and reviewing other contributions. 🤖 This is an automated message. |
There was a problem hiding this comment.
🟡 Changes recommended
Retry preparation can interfere with the previous attempt’s request, and cancellation cleanup needs regression coverage.
2 open issues
What changed in this PR
These provider review instructions are being used.
This PR corrects request handling in the provider’s internal GitHub HTTP client retry transport.
Changes:
- Copies requests and restores request bodies before retrying.
- Cleans up retained responses when retries end with an error.
- Adds body-rewind tests and checks request payloads and attempt counts.
| File | Description |
|---|---|
internal/ghclient/retry.go |
Updates request copying, body replay, and response cleanup. |
internal/ghclient/retry_test.go |
Verifies retry counts and request payload preservation. |
internal/ghclient/body.go |
Adds a helper to restore request bodies using GetBody. |
internal/ghclient/body_test.go |
Tests body restoration and invalid request handling. |
Review effort: Balanced
7e399c4 to
fb12af3
Compare
fb12af3 to
ee900bc
Compare
Signed-off-by: Steve Hipwell <steve.hipwell@gmail.com>
ee900bc to
9afae6a
Compare
robert-crandall
left a comment
There was a problem hiding this comment.
Thanks for cleaning this up! Splitting rewindableTransport out from retryTransport makes the transport stack much easier to follow. I left a couple of inline notes on transport.go and one small test comment in retry_test.go.
| @@ -87,6 +79,12 @@ func newTransport(tokenSource oauth2.TokenSource, opts ClientOptions) (http.Roun | |||
|
|
|||
| tr = newRateLimitTransport(tr) | |||
There was a problem hiding this comment.
Because newRateLimitTransport sits outside retryTransport, secondary rate-limit retries (403/429 with Retry-After) loop inside go-github-ratelimit without going back through retryTransport's if i 0 rewind. Over HTTP/2 (ForceAttemptHTTP2 = true), the second attempt sends an empty req.Body (request declared a Content-Length of ... but only wrote 0 bytes).
Here is a quick test sketch that reproduces it:
srv := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body) // attempt 1: `{"query":"mutation"}`, attempt 2: ""
if attempt.Add(1) == 1 {
w.Header().Set("Retry-After", "1")
w.WriteHeader(http.StatusForbidden)
w.Write([]byte(`{"message": "You have exceeded a secondary rate limit"}`))
return
}
if len(body) == 0 {
w.WriteHeader(http.StatusBadRequest)
return
}
w.WriteHeader(http.StatusOK)
}))
srv.EnableHTTP2 = true
srv.StartTLS()
tr := newRateLimitTransport(srv.Client().Transport)
tr, _ = newRewindableTransport(tr, 0)
req, _ := http.NewRequest(http.MethodPost, srv.URL, bytes.NewBufferString(`{"query":"mutation"}`))
resp, _ := tr.RoundTrip(req) // resp.StatusCode == 400 because attempt 2 had an empty bodyCan you rewind req.Body before every attempt that enters newRateLimitTransport's inner transport when req.GetBody != nil?
| tr = ctr | ||
| } | ||
|
|
||
| tr = logging.NewLoggingHTTPTransport(tr) |
There was a problem hiding this comment.
Moving logging.NewLoggingHTTPTransport to the outermost layer means httputil.DumpRequestOut(req, true) runs before rewindableTransport makes its shallow copy (reqCopy := *req) or throttleTransport acquires a concurrency slot. When TF_LOG is enabled, DumpRequestOut drains and replaces the caller's req.Body in place, and it also hides individual retry/rate-limit attempts from debug logs.
Here is a quick test sketch showing the caller mutation:
t.Setenv("TF_LOG", "DEBUG")
tr, _ := newRewindableTransport(srv.Client().Transport, 0)
tr = logging.NewLoggingHTTPTransport(tr)
origBody := io.NopCloser(strings.NewReader("hello"))
req, _ := http.NewRequest(http.MethodPost, srv.URL, origBody)
_, _ = tr.RoundTrip(req)
// req.Body != origBody (mutated in place by httputil.DumpRequestOut before rewindableTransport's reqCopy)Can you keep logging.NewLoggingHTTPTransport inside retryTransport (around the base/cache/OAuth transport, as before) so rewindableTransport protects the caller's req first and debug logs still show each wire attempt?



Resolves #ISSUE_NUMBER
Before the change?
After the change?
Pull request checklist
Does this introduce a breaking change?
Please see our docs on breaking changes to help!