feat: simplify HTTP client proxy pools and authenticated JSON calls

This commit is contained in:
2026-09-26 15:28:19 +08:00
parent 068e85132f
commit 96e7d9629b
27 changed files with 1611 additions and 143 deletions

153
httpc/README.md Normal file
View File

@@ -0,0 +1,153 @@
# httpc:复用连接的 HTTP 客户端
`httpc` 在标准 `net/http` 上提供独立连接池、网络阶段总超时、保守重试、请求追踪和有界 JSON 调用。普通接口可以直接带认证头调用;上传、下载、特殊协议仍使用标准 Request/Response。没有业务身份模型或全局认证信息。
## 创建一次,业务重复使用
```go
client, err := httpc.New(httpc.Config{
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
Timeout: 3 * time.Second,
DialTimeout: time.Second,
TLSHandshakeTimeout: time.Second,
ResponseHeaderTimeout: time.Second,
MaxConnsPerHost: 32,
MaxIdleConnsPerHost: 16,
MaxIdleConns: 64,
IdleConnTimeout: 60 * time.Second,
MaxResponseBytes: 1 << 20,
})
if err != nil { return err }
// 在 svc 初始化并持有 client,向生命周期登记 client.Close。
// 普通独立程序也可 defer client.Close();不要每次请求重新 New。
err = client.JSON(ctx, http.MethodGet, serviceURL, nil, &result,
httpc.WithHeader("Authorization", "Bearer "+token),
httpc.WithHeaders(http.Header{"Accept": {"application/json"}}),
)
```
Client 可并发使用。每个调用的 input/output 及其读写同步由调用方负责,不要让多个请求同时写同一个 output。
直接运行 [examples/httpClient](../examples/httpClient/README.md),只使用本地临时服务,无需业务 Token 或外部基础设施。
## 代理规则
| 配置 | 行为 |
| --- | --- |
| 不填写 Proxy | 保留被克隆 DefaultTransport 的代理策略;未改动的标准库默认使用环境代理 |
| `Mode: httpc.ProxyEnvironment` | 明确使用 HTTP_PROXY / HTTPS_PROXY / NO_PROXY(及小写形式) |
| `Mode: httpc.ProxyDirect` | 设置 Proxy=nil,明确直连,不受环境代理影响 |
| `Mode: httpc.ProxyURL, URL: "http://proxy.example:8080"` | 固定代理地址,不通过 NO_PROXY 排除 |
支持 http、https、socks5、socks5h;底层行为沿用 Go Transport。固定地址必须是代理 origin,不接受业务路径、查询、fragment 或无效端口。凭证可由 URL userinfo 提供,但配置错误不输出 URL/密码;不要打印整个 Config。环境代理规则由标准库读取并缓存,不提供运行时环境代理热更新。
内部服务要直连,只设置 `proxy.mode: direct`,无需创建 Transport。需要自定义路由、代理认证握手等复杂逻辑时注入自己的 Transport。
## 超时和连接池
除原有字段外,新字段都可选:**0 保留默认行为,正数覆盖,负数拒绝**。需要明确关闭某个阶段超时、关闭 TCP keepalive 或特殊 dialer 时,使用自定义 Transport。
| 字段 | 默认/含义 |
| --- | --- |
| Timeout | 10s;从 Do 开始,覆盖 DNS/建连、等待连接、重试、退避、响应头和读取响应体 |
| DialTimeout | 不填时保留原 dialer;标准库默认 30s;配置后使用标准 net.Dialer |
| DialKeepAlive | 标准 net.Dialer 的 TCP keepalive 探测间隔,标准默认 30s;不是 HTTP 空闲连接超时 |
| TLSHandshakeTimeout | 沿用克隆 Transport;标准库默认 10s |
| ResponseHeaderTimeout | 沿用 Transport,标准默认不单独限制;从完整发送请求后等响应头,总 Timeout 仍生效 |
| ExpectContinueTimeout | 沿用 Transport;标准默认 1s,等待 `100-continue` 的时间 |
| IdleConnTimeout | 沿用 Transport;标准默认 90s,只回收空闲连接 |
| MaxConnsPerHost | 默认 64;每个目标的建连中、活动与空闲连接总上限 |
| MaxIdleConnsPerHost | 默认等于 MaxConnsPerHost,可单独配置 |
| MaxIdleConns | 默认 MaxConnsPerHost×2,可单独配置;所有目标合计的空闲连接上限 |
| MaxResponseBytes | JSON 成功响应默认最多 1 MiB,最大允许配置 1 GiB;应按实际接口保持较小限额 |
每主机上限最多 65536,总空闲连接最多 131072。总空闲上限可以小于各主机上限之和。MaxConnsPerHost 不等于整个进程的连接上限,也不等于 HTTP/2 的请求并发上限。
配置 DialTimeout / DialKeepAlive 会创建标准 net.Dialer;未填的另一个值使用标准默认 30s。不要通过全局修改 DefaultTransport 来塞入项目的特殊拨号逻辑;请显式使用借用的 Transport。若 DefaultTransport 有自定义 TLS dialer,相关阶段覆盖会报错,避免 TLS timeout 被悄悄忽略。
调用者 context 的更早截止时间优先。总超时不能抢占 Go 的 JSON 编解码或不遵守 context 的自定义 RoundTripper;计时从 Do 开始,不包括调用前的 json.Marshal。TLS 校验默认保持开启,没有新增 InsecureSkipVerify 配置。
## JSON 的单次请求选项
```go
err := client.JSON(ctx, "POST", endpoint, input, &output,
httpc.WithHeader("Authorization", "Bearer "+token),
httpc.WithHeader("X-Tenant", tenant),
httpc.WithMaxResponseBytes(64<<10),
)
```
- `WithHeader` 设置一个头;`WithHeaders` 支持多值 http.Header。后出现的选项覆盖同名头,不影响其他头。
- WithHeaders 在创建选项时快照原 map,每次调用复制值。构建后修改原 map 不影响选项,选项可以并发复用;构建时原 map 的读写同步由调用方负责。
- 不提供 Client 全局 Authorization,避免跨用户调用共享凭证。框架不会把请求头写进日志。
- 有 input 时默认 Content-Type 为 application/json;显式请求头可覆盖。请求头名称/值会校验,拒绝 CRLF 等控制字符和歧义的大小写重复键。
- Host、Content-Length、Transfer-Encoding、Trailer 属于请求目标/分帧控制,请用标准 Request 配合 Do;JSON 选项中不允许伪装成普通头。
- `WithMaxResponseBytes` 是本次 JSON 的正数限额,不修改 Client 默认值,不改变 Do 的流式语义。
JSON 自动关闭响应体。仅 2xx 解码;空成功响应(例如 204)保留 output 原值。非空输入使用标准 json.Unmarshal,拒绝多个 JSON 文档,类型错误保留 Go 的错误信息与行为。
`output == nil` 按限额流式丢弃成功响应,不把整个 body 读进内存,也不检查 JSON 语法。响应限额按实际读取的 body 计算;标准库自动 gzip 解压后仍受上限约束。input 会先 json.Marshal 到内存,大文件上传不适合用 JSON helper。
非 2xx 返回 `*httpc.StatusError`,只包含状态码,不包含错误响应体。为连接复用,只尝试读完**声明长度已知且不超过 min(32 KiB, 本次响应限额)** 的小错误体,最多等待 25ms(或更早总截止时间);失败仍返回原 HTTP 状态。未知长度/大错误体立即关闭,优先及时返回。重试过程的错误体另受 32 KiB 大小及整体请求截止时间约束,不无界读取。
## 配置文件与生成项目
在项目 Config 中加一个字段:
```go
Outbound httpc.Config `json:"outbound"`
```
项目 Validate 中调用 `c.Outbound.Validate()`;创建时 New 也会验证。svc 初始化一次并通过生命周期登记 Close。文件例子:
```yaml
outbound:
timeout: 3s
proxy:
mode: direct
dialTimeout: 500ms
tlsHandshakeTimeout: 1s
responseHeaderTimeout: 1s
maxConnsPerHost: 32
maxIdleConnsPerHost: 16
maxIdleConns: 64
idleConnTimeout: 60s
maxResponseBytes: 1048576
```
现有统一加载入口可读取 JSON/YAML/XML,并自动覆盖 `GO_OMNI_OUTBOUND_PROXY_MODE`、`GO_OMNI_OUTBOUND_DIAL_TIMEOUT` 等环境变量。无需为每个参数手写环境解析。详见 [统一配置](../CONFIG.md)。
## 标准请求、特殊 TLS 与生命周期
```go
req, err := http.NewRequestWithContext(ctx, "GET", downloadURL, nil)
if err != nil { return err }
req.Header.Set("Authorization", "Bearer "+token)
res, err := client.Do(req)
if err != nil { return err }
defer res.Body.Close()
// 自行判断状态,按项目要求限流/限制大小,流式写入目标;不要无界 ReadAll 大文件。
_, err = io.Copy(destination, res.Body)
```
Do 不自动限制 body,允许下载/流式处理,调用方必须关闭。响应体消费/关闭前网络超时持续有效。
`Config.Transport` 可注入任意 http.RoundTripper,适配自定义 dialer、mTLS、根证书、监控包装或测试替身。框架借用它,**不修改、不调用其 CloseIdleConnections**。代理、连接池、TLS/建连阶段配置不能同时填入,避免无效配置;总 Timeout、JSON 限额、框架重试/日志仍有效。借用实现必须遵守标准请求和 body 所有权、并发及 context 约定。
`Close` 幂等:拒绝新调用,关闭自有池的空闲连接,不主动打断已开始的请求。服务关闭应先排空 HTTP/后台调用,再关闭 Client;它不是后台 drain 的替代品。生成项目可通过 svc 的生命周期登记清理。
## 重试、日志与兼容
MaxRetries 默认 0,最多 5。仅可重放的 GET/HEAD/OPTIONS 对网络错误、429/502/503/504 做框架重试。POST/PUT/PATCH/DELETE 不自动做应用层重试。标准库或自定义 Transport 自己的重放规则仍可能生效,不能承诺业务“恰好执行一次”。
Backoff 默认 100ms,MaxBackoff 默认 2s,采用指数退避与抖动。Retry-After 要求等待超过 MaxBackoff 时返回当前响应,不提前请求;总截止时间限制全部尝试。成功或无重试的调用不计算退避。
不跟随重定向,避免自动向其他地址转发认证信息;3xx 由调用者判断。默认传递 context 中的 X-Request-ID,显式请求头优先。日志只包含 method、hostname、status、attempt、收到响应头的耗时等,不打印路径、查询、用户名密码、认证头或响应体;标准 net/http 返回的 URL 错误可能携带地址,项目不要把 URL 中的密钥当作安全日志字段。
v0.21 兼容说明:
- 原来的 JSON 五参数直接调用继续有效;新方法末尾增加 `...httpc.RequestOption`,自行定义的 interface/方法值类型需要同步签名。
- 原默认代理、总超时、重试、池大小和成功响应含义保留;新字段未设置时不改变它们。
- `Transport` 与显式池/代理/阶段参数混用现在报错;此前 MaxConnsPerHost 在自定义 Transport 下实际被忽略。迁移时把参数放到自己的 Transport,或删掉 Transport 改用内置配置。
- 没有移除 Do、JSON、StatusError 或自定义 Transport,没有引入新第三方依赖。

100
httpc/benchmark_test.go Normal file
View File

@@ -0,0 +1,100 @@
package httpc
import (
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
type benchmarkTransport struct {
payload string
status int
}
func (t benchmarkTransport) RoundTrip(r *http.Request) (*http.Response, error) {
if r.Body != nil {
_ = r.Body.Close()
}
return &http.Response{StatusCode: t.status, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(t.payload)), ContentLength: int64(len(t.payload)), Request: r}, nil
}
func BenchmarkHTTPClient(b *testing.B) {
for _, size := range []int{128, 4096, 65536} {
for _, discard := range []bool{false, true} {
label := map[int]string{128: "128B", 4096: "4KiB", 65536: "64KiB"}[size]
if discard {
label += "/discard"
} else {
label += "/decode"
}
b.Run(label, func(b *testing.B) {
c, err := New(Config{Transport: benchmarkTransport{`{"value":"` + strings.Repeat("x", size) + `"}`, 200}, DisableLogging: true})
if err != nil {
b.Fatal(err)
}
defer c.Close()
var output struct {
Value string `json:"value"`
}
var target any = &output
if discard {
target = nil
}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if err := c.JSON(context.Background(), "GET", "http://example.invalid/item", nil, target); err != nil {
b.Fatal(err)
}
}
})
}
}
b.Run("Do", func(b *testing.B) {
c, _ := New(Config{Transport: benchmarkTransport{"", 200}, DisableLogging: true})
defer c.Close()
r, _ := http.NewRequest("GET", "http://example.invalid/item", nil)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
res, err := c.Do(r)
if err != nil {
b.Fatal(err)
}
res.Body.Close()
}
})
}
func BenchmarkHTTPPool(b *testing.B) {
for _, status := range []int{200, 401} {
b.Run(http.StatusText(status), func(b *testing.B) {
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(status); io.WriteString(w, `{"ok":true}`) }))
defer s.Close()
c, _ := New(Config{DisableLogging: true})
defer c.Close()
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
if status == 200 && err != nil {
b.Fatal(err)
}
}
})
}
}
func BenchmarkHTTPHeaders(b *testing.B) {
c, _ := New(Config{Transport: benchmarkTransport{`{"ok":true}`, 200}, DisableLogging: true})
defer c.Close()
option := WithHeaders(http.Header{"Authorization": []string{"Bearer example"}, "X-Request-Id": []string{"example-request"}})
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if err := c.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, option); err != nil {
b.Fatal(err)
}
}
}

View File

@@ -3,9 +3,7 @@
package httpc
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
@@ -22,18 +20,6 @@ import (
var ErrClosed = errors.New("httpc: closed")
var ErrResponseTooLarge = errors.New("httpc: response exceeds limit")
type Config struct {
Timeout time.Duration `json:"timeout"`
MaxRetries int `json:"maxRetries"`
Backoff time.Duration `json:"backoff"`
MaxBackoff time.Duration `json:"maxBackoff"`
MaxConnsPerHost int `json:"maxConnsPerHost"`
MaxResponseBytes int64 `json:"maxResponseBytes"`
DisableRequestID bool `json:"disableRequestId"`
DisableLogging bool `json:"disableLogging"`
Transport http.RoundTripper `json:"-"` // Borrowed. The caller owns its lifecycle.
}
type Client struct {
config Config
client *http.Client
@@ -41,43 +27,19 @@ type Client struct {
closed atomic.Bool
}
// New creates a reusable client. Close it after callers and response streams drain.
func New(cfg Config) (*Client, error) {
if cfg.Timeout < 0 || cfg.Backoff < 0 || cfg.MaxBackoff < 0 || cfg.MaxRetries < 0 || cfg.MaxRetries > 5 || cfg.MaxConnsPerHost < 0 || cfg.MaxResponseBytes < 0 {
return nil, errors.New("httpc: invalid configuration")
normalized, err := normalizeConfig(cfg)
if err != nil {
return nil, err
}
if cfg.Timeout == 0 {
cfg.Timeout = 10 * time.Second
}
if cfg.Backoff == 0 {
cfg.Backoff = 100 * time.Millisecond
}
if cfg.MaxBackoff == 0 {
cfg.MaxBackoff = 2 * time.Second
}
if cfg.Backoff > cfg.MaxBackoff {
return nil, errors.New("httpc: backoff exceeds maxBackoff")
}
if cfg.MaxConnsPerHost == 0 {
cfg.MaxConnsPerHost = 64
}
if cfg.MaxResponseBytes == 0 {
cfg.MaxResponseBytes = 1 << 20
}
if cfg.MaxResponseBytes > 1<<30 || cfg.MaxConnsPerHost > 65536 {
return nil, errors.New("httpc: response or connection limit exceeds supported capacity")
}
c := &Client{config: cfg}
c := &Client{config: normalized}
transport := cfg.Transport
if transport == nil {
// Own a private pool; never mutate the process-wide DefaultTransport.
base, ok := http.DefaultTransport.(*http.Transport)
if !ok {
return nil, errors.New("httpc: provide Transport when DefaultTransport is replaced")
c.owned, err = ownedTransport(normalized)
if err != nil {
return nil, err
}
c.owned = base.Clone()
c.owned.MaxConnsPerHost = cfg.MaxConnsPerHost
c.owned.MaxIdleConnsPerHost = cfg.MaxConnsPerHost
c.owned.MaxIdleConns = cfg.MaxConnsPerHost * 2
transport = c.owned
}
c.client = &http.Client{Transport: transport, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
@@ -114,7 +76,10 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
}
replayable := (base.Method == "" || base.Method == "GET" || base.Method == "HEAD" || base.Method == "OPTIONS") && (base.Body == nil || base.Body == http.NoBody || base.GetBody != nil)
for attempt := 0; ; attempt++ {
req := base.Clone(ctx)
req := base
if attempt > 0 {
req = base.Clone(ctx)
}
if attempt > 0 && base.Body != nil && base.Body != http.NoBody {
body, err := base.GetBody()
if err != nil {
@@ -133,23 +98,9 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
logx.InfoContext(ctx, "http.client", "method", req.Method, "host", req.URL.Hostname(), "status", status, "attempt", attempt+1, "durationMs", time.Since(started).Seconds()*1000, "failed", err != nil)
}
retry := replayable && attempt < c.config.MaxRetries && ctx.Err() == nil && (err != nil || status == 429 || status == 502 || status == 503 || status == 504)
delay := c.config.Backoff
for i := 0; i < attempt && delay < c.config.MaxBackoff; i++ {
if delay > c.config.MaxBackoff/2 {
delay = c.config.MaxBackoff
} else {
delay *= 2
}
}
if response != nil && response.Header.Get("Retry-After") != "" {
if value, ok := retryAfter(response.Header.Get("Retry-After")); ok {
delay = value
}
} else if delay > 1 {
delay = delay/2 + time.Duration(rand.Int63n(int64(delay/2)+1))
}
if delay > c.config.MaxBackoff {
retry = false
var delay time.Duration
if retry {
delay, retry = c.retryDelay(attempt, response)
}
if !retry {
if err != nil {
@@ -160,9 +111,7 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
return response, nil
}
if response != nil {
if response.ContentLength >= 0 && response.ContentLength <= 32<<10 {
_, _ = io.Copy(io.Discard, response.Body)
}
drainSmallBody(response, 32<<10)
_ = response.Body.Close()
}
timer := time.NewTimer(delay)
@@ -204,48 +153,6 @@ type StatusError struct{ StatusCode int }
func (e *StatusError) Error() string { return fmt.Sprintf("httpc: HTTP status %d", e.StatusCode) }
// JSON bounds response memory, closes the body, and decodes only 2xx responses.
// Error bodies, URLs, credentials and query strings are not included in logs.
func (c *Client) JSON(ctx context.Context, method, url string, input, output any) error {
if ctx == nil {
return errors.New("httpc: nil context")
}
var body io.Reader
if input != nil {
data, err := json.Marshal(input)
if err != nil {
return err
}
body = bytes.NewReader(data)
}
request, err := http.NewRequestWithContext(ctx, method, url, body)
if err != nil {
return err
}
if input != nil {
request.Header.Set("Content-Type", "application/json")
}
response, err := c.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode >= 300 {
return &StatusError{StatusCode: response.StatusCode}
}
data, err := io.ReadAll(io.LimitReader(response.Body, c.config.MaxResponseBytes+1))
if err != nil {
return err
}
if int64(len(data)) > c.config.MaxResponseBytes {
return ErrResponseTooLarge
}
if output == nil || len(bytes.TrimSpace(data)) == 0 {
return nil
}
return json.Unmarshal(data, output)
}
func (c *Client) Close() error {
c.closed.Store(true)
if c.owned != nil {
@@ -253,3 +160,34 @@ func (c *Client) Close() error {
}
return nil
}
// retryDelay is deliberately outside the success path: no jitter/random work
// for calls that cannot or do not need to retry.
func (c *Client) retryDelay(attempt int, response *http.Response) (time.Duration, bool) {
delay := c.config.Backoff
for i := 0; i < attempt && delay < c.config.MaxBackoff; i++ {
if delay > c.config.MaxBackoff/2 {
delay = c.config.MaxBackoff
} else {
delay *= 2
}
}
if response != nil {
if value, ok := retryAfter(response.Header.Get("Retry-After")); ok {
return value, value <= c.config.MaxBackoff
}
}
if delay > 1 {
delay = delay/2 + time.Duration(rand.Int63n(int64(delay/2)+1))
}
return delay, delay <= c.config.MaxBackoff
}
// Drain only a known small error body. Unknown/large bodies are closed promptly;
// never allocate or consume an unbounded error document just to retain a pool slot.
func drainSmallBody(response *http.Response, limit int64) {
limit = min(limit, 32<<10)
if response.Body != nil && response.ContentLength >= 0 && response.ContentLength <= limit {
_, _ = io.Copy(io.Discard, io.LimitReader(response.Body, limit+1))
}
}

205
httpc/config.go Normal file
View File

@@ -0,0 +1,205 @@
package httpc
import (
"errors"
"fmt"
"net"
"net/http"
"net/url"
"reflect"
"strconv"
"strings"
"time"
)
// ProxyMode controls the owned transport's proxy selection. Empty preserves
// DefaultTransport's policy (normally environment proxies).
type ProxyMode string
const (
ProxyEnvironment ProxyMode = "environment"
ProxyDirect ProxyMode = "direct"
ProxyURL ProxyMode = "url"
)
type ProxyConfig struct {
Mode ProxyMode `json:"mode"`
URL string `json:"url"` // May contain credentials; never log the configuration.
}
// Config owns one private connection pool unless Transport is supplied.
// New duration/pool fields are optional: zero retains existing defaults, positive
// values override them. Use a custom Transport for disabling timeouts, custom
// dialers, mTLS and other protocol-specific settings.
type Config struct {
Timeout time.Duration `json:"timeout"`
MaxRetries int `json:"maxRetries"`
Backoff time.Duration `json:"backoff"`
MaxBackoff time.Duration `json:"maxBackoff"`
MaxConnsPerHost int `json:"maxConnsPerHost"`
MaxIdleConns int `json:"maxIdleConns"`
MaxIdleConnsPerHost int `json:"maxIdleConnsPerHost"`
DialTimeout time.Duration `json:"dialTimeout"`
DialKeepAlive time.Duration `json:"dialKeepAlive"`
TLSHandshakeTimeout time.Duration `json:"tlsHandshakeTimeout"`
ResponseHeaderTimeout time.Duration `json:"responseHeaderTimeout"`
ExpectContinueTimeout time.Duration `json:"expectContinueTimeout"`
IdleConnTimeout time.Duration `json:"idleConnTimeout"`
Proxy ProxyConfig `json:"proxy"`
MaxResponseBytes int64 `json:"maxResponseBytes"`
DisableRequestID bool `json:"disableRequestId"`
DisableLogging bool `json:"disableLogging"`
Transport http.RoundTripper `json:"-"` // Borrowed; never modified or closed.
}
const maxResponseBytes int64 = 1 << 30
// Validate checks values without creating pools or making network requests.
func (cfg Config) Validate() error { _, err := normalizeConfig(cfg); return err }
func normalizeConfig(cfg Config) (Config, error) {
for _, field := range []struct {
name string
value time.Duration
}{
{"timeout", cfg.Timeout}, {"backoff", cfg.Backoff}, {"maxBackoff", cfg.MaxBackoff},
{"dialTimeout", cfg.DialTimeout}, {"dialKeepAlive", cfg.DialKeepAlive}, {"tlsHandshakeTimeout", cfg.TLSHandshakeTimeout},
{"responseHeaderTimeout", cfg.ResponseHeaderTimeout}, {"expectContinueTimeout", cfg.ExpectContinueTimeout}, {"idleConnTimeout", cfg.IdleConnTimeout},
} {
if field.value < 0 {
return Config{}, fmt.Errorf("httpc: %s must not be negative", field.name)
}
}
if cfg.MaxRetries < 0 || cfg.MaxRetries > 5 {
return Config{}, errors.New("httpc: maxRetries must be 0..5")
}
if cfg.MaxConnsPerHost < 0 || cfg.MaxConnsPerHost > 65536 || cfg.MaxIdleConnsPerHost < 0 || cfg.MaxIdleConnsPerHost > 65536 || cfg.MaxIdleConns < 0 || cfg.MaxIdleConns > 131072 {
return Config{}, errors.New("httpc: invalid connection pool limits")
}
if cfg.MaxResponseBytes < 0 || cfg.MaxResponseBytes > maxResponseBytes {
return Config{}, errors.New("httpc: maxResponseBytes must be 0..1 GiB")
}
if _, err := proxyFunction(cfg.Proxy); err != nil {
return Config{}, err
}
if cfg.Transport != nil {
value := reflect.ValueOf(cfg.Transport)
switch value.Kind() {
case reflect.Pointer, reflect.Map, reflect.Func, reflect.Slice, reflect.Interface, reflect.Chan:
if value.IsNil() {
return Config{}, errors.New("httpc: nil Transport")
}
}
if cfg.hasTransportOptions() {
return Config{}, errors.New("httpc: Transport cannot be combined with proxy, dial, TLS or connection pool options; configure the borrowed transport directly")
}
}
if cfg.Timeout == 0 {
cfg.Timeout = 10 * time.Second
}
if cfg.Backoff == 0 {
cfg.Backoff = 100 * time.Millisecond
}
if cfg.MaxBackoff == 0 {
cfg.MaxBackoff = 2 * time.Second
}
if cfg.Backoff > cfg.MaxBackoff {
return Config{}, errors.New("httpc: backoff exceeds maxBackoff")
}
if cfg.MaxConnsPerHost == 0 {
cfg.MaxConnsPerHost = 64
}
if cfg.MaxIdleConnsPerHost == 0 {
cfg.MaxIdleConnsPerHost = cfg.MaxConnsPerHost
}
if cfg.MaxIdleConns == 0 {
cfg.MaxIdleConns = cfg.MaxConnsPerHost * 2
}
if cfg.MaxResponseBytes == 0 {
cfg.MaxResponseBytes = 1 << 20
}
return cfg, nil
}
func (c Config) hasTransportOptions() bool {
return c.Proxy.Mode != "" || c.Proxy.URL != "" || c.MaxConnsPerHost != 0 || c.MaxIdleConns != 0 || c.MaxIdleConnsPerHost != 0 || c.DialTimeout != 0 || c.DialKeepAlive != 0 || c.TLSHandshakeTimeout != 0 || c.ResponseHeaderTimeout != 0 || c.ExpectContinueTimeout != 0 || c.IdleConnTimeout != 0
}
func proxyFunction(c ProxyConfig) (func(*http.Request) (*url.URL, error), error) {
if c.Mode != ProxyURL && c.URL != "" {
return nil, errors.New("httpc: proxy URL requires mode url")
}
switch c.Mode {
case "":
return nil, nil
case ProxyDirect:
return nil, nil
case ProxyEnvironment:
return http.ProxyFromEnvironment, nil
case ProxyURL:
u, err := url.Parse(c.URL)
if err != nil || u == nil || u.Hostname() == "" || u.Opaque != "" || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || (u.Path != "" && u.Path != "/") {
return nil, errors.New("httpc: proxy URL must be an absolute proxy origin without path, query or fragment")
}
switch u.Scheme {
case "http", "https", "socks5", "socks5h":
default:
return nil, errors.New("httpc: proxy scheme must be http, https, socks5 or socks5h")
}
if strings.HasSuffix(u.Host, ":") {
return nil, errors.New("httpc: proxy port must be 1..65535")
}
if port := u.Port(); port != "" {
n, e := strconv.Atoi(port)
if e != nil || n < 1 || n > 65535 {
return nil, errors.New("httpc: proxy port must be 1..65535")
}
}
u.Path = ""
u.RawPath = ""
return http.ProxyURL(u), nil
default:
return nil, errors.New("httpc: proxy mode must be environment, direct or url")
}
}
func ownedTransport(cfg Config) (*http.Transport, error) {
base, ok := http.DefaultTransport.(*http.Transport)
if !ok || base == nil {
return nil, errors.New("httpc: provide Transport when DefaultTransport is replaced")
}
if (base.DialTLSContext != nil || base.DialTLS != nil) && (cfg.DialTimeout > 0 || cfg.DialKeepAlive > 0 || cfg.TLSHandshakeTimeout > 0) {
return nil, errors.New("httpc: provide Transport when DefaultTransport has a custom TLS dialer")
}
t := base.Clone()
t.MaxConnsPerHost = cfg.MaxConnsPerHost
t.MaxIdleConnsPerHost = cfg.MaxIdleConnsPerHost
t.MaxIdleConns = cfg.MaxIdleConns
if cfg.Proxy.Mode != "" {
proxy, err := proxyFunction(cfg.Proxy)
if err != nil {
return nil, err
}
t.Proxy = proxy
}
if cfg.DialTimeout > 0 || cfg.DialKeepAlive > 0 {
dialer := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
if cfg.DialTimeout > 0 {
dialer.Timeout = cfg.DialTimeout
}
if cfg.DialKeepAlive > 0 {
dialer.KeepAlive = cfg.DialKeepAlive
}
t.DialContext = dialer.DialContext
t.Dial = nil
}
if cfg.TLSHandshakeTimeout > 0 {
t.TLSHandshakeTimeout = cfg.TLSHandshakeTimeout
}
if cfg.ResponseHeaderTimeout > 0 {
t.ResponseHeaderTimeout = cfg.ResponseHeaderTimeout
}
if cfg.ExpectContinueTimeout > 0 {
t.ExpectContinueTimeout = cfg.ExpectContinueTimeout
}
if cfg.IdleConnTimeout > 0 {
t.IdleConnTimeout = cfg.IdleConnTimeout
}
return t, nil
}

277
httpc/config_test.go Normal file
View File

@@ -0,0 +1,277 @@
package httpc
import (
"context"
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"os/exec"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
func testClient(t *testing.T, cfg Config) *Client {
t.Helper()
cfg.DisableLogging = true
c, err := New(cfg)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { c.Close() })
return c
}
func TestTransportConfigurationAndIsolation(t *testing.T) {
base := http.DefaultTransport.(*http.Transport)
oldConns, oldIdle, oldTimeout := base.MaxConnsPerHost, base.MaxIdleConns, base.ResponseHeaderTimeout
cfg := Config{Proxy: ProxyConfig{Mode: ProxyDirect}, DialTimeout: time.Second, DialKeepAlive: 20 * time.Second, TLSHandshakeTimeout: 2 * time.Second, ResponseHeaderTimeout: 3 * time.Second, ExpectContinueTimeout: 4 * time.Second, IdleConnTimeout: 5 * time.Second, MaxConnsPerHost: 8, MaxIdleConns: 12, MaxIdleConnsPerHost: 3}
c := testClient(t, cfg)
if c.owned == base || c.owned.Proxy != nil || c.owned.DialContext == nil || c.owned.MaxConnsPerHost != 8 || c.owned.MaxIdleConns != 12 || c.owned.MaxIdleConnsPerHost != 3 || c.owned.TLSHandshakeTimeout != 2*time.Second || c.owned.ResponseHeaderTimeout != 3*time.Second || c.owned.ExpectContinueTimeout != 4*time.Second || c.owned.IdleConnTimeout != 5*time.Second {
t.Fatal("transport options not applied")
}
if base.MaxConnsPerHost != oldConns || base.MaxIdleConns != oldIdle || base.ResponseHeaderTimeout != oldTimeout {
t.Fatal("modified shared transport")
}
if c.owned.TLSClientConfig != nil && c.owned.TLSClientConfig.InsecureSkipVerify {
t.Fatal("TLS verification disabled")
}
d := testClient(t, Config{})
if d.owned.MaxConnsPerHost != 64 || d.owned.MaxIdleConnsPerHost != 64 || d.owned.MaxIdleConns != 128 || d.owned.IdleConnTimeout != base.IdleConnTimeout || d.owned.TLSHandshakeTimeout != base.TLSHandshakeTimeout || d.owned.ResponseHeaderTimeout != base.ResponseHeaderTimeout {
t.Fatal("default behavior changed")
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if conn, err := c.owned.DialContext(ctx, "tcp", "127.0.0.1:1"); err == nil {
conn.Close()
t.Fatal("dial ignored context")
}
}
func TestProxyValidation(t *testing.T) {
for _, scheme := range []string{"http", "https", "socks5", "socks5h"} {
c := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyURL, URL: scheme + "://user:private-value@proxy.invalid:8080"}})
req, _ := http.NewRequest("GET", "https://target.invalid", nil)
u, err := c.owned.Proxy(req)
if err != nil || u.Host != "proxy.invalid:8080" || u.Scheme != scheme {
t.Fatal("explicit proxy unavailable")
}
}
bad := []Config{
{Proxy: ProxyConfig{Mode: "private-value"}},
{Proxy: ProxyConfig{URL: "http://private-value"}},
{Proxy: ProxyConfig{Mode: ProxyDirect, URL: "http://private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://user:private-value@proxy.invalid:bad"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "file:///private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid/private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid?token=private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid:65536"}},
{DialTimeout: -1}, {DialKeepAlive: -1}, {TLSHandshakeTimeout: -1}, {ResponseHeaderTimeout: -1}, {ExpectContinueTimeout: -1}, {IdleConnTimeout: -1}, {MaxIdleConns: -1}, {MaxIdleConnsPerHost: 65537},
{Transport: benchmarkTransport{}, Proxy: ProxyConfig{Mode: ProxyDirect}},
{Transport: benchmarkTransport{}, DialTimeout: time.Second}, {Transport: benchmarkTransport{}, MaxConnsPerHost: 2},
}
for i, cfg := range bad {
if _, err := New(cfg); err == nil || strings.Contains(err.Error(), "private-value") {
t.Fatalf("case %d: unsafe or missing error %v", i, err)
}
}
var nilTransport *http.Transport
if _, err := New(Config{Transport: nilTransport}); err == nil {
t.Fatal("typed nil accepted")
}
}
func TestProxyRouting(t *testing.T) {
var proxied, target atomic.Int32
proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxied.Add(1)
if r.URL.Scheme != "http" || r.URL.Host != "target.invalid" {
t.Error("not an absolute proxy request")
}
io.WriteString(w, `{"proxy":true}`)
}))
defer proxy.Close()
origin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { target.Add(1); io.WriteString(w, `{}`) }))
defer origin.Close()
// Isolate the standard library's process-cached environment proxy lookup.
executable, err := os.Executable()
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, executable, "-test.run=^TestProxyEnvironmentProcess$")
for _, value := range os.Environ() {
key := strings.SplitN(value, "=", 2)[0]
switch strings.ToUpper(key) {
case "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "ALL_PROXY", "GO_OMNI_HTTPC_PROXY_CHILD", "GO_OMNI_HTTPC_TARGET":
continue
}
cmd.Env = append(cmd.Env, value)
}
cmd.Env = append(cmd.Env, "GO_OMNI_HTTPC_PROXY_CHILD=1", "GO_OMNI_HTTPC_TARGET="+origin.URL, "HTTP_PROXY="+proxy.URL, "HTTPS_PROXY="+proxy.URL, "NO_PROXY=bypass.invalid")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(err, string(out))
}
if proxied.Load() != 3 || target.Load() != 1 {
t.Fatalf("proxy=%d direct=%d", proxied.Load(), target.Load())
}
}
func TestProxyEnvironmentProcess(t *testing.T) {
if os.Getenv("GO_OMNI_HTTPC_PROXY_CHILD") != "1" {
t.Skip("subprocess only")
}
for _, proxy := range []ProxyConfig{{}, {Mode: ProxyEnvironment}, {Mode: ProxyURL, URL: os.Getenv("HTTP_PROXY")}} {
c := testClient(t, Config{Proxy: proxy})
if err := c.JSON(context.Background(), "GET", "http://target.invalid/path", nil, nil); err != nil {
t.Fatal(err)
}
}
environment := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyEnvironment}})
req, _ := http.NewRequest("GET", "http://bypass.invalid", nil)
u, err := environment.owned.Proxy(req)
if err != nil || u != nil {
t.Fatal("NO_PROXY ignored")
}
direct := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyDirect}})
if direct.owned.Proxy != nil {
t.Fatal("direct still has proxy")
}
if err := direct.JSON(context.Background(), "GET", os.Getenv("GO_OMNI_HTTPC_TARGET"), nil, nil); err != nil {
t.Fatal(err)
}
}
func TestResponseHeaderTimeoutRecovery(t *testing.T) {
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/slow" {
<-r.Context().Done()
return
}
io.WriteString(w, `{}`)
}))
defer s.Close()
c := testClient(t, Config{Timeout: 2 * time.Second, ResponseHeaderTimeout: 30 * time.Millisecond})
start := time.Now()
err := c.JSON(context.Background(), "GET", s.URL+"/slow", nil, nil)
var network net.Error
if !errors.As(err, &network) || !network.Timeout() || time.Since(start) > time.Second {
t.Fatal("header timeout", err)
}
if err = c.JSON(context.Background(), "GET", s.URL+"/ok", nil, nil); err != nil {
t.Fatal("did not recover", err)
}
}
func TestTLSHandshakeTimeout(t *testing.T) {
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer listener.Close()
done := make(chan struct{})
go func() {
defer close(done)
conn, err := listener.Accept()
if err != nil {
return
}
defer conn.Close()
_, _ = io.Copy(io.Discard, conn)
}()
c := testClient(t, Config{Timeout: 2 * time.Second, Proxy: ProxyConfig{Mode: ProxyDirect}, TLSHandshakeTimeout: 30 * time.Millisecond})
start := time.Now()
err = c.JSON(context.Background(), "GET", "https://"+listener.Addr().String(), nil, nil)
var network net.Error
if !errors.As(err, &network) || !network.Timeout() || time.Since(start) > time.Second {
t.Fatal("handshake timeout", err)
}
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("TLS connection leaked")
}
}
func TestPoolLimitsAndIdleExpiry(t *testing.T) {
var connections atomic.Int32
entered := make(chan struct{}, 12)
release := make(chan struct{})
closed := make(chan struct{}, 12)
s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
entered <- struct{}{}
select {
case <-release:
case <-r.Context().Done():
return
}
io.WriteString(w, `{}`)
}))
s.Config.ConnState = func(_ net.Conn, state http.ConnState) {
if state == http.StateNew {
connections.Add(1)
}
if state == http.StateClosed {
closed <- struct{}{}
}
}
s.Start()
defer s.Close()
c := testClient(t, Config{Timeout: 2 * time.Second, MaxConnsPerHost: 2, MaxIdleConnsPerHost: 2, MaxIdleConns: 2, IdleConnTimeout: 40 * time.Millisecond})
var wg sync.WaitGroup
for i := 0; i < 12; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); err != nil {
t.Error(err)
}
}()
}
for i := 0; i < 2; i++ {
select {
case <-entered:
case <-time.After(time.Second):
close(release)
t.Fatal("requests did not start")
}
}
if connections.Load() != 2 {
t.Error("connection cap", connections.Load())
}
close(release)
wg.Wait()
if connections.Load() != 2 {
t.Fatal("pool was not reused", connections.Load())
}
for i := 0; i < 2; i++ {
select {
case <-closed:
case <-time.After(time.Second):
t.Fatal("idle pool was not reclaimed")
}
}
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); err != nil {
t.Fatal(err)
}
if connections.Load() != 3 {
t.Fatal("new connection not established after idle timeout")
}
c.Close()
select {
case <-closed:
case <-time.After(time.Second):
t.Fatal("Close left idle connections open")
}
}
func TestHTTPSVerificationRetained(t *testing.T) {
s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, `{}`) }))
defer s.Close()
c := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyDirect}})
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
var urlError *url.Error
if !errors.As(err, &urlError) {
t.Fatal("self-signed server unexpectedly trusted", err)
}
}

119
httpc/json.go Normal file
View File

@@ -0,0 +1,119 @@
package httpc
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"net/http"
"time"
)
// JSON bounds response memory, closes the body, and decodes only 2xx responses.
// Optional headers/limits belong to this call and never alter the client.
// output=nil discards a bounded stream; it does not buffer or validate JSON.
// Empty successful bodies leave output unchanged. StatusError contains no body.
func (c *Client) JSON(ctx context.Context, method, url string, input, output any, options ...RequestOption) error {
if ctx == nil {
return errors.New("httpc: nil context")
}
if c.closed.Load() {
return ErrClosed
}
if err := ctx.Err(); err != nil {
return err
}
settings := requestOptions{maxResponseBytes: c.config.MaxResponseBytes}
if len(options) > 0 {
var err error
settings, err = jsonOptions(c.config.MaxResponseBytes, options)
if err != nil {
return err
}
}
var body io.Reader
if input != nil {
data, err := json.Marshal(input)
if err != nil {
return err
}
body = bytes.NewReader(data)
}
request, err := http.NewRequestWithContext(ctx, method, url, body)
if err != nil {
return err
}
if settings.headers != nil {
request.Header = settings.headers
}
if input != nil {
if _, exists := request.Header["Content-Type"]; !exists {
request.Header.Set("Content-Type", "application/json")
}
}
response, err := c.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode >= 300 {
drainStatusBody(response, settings.maxResponseBytes)
return &StatusError{StatusCode: response.StatusCode}
}
if response.ContentLength > settings.maxResponseBytes {
return ErrResponseTooLarge
}
limited := io.LimitReader(response.Body, settings.maxResponseBytes+1)
if output == nil {
count, err := io.Copy(io.Discard, limited)
if err != nil {
return err
}
if count > settings.maxResponseBytes {
return ErrResponseTooLarge
}
return nil
}
data, err := io.ReadAll(limited)
if err != nil {
return err
}
if int64(len(data)) > settings.maxResponseBytes {
return ErrResponseTooLarge
}
if len(bytes.TrimSpace(data)) == 0 {
return nil
}
return json.Unmarshal(data, output)
}
// A known small status body may still be slow. Spend at most a short grace
// period trying to retain the connection, then cancel the HTTP read. The status
// remains authoritative and the deferred Close still owns body cleanup.
func drainStatusBody(response *http.Response, limit int64) {
if response.ContentLength < 0 || response.ContentLength > min(limit, 32<<10) {
return
}
body, ok := response.Body.(*responseBody)
if !ok {
return
}
timer := time.AfterFunc(25*time.Millisecond, body.cancel)
defer timer.Stop()
drainSmallBody(response, limit)
}
// Keep the option builder out of the allocation path of existing plain calls.
func jsonOptions(limit int64, options []RequestOption) (requestOptions, error) {
settings := requestOptions{maxResponseBytes: limit}
for _, option := range options {
if option == nil {
return requestOptions{}, errors.New("httpc: nil request option")
}
if err := option(&settings); err != nil {
return requestOptions{}, err
}
}
return settings, nil
}

99
httpc/request.go Normal file
View File

@@ -0,0 +1,99 @@
package httpc
import (
"errors"
"net/http"
"strings"
)
type requestOptions struct {
headers http.Header
maxResponseBytes int64
}
// RequestOption applies only to one JSON call. Returned options can be reused
// concurrently; request headers are independent, with no client-wide credentials.
type RequestOption func(*requestOptions) error
// WithHeader sets a header for this call; later options replace earlier values.
// Framing and Host must be controlled through a standard Request passed to Do.
func WithHeader(name, value string) RequestOption {
name = http.CanonicalHeaderKey(name)
err := checkHeader(name, []string{value})
return func(o *requestOptions) error {
if err != nil {
return err
}
if o.headers == nil {
o.headers = make(http.Header)
}
o.headers[name] = []string{value}
return nil
}
}
// WithHeaders snapshots headers immediately. Editing the original map after
// construction does not change the option; each invocation gets separate slices.
func WithHeaders(headers http.Header) RequestOption {
snapshot := make(http.Header, len(headers))
var err error
for key, values := range headers {
name := http.CanonicalHeaderKey(key)
if err = checkHeader(name, values); err != nil {
break
}
if _, exists := snapshot[name]; exists {
err = errors.New("httpc: duplicate canonical header name")
break
}
snapshot[name] = append([]string(nil), values...)
}
return func(o *requestOptions) error {
if err != nil {
return err
}
if o.headers == nil {
o.headers = make(http.Header, len(snapshot))
}
for name, values := range snapshot {
o.headers[name] = append([]string(nil), values...)
}
return nil
}
}
// WithMaxResponseBytes overrides JSON's bounded response read for one call.
// The size is positive and at most 1 GiB. Do remains a caller-owned stream.
func WithMaxResponseBytes(limit int64) RequestOption {
return func(o *requestOptions) error {
if limit <= 0 || limit > maxResponseBytes {
return errors.New("httpc: response limit must be positive and at most 1 GiB")
}
o.maxResponseBytes = limit
return nil
}
}
func checkHeader(name string, values []string) error {
if name == "" {
return errors.New("httpc: header name must not be empty")
}
for _, c := range []byte(name) {
if c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || strings.ContainsRune("!#$%&'*+-.^_`|~", rune(c)) {
continue
}
return errors.New("httpc: invalid header name")
}
switch name {
case "Host", "Content-Length", "Transfer-Encoding", "Trailer":
return errors.New("httpc: use Do for Host or request framing controls")
}
for _, value := range values {
for i := 0; i < len(value); i++ {
c := value[i]
if c == '\x7f' || c < ' ' && c != '\t' {
return errors.New("httpc: invalid header value")
}
}
}
return nil
}

316
httpc/request_test.go Normal file
View File

@@ -0,0 +1,316 @@
package httpc
import (
"bytes"
"compress/gzip"
"context"
"errors"
"fmt"
"git.yuntuo.hk/k2safe/go-omni/logx"
"io"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
func TestJSONHeadersConcurrentSnapshot(t *testing.T) {
var leaked atomic.Bool
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer "+r.URL.Query().Get("caller") {
leaked.Store(true)
}
if r.Header.Get("X-Shared") != "original" || len(r.Header.Values("X-Tag")) != 2 {
leaked.Store(true)
}
if r.Header.Get("Content-Type") != "application/vnd.example+json" {
leaked.Store(true)
}
io.WriteString(w, `{"ok":true}`)
}))
defer s.Close()
c := testClient(t, Config{})
headers := http.Header{"X-Shared": []string{"original"}, "X-Tag": []string{"one", "two"}}
snapshot := WithHeaders(headers)
headers.Set("X-Shared", "changed")
headers["X-Tag"][0] = "changed"
var wg sync.WaitGroup
for i := 0; i < 24; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
id := fmt.Sprint(i)
var output struct {
OK bool `json:"ok"`
}
if err := c.JSON(context.Background(), "POST", s.URL+"?caller="+id, map[string]bool{"hello": true}, &output, snapshot, WithHeader("Authorization", "Bearer "+id), WithHeader("content-type", "application/vnd.example+json")); err != nil || !output.OK {
t.Errorf("JSON failed: %v", err)
}
}(i)
}
wg.Wait()
if leaked.Load() {
t.Fatal("headers leaked across requests or changed with source map")
}
}
func TestRequestOptionsValidationBeforeNetwork(t *testing.T) {
var calls atomic.Int32
c := testClient(t, Config{Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
calls.Add(1)
return benchmarkTransport{`{}`, 200}.RoundTrip(r)
})})
invalid := []RequestOption{nil, WithHeader("bad name", "private-token"), WithHeader("Authorization", "private-token\r\nInjected: yes"), WithHeader("Host", "example.invalid"), WithHeader("Content-Length", "1"), WithHeader("Transfer-Encoding", "chunked"), WithHeader("Trailer", "X-Signature"), WithHeaders(http.Header{"X-Name": []string{"one"}, "x-name": []string{"two"}}), WithMaxResponseBytes(0), WithMaxResponseBytes(maxResponseBytes + 1)}
for _, option := range invalid {
err := c.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, option)
if err == nil || strings.Contains(err.Error(), "private-token") {
t.Fatal("missing or unsafe error", err)
}
}
if calls.Load() != 0 {
t.Fatal("invalid options reached network")
}
}
func TestJSONResponseLimitsAndStreaming(t *testing.T) {
var calls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
switch r.URL.Path {
case "/empty":
w.WriteHeader(204)
case "/chunked":
w.(http.Flusher).Flush()
io.WriteString(w, strings.Repeat("x", 17))
case "/status-stream":
w.WriteHeader(401)
w.(http.Flusher).Flush()
<-r.Context().Done()
case "/multiple":
io.WriteString(w, `{} {}`)
default:
io.WriteString(w, `{"ok":true}`)
}
}))
defer server.Close()
c := testClient(t, Config{MaxResponseBytes: 8, Timeout: time.Second})
var out struct {
OK bool `json:"ok"`
}
if err := c.JSON(context.Background(), "GET", server.URL, nil, &out, WithMaxResponseBytes(32)); err != nil || !out.OK {
t.Fatal(err)
}
if err := c.JSON(context.Background(), "GET", server.URL, nil, nil); !errors.Is(err, ErrResponseTooLarge) {
t.Fatal("limit changed permanently", err)
}
for _, target := range []any{nil, &out} {
if err := c.JSON(context.Background(), "GET", server.URL+"/chunked", nil, target, WithMaxResponseBytes(16)); !errors.Is(err, ErrResponseTooLarge) {
t.Fatal("unbounded chunked body", err)
}
}
if err := c.JSON(context.Background(), "GET", server.URL+"/empty", nil, &out); err != nil || !out.OK {
t.Fatal("empty success changed output", err)
}
if err := c.JSON(context.Background(), "GET", server.URL+"/multiple", nil, &out); err == nil {
t.Fatal("multiple JSON documents accepted")
}
start := time.Now()
err := c.JSON(context.Background(), "GET", server.URL+"/status-stream", nil, nil)
var status *StatusError
if !errors.As(err, &status) || status.StatusCode != 401 || time.Since(start) > 500*time.Millisecond {
t.Fatal("waited for unknown status body", err)
}
if calls.Load() != 7 {
t.Fatal(calls.Load())
}
}
func TestErrorBodyPoolReuseAndDrainBound(t *testing.T) {
var connections atomic.Int32
s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(401)
io.WriteString(w, "private-error-body")
}))
s.Config.ConnState = func(_ net.Conn, state http.ConnState) {
if state == http.StateNew {
connections.Add(1)
}
}
s.Start()
defer s.Close()
c := testClient(t, Config{})
for i := 0; i < 10; i++ {
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
var status *StatusError
if !errors.As(err, &status) || strings.Contains(err.Error(), "private-error-body") {
t.Fatal(err)
}
}
if connections.Load() != 1 {
t.Fatal("small status response prevented reuse", connections.Load())
}
body := &countedBody{Reader: strings.NewReader(strings.Repeat("x", 1<<20))}
custom := testClient(t, Config{Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
return &http.Response{StatusCode: 401, ContentLength: 1, Body: body, Header: make(http.Header)}, nil
})})
err := custom.JSON(context.Background(), "GET", "http://example.invalid", nil, nil)
var status *StatusError
if !errors.As(err, &status) || !body.closed || body.read > (32<<10)+1 {
t.Fatal("unbounded/missing drain", body.read, body.closed, err)
}
}
type countedBody struct {
io.Reader
read int
closed bool
}
func (b *countedBody) Read(p []byte) (int, error) {
n, err := b.Reader.Read(p)
b.read += n
return n, err
}
func (b *countedBody) Close() error { b.closed = true; return nil }
type roundTripperFunc func(*http.Request) (*http.Response, error)
func (f roundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
type borrowedTransport struct{ closed atomic.Bool }
func (t *borrowedTransport) RoundTrip(r *http.Request) (*http.Response, error) {
return benchmarkTransport{`{}`, 200}.RoundTrip(r)
}
func (t *borrowedTransport) CloseIdleConnections() { t.closed.Store(true) }
func TestBorrowedTransportAndHeaderRetry(t *testing.T) {
owned := &borrowedTransport{}
c := testClient(t, Config{Transport: owned})
c.Close()
if owned.closed.Load() {
t.Fatal("closed caller's transport")
}
var attempts atomic.Int32
retry := testClient(t, Config{MaxRetries: 1, Backoff: time.Millisecond, Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
if r.Header.Get("Authorization") != "Bearer test" {
t.Error("retry lost header")
}
status := 200
if attempts.Add(1) == 1 {
status = 503
}
return benchmarkTransport{`{}`, status}.RoundTrip(r)
})})
if err := retry.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, WithHeader("Authorization", "Bearer test")); err != nil || attempts.Load() != 2 {
t.Fatal(err, attempts.Load())
}
}
func FuzzRequestHeaderOptions(f *testing.F) {
f.Add("Authorization", "Bearer example")
f.Add("X-Name", "one\r\nX-Test: bad")
f.Fuzz(func(t *testing.T, name, value string) {
if len(name)+len(value) > 8192 {
t.Skip()
}
option := WithHeader(name, value)
var settings requestOptions
if err := option(&settings); err != nil {
return
}
req, err := http.NewRequest("GET", "http://example.invalid", nil)
if err != nil {
t.Fatal(err)
}
req.Header = settings.headers
// The standard Request writer must be able to serialize accepted headers.
if err = req.Write(io.Discard); err != nil {
t.Fatal("accepted malformed header", err)
}
})
}
func TestHTTPLogDoesNotContainCredentials(t *testing.T) {
var buffer bytes.Buffer
logger, err := logx.New(logx.Config{Writer: &buffer})
if err != nil {
t.Fatal(err)
}
defer logger.Close()
c, err := New(Config{Transport: benchmarkTransport{"private-response", 401}})
if err != nil {
t.Fatal(err)
}
defer c.Close()
ctx := logx.WithLogger(context.Background(), logger)
err = c.JSON(ctx, "GET", "http://private-user:private-password@example.invalid/private-path?token=private-query", nil, nil, WithHeader("Authorization", "Bearer private-token"))
if err == nil {
t.Fatal("expected status error")
}
for _, secret := range []string{"private-user", "private-password", "private-path", "private-query", "private-token", "private-response"} {
if strings.Contains(buffer.String(), secret) || strings.Contains(err.Error(), secret) {
t.Fatal("sensitive field leaked")
}
}
if !strings.Contains(buffer.String(), "http.client") {
t.Fatal("missing access log")
}
}
func TestCloseAllowsInflightCallToFinish(t *testing.T) {
entered := make(chan struct{})
release := make(chan struct{})
var once sync.Once
releaseNow := func() { once.Do(func() { close(release) }) }
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { close(entered); <-release; io.WriteString(w, `{}`) }))
defer s.Close()
c := testClient(t, Config{Timeout: time.Second})
done := make(chan error, 1)
go func() { done <- c.JSON(context.Background(), "GET", s.URL, nil, nil) }()
defer releaseNow()
select {
case <-entered:
case err := <-done:
t.Fatal("request failed before handler", err)
case <-time.After(2 * time.Second):
t.Fatal("handler not entered")
}
c.Close()
releaseNow()
if err := <-done; err != nil {
t.Fatal("Close interrupted in-flight call", err)
}
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); !errors.Is(err, ErrClosed) {
t.Fatal("accepted new call after Close", err)
}
}
func TestSlowKnownErrorBodyHasShortDrainBudget(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Length", "100")
w.WriteHeader(503)
w.(http.Flusher).Flush()
<-r.Context().Done()
}))
defer server.Close()
c := testClient(t, Config{Timeout: 2 * time.Second})
start := time.Now()
err := c.JSON(context.Background(), "GET", server.URL, nil, nil)
var status *StatusError
if !errors.As(err, &status) || status.StatusCode != 503 || time.Since(start) > 500*time.Millisecond {
t.Fatal("slow error body delayed status", err, time.Since(start))
}
}
func TestJSONLimitAppliesAfterGzipExpansion(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Encoding", "gzip")
writer := gzip.NewWriter(w)
_, _ = io.WriteString(writer, strings.Repeat("x", 4096))
_ = writer.Close()
}))
defer server.Close()
c := testClient(t, Config{MaxResponseBytes: 64})
if err := c.JSON(context.Background(), "GET", server.URL, nil, nil); !errors.Is(err, ErrResponseTooLarge) {
t.Fatal("gzip bypassed response limit", err)
}
}