Compare commits
4 Commits
v0.19.0-pr
...
v0.21.0-pr
| Author | SHA1 | Date | |
|---|---|---|---|
| 96e7d9629b | |||
| 068e85132f | |||
| c60841f901 | |||
| 91a6beb2f3 |
18
CHANGELOG.md
18
CHANGELOG.md
@@ -1,5 +1,23 @@
|
||||
# Changelog
|
||||
|
||||
## v0.21.0-preview(2026-09-26)
|
||||
|
||||
- httpc 可配置环境/直连/固定代理、常用连接阶段超时和独立空闲池参数。
|
||||
- JSON 新增逐请求 Header 快照与响应大小选项,保持有界读取、关闭和认证隔离。
|
||||
- 成功请求少复制和退避计算,丢弃响应流式处理,错误体有界回收以改善连接复用。
|
||||
- 增加真实本地代理、超时、连接限制/回收、并发、日志脱敏、gzip、fuzz 和修改前后基准。
|
||||
- JSON 自定义方法接口签名需同步 variadic 参数;借用 Transport 的参数混配现在报错。详见 httpc/README.md。
|
||||
|
||||
|
||||
## v0.20.0-preview(2026-09-26)
|
||||
|
||||
- 生成项目配置加载入口持续由生成器维护;保留业务字段、默认值、校验与 NewLoader 扩展文件。
|
||||
- config.LoadProject 统一 JSON/YAML/XML、环境覆盖和校验;提供实例级 Decoder 注册。
|
||||
- 文件/字段/环境/行列的安全诊断,Invalid/AtField 与原始错误链。
|
||||
- 增加格式等价、精度、null/空数组、并发、资源限制、默认值隔离、真实生成项目与性能验收。
|
||||
- 旧手写 Load 必须显式迁移;低层 API 保留。固定数组严格长度、YAML alias/merge/custom tag 拒绝。用法及限制见 CONFIG.md。
|
||||
|
||||
|
||||
## v0.19.0-preview(2026-09-25)
|
||||
|
||||
- 新增通用可选 copyx:构造器、浅/深拷贝、精确类型转换、字段规则、nil/零值策略、并发复用。
|
||||
|
||||
178
CONFIG.md
Normal file
178
CONFIG.md
Normal file
@@ -0,0 +1,178 @@
|
||||
# 统一配置加载
|
||||
|
||||
v0.20 起,新生成服务的 `main.go` 调用生成器维护的 `config.Load`。项目只需要定义业务字段、默认值和校验;日志、HTTP、数据库等配置使用同一流程,配置加载时不打开数据库、监听端口或创建日志文件。
|
||||
|
||||
```text
|
||||
代码默认值 → envDefault(仅填零值)→ 可选配置文件
|
||||
→ 完整结构体环境覆盖(含别名)→ 框架规范化 → 项目 Validate
|
||||
→ 初始化日志及项目依赖 → 启动服务
|
||||
```
|
||||
|
||||
## 生成项目要维护什么
|
||||
|
||||
| 文件 | 维护方 | 用途 |
|
||||
| --- | --- | --- |
|
||||
| `main.go` | 生成器 | 启动、信号、请求排空、依赖关闭 |
|
||||
| `internal/config/loadGen.go` | 生成器 | 调用统一加载入口;纳入 `generate --check` |
|
||||
| `internal/config/config.go` | 项目 | `Config`、`Default()`、`Validate()` |
|
||||
| `internal/config/loader.go` | 项目 | `NewLoader()`,只有增加文件格式时才需修改 |
|
||||
| `etc/config.yaml` | 项目 | 部署配置;可改用 JSON/XML 并传 `-config` |
|
||||
| `internal/svc` | 项目与生成器分文件 | 初始化依赖、登记清理、AppOptions |
|
||||
|
||||
比如增加业务的批次数量:在 Config 增加一个字段,在 Validate 加范围校验;需要部署值时改配置文件或环境变量。不修改 main、Load 或 Handler。
|
||||
|
||||
```go
|
||||
// 添加到项目 Config。
|
||||
BatchSize int `json:"batchSize" envDefault:"100"`
|
||||
|
||||
// 添加到项目 Validate。
|
||||
if c.BatchSize < 1 || c.BatchSize > 1000 {
|
||||
return omniConfig.Invalid("batchSize", "must be between 1 and 1000")
|
||||
}
|
||||
```
|
||||
|
||||
自动支持 `GO_OMNI_BATCH_SIZE`。生成服务的空环境值策略为 `ignore`,0 和 false 是有效覆盖;范围由 Validate 决定。
|
||||
重新生成保留 config.go、loader.go 和 svc 的手写扩展。
|
||||
|
||||
## 手写项目用法
|
||||
|
||||
```go
|
||||
package main
|
||||
|
||||
import (
|
||||
"time"
|
||||
"git.yuntuo.hk/k2safe/go-omni/config"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Timeout time.Duration `json:"timeout" envDefault:"3s"`
|
||||
Limit int `json:"limit" envDefault:"100"`
|
||||
Token string `json:"-" env:"APP_TOKEN,secret"`
|
||||
}
|
||||
func (c Config) Validate() error {
|
||||
if c.Timeout <= 0 {
|
||||
return config.Invalid("timeout", "must be positive")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// cfg, err := config.LoadProject("etc/config.yaml", Config{},
|
||||
// config.EnvOptions{Prefix: "APP_", Empty: "reject"})
|
||||
```
|
||||
|
||||
`LoadProject` 接收结构体默认值,可实现值接收器或指针接收器的 `Validate() error`。空路径表示不读文件。
|
||||
`DecodeFile` 只做文件覆盖,不做 envDefault、环境覆盖或业务 Validate;旧的两参数调用仍有效,可用结构体或结构体指针。低层 `ApplyDefaults`、`OverlayEnv`、`LoadFile` 继续存在。
|
||||
|
||||
嵌套校验用 `config.AtField("client", err)` 加路径前缀。公共原因通过 `config.Invalid` 提供,原因中不要包含密码、DSN、Token 或输入原值。
|
||||
|
||||
## JSON / YAML / XML
|
||||
|
||||
按文件扩展名选择解析器:`.json`、`.yaml`、`.yml`、`.xml`,大小写不敏感。所有格式使用 Go 结构体的 **json 字段名**,无需为 XML 复制一份结构或标签。
|
||||
以下配置等价:
|
||||
|
||||
```json
|
||||
{"client":{"timeout":"2s"},"enabled":false,"ports":[8080,8081],"optional":null}
|
||||
```
|
||||
|
||||
```yaml
|
||||
client:
|
||||
timeout: 2s
|
||||
enabled: false
|
||||
ports: [8080, 8081]
|
||||
optional: null
|
||||
```
|
||||
|
||||
```xml
|
||||
<config>
|
||||
<client><timeout>2s</timeout></client>
|
||||
<enabled>false</enabled>
|
||||
<ports type="array"><item>8080</item><item>8081</item></ports>
|
||||
<optional type="null"/>
|
||||
</config>
|
||||
```
|
||||
|
||||
XML 必须使用一个 `<config>` 根;普通叶节点按目标 Go 类型转换。数组显式 `type="array"` 加 `<item>`,不根据重复元素猜测类型。`type="array"` 空元素表示空数组,`type="object"` 表示空对象,`type="null"` 表示 null。`any` 中的 XML 叶节点默认是字符串,需要动态数字/布尔时标注 `type="number"` / `type="bool"`;也可用 `type="string"`。
|
||||
|
||||
类型和覆盖规则:
|
||||
|
||||
- 缺失字段保留默认值;显式 0、false、空字符串正常覆盖。
|
||||
- null 清空指针、map、slice、interface;普通值字段保持默认值,沿用 Go JSON 解码语义。需要区分 null 时用指针。
|
||||
- 数组整体替换;空 slice 保持非 nil;固定长度 Go 数组要求长度一致,避免丢弃多余配置。
|
||||
- map 合并键,已提供的键整体覆盖对应值;map 中未提供的键保留。
|
||||
- duration 支持 `250ms`、`3s`;整数保持精度,动态 JSON 数字使用 `json.Number`,不先转 float64。
|
||||
- 自定义 `json.Unmarshaler` / `encoding.TextUnmarshaler` 仍可接入。这些回调是可信项目代码,必须自己保证并发和副作用安全;不要修改共享外部状态。
|
||||
- JSON `,string` 字段、嵌入字段遵循 JSON 字段选择;未知字段、重复绑定、冲突均拒绝。
|
||||
|
||||
内置解析器限制:单文件最多 1 MiB,解析最多 64 层、100000 个节点;绑定和默认值复制也有深度限制。只接受普通文件,不读取设备或命名管道。不支持多文档、重复字段、无效 UTF-8;YAML 拒绝 alias/merge/custom tag,XML 拒绝 DTD/外部实体、命名空间、业务属性和混合内容。XML 仅支持配置所需的子集,不是通用 XML 文档映射器。
|
||||
|
||||
这些限制针对启动配置,避免意外超大文件、别名扩展和模糊覆盖。自定义值解码器的内部字段校验由该解码器负责;例如现有错误目录使用其专有解码规则。
|
||||
|
||||
## 环境覆盖
|
||||
|
||||
统一入口的 EnvOptions 控制前缀、空值策略、别名和 Lookup。完整 JSON 路径自动转换为 UPPER_SNAKE_CASE:`client.timeout` → `APP_CLIENT_TIMEOUT`。
|
||||
|
||||
```go
|
||||
config.EnvOptions{
|
||||
Prefix: "GO_OMNI_", Empty: "ignore",
|
||||
Aliases: config.ServerEnvAliases("GO_OMNI_", "server"),
|
||||
}
|
||||
```
|
||||
|
||||
生成项目同时接受完整的 `GO_OMNI_SERVER_HTTP_READ_TIMEOUT` 与短别名 `GO_OMNI_HTTP_READ_TIMEOUT`。完整名存在时优先,即便其空值被 ignore,也不回退别名。不会再二次调用 ApplyEnv 覆盖业务字段。
|
||||
|
||||
显式 `env:"CUSTOM_NAME"` 是绝对名称;`env:"-"` 跳过子树。`json:"-"` 只在声明显式 env 名时参与,可让密钥只能从环境加载。`secret` 标签表达用途,解析错误对所有字段都不打印值。slice/map 的环境值采用 JSON。`required` 检查字符串非空、指针/集合非 nil;复杂范围由 Validate 负责。详见 [完整业务环境覆盖](DEVELOPMENT.md#完整业务配置的环境覆盖)。
|
||||
|
||||
显式环境选择日志 stdout/stderr 会清除文件 profile 的 File/Rotation,除非同时提供非空 File 覆盖。该兼容规则由框架处理。
|
||||
|
||||
## 扩展解析器
|
||||
|
||||
内置格式直接可用。新增格式只在 `internal/config/loader.go` 的 `NewLoader` 注册:
|
||||
|
||||
```go
|
||||
func NewLoader() (*omniConfig.Loader, error) {
|
||||
return omniConfig.NewLoader(
|
||||
omniConfig.WithDecoder("custom", omniConfig.DecoderFunc(
|
||||
func(data []byte) (omniConfig.Document, error) {
|
||||
// 演示注册扩展名;实际可在这里调用项目选定的 TOML 等解析库。
|
||||
return (omniConfig.JSONDecoder{}).Decode(data)
|
||||
},
|
||||
)),
|
||||
)
|
||||
}
|
||||
```
|
||||
|
||||
Decoder 返回 Document:Value 是 map[string]any 根,子值为 string/bool/json.Number/nil/[]any/map[string]any;无自身类型的叶节点可返回 config.Text。Positions 可选,用 `client.timeout` 或 `servers[0].timeout` 标记 1-based 行列。不提供位置时仍有文件、字段、原因。
|
||||
|
||||
自己的语法错误可返回 `&config.Error{Field: "client.timeout", Line: 3, Column: 5, Reason: "expected a duration"}`。Reason 必须安全;普通第三方错误仅公开 `invalid configuration syntax`,内部 cause 通过 errors.Is/As 保留。
|
||||
|
||||
注册表属于 Loader 实例,无全局 init 注册;不能覆盖已有扩展名。Loader 可共享,但 Decoder 必须支持并发且不保留/修改输入。解析插件是可信 Go 代码,框架不是插件沙箱。新格式依赖由选择它的项目显式引入,核心不强制引入 TOML 等库。
|
||||
|
||||
## 定位配置错误
|
||||
|
||||
```text
|
||||
config "etc/config.yaml":4:12 field "client.timeout": expected a duration such as 5s
|
||||
config env "APP_CLIENT_TIMEOUT" field "client.timeout": expected a valid time.Duration
|
||||
```
|
||||
|
||||
`errors.As(err, &diagnostic)` 可获取 `*config.Error` 的 File/Field/Env/Line/Column/Reason;原 `DecodeError` 名称保留为别名。`errors.Is` 可判断文件不存在等 I/O 原因。
|
||||
|
||||
解析失败显示文件和可获取的行列;Validate 使用 Invalid/AtField 才能定位业务字段。环境覆盖导致的错误优先标环境变量,不错误地归因到文件;数组元素验证也会追溯到覆盖它的集合环境变量。默认值未出现在文件里时没有行列。YAML 语法错误通常只有行号,列显示 1。
|
||||
|
||||
不打印底层错误原文或配置值;任意普通 `errors.New(...)` 校验失败的公开原因是 `validation failed`,内部原因仍在链中。项目应改用 Invalid 给出安全的原因。第三方子配置校验若没有字段元数据,只定位到对应子树,框架不会从任意错误文案猜字段。
|
||||
|
||||
不要把 Unwrap 后的原始错误、整个配置对象写入公开日志/HTTP;自定义解析器与校验器传入的 Reason 由项目负责脱敏。
|
||||
|
||||
## v0.19 及更早项目迁移
|
||||
|
||||
先提交现有代码,再升级匹配的 CLI 与框架。旧项目的 `internal/config/config.go` 含手写 Load,新生成器会明确阻止 generate/check/dry-run,不会直接覆盖它。
|
||||
|
||||
1. 保留 Config、Default、Validate、业务配置结构及辅助函数。
|
||||
2. 删除旧 Load。普通的 defaults/file/env/Validate 流程已由 LoadProject 接管;移除由此不再使用的 import。将自定义默认值和校验分别迁入 Default / Validate,自定义文件格式迁入 NewLoader。
|
||||
3. 框架只在 `loader.go` 不存在时创建 NewLoader;已有项目文件同名时需要保证它提供同签名函数。无需手写 loadGen.go。
|
||||
4. 重新生成,执行 `generate --check`、`go test ./...`、`go build .`,再用真实配置验证启动。
|
||||
|
||||
保留业务需要的显式 env 标签/规则;如果原 Load 有解密或其他项目特殊步骤,不要直接删除,应先明确其归属,必要时在自定义 Decoder 中处理。生成器不自动搬动或猜测手写行为。`api upgrade` 仍可预览较早版本的基础设施差异,但已修改的配置必须按上述步骤迁移。
|
||||
|
||||
不会移除低层配置 API;本次移除的是**新项目里重复手写的加载流程**,不删除旧项目的业务文件。
|
||||
|
||||
完整可运行例子:[examples/configuration](examples/configuration/README.md)。
|
||||
@@ -1,4 +1,4 @@
|
||||
# 通用业务开发指南(v0.19.0-preview)
|
||||
# 通用业务开发指南(v0.21.0-preview)
|
||||
|
||||
从 [README](README.md) 创建服务后,业务主要维护 `.api`、配置、`svc.Init`、Logic、Repository 和项目中间件。本页说明通用接入能力;字段、错误码、权限、表结构和事务边界都由项目定义。
|
||||
|
||||
@@ -102,7 +102,9 @@ defer reader.Close()
|
||||
|
||||
## 完整业务配置的环境覆盖
|
||||
|
||||
生成项目默认执行:代码与 `envDefault` 默认值 → JSON/YAML 文件 → 全结构体环境覆盖 → 既有框架环境别名 → `Validate`。增加业务字段不需要另外写 `os.Getenv`。
|
||||
生成器维护 `internal/config/loadGen.go`;项目维护 Config/Default/Validate 和可选 NewLoader。三种格式、可扩展解析器、字段错误定位及旧 Load 迁移见 [统一配置指南](CONFIG.md)。
|
||||
|
||||
生成项目默认执行:代码与 `envDefault` 默认值 → JSON/YAML/XML 文件 → 全结构体环境覆盖(含别名)→ 规范化 → `Validate`。增加业务字段不需要另外写 `os.Getenv`。
|
||||
|
||||
```go
|
||||
type ClientConfig struct {
|
||||
@@ -278,7 +280,7 @@ go-omni workspace check --file omni.workspace.json
|
||||
|
||||
## v0.13 项目升级
|
||||
|
||||
先使用与目标框架匹配的 CLI;CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.19.0-preview(或显式引用当前源码)。
|
||||
先使用与目标框架匹配的 CLI;CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.20.0-preview(或显式引用当前源码)。
|
||||
|
||||
```sh
|
||||
go-omni api upgrade --dir . --dry-run
|
||||
|
||||
46
MODULES.md
46
MODULES.md
@@ -6,6 +6,24 @@
|
||||
[examples/modules](examples/modules/README.md)。业务入口继续使用 `omni.New`,中间件依赖
|
||||
`httpx`,不依赖具体服务端网络实现。各模块按需创建、注入 Service,再注册到应用关闭流程。
|
||||
|
||||
## 请求追踪与基础指标
|
||||
|
||||
`omni.New()` 的默认链包含 RequestID、访问日志(可配置开关)和 Recovery;无需重复挂载。业务通过 `c.Context()` 传递 requestId,`logx.FromContext(ctx)` 取得该 App 的 Logger。手动组装 httpx 链时,也可使用 `middleware.RequestID("X-Request-ID")`、`middleware.AccessLog()` 和 `middleware.Recovery()`。
|
||||
|
||||
轻量进程指标直接使用 `metrics`,无需 OTel collector:
|
||||
|
||||
```go
|
||||
// import omni "git.yuntuo.hk/k2safe/go-omni"
|
||||
// import "git.yuntuo.hk/k2safe/go-omni/metrics"
|
||||
app := omni.New()
|
||||
measurements := metrics.New()
|
||||
app.Use(measurements.Middleware())
|
||||
app.GET("/metrics", measurements.Handler())
|
||||
// 也可以直接读取 measurements.Snapshot()。
|
||||
```
|
||||
|
||||
记录请求数、失败数、处理中数量、累计耗时与 HTTP 状态分布;返回的是 JSON 快照,并非 Prometheus 文本格式。指标接口是否暴露以及访问权限由项目配置。需要 HTTP/SQL/Redis spans 和标准 OTel exporter 时,使用 [telemetry 示例](examples/platform/main.go)。CORS 由项目中间件或代理配置,当前没有内置 CORS 模块。
|
||||
|
||||
## 缓存:cache 与 cache/redis
|
||||
|
||||
```go
|
||||
@@ -52,30 +70,26 @@ application.OnShutdown(client.Close)
|
||||
|
||||
```go
|
||||
client, err := httpc.New(httpc.Config{
|
||||
Timeout: 3*time.Second, MaxRetries: 2, MaxConnsPerHost: 32,
|
||||
MaxResponseBytes: 1 << 20,
|
||||
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
|
||||
Timeout: 3*time.Second, DialTimeout: time.Second,
|
||||
TLSHandshakeTimeout: time.Second, ResponseHeaderTimeout: time.Second,
|
||||
MaxConnsPerHost: 32, MaxIdleConnsPerHost: 16, MaxIdleConns: 64,
|
||||
IdleConnTimeout: 60*time.Second, MaxResponseBytes: 1<<20,
|
||||
})
|
||||
if err != nil { return err }
|
||||
application.OnShutdown(client.Close)
|
||||
err = client.JSON(ctx, "GET", serviceURL, nil, &result)
|
||||
err = client.JSON(ctx, "GET", serviceURL, nil, &result,
|
||||
httpc.WithHeader("Authorization", "Bearer "+token),
|
||||
)
|
||||
```
|
||||
|
||||
客户端应在服务启动时创建并复用。默认使用独立的 `net/http.Transport` 连接池,每个目标主机
|
||||
有连接上限;该上限不是整个进程的全局连接数上限。也可以注入自有 `http.RoundTripper`,
|
||||
其连接限制和生命周期由调用方负责。这样出站实现同样可以替换。
|
||||
启动时创建并复用 Client。代理可选环境、直连、固定地址;不配置时保留标准库原代理策略。JSON 支持单次请求头和响应限额,不保存 Client 全局 Token。新阶段超时与空闲连接参数可选,0 沿用默认。
|
||||
|
||||
总超时覆盖建连、重试、等待和读取响应体。`Do` 返回后调用方必须读取/关闭 `response.Body`;
|
||||
`JSON` 自动关闭,并限制解码前读取的字节数。`Do` 允许流式读取,本身不限制响应体大小。
|
||||
`Close` 拒绝新请求、关闭自有池的空闲连接;在途请求应先排空。
|
||||
成功响应读取有界;output=nil 流式丢弃,不整体缓存。小错误体只在大小/时间预算内尝试读完以复用连接;未知或大错误体及时关闭。Do 保留标准流式 Request/Response,由调用者限量读取并关闭 body。Close 拒绝新调用并关闭自有池空闲连接,项目先排空在途工作。
|
||||
|
||||
`MaxRetries` 默认 0,最多 5。封装只对可重放的 GET/HEAD/OPTIONS 重试网络错误及
|
||||
429/502/503/504;POST/PUT/PATCH/DELETE 不做应用层自动重试。标准库 Transport 和借用的
|
||||
RoundTripper 自身的重放规则仍生效,不能据此承诺“只执行一次”。退避带抖动,尊重
|
||||
`Retry-After`;服务器要求等待超过 `MaxBackoff` 时直接返回该响应,不提前重试。
|
||||
重定向交给业务处理,3xx 不会自动跟随。
|
||||
自定义 Transport 借用、不修改、不关闭;不能同时指定池/代理/阶段选项。总超时持续覆盖 HTTP 等待、重试和读取 body,但不抢占 JSON 编解码或不遵守 context 的自定义代码。MaxRetries 默认 0,只对可重放 GET/HEAD/OPTIONS 重试;3xx 不跟随。标准库和借用 Transport 自身仍可能重放,不能据此承诺业务只执行一次。
|
||||
|
||||
默认从 context 传递 `X-Request-ID`,日志只包含方法、主机、状态、尝试次数和收到响应头的耗时,
|
||||
不记录完整 URL、查询参数、凭证和响应体。非 2xx 的 JSON 调用返回只带状态码的 `StatusError`。
|
||||
完整参数、NO_PROXY、TLS、安全日志、重试、签名兼容说明见 [httpc 指南](httpc/README.md),可运行 [HTTP Client 示例](examples/httpClient/README.md)。
|
||||
|
||||
## 请求保护:middleware
|
||||
|
||||
|
||||
13
PROJECT.md
13
PROJECT.md
@@ -1,6 +1,6 @@
|
||||
# 生成项目与文件维护边界
|
||||
|
||||
v0.19.0-preview 提供受管理的生成项目。安装同版远程 CLI,生成的项目直接引用该版本,无需本地 replace。根目录入口支持 `go run .` 和 `go build .`;框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。
|
||||
v0.21.0-preview 提供受管理的生成项目。安装同版远程 CLI,生成的项目直接引用该版本,无需本地 replace。根目录入口支持 `go run .` 和 `go build .`;框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。
|
||||
|
||||
## 创建和日常开发
|
||||
|
||||
@@ -38,7 +38,7 @@ ERROR [logicSignature] internal/logic/greet/greet.go:12 (GET /api/greet/:name):
|
||||
|
||||
这是轻量源码分析,不运行项目 init、不执行 Go 命令、不下载依赖。项目自定义类型别名、嵌入方法或 build tags 等无法可靠判断时,会给 REVIEW 并交给 Go 编译器验证;不把这些情况强行判为签名错误。它不检查方法体是否引用已删除字段,也不证明业务逻辑正确。CI 中应同时执行 `go-omni api generate --check` 和 `go test ./...`,按项目实际构建参数测试。当前版本首次接手缺少快照的 manifest v2 时,会明确提示建立基线,不猜测旧契约;Logic 签名检查仍会执行。
|
||||
|
||||
本版新增生成选项、公共字段组合和映射,详见 [生成能力指南](GENERATION.md)。
|
||||
生成选项、公共字段组合和可选 Go 拷贝工具,详见 [生成能力指南](GENERATION.md)。
|
||||
|
||||
## 哪些文件由用户维护
|
||||
|
||||
@@ -46,7 +46,8 @@ ERROR [logicSignature] internal/logic/greet/greet.go:12 (GET /api/greet/:name):
|
||||
| --- | --- |
|
||||
| `api/*.api` | 路由、DTO、具名中间件、响应模板、请求/响应模式 |
|
||||
| `etc/config.yaml` | 运行参数;密钥通过环境变量提供 |
|
||||
| `internal/config/config.go` | 自定义配置字段、加载、默认值与校验 |
|
||||
| `internal/config/config.go` | 自定义配置字段、默认值与校验 |
|
||||
| `internal/config/loader.go` | 需要额外文件格式时注册 Decoder |
|
||||
| `internal/svc/serviceContext.go` | 依赖字段和初始化,持有加载后的 Config |
|
||||
| `internal/svc/options.go` | AppOptions、GlobalMiddleware;注册 Go 错误目录和全局扩展 |
|
||||
| `internal/svc/databaseOptions.go` | 可选 MySQL 驱动与项目启动检查扩展 |
|
||||
@@ -62,7 +63,7 @@ ERROR [logicSignature] internal/logic/greet/greet.go:12 (GET /api/greet/:name):
|
||||
|
||||
以上骨架只补缺失文件,既有内容按字节保留。删除已废弃接口后,对应业务文件也不会被自动删除;由开发者整理编译器指出的失效类型或签名。
|
||||
|
||||
生成器持续管理:`internal/svc/runtimeGen.go`、`main.go`、`internal/handler/*Gen.go`、分组的 `handlersGen.go`、`internal/types/*Gen.go`、中间件 `registryGen.go`、`docs/openapi.json`、`.goOmni/apiManifest.json`。如果在 `.api` 里选择声明错误,还会管理 `internal/apierrors/errorsGen.go`。文件标注 DO NOT EDIT;手改会被拒绝覆盖,需先移走自定义内容。
|
||||
生成器持续管理:`internal/config/loadGen.go`、`internal/svc/runtimeGen.go`、`main.go`、`internal/handler/*Gen.go`、分组的 `handlersGen.go`、`internal/types/*Gen.go`、中间件 `registryGen.go`、`docs/openapi.json`、`.goOmni/apiManifest.json`。如果在 `.api` 里选择声明错误,还会管理 `internal/apierrors/errorsGen.go`。文件标注 DO NOT EDIT;手改会被拒绝覆盖,需先移走自定义内容。
|
||||
|
||||
Logic 实例在注册时创建并由请求并发复用;请求数据用方法参数和局部变量传递,不存入共享 Logic/svc 字段。
|
||||
|
||||
@@ -114,6 +115,8 @@ c.RequestShutdown(err)
|
||||
|
||||
`Lifecycle *lifecycle.Controller` 是 svc 持有的通用控制器,初始化时创建,生成入口使用它的 Context。该 Context 表示服务正在停止,不代表数据库已经关闭。不能把它当作在途请求的 Context,也不要在它取消时立刻关闭请求仍依赖的资源。发号器失去租约后拒绝发号等业务安全规则仍由项目实现。
|
||||
|
||||
配置加载入口 `internal/config/loadGen.go` 也由生成器维护。业务扩展 Config/Default/Validate;只有新格式才需要修改 NewLoader。JSON/YAML/XML、错误定位和旧 Load 迁移见 [配置指南](CONFIG.md)。
|
||||
|
||||
## 配置内置接口
|
||||
|
||||
生成项目在 `etc/config.yaml` 明确列出以下默认配置;可以独立关闭、修改路径,或让健康接口返回原始 JSON:
|
||||
@@ -212,7 +215,7 @@ get /download (DownloadReq)
|
||||
|
||||
## 已有服务接入步骤
|
||||
|
||||
每个独立服务生成一个项目,module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.19.0-preview:
|
||||
每个独立服务生成一个项目,module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.21.0-preview:
|
||||
|
||||
1. 先保留旧服务,整理实际路由及现有响应、鉴权和超时要求。在新目录执行 `go-omni new --api /path/to/service.api --module example.com/service ./service`;不对旧入口强制覆盖。
|
||||
2. 在 `.api` 中选择 `@app(defaultResponse: ...)`,显式保留原项目的字段、成功码、错误码和 `dataMode`。需要独立回执的路由单独声明 raw/produces。普通路由、404/405 和提前返回的中间件一起验收。
|
||||
|
||||
66
README.md
66
README.md
@@ -2,7 +2,7 @@
|
||||
|
||||
GoOmni 是面向多个业务项目复用的 Go 后端框架:用 `.api` 定义接口、请求和返回类型,生成项目目录、HTTP 接入代码及 OpenAPI,业务代码负责 Logic 和 Repository。也可以在已有项目中直接使用 `omni.New()`,或单独引入日志、存储等模块。
|
||||
|
||||
**从这份 README 开始接入。** 当前版本 **`v0.19.0-preview`**,新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入;文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md),版本变化与验收见 [发布说明](RELEASE.md)。
|
||||
**从这份 README 开始接入。** 当前版本 **`v0.21.0-preview`**,新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入;文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md),版本变化与验收见 [发布说明](RELEASE.md)。
|
||||
|
||||
**v0.19 改动**:新增可选 `copyx` 工具,用普通 Go 调用完成浅拷贝、深拷贝和显式类型转换;深拷贝会拒绝自定义转换中残留的源对象可变引用。**移除了 `mappings` / `mappingFiles` 和映射代码生成**,已有使用者须按 [迁移步骤](COPY.md#从旧-mapping-迁移) 调整,不能直接删除仍在调用的旧文件。
|
||||
|
||||
@@ -37,23 +37,71 @@ Go **1.22+**。框架不依赖 Gin,默认使用标准库网络层,提供可
|
||||
| 公共响应与特殊协议 | `.api` 定义 response 并 `@app(defaultResponse: ...)`;Go 用 `WithResponse` / Encoder | [响应配置](RESPONSE.md)、[响应模板契约](examples/contracts/responses.api) |
|
||||
| 错误目录与动态提示 | `WithErrors(catalog)` 注册一次;`ByCode(...).WithParams(...).WithCause(err)` | [错误目录](errors/README.md)、[模板参数示例](examples/contracts/errorParams.api) |
|
||||
| OpenAPI 与错误公共引用 | `.api` 自动生成;手写接口可使用 `openapi` 注册器;Go 错误目录显式导出 | [契约工具](DEVELOPMENT.md)、[注册示例](examples/platform/main.go) |
|
||||
| 结构化业务日志 | `logx.Info/Error`、`logx.From(ctx)`、`logx.New`;支持轮转、脱敏、慢请求和级别更新 | [下方日志示例](#日志)、[完整配置](logx/README.md) |
|
||||
| 文件配置与环境覆盖 | 默认值→JSON/YAML→环境;完整业务结构用 `config.LoadProject[T]` / 环境别名 | [配置详解与例子](DEVELOPMENT.md)、[动态级别](ADVANCED.md#配置与日志热更新) |
|
||||
| 结构化业务日志 | `logx.Info/Error`、`logx.FromContext(ctx)`、`logx.New`;支持轮转、脱敏、慢请求和级别更新 | [下方日志示例](#日志)、[完整配置](logx/README.md) |
|
||||
| 文件配置与环境覆盖 | 默认值→JSON/YAML/XML→环境→校验;完整业务结构用 `config.LoadProject[T]` / 环境别名 | [统一配置指南](CONFIG.md)、[完整可运行示例](examples/configuration/README.md)、[动态级别](ADVANCED.md#配置与日志热更新) |
|
||||
| 生命周期与健康检查 | svc 登记 OnClose/OnStop/OnDrain,RequestShutdown 通知故障;健康路径/开关可配 | [生命周期用法](lifecycle/README.md)、[健康检查](health/README.md)、[生成项目扩展](PROJECT.md) |
|
||||
| 数据库、连接池与事务 | `database.Open` / `OpenConnector`;事务与检查由项目显式调用 | [通用数据库](database/README.md)、[MySQL TLS/Logger](database/mysql/README.md) |
|
||||
| 可选 GORM 与迁移 | gormx 复用已有连接池;`go-omni migrate` 预览,显式 apply | [GORM 示例](database/gormx/README.md)、[迁移用法](ADVANCED.md#迁移审计与分页) |
|
||||
| 认证与权限接口 | 注入 Authenticator / Policy;JWT 或 Session 只是可选实现 | [认证说明](auth/README.md)、[鉴权完整例子](examples/platform/README.md) |
|
||||
| 缓存 | `cache.NewMemory` 或 Redis 实现;业务面向 Store 的 Get/Set/Delete | [缓存配置/代码](MODULES.md)、[可运行模块服务](examples/modules/README.md) |
|
||||
| 出站 HTTP、熔断 | `httpc.New` 复用连接池与截止时间;`resilience.New` 隔离故障依赖 | [HTTP 用法](MODULES.md)、[熔断/观测](ADVANCED.md#熔断与观测) |
|
||||
| CORS、日志、恢复、限流、幂等、超时 | `app.Use(...)` 按需组合 middleware;跨副本选择 Redis 实现 | [模块说明与限制](MODULES.md)、[分布式实现](ADVANCED.md#分布式限流与幂等) |
|
||||
| 出站 HTTP、熔断 | `httpc.New` 配置代理、连接池和阶段超时;JSON 支持单次请求头/限额;`resilience.New` 隔离故障 | [HTTP 完整用法](httpc/README.md)、[可运行示例](examples/httpClient/README.md)、[熔断/观测](ADVANCED.md#熔断与观测) |
|
||||
| 请求追踪、访问日志、恢复、限流、幂等、超时 | `app.Use(...)` 按需组合 middleware;跨副本选择 Redis 实现 | [模块说明与限制](MODULES.md#请求追踪与基础指标)、[分布式实现](ADVANCED.md#分布式限流与幂等) |
|
||||
| 本地后台任务与调度 | `task.New`、Submit、Every;有界队列、排空关闭 | [task 示例与生命周期](MODULES.md#后台执行task) |
|
||||
| 持久队列 | 业务使用 Publisher / Consumer;Redis Streams 提供至少一次交付 | [队列初始化/消费示例](ADVANCED.md#可靠队列) |
|
||||
| 文件存储、直传、云端分片 | Store 的 Put/Get/Delete;Local、S3/R2、OSS 按配置装配;扩展接口提供分片续传 | [存储接口与完整例子](storage/README.md)、[本地存储服务](examples/platform/README.md) |
|
||||
| 日志关联、指标与链路 | `metrics.New().Middleware()`;telemetry 注入 OTel exporter/reader | [观测用法](ADVANCED.md#熔断与观测)、[完整接入](examples/platform/main.go) |
|
||||
| 日志关联、指标与链路 | `metrics.New().Middleware()`;telemetry 注入 OTel exporter/reader | [基础指标](MODULES.md#请求追踪与基础指标)、[OTel 用法](ADVANCED.md#熔断与观测)、[完整接入](examples/platform/main.go) |
|
||||
| 审计与分页 | `audit.New` 注入 Sink/白名单;`pagination.Parse` / `NewResult` | [解释与调用例子](ADVANCED.md#迁移审计与分页) |
|
||||
| 可选结构转换与深拷贝 | `copyx.Copy` / `Clone` / `New`;Go converter 处理差异 | [下方示例](#可选结构转换与深拷贝)、[完整 copyx 指南](COPY.md)、[可运行代码](examples/copying) |
|
||||
| 自定义适配器验收 | `transporttest.Run`、`storagetest.Run` / `RunUploads` 验证扩展契约 | [网络契约测试入口](transport/transporttest/router.go)、[存储契约测试入口](storage/storagetest/contract.go) |
|
||||
|
||||
## 带认证的出站 HTTP
|
||||
|
||||
HTTP Client 在 svc 初始化一次,业务重复使用。普通内部请求可直接选择直连和连接参数,无需手写 Transport:
|
||||
|
||||
```go
|
||||
client, err := httpc.New(httpc.Config{
|
||||
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
|
||||
Timeout: 3*time.Second, DialTimeout: time.Second,
|
||||
TLSHandshakeTimeout: time.Second, ResponseHeaderTimeout: time.Second,
|
||||
MaxConnsPerHost: 32, MaxIdleConnsPerHost: 16, MaxIdleConns: 64,
|
||||
})
|
||||
if err != nil { return err }
|
||||
// 在 svc 生命周期登记 client.Close;关闭前先排空使用它的工作。
|
||||
err = client.JSON(ctx, "GET", endpoint, nil, &result,
|
||||
httpc.WithHeader("Authorization", "Bearer "+token),
|
||||
httpc.WithMaxResponseBytes(64<<10),
|
||||
)
|
||||
```
|
||||
|
||||
请求头只作用于本次调用,不把用户 Token 留在共享 Client。默认代理/参数保持原行为;也可选择环境代理或固定代理。JSON 自动关闭并限制响应,Do 保留下载/流式读取扩展。完整配置、重试边界和已有自定义 Transport 的迁移见 [httpc](httpc/README.md)。
|
||||
|
||||
```sh
|
||||
go run ./examples/httpClient
|
||||
```
|
||||
|
||||
## 统一配置加载
|
||||
|
||||
新生成项目的加载流程由框架管理。业务在 `internal/config/config.go` 增加字段、默认值和 `Validate`,无需改 main 或复制环境变量解析。JSON、YAML、XML 共用字段和校验;扩展格式在 `loader.go` 注册。
|
||||
|
||||
```go
|
||||
// 项目 Config 中增加字段,自动支持 GO_OMNI_BATCH_SIZE:
|
||||
BatchSize int `json:"batchSize" envDefault:"100"`
|
||||
|
||||
// 项目 Validate 中增加业务范围检查:
|
||||
if c.BatchSize < 1 {
|
||||
return omniConfig.Invalid("batchSize", "must be positive")
|
||||
}
|
||||
```
|
||||
|
||||
错误包含文件、字段、可用的行列或环境变量名;输入原值不自动写入错误。
|
||||
|
||||
```sh
|
||||
go run ./examples/configuration -format xml
|
||||
DEMO_CLIENT_TIMEOUT=500ms go run ./examples/configuration
|
||||
```
|
||||
|
||||
详见 [统一加载、解析器、迁移](CONFIG.md) 和 [完整示例](examples/configuration/README.md)。**旧项目须先迁移手写 Load,再重新生成**;生成器会保留业务配置并拒绝冲突,不静默覆盖。
|
||||
|
||||
## 选择接入方式
|
||||
|
||||
| 你的项目 | 从哪里开始 |
|
||||
@@ -74,7 +122,7 @@ module 地址是 `git.yuntuo.hk/k2safe/go-omni`。先确保本机 Go 和 Git 可
|
||||
|
||||
```sh
|
||||
export GOPRIVATE="$(go env GOPRIVATE),git.yuntuo.hk"
|
||||
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.19.0-preview
|
||||
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.21.0-preview
|
||||
export PATH="$(go env GOPATH)/bin:$PATH"
|
||||
go-omni --help
|
||||
```
|
||||
@@ -231,7 +279,7 @@ go-omni new --api catalog.api \
|
||||
在业务项目的 **go.mod 所在目录**执行:
|
||||
|
||||
```sh
|
||||
go get git.yuntuo.hk/k2safe/go-omni@v0.19.0-preview
|
||||
go get git.yuntuo.hk/k2safe/go-omni@v0.21.0-preview
|
||||
```
|
||||
|
||||
已有项目可直接复用现有目录、Service、Repository 和数据库连接,先选择一个接口接入。也可以只用 `logx`、`storage` 等包,暂时保留原来的 HTTP 框架。希望入口和 HTTP 接入都交给生成器的项目,按 [已有服务接入步骤](PROJECT.md#已有服务接入步骤) 在新目录生成后迁入业务代码。
|
||||
@@ -533,7 +581,7 @@ go test ./...
|
||||
|
||||
`main.go`、`Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理,手动修改会报告冲突。Logic、Repository、中间件实现、配置和依赖装配属于项目,重复生成保留手写内容。契约变化会提示受影响的 Logic 文件和 DTO 字段,明确的签名错误会给出原签名与期望签名;生成后 `--check` 仍能检出未修复的问题。将 `.api`、`omni.gen.json`、生成文件和 `.goOmni/apiManifest.json` 一起提交,CI 同时运行 `generate --check` 与 `go test ./...`。详见 [契约变更提示与检查边界](PROJECT.md#契约变更提示)。
|
||||
|
||||
CLI 与业务 module 固定使用 `v0.19.0-preview`。v0.14.x 项目升级依赖与 CLI 后重新生成,框架维护的 main/runtimeGen 自动更新;v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理,不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移),旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。
|
||||
CLI 与业务 module 固定使用 `v0.21.0-preview`。v0.14–v0.19 项目先按 [配置入口迁移](CONFIG.md#v019-及更早项目迁移) 整理旧 Load,再升级依赖与 CLI 并重新生成,框架维护的 main/runtimeGen 自动更新;v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理,不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移),旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。
|
||||
|
||||
## 按需深入
|
||||
|
||||
|
||||
49
RELEASE.md
49
RELEASE.md
@@ -1,4 +1,49 @@
|
||||
# v0.19.0-preview
|
||||
# v0.21.0-preview
|
||||
|
||||
让带认证的 HTTP 调用和内部直连可以直接使用 httpc,减少手写 Transport 与 JSON 响应处理。
|
||||
|
||||
- Proxy 支持环境、直连、固定代理;默认保留既有策略。
|
||||
- 建连、TCP keepalive、TLS 握手、响应头、100-continue、空闲超时和独立空闲连接数量可选配置;不修改全局 Transport,不关闭借用实例。
|
||||
- JSON 支持 WithHeader / WithHeaders / WithMaxResponseBytes;认证信息逐请求隔离,自动关闭和有界响应处理保留。
|
||||
- 成功路径减少请求复制和无用退避;output=nil 按限额流式丢弃;小错误体在大小和时间预算内尝试复用连接。
|
||||
- 不新增依赖、不增加业务规则;Do 保留标准流式扩展。CLI 最新版本与模板最低版本分开,本轮模板兼容门槛仍是 v0.20。
|
||||
|
||||
```sh
|
||||
go get git.yuntuo.hk/k2safe/go-omni@v0.21.0-preview
|
||||
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.21.0-preview
|
||||
```
|
||||
|
||||
**兼容注意**:JSON 旧的直接调用仍有效,自定义 interface/函数值需要同步新增的 variadic 参数。自定义 Transport 与显式连接/代理/阶段参数混用现在报错,避免参数被忽略;迁移时在自己的 Transport 配置,或改用内置字段。没有删除原调用入口。
|
||||
|
||||
[参数与使用说明](httpc/README.md)、[可运行示例](examples/httpClient/README.md)、[场景与性能验收](reports/httpcAcceptance.md) 已同步到 README。48 个测试包全量 race、Go 1.22 兼容、vet/build、约 110 万次请求头 fuzz 通过。性能来自本机同条件基准,不能代替项目实际流量验证;仍为 preview。
|
||||
|
||||
---
|
||||
|
||||
# v0.20.0-preview(历史版本)
|
||||
|
||||
统一配置加载,减少生成项目重复维护的启动代码。
|
||||
|
||||
- 生成器维护 `internal/config/loadGen.go`,调用 `config.LoadProject`;项目只扩展 Config/Default/Validate,可选在 loader.go/NewLoader 注册新格式。入口、环境覆盖、日志输出规范化无需业务重复实现。
|
||||
- JSON/YAML/XML 使用同一个加载与绑定流程;Decoder 通过实例注册,无全局 init 或额外强制依赖。内置格式、字段、文件大小与层数均有明确限制。
|
||||
- `config.Error` 携带文件、字段、环境变量、可用行列及安全原因;`Invalid` / `AtField` 支持业务验证。保留内部错误链,不自动输出原值;大整数、空集合、null、默认值隔离和自定义反序列化有回归覆盖。
|
||||
- 配置解析在日志、依赖和监听初始化之前;生成/check/dry-run 使用一致入口和文件所有权规则。
|
||||
- **迁移要求:旧项目先整理手写 Load,再重新生成。** 不静默覆盖业务文件。低层配置 API 保留;新项目不再复制手写加载流程。固定 Go 数组严格检查长度,YAML alias/merge/custom tag 不再接受,避免模糊覆盖和扩展行为。
|
||||
|
||||
```sh
|
||||
go get git.yuntuo.hk/k2safe/go-omni@v0.20.0-preview
|
||||
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.20.0-preview
|
||||
# 先按 CONFIG.md 迁移旧 Load,再执行:
|
||||
go-omni api generate
|
||||
go-omni api generate --check
|
||||
go test ./...
|
||||
go build .
|
||||
```
|
||||
|
||||
完整用法、格式、脱敏边界和迁移见 [统一配置指南](CONFIG.md),可运行例子见 [examples/configuration](examples/configuration/README.md)。回归、模糊测试与启动加载性能见 [验收记录](reports/configurationAcceptance.md)。47 个测试包的全量 race、Go 1.22 兼容、vet/build、30 秒 fuzz 与 5 个基准场景三轮测量通过。空缓存远程安装和 8 组生成消费者回归通过,无本地 replace;标签固定在 `c60841f901a3bcc3e9582eece6cfb2cad008fb9e`,见 [远程记录](reports/configurationRemote.json)。仍为 preview;项目的实际配置和业务依赖应自行验收。
|
||||
|
||||
---
|
||||
|
||||
# v0.19.0-preview(历史版本)
|
||||
|
||||
将可选结构转换改为直接 Go 调用,减少业务项目需要维护的配置。普通接口继续使用 `.api` 生成 DTO,不要求增加独立模型或任何拷贝规则。
|
||||
|
||||
@@ -19,7 +64,7 @@ go test ./...
|
||||
|
||||
迁移步骤和语义边界见 [copyx](COPY.md#从旧-mapping-迁移)。新工具做运行时检查,不能声称具备旧生成器的编译期字段覆盖保证;实际类型对必须有测试。独立三端工资示例验证金额精度、缺失/null/零值、空集合、错误链和字段新增;不修改或代替接入项目的完整业务回归。
|
||||
|
||||
46 个测试包的全量 race(含实际生成项目)、vet/build、Go 1.22 兼容测试通过;30 秒 fuzz 无失败,14 种基准场景各测量三轮。验收条件、性能数据及本轮未覆盖的外部环境见 [v0.19 验收记录](reports/copyxAcceptance.md)。本版仍为 preview。
|
||||
46 个测试包的全量 race(含实际生成项目)、vet/build、Go 1.22 兼容测试通过;30 秒 fuzz 无失败,14 种基准场景各测量三轮。验收条件、性能数据及本轮未覆盖的外部环境见 [v0.19 验收记录](reports/copyxAcceptance.md)。本版仍为 preview。 空缓存远程 CLI 与 7 组真实生成消费者验收通过,无本地 replace;标签固定在 `fd83235ce7fc15ce88a482d385d70e61f25d7b32`,详见 [远程记录](reports/copyxRemote.json)。
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ v0.14 新增 `api format [--check]`、`api upgrade [--dry-run|--preview <空目
|
||||
|
||||
```sh
|
||||
# 仓库读取权限与 PATH 配置见根目录 README
|
||||
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.19.0-preview
|
||||
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.21.0-preview
|
||||
go-omni new --module example.com/hello ./hello
|
||||
|
||||
# 从自己的契约生成;service.api 及其相对 import 文件需要在本地存在
|
||||
@@ -49,14 +49,14 @@ main、Gen.go、OpenAPI 和 manifest 由生成器维护;业务文件按字节
|
||||
|
||||
## 版本与迁移
|
||||
|
||||
CLI 与框架依赖固定使用 v0.19.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod,不添加本地 replace;`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换:
|
||||
CLI 与框架依赖固定使用 v0.20.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod,不添加本地 replace;`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换:
|
||||
|
||||
```sh
|
||||
# 在 go-omni 源码仓库中执行
|
||||
go run ./cmd/go-omni new --module example.com/hello --framework-dir . ../hello
|
||||
```
|
||||
|
||||
当前模板与依赖以 v0.19.0-preview 为基线,继续使用 manifest v2;GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级):原始模板可自动替换,含业务修改的基础文件按预览合并;通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务,不使用旧 `add` 命令的手动路由装配。
|
||||
当前模板与依赖以 v0.20.0-preview 为基线,继续使用 manifest v2;GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级):原始模板可自动替换,含业务修改的基础文件按预览合并;通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务,不使用旧 `add` 命令的手动路由装配。
|
||||
|
||||
## 数据库迁移命令
|
||||
|
||||
@@ -69,3 +69,7 @@ go-omni migrate --dir migrations --apply
|
||||
```
|
||||
|
||||
SQL 命名示例 `0001_createUsers.up.sql`;预览只显示版本和 SHA-256,不连接数据库。失败保留 dirty 状态,不自动 down/force。细节见 [扩展模块](../../ADVANCED.md)。
|
||||
|
||||
### v0.20 配置入口
|
||||
|
||||
新生成 `internal/config/loadGen.go` 调用统一加载,纳入生成检查;项目 config.go 只维护字段、默认值和校验,loader.go 用于注册额外格式。JSON/YAML/XML 内置。旧手写 Load 会阻止生成,请先按 [迁移步骤](../../CONFIG.md#v019-及更早项目迁移) 整理,不会自动覆盖业务代码。
|
||||
|
||||
113
cmd/go-omni/apigen/configLoader_test.go
Normal file
113
cmd/go-omni/apigen/configLoader_test.go
Normal file
@@ -0,0 +1,113 @@
|
||||
package apigen
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestGeneratedConfigurationConsumer(t *testing.T) {
|
||||
if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") != "1" {
|
||||
t.Skip("set GO_OMNI_SCAFFOLD_INTEGRATION=1")
|
||||
}
|
||||
dir := filepath.Join(t.TempDir(), "app")
|
||||
api := filepath.Join(t.TempDir(), "service.api")
|
||||
writeFile(t, api, "service configuration {\n@handler ping\nget /ping\n}\n")
|
||||
o := Options{API: api, Dir: dir, Module: "example.com/configuration", FrameworkDir: frameworkRoot(t)}
|
||||
if err := generateManagedTestProject(t, o); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
main := readFile(t, filepath.Join(dir, "main.go"))
|
||||
// The business only changes fields/defaults/validation, not the loading flow.
|
||||
writeFile(t, filepath.Join(dir, "internal/config/config.go"), `package config
|
||||
import("git.yuntuo.hk/k2safe/go-omni/config";"git.yuntuo.hk/k2safe/go-omni/logx")
|
||||
type Config struct { Server config.Config `+"`json:\"server\"`"+`;Limit int `+"`json:\"limit\" envDefault:\"10\"`"+` }
|
||||
func Default()Config{ c:=config.Default();c.Log.File="./var/log/app.log";c.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:10};return Config{Server:c} }
|
||||
func(c Config)Validate()error{if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)};if c.Limit<0{return config.Invalid("limit","must not be negative")};return nil}
|
||||
`)
|
||||
writeFile(t, filepath.Join(dir, "internal/config/loader.go"), `package config
|
||||
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
|
||||
func NewLoader()(*omniConfig.Loader,error){return omniConfig.NewLoader(omniConfig.WithDecoder("custom",omniConfig.JSONDecoder{}))}
|
||||
`)
|
||||
svc := filepath.Join(dir, "internal/svc/serviceContext.go")
|
||||
writeFile(t, svc, strings.Replace(readFile(t, svc), "return nil", "panic(\"dependencies must not initialize on invalid config\")", 1))
|
||||
writeFile(t, filepath.Join(dir, "consumer_test.go"), readFile(t, "testdata/configuration/consumer_test.go"))
|
||||
runConsumerGo(t, dir, "mod", "tidy")
|
||||
runConsumerGo(t, dir, "test", "-race", "-count=1", "./...")
|
||||
runConsumerGo(t, dir, "build", ".")
|
||||
o.API = filepath.Join(dir, "api/service.api")
|
||||
before := snapshot(t, dir)
|
||||
for _, mode := range []string{"dry", "generate", "check"} {
|
||||
o.DryRun = mode == "dry"
|
||||
o.Check = mode == "check"
|
||||
if err := generateManagedTestProject(t, o); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if !sameFiles(before, snapshot(t, dir)) {
|
||||
t.Fatal("business files overwritten")
|
||||
}
|
||||
if main != readFile(t, filepath.Join(dir, "main.go")) {
|
||||
t.Fatal("main changed")
|
||||
}
|
||||
load := filepath.Join(dir, "internal/config/loadGen.go")
|
||||
original := readFile(t, load)
|
||||
writeFile(t, load, original+"\n// drift\n")
|
||||
if err := generateManagedTestProject(t, o); err == nil {
|
||||
t.Fatal("loadGen drift accepted")
|
||||
}
|
||||
writeFile(t, load, original)
|
||||
if os.Getenv("GO_OMNI_RELEASE_CLI") != "" {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, "go", "list", "-m", "-json", frameworkModule)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), "GOWORK=off")
|
||||
data, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatal(err, string(data))
|
||||
}
|
||||
var dependency struct {
|
||||
Version, Sum string
|
||||
Replace any
|
||||
}
|
||||
if err = json.Unmarshal(data, &dependency); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if dependency.Version != os.Getenv("GO_OMNI_RELEASE_VERSION") || dependency.Replace != nil || dependency.Sum == "" {
|
||||
t.Fatal("not a remote consumer")
|
||||
}
|
||||
t.Logf("remote framework %s checksum %s; no local replace", dependency.Version, dependency.Sum)
|
||||
}
|
||||
}
|
||||
func TestConfigLoadMigrationGuard(t *testing.T) {
|
||||
dir := filepath.Join(t.TempDir(), "app")
|
||||
o := Options{Dir: dir, Module: "example.com/upgrade", FrameworkDir: frameworkRoot(t)}
|
||||
if _, err := New(o); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
o.API = filepath.Join(dir, "api/service.api")
|
||||
matches, err := filepath.Glob(filepath.Join(dir, "api/*.api"))
|
||||
if err != nil || len(matches) != 1 {
|
||||
t.Fatal(matches, err)
|
||||
}
|
||||
o.API = matches[0]
|
||||
writeFile(t, filepath.Join(dir, "internal/config/legacy.go"), "package config\nfunc Load(path string)(Config,error){return Default(),nil}\n")
|
||||
before := snapshot(t, dir)
|
||||
for _, mode := range []string{"dry", "generate", "check"} {
|
||||
o.DryRun = mode == "dry"
|
||||
o.Check = mode == "check"
|
||||
_, err := Generate(o)
|
||||
if err == nil || !strings.Contains(err.Error(), "CONFIG.md") {
|
||||
t.Fatal("missing migration diagnostic", err)
|
||||
}
|
||||
if !sameFiles(before, snapshot(t, dir)) {
|
||||
t.Fatal("migration guard changed files")
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2,6 +2,7 @@ package apigen
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"os"
|
||||
"os/exec"
|
||||
@@ -92,4 +93,26 @@ func(s *Logic)Save(ctx context.Context,req *types.%sRequest)(*types.EmployeeResp
|
||||
}
|
||||
runConsumerGo(t, dir, "test", "-race", "-count=1", "./...")
|
||||
runConsumerGo(t, dir, "build", ".")
|
||||
if os.Getenv("GO_OMNI_RELEASE_CLI") != "" {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, "go", "list", "-m", "-json", frameworkModule)
|
||||
cmd.Dir = dir
|
||||
cmd.Env = append(os.Environ(), "GOWORK=off")
|
||||
data, err := cmd.CombinedOutput()
|
||||
if err != nil {
|
||||
t.Fatal(err, string(data))
|
||||
}
|
||||
var dependency struct {
|
||||
Version, Sum string
|
||||
Replace any
|
||||
}
|
||||
if err = json.Unmarshal(data, &dependency); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if dependency.Version != os.Getenv("GO_OMNI_RELEASE_VERSION") || dependency.Replace != nil || dependency.Sum == "" {
|
||||
t.Fatal("not a fixed remote consumer")
|
||||
}
|
||||
t.Logf("remote framework %s checksum %s; no local replace", dependency.Version, dependency.Sum)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -19,7 +19,10 @@ import (
|
||||
)
|
||||
|
||||
const frameworkModule = "git.yuntuo.hk/k2safe/go-omni"
|
||||
const minimumVersion = "v0.19.0-preview"
|
||||
|
||||
// Runtime-only releases need not raise the managed template compatibility floor.
|
||||
const minimumVersion = "v0.20.0-preview"
|
||||
const currentVersion = "v0.21.0-preview"
|
||||
const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n"
|
||||
|
||||
type Options struct {
|
||||
@@ -106,6 +109,9 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
|
||||
if err = checkRuntimeExtension(directory); err != nil {
|
||||
return Output{}, err
|
||||
}
|
||||
if err = checkConfigLoadExtension(directory); err != nil {
|
||||
return Output{}, err
|
||||
}
|
||||
for _, r := range c.Spec.Routes {
|
||||
if !r.Manual {
|
||||
path := filepath.Join(directory, "internal/handler", r.Group, lower(r.Handler)+"Manual.go")
|
||||
@@ -152,7 +158,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
|
||||
if !filepath.IsAbs(path) {
|
||||
path = filepath.Join(directory, path)
|
||||
}
|
||||
if source, err := os.ReadFile(filepath.Join(path, "config", "env.go")); err == nil && strings.Contains(string(source), "func ServerEnvAliases(") {
|
||||
if source, err := os.ReadFile(filepath.Join(path, "config", "loader.go")); err == nil && strings.Contains(string(source), "func NewLoader(") {
|
||||
localCurrent = true
|
||||
}
|
||||
}
|
||||
@@ -262,7 +268,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
|
||||
return Output{}, errors.New("generation configuration changed during generation; retry")
|
||||
}
|
||||
out, err := apply(directory, o.Module, entry, generated, seeds, o.Check, o.DryRun, o.profile(), snapshotContract(c.Spec), o.lockHeld)
|
||||
expectedVersion := minimumVersion
|
||||
expectedVersion := currentVersion
|
||||
if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" && semver.IsValid(info.Main.Version) {
|
||||
expectedVersion = info.Main.Version
|
||||
}
|
||||
@@ -301,7 +307,7 @@ func moduleFile(o Options) ([]byte, error) {
|
||||
if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" {
|
||||
version = info.Main.Version
|
||||
} else {
|
||||
version = minimumVersion
|
||||
version = currentVersion
|
||||
}
|
||||
}
|
||||
if !semver.IsValid(version) || semver.Major(version) != "v0" && semver.Major(version) != "v1" {
|
||||
|
||||
@@ -58,6 +58,8 @@ Unimplemented/nil factories fail startup and clean registered dependencies.
|
||||
Existing projects retain middleware: handlers until explicitly migrated.
|
||||
Ordinary Logic uses the generated Request/Response types directly. Where an
|
||||
independent business type is needed, use ordinary Go or the optional copyx package.
|
||||
Configuration loading is generator-managed in internal/config/loadGen.go. Extend fields/defaults/Validate in config.go; additional decoders belong in loader.go/NewLoader. JSON/YAML/XML use one pipeline. See the framework CONFIG.md.
|
||||
|
||||
There are no mappings/mappingFiles settings or generated mapping files in v0.19.
|
||||
See COPY.md in the framework repository for copy policies and migration steps,
|
||||
and GENERATION.md for field composition and middleware factories.
|
||||
|
||||
59
cmd/go-omni/apigen/testdata/configuration/consumer_test.go
vendored
Normal file
59
cmd/go-omni/apigen/testdata/configuration/consumer_test.go
vendored
Normal file
@@ -0,0 +1,59 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"example.com/configuration/internal/config"
|
||||
omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGeneratedConfiguration(t *testing.T) {
|
||||
for ext, source := range map[string]string{
|
||||
"json": `{"limit":7}`, "yaml": "limit: 7\n", "xml": "<config><limit>7</limit></config>", "custom": `{"limit":7}`,
|
||||
} {
|
||||
t.Run(ext, func(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "app."+ext)
|
||||
if err := os.WriteFile(path, []byte(source), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cfg, err := config.Load(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.Limit != 7 {
|
||||
t.Fatal(cfg.Limit)
|
||||
}
|
||||
t.Setenv("GO_OMNI_LIMIT", "0")
|
||||
cfg, err = config.Load(path)
|
||||
if err != nil || cfg.Limit != 0 {
|
||||
t.Fatal(cfg.Limit, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
t.Setenv("GO_OMNI_LOG_OUTPUT", "stderr")
|
||||
cfg, err := config.Load("")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.Limit != 10 || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled {
|
||||
t.Fatal("defaults/log normalization", cfg, err)
|
||||
}
|
||||
path := filepath.Join(t.TempDir(), "bad.yaml")
|
||||
os.WriteFile(path, []byte("limit: -1\n"), 0600)
|
||||
// Validation must run before svc.Initialize; the project initializer panics
|
||||
// below if reached, so this also detects accidentally starting dependencies.
|
||||
err = run(context.Background(), path)
|
||||
var diagnostic *omniConfig.Error
|
||||
if !errors.As(err, &diagnostic) || diagnostic.Field != "limit" || diagnostic.File != path || diagnostic.Line != 1 {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Setenv("GO_OMNI_LIMIT", "private-not-a-number")
|
||||
_, err = config.Load(path)
|
||||
if !errors.As(err, &diagnostic) || diagnostic.Env != "GO_OMNI_LIMIT" || strings.Contains(err.Error(), "private-not-a-number") {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -30,6 +30,39 @@ type UpgradePlan struct {
|
||||
Module, Entry string
|
||||
}
|
||||
|
||||
// Handwritten old loaders must be migrated explicitly, never silently replaced
|
||||
// or left competing with the generated entry point.
|
||||
func checkConfigLoadExtension(root string) error {
|
||||
dir := filepath.Join(root, "internal/config")
|
||||
entries, err := os.ReadDir(dir)
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
return nil
|
||||
}
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, entry := range entries {
|
||||
name := entry.Name()
|
||||
if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") || name == "loadGen.go" {
|
||||
continue
|
||||
}
|
||||
rel := "internal/config/" + name
|
||||
if err := safePath(root, rel); err != nil {
|
||||
return err
|
||||
}
|
||||
f, err := goparser.ParseFile(gotoken.NewFileSet(), filepath.Join(root, rel), nil, 0)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse %s: %w", rel, err)
|
||||
}
|
||||
for _, decl := range f.Decls {
|
||||
if fn, ok := decl.(*ast.FuncDecl); ok && fn.Recv == nil && fn.Name.Name == "Load" {
|
||||
return fmt.Errorf("%s: migrate the project-owned Load function before generating internal/config/loadGen.go; retain Config/Default/Validate and move decoder options to NewLoader; see CONFIG.md", rel)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// PlanUpgrade compares infrastructure against the exact previous release seeds.
|
||||
// A custom file is never classified as safe to replace. Logic, middleware,
|
||||
// options and repositories outside the changed templates are left untouched.
|
||||
|
||||
@@ -39,7 +39,7 @@ func oldInfrastructureProject(t *testing.T) string {
|
||||
raw, _ := json.Marshal(m)
|
||||
writeFile(t, filepath.Join(dir, manifestPath), string(raw))
|
||||
path := filepath.Join(dir, "go.mod")
|
||||
writeFile(t, path, strings.ReplaceAll(readFile(t, path), minimumVersion, "v0.13.0-preview"))
|
||||
writeFile(t, path, strings.ReplaceAll(readFile(t, path), currentVersion, "v0.13.0-preview"))
|
||||
return dir
|
||||
}
|
||||
|
||||
@@ -74,7 +74,7 @@ func TestUpgradeV014PreservesExtensions(t *testing.T) {
|
||||
raw, _ := json.Marshal(m)
|
||||
writeFile(t, filepath.Join(dir, manifestPath), string(raw))
|
||||
mod := filepath.Join(dir, "go.mod")
|
||||
writeFile(t, mod, strings.ReplaceAll(readFile(t, mod), minimumVersion, "v0.14.1-preview"))
|
||||
writeFile(t, mod, strings.ReplaceAll(readFile(t, mod), currentVersion, "v0.14.1-preview"))
|
||||
owned := map[string]string{}
|
||||
for _, name := range []string{"internal/svc/serviceContext.go", "internal/config/config.go", "etc/config.yaml"} {
|
||||
path := filepath.Join(dir, name)
|
||||
|
||||
@@ -284,6 +284,9 @@ func apply(root, module, entry string, generated, seeds map[string][]byte, check
|
||||
}
|
||||
func digest(data []byte) string { sum := sha256.Sum256(data); return hex.EncodeToString(sum[:]) }
|
||||
func ownedPath(name string) bool {
|
||||
if name == "internal/config/loadGen.go" {
|
||||
return true
|
||||
}
|
||||
if strings.HasPrefix(name, "internal/types/dto") && strings.HasSuffix(name, "Gen.go") && goName.MatchString(strings.TrimSuffix(filepath.Base(name), ".go")) {
|
||||
return true
|
||||
}
|
||||
|
||||
@@ -8,7 +8,7 @@ import (
|
||||
)
|
||||
|
||||
func TestNewWithRemoteFrameworkVersion(t *testing.T) {
|
||||
for _, flag := range [][]string{{"--framework-version", "v0.19.0-preview"}, {"--framework-version=v0.19.0-preview"}} {
|
||||
for _, flag := range [][]string{{"--framework-version", "v0.21.0-preview"}, {"--framework-version=v0.21.0-preview"}} {
|
||||
project := filepath.Join(t.TempDir(), "billing")
|
||||
args := append([]string{"--module", "example.com/billing"}, flag...)
|
||||
if err := runNew(append(args, project)); err != nil {
|
||||
@@ -18,7 +18,7 @@ func TestNewWithRemoteFrameworkVersion(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !strings.Contains(string(content), "go-omni v0.19.0-preview") || strings.Contains(string(content), "replace ") {
|
||||
if !strings.Contains(string(content), "go-omni v0.21.0-preview") || strings.Contains(string(content), "replace ") {
|
||||
t.Fatalf("CLI did not pin remote dependency: %s", content)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -21,6 +21,8 @@ func apiInfrastructure(module, database, storage, logOutput, logFormat string, l
|
||||
"internal/svc/runtimeGen.go": "templates/api-runtime.go.tmpl",
|
||||
"internal/svc/options.go": "templates/api-options.go.tmpl",
|
||||
"internal/config/config.go": "templates/api-config.go.tmpl",
|
||||
"internal/config/loadGen.go": "templates/api-load.go.tmpl",
|
||||
"internal/config/loader.go": "templates/api-loader.go.tmpl",
|
||||
"etc/config.yaml": "templates/modules-config.yaml.tmpl",
|
||||
"internal/svc/serviceContext.go": "templates/api-context.go.tmpl",
|
||||
}
|
||||
@@ -31,6 +33,8 @@ func apiInfrastructure(module, database, storage, logOutput, logFormat string, l
|
||||
files["internal/svc/storage.go"] = "templates/modules-storage.go.tmpl"
|
||||
}
|
||||
if legacy {
|
||||
delete(files, "internal/config/loadGen.go")
|
||||
delete(files, "internal/config/loader.go")
|
||||
delete(files, "internal/svc/runtimeGen.go")
|
||||
delete(files, "internal/svc/databaseOptions.go")
|
||||
files["internal/svc/close.go"] = "templates/api-close.go.tmpl"
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
package settings
|
||||
import (
|
||||
"os"
|
||||
{{if and (not .API) (ne .Storage "none")}}"strings"{{end}}
|
||||
"time"
|
||||
"git.yuntuo.hk/k2safe/go-omni/config"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
{{if eq .LogOutput "file"}}"git.yuntuo.hk/k2safe/go-omni/logx"{{end}}
|
||||
{{if eq .Database "mysql"}}mysqladapter "git.yuntuo.hk/k2safe/go-omni/database/mysql"{{end}}
|
||||
{{if ne .Storage "none"}}"errors"
|
||||
{{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}}
|
||||
@@ -35,22 +34,10 @@ func Default() Config {
|
||||
}{{end}}}
|
||||
}
|
||||
|
||||
// Load applies defaults -> file -> typed environment -> validation.
|
||||
// New fields automatically use GO_OMNI_<JSON_PATH>, or an explicit env tag.
|
||||
func Load(path string)(Config,error){
|
||||
cfg:=Default();var err error
|
||||
cfg,err=config.ApplyDefaults(cfg);if err!=nil{return Config{},err}
|
||||
if path!="" {cfg,err=config.DecodeFile(path,cfg);if err!=nil{return Config{},err}}
|
||||
cfg,err=config.OverlayEnv(cfg,config.EnvOptions{Prefix:"GO_OMNI_",Empty:"ignore",Aliases:config.ServerEnvAliases("GO_OMNI_","server")});if err!=nil{return Config{},err}
|
||||
output,outputSet:=os.LookupEnv("GO_OMNI_SERVER_LOG_OUTPUT");if !outputSet {output=os.Getenv("GO_OMNI_LOG_OUTPUT")}
|
||||
file,fileSet:=os.LookupEnv("GO_OMNI_SERVER_LOG_FILE");if !fileSet {file=os.Getenv("GO_OMNI_LOG_FILE")}
|
||||
if output!="" && file=="" {cfg.Server.Log.File="";cfg.Server.Log.Rotation=logx.RotationConfig{}}
|
||||
return cfg,cfg.Validate()
|
||||
}
|
||||
func(c Config)Validate()error{
|
||||
if err:=c.Server.Validate();err!=nil{return err}
|
||||
{{if eq .Database "mysql"}}if err:=c.Database.Validate();err!=nil{return err}{{end}}
|
||||
{{if ne .Storage "none"}}if err:=c.Storage.Validate();err!=nil{return err}{{end}}
|
||||
if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)}
|
||||
{{if eq .Database "mysql"}}if err:=c.Database.Validate();err!=nil{return config.AtField("database",err)}{{end}}
|
||||
{{if ne .Storage "none"}}if err:=c.Storage.Validate();err!=nil{return config.AtField("storage",err)}{{end}}
|
||||
return nil
|
||||
}
|
||||
{{if ne .Storage "none"}}
|
||||
|
||||
13
cmd/go-omni/scaffold/templates/api-load.go.tmpl
Normal file
13
cmd/go-omni/scaffold/templates/api-load.go.tmpl
Normal file
@@ -0,0 +1,13 @@
|
||||
package config
|
||||
|
||||
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
|
||||
|
||||
// Load is maintained by the generator. Extend Config/Default/Validate in
|
||||
// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline.
|
||||
func Load(path string) (Config, error) {
|
||||
loader,err:=NewLoader()
|
||||
if err!=nil{return Config{},err}
|
||||
return omniConfig.LoadProject(path,Default(),omniConfig.EnvOptions{
|
||||
Prefix:"GO_OMNI_",Empty:"ignore",Aliases:omniConfig.ServerEnvAliases("GO_OMNI_","server"),
|
||||
},loader)
|
||||
}
|
||||
7
cmd/go-omni/scaffold/templates/api-loader.go.tmpl
Normal file
7
cmd/go-omni/scaffold/templates/api-loader.go.tmpl
Normal file
@@ -0,0 +1,7 @@
|
||||
package config
|
||||
|
||||
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
|
||||
|
||||
// NewLoader is project-owned. JSON/YAML/XML work without changes. Register
|
||||
// additional format decoders here only when the project needs them.
|
||||
func NewLoader() (*omniConfig.Loader,error) { return omniConfig.NewLoader() }
|
||||
@@ -10,7 +10,7 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni"
|
||||
// Source builds can carry an unpublished VCS version (including +dirty).
|
||||
// Keep this fallback in sync with the next release; installed CLIs use their
|
||||
// downloaded module version, which also has a module checksum.
|
||||
const defaultFrameworkVersion = "v0.19.0-preview"
|
||||
const defaultFrameworkVersion = "v0.21.0-preview"
|
||||
|
||||
// This target is not a claim that the release already exists. Source users
|
||||
// select a local framework checkout until the module templates are published.
|
||||
|
||||
320
config/bind.go
Normal file
320
config/bind.go
Normal file
@@ -0,0 +1,320 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"sort"
|
||||
"strings"
|
||||
)
|
||||
|
||||
var jsonUnmarshalerType = reflect.TypeOf((*json.Unmarshaler)(nil)).Elem()
|
||||
|
||||
type documentBinder struct {
|
||||
doc Document
|
||||
nodes int
|
||||
}
|
||||
|
||||
func (b *documentBinder) step(path string, depth int) error {
|
||||
b.nodes++
|
||||
if depth > maxConfigDepth || b.nodes > maxConfigNodes {
|
||||
return b.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func bindDocument[T any](doc Document, defaults T) (T, error) {
|
||||
var zero T
|
||||
base := reflect.ValueOf(&defaults).Elem()
|
||||
if base.Kind() != reflect.Struct && (base.Kind() != reflect.Pointer || base.Type().Elem().Kind() != reflect.Struct) {
|
||||
return zero, &Error{Reason: "project configuration must be a struct"}
|
||||
}
|
||||
result, err := cloneConfiguration(base, 0)
|
||||
if err != nil {
|
||||
return zero, &Error{Reason: "cannot clone configuration defaults", cause: err}
|
||||
}
|
||||
b := documentBinder{doc: doc}
|
||||
if err = b.bind(doc.Value, result, "", 0, false); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
return result.Interface().(T), nil
|
||||
}
|
||||
|
||||
func (b *documentBinder) decode(v any, dst reflect.Value, path string) (err error) {
|
||||
defer func() {
|
||||
if recover() != nil {
|
||||
err = b.doc.diagnostic(path, "custom value decoder panicked", nil)
|
||||
}
|
||||
}()
|
||||
raw, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return b.doc.diagnostic(path, "unsupported configuration value", err)
|
||||
}
|
||||
if err = json.Unmarshal(raw, dst.Addr().Interface()); err != nil {
|
||||
return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int, quoted bool) error {
|
||||
if err := b.step(path, depth); err != nil {
|
||||
return err
|
||||
}
|
||||
if v == nil {
|
||||
return b.decode(nil, dst, path)
|
||||
} // Preserve encoding/json null semantics.
|
||||
if dst.Kind() == reflect.Pointer {
|
||||
if dst.IsNil() {
|
||||
dst.Set(reflect.New(dst.Type().Elem()))
|
||||
}
|
||||
return b.bind(v, dst.Elem(), path, depth+1, quoted)
|
||||
}
|
||||
if dst.Type() == durationType {
|
||||
var raw string
|
||||
switch x := v.(type) {
|
||||
case string:
|
||||
raw = x
|
||||
case Text:
|
||||
raw = string(x)
|
||||
default:
|
||||
return b.decode(v, dst, path)
|
||||
}
|
||||
if err := parseEnvValue(dst, strings.TrimSpace(raw)); err != nil {
|
||||
return b.doc.diagnostic(path, "expected a duration such as 5s", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if dst.CanAddr() && dst.Addr().Type().Implements(jsonUnmarshalerType) {
|
||||
normalized, err := b.dynamic(v, path, depth+1)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return b.decode(normalized, dst, path)
|
||||
}
|
||||
if dst.CanAddr() && dst.Addr().Type().Implements(textType) {
|
||||
if text, ok := v.(Text); ok {
|
||||
v = string(text)
|
||||
}
|
||||
return b.decode(v, dst, path)
|
||||
}
|
||||
if text, ok := v.(Text); ok {
|
||||
if dst.Kind() == reflect.String || dst.Kind() == reflect.Interface || (dst.Kind() == reflect.Slice && dst.Type().Elem().Kind() == reflect.Uint8) {
|
||||
return b.decode(string(text), dst, path)
|
||||
}
|
||||
switch dst.Kind() {
|
||||
case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64:
|
||||
if err := parseEnvValue(dst, strings.TrimSpace(string(text))); err != nil {
|
||||
return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err)
|
||||
}
|
||||
return nil
|
||||
default:
|
||||
return b.doc.diagnostic(path, "expected an object or array, not scalar text", nil)
|
||||
}
|
||||
}
|
||||
if quoted {
|
||||
s, ok := v.(string)
|
||||
if !ok {
|
||||
return b.doc.diagnostic(path, "expected a JSON-encoded string", nil)
|
||||
}
|
||||
if err := json.Unmarshal([]byte(s), dst.Addr().Interface()); err != nil {
|
||||
return b.doc.diagnostic(path, "invalid string-encoded value", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
switch dst.Kind() {
|
||||
case reflect.Struct:
|
||||
m, ok := v.(map[string]any)
|
||||
if !ok {
|
||||
return b.doc.diagnostic(path, "expected an object", nil)
|
||||
}
|
||||
fields := configurationFields(dst.Type())
|
||||
used := map[string]bool{}
|
||||
for _, name := range sortedKeys(m) {
|
||||
child := joinField(path, name)
|
||||
f, ok := fields[name]
|
||||
canonical := name
|
||||
if !ok {
|
||||
for n, field := range fields {
|
||||
if strings.EqualFold(n, name) {
|
||||
if ok {
|
||||
return b.doc.diagnostic(child, "ambiguous field name", nil)
|
||||
}
|
||||
f, ok, canonical = field, true, n
|
||||
}
|
||||
}
|
||||
}
|
||||
if !ok {
|
||||
return b.doc.diagnostic(child, "unknown field", nil)
|
||||
}
|
||||
if used[canonical] {
|
||||
return b.doc.diagnostic(child, "duplicate field binding", nil)
|
||||
}
|
||||
used[canonical] = true
|
||||
target := dst
|
||||
for _, index := range f.Index {
|
||||
if target.Kind() == reflect.Pointer {
|
||||
if target.IsNil() {
|
||||
if !target.CanSet() {
|
||||
return b.doc.diagnostic(child, "field cannot be assigned", nil)
|
||||
}
|
||||
target.Set(reflect.New(target.Type().Elem()))
|
||||
}
|
||||
target = target.Elem()
|
||||
}
|
||||
target = target.Field(index)
|
||||
}
|
||||
if !target.CanSet() {
|
||||
return b.doc.diagnostic(child, "field cannot be assigned", nil)
|
||||
}
|
||||
if err := b.bind(m[name], target, child, depth+1, strings.Contains(","+f.Tag.Get("json")+",", ",string,")); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
case reflect.Map:
|
||||
m, ok := v.(map[string]any)
|
||||
if !ok || dst.Type().Key().Kind() != reflect.String {
|
||||
return b.doc.diagnostic(path, "expected an object with string keys", nil)
|
||||
}
|
||||
if dst.IsNil() {
|
||||
dst.Set(reflect.MakeMap(dst.Type()))
|
||||
}
|
||||
for _, key := range sortedKeys(m) {
|
||||
value := reflect.New(dst.Type().Elem()).Elem()
|
||||
if err := b.bind(m[key], value, joinField(path, key), depth+1, false); err != nil {
|
||||
return err
|
||||
}
|
||||
k := reflect.New(dst.Type().Key()).Elem()
|
||||
k.SetString(key)
|
||||
dst.SetMapIndex(k, value)
|
||||
}
|
||||
return nil
|
||||
case reflect.Slice, reflect.Array:
|
||||
if _, ok := v.(string); ok && dst.Kind() == reflect.Slice && dst.Type().Elem().Kind() == reflect.Uint8 {
|
||||
return b.decode(v, dst, path)
|
||||
}
|
||||
items, ok := v.([]any)
|
||||
if !ok {
|
||||
return b.doc.diagnostic(path, "expected an array", nil)
|
||||
}
|
||||
if dst.Kind() == reflect.Array && len(items) != dst.Len() {
|
||||
return b.doc.diagnostic(path, "array length does not match its declared type", nil)
|
||||
}
|
||||
if dst.Kind() == reflect.Slice {
|
||||
dst.Set(reflect.MakeSlice(dst.Type(), len(items), len(items)))
|
||||
}
|
||||
for i, value := range items {
|
||||
if err := b.bind(value, dst.Index(i), fmt.Sprintf("%s[%d]", path, i), depth+1, false); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
case reflect.Interface:
|
||||
v, err := b.dynamic(v, path, depth+1)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if dst.Type().NumMethod() == 0 {
|
||||
dst.Set(reflect.ValueOf(v))
|
||||
return nil
|
||||
}
|
||||
return b.decode(v, dst, path)
|
||||
default:
|
||||
return b.decode(v, dst, path)
|
||||
}
|
||||
}
|
||||
func sortedKeys(m map[string]any) []string {
|
||||
keys := make([]string, 0, len(m))
|
||||
for k := range m {
|
||||
keys = append(keys, k)
|
||||
}
|
||||
sort.Strings(keys)
|
||||
return keys
|
||||
}
|
||||
func (b *documentBinder) dynamic(v any, path string, depth int) (any, error) {
|
||||
if err := b.step(path, depth); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
switch x := v.(type) {
|
||||
case Text:
|
||||
return string(x), nil
|
||||
case map[string]any:
|
||||
out := make(map[string]any, len(x))
|
||||
for _, k := range sortedKeys(x) {
|
||||
c, err := b.dynamic(x[k], joinField(path, k), depth+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[k] = c
|
||||
}
|
||||
return out, nil
|
||||
case []any:
|
||||
out := make([]any, len(x))
|
||||
for i, v := range x {
|
||||
c, err := b.dynamic(v, fmt.Sprintf("%s[%d]", path, i), depth+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[i] = c
|
||||
}
|
||||
return out, nil
|
||||
default:
|
||||
return v, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Match encoding/json's field promotion: the shallowest declaration wins;
|
||||
// explicit tags win ties, and ambiguous fields remain unavailable.
|
||||
func configurationFields(t reflect.Type) map[string]reflect.StructField {
|
||||
type candidate struct {
|
||||
field reflect.StructField
|
||||
tagged bool
|
||||
}
|
||||
candidates := map[string][]candidate{}
|
||||
var visit func(reflect.Type, []int, map[reflect.Type]bool)
|
||||
visit = func(current reflect.Type, prefix []int, visiting map[reflect.Type]bool) {
|
||||
if visiting[current] {
|
||||
return
|
||||
}
|
||||
visiting[current] = true
|
||||
defer delete(visiting, current)
|
||||
for i := 0; i < current.NumField(); i++ {
|
||||
f := current.Field(i)
|
||||
name := strings.Split(f.Tag.Get("json"), ",")[0]
|
||||
if name == "-" {
|
||||
continue
|
||||
}
|
||||
child := f.Type
|
||||
if child.Kind() == reflect.Pointer {
|
||||
child = child.Elem()
|
||||
}
|
||||
if f.PkgPath != "" && (!f.Anonymous || child.Kind() != reflect.Struct) {
|
||||
continue
|
||||
}
|
||||
f.Index = append(append([]int(nil), prefix...), i)
|
||||
if name == "" && f.Anonymous && child.Kind() == reflect.Struct {
|
||||
visit(child, f.Index, visiting)
|
||||
continue
|
||||
}
|
||||
tagged := name != ""
|
||||
if name == "" {
|
||||
name = f.Name
|
||||
}
|
||||
candidates[name] = append(candidates[name], candidate{f, tagged})
|
||||
}
|
||||
}
|
||||
visit(t, nil, map[reflect.Type]bool{})
|
||||
fields := map[string]reflect.StructField{}
|
||||
for name, list := range candidates {
|
||||
sort.Slice(list, func(i, j int) bool {
|
||||
if len(list[i].field.Index) != len(list[j].field.Index) {
|
||||
return len(list[i].field.Index) < len(list[j].field.Index)
|
||||
}
|
||||
return list[i].tagged && !list[j].tagged
|
||||
})
|
||||
if len(list) > 1 && len(list[0].field.Index) == len(list[1].field.Index) && list[0].tagged == list[1].tagged {
|
||||
continue
|
||||
}
|
||||
fields[name] = list[0].field
|
||||
}
|
||||
return fields
|
||||
}
|
||||
@@ -62,19 +62,19 @@ func Default() Config {
|
||||
// Validate checks configuration values before an application starts.
|
||||
func (c Config) Validate() error {
|
||||
if err := c.Lifecycle.Validate(); err != nil {
|
||||
return err
|
||||
return AtField("lifecycle", err)
|
||||
}
|
||||
if err := c.Request.Validate(); err != nil {
|
||||
return err
|
||||
return AtField("request", err)
|
||||
}
|
||||
if err := c.Log.Validate(); err != nil {
|
||||
return err
|
||||
return AtField("log", err)
|
||||
}
|
||||
if err := c.Endpoints.Validate(); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := c.Errors.Validate(); err != nil {
|
||||
return err
|
||||
return AtField("errors", err)
|
||||
}
|
||||
encoder := c.ResponseEncoder
|
||||
if encoder == nil {
|
||||
@@ -90,7 +90,7 @@ func (c Config) Validate() error {
|
||||
return err
|
||||
}
|
||||
if err := c.Response.Validate(); err != nil {
|
||||
return err
|
||||
return AtField("response", err)
|
||||
}
|
||||
if c.SlowRequestThreshold < 0 {
|
||||
return &ConfigError{Field: "slowRequestThreshold", Message: "must be nonnegative"}
|
||||
@@ -99,13 +99,13 @@ func (c Config) Validate() error {
|
||||
return &ConfigError{Field: "address", Message: "is required"}
|
||||
}
|
||||
if strings.TrimSpace(c.RequestIDHeader) == "" {
|
||||
return &ConfigError{Field: "request_id_header", Message: "is required"}
|
||||
return &ConfigError{Field: "requestIdHeader", Message: "is required"}
|
||||
}
|
||||
if c.ShutdownTimeout <= 0 {
|
||||
return &ConfigError{Field: "shutdown_timeout", Message: "must be positive"}
|
||||
return &ConfigError{Field: "shutdownTimeout", Message: "must be positive"}
|
||||
}
|
||||
if _, err := httpx.NewClientIPResolver(c.TrustedProxies...); err != nil {
|
||||
return &ConfigError{Field: "trusted_proxies", Message: err.Error()}
|
||||
return &Error{Field: "trustedProxies", Reason: "expected IP addresses or CIDRs", cause: err}
|
||||
}
|
||||
return c.ValidateTransport()
|
||||
}
|
||||
|
||||
@@ -112,7 +112,7 @@ func TestLoadFromEnvRejectsInvalidValues(t *testing.T) {
|
||||
|
||||
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "1s")
|
||||
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "proxy.example.com")
|
||||
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "trusted_proxies") {
|
||||
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "trustedProxies") {
|
||||
t.Fatalf("expected invalid trusted proxy error, got %v", err)
|
||||
}
|
||||
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "")
|
||||
@@ -123,7 +123,7 @@ func TestLoadFromEnvRejectsInvalidValues(t *testing.T) {
|
||||
}
|
||||
cfg = Default()
|
||||
cfg.RequestIDHeader = " "
|
||||
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "request_id_header") {
|
||||
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "requestIdHeader") {
|
||||
t.Fatalf("expected invalid request header error, got %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
398
config/decoders.go
Normal file
398
config/decoders.go
Normal file
@@ -0,0 +1,398 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"encoding/xml"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"regexp"
|
||||
"sort"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
type JSONDecoder struct{}
|
||||
type YAMLDecoder struct{}
|
||||
type XMLDecoder struct{}
|
||||
|
||||
type parseState struct {
|
||||
doc Document
|
||||
nodes int
|
||||
}
|
||||
|
||||
func newParseState() *parseState { return &parseState{doc: Document{Positions: map[string]Position{}}} }
|
||||
func (p *parseState) enter(path string, pos Position, depth int) error {
|
||||
p.nodes++
|
||||
p.doc.Positions[path] = pos
|
||||
if depth > maxConfigDepth || p.nodes > maxConfigNodes {
|
||||
return p.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
type sourcePositions struct{ starts []int }
|
||||
|
||||
func newSourcePositions(data []byte) sourcePositions {
|
||||
p := sourcePositions{starts: []int{0}}
|
||||
for i, c := range data {
|
||||
if c == '\n' {
|
||||
p.starts = append(p.starts, i+1)
|
||||
}
|
||||
}
|
||||
return p
|
||||
}
|
||||
func (p sourcePositions) at(offset int64) Position {
|
||||
if offset < 0 {
|
||||
offset = 0
|
||||
}
|
||||
i := sort.Search(len(p.starts), func(i int) bool { return int64(p.starts[i]) > offset }) - 1
|
||||
return Position{Line: i + 1, Column: int(offset) - p.starts[i] + 1}
|
||||
}
|
||||
|
||||
var yamlLine = regexp.MustCompile(`line ([0-9]+)`)
|
||||
|
||||
func yamlSyntax(err error) *Error {
|
||||
d := &Error{Reason: "invalid YAML syntax", cause: err}
|
||||
if m := yamlLine.FindStringSubmatch(err.Error()); len(m) == 2 {
|
||||
d.Line, _ = strconv.Atoi(m[1])
|
||||
d.Column = 1
|
||||
}
|
||||
return d
|
||||
}
|
||||
func (JSONDecoder) Decode(data []byte) (Document, error) {
|
||||
p := newParseState()
|
||||
if len(data) > maxConfigBytes {
|
||||
return p.doc, &Error{Reason: "file exceeds 1 MiB"}
|
||||
}
|
||||
if !utf8.Valid(data) {
|
||||
return p.doc, &Error{Reason: "configuration must be UTF-8"}
|
||||
}
|
||||
positions := newSourcePositions(data)
|
||||
d := json.NewDecoder(bytes.NewReader(data))
|
||||
d.UseNumber()
|
||||
var parse func(string, int) (any, error)
|
||||
parse = func(path string, depth int) (any, error) {
|
||||
offset := d.InputOffset()
|
||||
for offset < int64(len(data)) && strings.ContainsRune(" \t\r\n:,", rune(data[offset])) {
|
||||
offset++
|
||||
}
|
||||
if err := p.enter(path, positions.at(offset), depth); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
token, err := d.Token()
|
||||
if err != nil {
|
||||
pos := positions.at(d.InputOffset())
|
||||
var syntax *json.SyntaxError
|
||||
if errors.As(err, &syntax) {
|
||||
pos = positions.at(syntax.Offset - 1)
|
||||
}
|
||||
return nil, &Error{Field: path, Line: pos.Line, Column: pos.Column, Reason: "invalid JSON syntax", cause: err}
|
||||
}
|
||||
if delim, ok := token.(json.Delim); ok {
|
||||
switch delim {
|
||||
case '{':
|
||||
out := map[string]any{}
|
||||
for d.More() {
|
||||
k, e := d.Token()
|
||||
if e != nil {
|
||||
return nil, p.doc.diagnostic(path, "invalid object key", e)
|
||||
}
|
||||
key, ok := k.(string)
|
||||
if !ok {
|
||||
return nil, p.doc.diagnostic(path, "object key must be a string", nil)
|
||||
}
|
||||
child := joinField(path, key)
|
||||
if _, exists := out[key]; exists {
|
||||
return nil, p.doc.diagnostic(child, "duplicate field", nil)
|
||||
}
|
||||
out[key], e = parse(child, depth+1)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
}
|
||||
if _, err = d.Token(); err != nil {
|
||||
return nil, p.doc.diagnostic(path, "unterminated object", err)
|
||||
}
|
||||
return out, nil
|
||||
case '[':
|
||||
out := []any{}
|
||||
for d.More() {
|
||||
v, e := parse(fmt.Sprintf("%s[%d]", path, len(out)), depth+1)
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
out = append(out, v)
|
||||
}
|
||||
if _, err = d.Token(); err != nil {
|
||||
return nil, p.doc.diagnostic(path, "unterminated array", err)
|
||||
}
|
||||
return out, nil
|
||||
default:
|
||||
return nil, p.doc.diagnostic(path, "unexpected JSON delimiter", nil)
|
||||
}
|
||||
}
|
||||
return token, nil
|
||||
}
|
||||
var err error
|
||||
p.doc.Value, err = parse("", 0)
|
||||
if err == nil {
|
||||
if _, e := d.Token(); e != io.EOF {
|
||||
err = p.doc.diagnostic("", "exactly one JSON document required", e)
|
||||
}
|
||||
}
|
||||
return p.doc, err
|
||||
}
|
||||
|
||||
func (YAMLDecoder) Decode(data []byte) (Document, error) {
|
||||
p := newParseState()
|
||||
if len(data) > maxConfigBytes {
|
||||
return p.doc, &Error{Reason: "file exceeds 1 MiB"}
|
||||
}
|
||||
if !utf8.Valid(data) {
|
||||
return p.doc, &Error{Reason: "configuration must be UTF-8"}
|
||||
}
|
||||
d := yaml.NewDecoder(bytes.NewReader(data))
|
||||
var root yaml.Node
|
||||
if err := d.Decode(&root); err != nil {
|
||||
return p.doc, yamlSyntax(err)
|
||||
}
|
||||
var extra yaml.Node
|
||||
if err := d.Decode(&extra); err != io.EOF {
|
||||
return p.doc, &Error{Reason: "exactly one YAML document required", cause: err}
|
||||
}
|
||||
var walk func(*yaml.Node, string, int) (any, error)
|
||||
walk = func(n *yaml.Node, path string, depth int) (any, error) {
|
||||
if err := p.enter(path, Position{n.Line, n.Column}, depth); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
switch n.Kind {
|
||||
case yaml.MappingNode:
|
||||
out := map[string]any{}
|
||||
for i := 0; i < len(n.Content); i += 2 {
|
||||
k, v := n.Content[i], n.Content[i+1]
|
||||
if k.Tag != "!!str" {
|
||||
return nil, p.doc.diagnostic(path, "mapping keys must be strings; merge keys are not supported", nil)
|
||||
}
|
||||
child := joinField(path, k.Value)
|
||||
if _, ok := out[k.Value]; ok {
|
||||
return nil, &Error{Field: child, Line: k.Line, Column: k.Column, Reason: "duplicate field"}
|
||||
}
|
||||
x, err := walk(v, child, depth+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
out[k.Value] = x
|
||||
}
|
||||
return out, nil
|
||||
case yaml.SequenceNode:
|
||||
out := make([]any, len(n.Content))
|
||||
for i, c := range n.Content {
|
||||
var err error
|
||||
out[i], err = walk(c, fmt.Sprintf("%s[%d]", path, i), depth+1)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return out, nil
|
||||
case yaml.AliasNode:
|
||||
return nil, p.doc.diagnostic(path, "YAML aliases are not supported", nil)
|
||||
case yaml.ScalarNode:
|
||||
switch n.Tag {
|
||||
case "!!str", "!!timestamp":
|
||||
return n.Value, nil
|
||||
case "!!null":
|
||||
return nil, nil
|
||||
case "!!bool", "!!int", "!!float":
|
||||
var v any
|
||||
if err := n.Decode(&v); err != nil {
|
||||
return nil, p.doc.diagnostic(path, "invalid YAML scalar", err)
|
||||
}
|
||||
raw, err := json.Marshal(v)
|
||||
if err != nil {
|
||||
return nil, p.doc.diagnostic(path, "number must be finite", err)
|
||||
}
|
||||
var out any
|
||||
dec := json.NewDecoder(bytes.NewReader(raw))
|
||||
dec.UseNumber()
|
||||
if err = dec.Decode(&out); err != nil {
|
||||
return nil, p.doc.diagnostic(path, "invalid YAML scalar", err)
|
||||
}
|
||||
return out, nil
|
||||
default:
|
||||
return nil, p.doc.diagnostic(path, "unsupported YAML tag", nil)
|
||||
}
|
||||
}
|
||||
return nil, p.doc.diagnostic(path, "unsupported YAML node", nil)
|
||||
}
|
||||
if len(root.Content) != 1 {
|
||||
return p.doc, &Error{Reason: "one nonempty YAML document required"}
|
||||
}
|
||||
var err error
|
||||
p.doc.Value, err = walk(root.Content[0], "", 0)
|
||||
return p.doc, err
|
||||
}
|
||||
|
||||
// XML uses a config root, element field names, type="array" with item children,
|
||||
// type="object" for empty objects, and type="null" for explicit null. Scalar
|
||||
// text is bound using the destination type; optional number/bool/string types
|
||||
// let dynamic any values preserve their intended JSON type.
|
||||
func (XMLDecoder) Decode(data []byte) (Document, error) {
|
||||
p := newParseState()
|
||||
if len(data) > maxConfigBytes {
|
||||
return p.doc, &Error{Reason: "file exceeds 1 MiB"}
|
||||
}
|
||||
if !utf8.Valid(data) {
|
||||
return p.doc, &Error{Reason: "configuration must be UTF-8"}
|
||||
}
|
||||
d := xml.NewDecoder(bytes.NewReader(data))
|
||||
var parse func(xml.StartElement, string, int, Position) (any, error)
|
||||
parse = func(start xml.StartElement, path string, depth int, pos Position) (any, error) {
|
||||
if err := p.enter(path, pos, depth); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if start.Name.Space != "" {
|
||||
return nil, p.doc.diagnostic(path, "XML namespaces are not supported", nil)
|
||||
}
|
||||
kind := ""
|
||||
attrSeen := false
|
||||
for _, a := range start.Attr {
|
||||
if a.Name.Space != "" || a.Name.Local != "type" || attrSeen {
|
||||
return nil, p.doc.diagnostic(path, "only one XML type attribute is supported", nil)
|
||||
}
|
||||
kind = a.Value
|
||||
attrSeen = true
|
||||
}
|
||||
if depth == 0 && kind == "" {
|
||||
kind = "object"
|
||||
}
|
||||
switch kind {
|
||||
case "", "object", "array", "null", "string", "number", "bool":
|
||||
default:
|
||||
return nil, p.doc.diagnostic(path, "unsupported XML type", nil)
|
||||
}
|
||||
children := map[string]any{}
|
||||
array := []any{}
|
||||
var text strings.Builder
|
||||
for {
|
||||
line, column := d.InputPos()
|
||||
token, err := d.Token()
|
||||
if err != nil {
|
||||
return nil, &Error{Field: path, Line: line, Column: column, Reason: "invalid XML syntax", cause: err}
|
||||
}
|
||||
switch x := token.(type) {
|
||||
case xml.StartElement:
|
||||
if kind != "" && kind != "object" && kind != "array" {
|
||||
return nil, p.doc.diagnostic(path, "scalar XML element cannot contain children", nil)
|
||||
}
|
||||
child := joinField(path, x.Name.Local)
|
||||
if kind == "array" {
|
||||
if x.Name.Local != "item" {
|
||||
return nil, p.doc.diagnostic(path, "XML array children must be item elements", nil)
|
||||
}
|
||||
child = fmt.Sprintf("%s[%d]", path, len(array))
|
||||
} else if _, ok := children[x.Name.Local]; ok {
|
||||
return nil, p.doc.diagnostic(child, "duplicate field; use type=\"array\" for lists", nil)
|
||||
}
|
||||
v, e := parse(x, child, depth+1, Position{line, column})
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if kind == "array" {
|
||||
array = append(array, v)
|
||||
} else {
|
||||
children[x.Name.Local] = v
|
||||
}
|
||||
case xml.CharData:
|
||||
text.Write([]byte(x))
|
||||
case xml.Comment:
|
||||
case xml.EndElement:
|
||||
raw := text.String()
|
||||
if len(children) > 0 || kind == "array" || kind == "object" {
|
||||
if strings.TrimSpace(raw) != "" {
|
||||
return nil, p.doc.diagnostic(path, "mixed XML text and elements are not supported", nil)
|
||||
}
|
||||
if kind == "array" {
|
||||
return array, nil
|
||||
}
|
||||
return children, nil
|
||||
}
|
||||
switch kind {
|
||||
case "null":
|
||||
if strings.TrimSpace(raw) != "" {
|
||||
return nil, p.doc.diagnostic(path, "null element must be empty", nil)
|
||||
}
|
||||
return nil, nil
|
||||
case "string":
|
||||
return raw, nil
|
||||
case "number":
|
||||
var n any
|
||||
dec := json.NewDecoder(strings.NewReader(strings.TrimSpace(raw)))
|
||||
dec.UseNumber()
|
||||
if err := dec.Decode(&n); err != nil {
|
||||
return nil, p.doc.diagnostic(path, "invalid number", err)
|
||||
}
|
||||
if _, ok := n.(json.Number); !ok {
|
||||
return nil, p.doc.diagnostic(path, "expected number", nil)
|
||||
}
|
||||
if _, e := dec.Token(); e != io.EOF {
|
||||
return nil, p.doc.diagnostic(path, "invalid number", e)
|
||||
}
|
||||
return n, nil
|
||||
case "bool":
|
||||
b, e := strconv.ParseBool(strings.TrimSpace(raw))
|
||||
if e != nil {
|
||||
return nil, p.doc.diagnostic(path, "expected boolean", e)
|
||||
}
|
||||
return b, nil
|
||||
}
|
||||
return Text(raw), nil
|
||||
default:
|
||||
return nil, p.doc.diagnostic(path, "XML directives and processing instructions are not supported", nil)
|
||||
}
|
||||
}
|
||||
}
|
||||
found := false
|
||||
prolog := false
|
||||
for {
|
||||
line, column := d.InputPos()
|
||||
token, err := d.Token()
|
||||
if err == io.EOF {
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
return p.doc, &Error{Line: line, Column: column, Reason: "invalid XML syntax", cause: err}
|
||||
}
|
||||
switch x := token.(type) {
|
||||
case xml.StartElement:
|
||||
if found || x.Name.Local != "config" {
|
||||
return p.doc, &Error{Line: line, Column: column, Reason: "exactly one config root required"}
|
||||
}
|
||||
found = true
|
||||
p.doc.Value, err = parse(x, "", 0, Position{line, column})
|
||||
if err != nil {
|
||||
return p.doc, err
|
||||
}
|
||||
case xml.CharData:
|
||||
if strings.TrimSpace(string(x)) != "" {
|
||||
return p.doc, &Error{Reason: "unexpected text outside XML root"}
|
||||
}
|
||||
case xml.Comment:
|
||||
case xml.ProcInst:
|
||||
if found || prolog || x.Target != "xml" {
|
||||
return p.doc, &Error{Reason: "XML processing instructions are not supported"}
|
||||
}
|
||||
prolog = true
|
||||
default:
|
||||
return p.doc, &Error{Reason: "XML directives are not supported"}
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return p.doc, &Error{Reason: "config root required"}
|
||||
}
|
||||
return p.doc, nil
|
||||
}
|
||||
94
config/diagnostic.go
Normal file
94
config/diagnostic.go
Normal file
@@ -0,0 +1,94 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Error is a value-free diagnostic. Cause may contain private parser details;
|
||||
// errors.Is/As remain available, but Error never formats Cause automatically.
|
||||
type Error struct {
|
||||
File, Field, Env, Reason string
|
||||
Line, Column int
|
||||
cause error
|
||||
}
|
||||
|
||||
func (e *Error) Error() string {
|
||||
where := "config"
|
||||
if e.File != "" {
|
||||
where = fmt.Sprintf("config %q", e.File)
|
||||
}
|
||||
if e.Line > 0 {
|
||||
where += fmt.Sprintf(":%d:%d", e.Line, e.Column)
|
||||
}
|
||||
if e.Env != "" {
|
||||
where += fmt.Sprintf(" env %q", e.Env)
|
||||
}
|
||||
if e.Field != "" {
|
||||
where += fmt.Sprintf(" field %q", e.Field)
|
||||
}
|
||||
return where + ": " + e.Reason
|
||||
}
|
||||
func (e *Error) Unwrap() error { return e.cause }
|
||||
|
||||
// DecodeError is retained for callers of errors.As from earlier releases.
|
||||
type DecodeError = Error
|
||||
|
||||
// Invalid reports a public, value-free validation reason for a configuration field.
|
||||
func Invalid(field, reason string) error { return &ConfigError{Field: field, Message: reason} }
|
||||
|
||||
// AtField prefixes a nested validation error without losing its original cause.
|
||||
// Unstructured validation errors are deliberately not echoed; use Invalid when
|
||||
// a custom validator needs a public reason rather than a private error message.
|
||||
func AtField(field string, err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
d := validationError(err)
|
||||
d.Field = joinField(field, d.Field)
|
||||
return d
|
||||
}
|
||||
func validationError(err error) *Error {
|
||||
var d *Error
|
||||
if errors.As(err, &d) {
|
||||
out := *d
|
||||
out.cause = err
|
||||
return &out
|
||||
}
|
||||
var c *ConfigError
|
||||
if errors.As(err, &c) {
|
||||
return &Error{Field: c.Field, Reason: c.Message, cause: err}
|
||||
}
|
||||
return &Error{Reason: "validation failed", cause: err}
|
||||
}
|
||||
func joinField(parent, child string) string {
|
||||
if parent == "" {
|
||||
return child
|
||||
}
|
||||
if child == "" {
|
||||
return parent
|
||||
}
|
||||
if strings.HasPrefix(child, "[") {
|
||||
return parent + child
|
||||
}
|
||||
return parent + "." + child
|
||||
}
|
||||
|
||||
// Position uses one-based source line/column numbers. Zero means unavailable.
|
||||
type Position struct{ Line, Column int }
|
||||
|
||||
// Document is the common decoder result. Values use JSON-compatible data:
|
||||
// string, bool, json.Number, nil, []any, map[string]any. Text is available for
|
||||
// formats such as XML whose scalar tokens have no inherent JSON type.
|
||||
type Document struct {
|
||||
Value any
|
||||
Positions map[string]Position
|
||||
}
|
||||
|
||||
type Text string
|
||||
|
||||
func (d Document) diagnostic(field, reason string, cause error) *Error {
|
||||
p := d.Positions[field]
|
||||
return &Error{Field: field, Reason: reason, Line: p.Line, Column: p.Column, cause: cause}
|
||||
}
|
||||
@@ -1,100 +1,16 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"gopkg.in/yaml.v3"
|
||||
)
|
||||
|
||||
// DecodeFile merges a bounded JSON/YAML document into defaults. It uses JSON
|
||||
// field names for both formats, rejects unknown fields and accepts durations
|
||||
// such as "3s". Application-specific validation remains with the caller.
|
||||
func DecodeFile[T any](path string, defaults T) (T, error) {
|
||||
result, err := decodeFile(path, defaults)
|
||||
if err != nil {
|
||||
var zero T
|
||||
return zero, &DecodeError{cause: err}
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
// DecodeError keeps input values out of public diagnostics while preserving
|
||||
// errors.Is/As for explicitly handled underlying I/O and syntax errors.
|
||||
type DecodeError struct{ cause error }
|
||||
|
||||
func (e *DecodeError) Error() string {
|
||||
return "config: cannot decode file; check its path, format, field names and value types"
|
||||
}
|
||||
func (e *DecodeError) Unwrap() error { return e.cause }
|
||||
func decodeFile[T any](path string, defaults T) (T, error) {
|
||||
var zero T
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
defer f.Close()
|
||||
data, err := io.ReadAll(io.LimitReader(f, (1<<20)+1))
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
if len(data) > 1<<20 {
|
||||
return zero, errors.New("config: file exceeds 1 MiB")
|
||||
}
|
||||
var value any
|
||||
switch strings.ToLower(filepath.Ext(path)) {
|
||||
case ".yaml", ".yml":
|
||||
dec := yaml.NewDecoder(bytes.NewReader(data))
|
||||
if err := dec.Decode(&value); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
var extra any
|
||||
if err := dec.Decode(&extra); err != io.EOF {
|
||||
return zero, errors.New("config: one YAML document required")
|
||||
}
|
||||
case ".json":
|
||||
if err := apierrors.DecodeCatalogJSON(data, &value); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
default:
|
||||
return zero, errors.New("config: expected .json, .yaml or .yml")
|
||||
}
|
||||
if _, ok := value.(map[string]any); !ok {
|
||||
return zero, errors.New("config: root must be an object")
|
||||
}
|
||||
value, err = durations(value, reflect.TypeOf(defaults))
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
encoded, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
// Clone configuration without serializing borrowed json:"-" capabilities.
|
||||
var result T
|
||||
original, err := cloneConfiguration(reflect.ValueOf(&defaults).Elem(), 0)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
reflect.ValueOf(&result).Elem().Set(original)
|
||||
dec := json.NewDecoder(bytes.NewReader(encoded))
|
||||
dec.DisallowUnknownFields()
|
||||
if err = dec.Decode(&result); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
func durations(v any, t reflect.Type) (any, error) {
|
||||
if t == nil {
|
||||
return v, nil
|
||||
|
||||
175
config/loader.go
Normal file
175
config/loader.go
Normal file
@@ -0,0 +1,175 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
)
|
||||
|
||||
const maxConfigBytes = 1 << 20
|
||||
const maxConfigDepth = 64
|
||||
const maxConfigNodes = 100000
|
||||
|
||||
// Decoder must not retain or mutate input and must support concurrent calls
|
||||
// when its Loader is shared. It is trusted application code, not a sandbox.
|
||||
type Decoder interface {
|
||||
Decode([]byte) (Document, error)
|
||||
}
|
||||
type DecoderFunc func([]byte) (Document, error)
|
||||
|
||||
func (f DecoderFunc) Decode(data []byte) (Document, error) { return f(data) }
|
||||
|
||||
// Loader owns an immutable per-instance decoder registry. No global registration.
|
||||
type Loader struct {
|
||||
decoders map[string]Decoder
|
||||
sealed bool
|
||||
}
|
||||
type LoaderOption func(*Loader) error
|
||||
|
||||
// WithDecoder adds a new extension; builtin/duplicate extensions cannot be replaced.
|
||||
func WithDecoder(extension string, decoder Decoder) LoaderOption {
|
||||
return func(l *Loader) error {
|
||||
if l == nil || l.decoders == nil || l.sealed {
|
||||
return errors.New("config: decoder registry cannot be changed after construction")
|
||||
}
|
||||
ext := strings.ToLower(strings.TrimPrefix(extension, "."))
|
||||
if ext == "" || strings.ContainsAny(ext, "./\\ \t\r\n") {
|
||||
return errors.New("config: invalid decoder extension")
|
||||
}
|
||||
if decoder == nil {
|
||||
return errors.New("config: nil decoder")
|
||||
}
|
||||
v := reflect.ValueOf(decoder)
|
||||
if (v.Kind() == reflect.Pointer || v.Kind() == reflect.Func || v.Kind() == reflect.Map || v.Kind() == reflect.Slice || v.Kind() == reflect.Interface) && v.IsNil() {
|
||||
return errors.New("config: nil decoder")
|
||||
}
|
||||
if _, exists := l.decoders[ext]; exists {
|
||||
return errors.New("config: decoder extension already registered")
|
||||
}
|
||||
l.decoders[ext] = decoder
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
func NewLoader(options ...LoaderOption) (*Loader, error) {
|
||||
l := &Loader{decoders: map[string]Decoder{"json": JSONDecoder{}, "yaml": YAMLDecoder{}, "yml": YAMLDecoder{}, "xml": XMLDecoder{}}}
|
||||
for _, o := range options {
|
||||
if o == nil {
|
||||
return nil, errors.New("config: nil loader option")
|
||||
}
|
||||
if err := o(l); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
l.sealed = true
|
||||
return l, nil
|
||||
}
|
||||
|
||||
var defaultLoader, _ = NewLoader()
|
||||
|
||||
func selectLoader(loaders []*Loader) (*Loader, error) {
|
||||
if len(loaders) == 0 {
|
||||
return defaultLoader, nil
|
||||
}
|
||||
if len(loaders) != 1 || loaders[0] == nil {
|
||||
return nil, errors.New("config: expected one non-nil loader")
|
||||
}
|
||||
return loaders[0], nil
|
||||
}
|
||||
func (l *Loader) read(path string) (doc Document, err error) {
|
||||
defer func() {
|
||||
if err != nil {
|
||||
var d *Error
|
||||
if errors.As(err, &d) {
|
||||
c := *d
|
||||
c.File = path
|
||||
err = &c
|
||||
} else {
|
||||
reason := "cannot read configuration file"
|
||||
if errors.Is(err, os.ErrNotExist) {
|
||||
reason = "configuration file does not exist"
|
||||
} else if errors.Is(err, os.ErrPermission) {
|
||||
reason = "permission denied reading configuration file"
|
||||
}
|
||||
err = &Error{File: path, Reason: reason, cause: err}
|
||||
}
|
||||
}
|
||||
}()
|
||||
decoder := l.decoders[strings.TrimPrefix(strings.ToLower(filepath.Ext(path)), ".")]
|
||||
if decoder == nil {
|
||||
return doc, &Error{Reason: "unsupported configuration format"}
|
||||
}
|
||||
info, e := os.Stat(path)
|
||||
if e != nil {
|
||||
return doc, e
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return doc, &Error{Reason: "configuration must be a regular file"}
|
||||
}
|
||||
f, e := os.Open(path)
|
||||
if e != nil {
|
||||
return doc, e
|
||||
}
|
||||
defer f.Close()
|
||||
info, e = f.Stat()
|
||||
if e != nil {
|
||||
return doc, e
|
||||
}
|
||||
if !info.Mode().IsRegular() {
|
||||
return doc, &Error{Reason: "configuration must be a regular file"}
|
||||
}
|
||||
data, e := io.ReadAll(io.LimitReader(f, maxConfigBytes+1))
|
||||
if e != nil {
|
||||
return doc, e
|
||||
}
|
||||
if len(data) > maxConfigBytes {
|
||||
return doc, &Error{Reason: "file exceeds 1 MiB"}
|
||||
}
|
||||
doc, err = decodeSafely(decoder, data)
|
||||
if err != nil {
|
||||
return doc, err
|
||||
}
|
||||
if _, ok := doc.Value.(map[string]any); !ok {
|
||||
return doc, doc.diagnostic("", "root must be an object", nil)
|
||||
}
|
||||
return doc, nil
|
||||
}
|
||||
func decodeSafely(decoder Decoder, data []byte) (doc Document, err error) {
|
||||
defer func() {
|
||||
if recover() != nil {
|
||||
err = &Error{Reason: "decoder panicked"}
|
||||
}
|
||||
}()
|
||||
doc, err = decoder.Decode(data)
|
||||
if err != nil {
|
||||
var d *Error
|
||||
if !errors.As(err, &d) {
|
||||
err = &Error{Reason: "invalid configuration syntax", cause: err}
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// DecodeFile overlays a bounded document onto independent defaults. With no
|
||||
// loader argument it accepts JSON/YAML/XML. It never opens dependencies.
|
||||
func DecodeFile[T any](path string, defaults T, loaders ...*Loader) (T, error) {
|
||||
var zero T
|
||||
l, err := selectLoader(loaders)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
doc, err := l.read(path)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
result, err := bindDocument(doc, defaults)
|
||||
if err != nil {
|
||||
d := validationError(err)
|
||||
d.File = path
|
||||
return zero, d
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
397
config/loader_test.go
Normal file
397
config/loader_test.go
Normal file
@@ -0,0 +1,397 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
type loaderExample struct {
|
||||
Server Config `json:"server"`
|
||||
Upload struct {
|
||||
MaxBytes int64 `json:"maxBytes" envDefault:"1024"`
|
||||
} `json:"upload"`
|
||||
Values []int64 `json:"values"`
|
||||
Optional *string `json:"optional"`
|
||||
Enabled bool `json:"enabled"`
|
||||
Dynamic any `json:"dynamic"`
|
||||
}
|
||||
|
||||
func (c loaderExample) Validate() error {
|
||||
if err := c.Server.Validate(); err != nil {
|
||||
return AtField("server", err)
|
||||
}
|
||||
if c.Upload.MaxBytes < 0 {
|
||||
return Invalid("upload.maxBytes", "must be nonnegative")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func noEnvironment(string) (string, bool) { return "", false }
|
||||
func configFile(t *testing.T, ext, raw string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "service."+ext)
|
||||
if err := os.WriteFile(path, []byte(raw), 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func TestUnifiedFormatAndPrecedence(t *testing.T) {
|
||||
formats := map[string]string{
|
||||
"json": `{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":7},"values":[9007199254740993],"optional":null,"enabled":true,"dynamic":{"id":9007199254740993}}`,
|
||||
"yaml": "server:\n http:\n readTimeout: 2s\nupload:\n maxBytes: 7\nvalues: [9007199254740993]\noptional: null\nenabled: true\ndynamic:\n id: 9007199254740993\n",
|
||||
"xml": `<config><server><http><readTimeout>2s</readTimeout></http></server><upload><maxBytes>7</maxBytes></upload><values type="array"><item>9007199254740993</item></values><optional type="null"/><enabled>true</enabled><dynamic><id type="number">9007199254740993</id></dynamic></config>`,
|
||||
}
|
||||
var reference loaderExample
|
||||
for _, ext := range []string{"json", "yaml", "xml"} {
|
||||
t.Run(ext, func(t *testing.T) {
|
||||
old := "old"
|
||||
base := loaderExample{Server: Default(), Optional: &old}
|
||||
cfg, err := LoadProject(configFile(t, ext, formats[ext]), base, EnvOptions{Prefix: "APP_", Lookup: func(key string) (string, bool) {
|
||||
v, ok := map[string]string{"APP_UPLOAD_MAX_BYTES": "0", "APP_ENABLED": "false"}[key]
|
||||
return v, ok
|
||||
}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if cfg.Upload.MaxBytes != 0 || cfg.Enabled || cfg.Optional != nil || cfg.Values[0] != 9007199254740993 || cfg.Server.HTTP.ReadTimeout != 2*time.Second {
|
||||
t.Fatal("values changed", cfg)
|
||||
}
|
||||
if cfg.Dynamic.(map[string]any)["id"] != json.Number("9007199254740993") {
|
||||
t.Fatal("dynamic integer precision lost")
|
||||
}
|
||||
if ext == "json" {
|
||||
reference = cfg
|
||||
} else if !reflect.DeepEqual(reference, cfg) {
|
||||
t.Fatal("format-dependent semantics")
|
||||
}
|
||||
if base.Optional == nil || *base.Optional != "old" {
|
||||
t.Fatal("defaults mutated")
|
||||
}
|
||||
})
|
||||
}
|
||||
cfg, err := LoadProject("", loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment})
|
||||
if err != nil || cfg.Upload.MaxBytes != 1024 {
|
||||
t.Fatal(cfg, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfigDiagnosticsAndCauses(t *testing.T) {
|
||||
for _, tc := range []struct {
|
||||
ext, body, field string
|
||||
line bool
|
||||
}{
|
||||
{"json", "{\n\"upload\": {\"maxBytes\":\"private-value\"}}", "upload.maxBytes", true},
|
||||
{"yaml", "server:\n http:\n readTimeout: private-value", "server.http.readTimeout", true},
|
||||
{"xml", "<config>\n<upload><maxBytes>private-value</maxBytes></upload></config>", "upload.maxBytes", true},
|
||||
{"json", `{"upload":{"maxBytes":-1}}`, "upload.maxBytes", true},
|
||||
{"yaml", "server:\n http:\n readTimeout: -1s", "server.http.readTimeout", true},
|
||||
{"json", `{"upload":{"typo":3}}`, "upload.typo", true},
|
||||
} {
|
||||
t.Run(tc.ext+tc.field, func(t *testing.T) {
|
||||
path := configFile(t, tc.ext, tc.body)
|
||||
_, err := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment})
|
||||
var d *Error
|
||||
if !errors.As(err, &d) || d.File != path || d.Field != tc.field || d.Reason == "" || tc.line && d.Line < 1 {
|
||||
t.Fatalf("missing location: %#v %v", d, err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "private-value") {
|
||||
t.Fatal("secret leaked", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
path := filepath.Join(t.TempDir(), "missing.json")
|
||||
_, err := LoadProject(path, loaderExample{}, EnvOptions{})
|
||||
if !errors.Is(err, os.ErrNotExist) {
|
||||
t.Fatal("I/O cause discarded", err)
|
||||
}
|
||||
_, err = DecodeFile(configFile(t, "json", `{"count":"private-value"}`), struct {
|
||||
Count int `json:"count"`
|
||||
}{})
|
||||
var typed *json.UnmarshalTypeError
|
||||
if !errors.As(err, &typed) {
|
||||
t.Fatal("type cause discarded", err)
|
||||
}
|
||||
_, err = LoadProject(configFile(t, "yaml", "upload:\n maxBytes: 7"), loaderExample{Server: Default()}, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { return "-1", k == "APP_UPLOAD_MAX_BYTES" }})
|
||||
var d *Error
|
||||
if !errors.As(err, &d) || d.Env != "APP_UPLOAD_MAX_BYTES" || d.Field != "upload.maxBytes" || d.File != "" || d.Line != 0 {
|
||||
t.Fatal("environment validation wrongly blamed file", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMalformedFormatsAndLimits(t *testing.T) {
|
||||
for ext, bodies := range map[string][]string{
|
||||
"json": {`{"enabled":true,"enabled":false}`, `{} {}`, `{"upload":`, `{"enabled":true,"Enabled":false}`, strings.Repeat(`{"a":`, 70) + `0` + strings.Repeat(`}`, 70)},
|
||||
"yaml": {"enabled: true\nenabled: false", "---\n{}\n---\n{}", "a: &x [1]\nb: *x", "a:\n <<: {}", "upload: [\n"},
|
||||
"xml": {`<!DOCTYPE config><config/>`, `<config>&private-value;</config>`, `<config/><config/>`, `<config xmlns="x"/>`, `<config type="" type=""/>`, `<config><enabled>true</enabled><enabled>false</enabled></config>`, `<config>text<enabled>true</enabled></config>`, `<config><values type="array"><bad>1</bad></values></config>`},
|
||||
} {
|
||||
for i, raw := range bodies {
|
||||
t.Run(fmt.Sprintf("%s%d", ext, i), func(t *testing.T) {
|
||||
_, err := LoadProject(configFile(t, ext, raw), loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment})
|
||||
if err == nil || strings.Contains(err.Error(), "private-value") {
|
||||
t.Fatal(err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
if _, err := DecodeFile(configFile(t, "json", strings.Repeat(" ", maxConfigBytes+1)), loaderExample{}); err == nil {
|
||||
t.Fatal("unbounded file")
|
||||
}
|
||||
if _, err := DecodeFile(configFile(t, "json", string([]byte{'{', '"', 0xff, '"', ':', '0', '}'})), loaderExample{}); err == nil {
|
||||
t.Fatal("invalid UTF-8")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecoderRegistryAndConcurrentUse(t *testing.T) {
|
||||
decoder := DecoderFunc(func([]byte) (Document, error) {
|
||||
return Document{Value: map[string]any{"upload": map[string]any{"maxBytes": json.Number("42")}}}, nil
|
||||
})
|
||||
l, err := NewLoader(WithDecoder("custom", decoder))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
path := configFile(t, "custom", "ignored")
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 24; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
cfg, e := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}, l)
|
||||
if e != nil || cfg.Upload.MaxBytes != 42 {
|
||||
t.Error(e)
|
||||
}
|
||||
}()
|
||||
}
|
||||
wg.Wait()
|
||||
if _, err := DecodeFile(path, loaderExample{}); err == nil {
|
||||
t.Fatal("decoder registration leaked globally")
|
||||
}
|
||||
for _, options := range [][]LoaderOption{{WithDecoder("json", decoder)}, {WithDecoder("x", decoder), WithDecoder("x", decoder)}, {WithDecoder("x", DecoderFunc(nil))}} {
|
||||
if _, err = NewLoader(options...); err == nil {
|
||||
t.Fatal("invalid registration")
|
||||
}
|
||||
}
|
||||
l, _ = NewLoader(WithDecoder("panic", DecoderFunc(func([]byte) (Document, error) { panic("private-value") })))
|
||||
_, err = DecodeFile(configFile(t, "panic", ""), loaderExample{}, l)
|
||||
if err == nil || strings.Contains(err.Error(), "private-value") {
|
||||
t.Fatal(err)
|
||||
}
|
||||
cycle := map[string]any{}
|
||||
cycle["self"] = cycle
|
||||
l, _ = NewLoader(WithDecoder("cycle", DecoderFunc(func([]byte) (Document, error) { return Document{Value: map[string]any{"dynamic": cycle}}, nil })))
|
||||
if _, err = DecodeFile(configFile(t, "cycle", ""), loaderExample{}, l); err == nil {
|
||||
t.Fatal("decoder bypassed traversal limit")
|
||||
}
|
||||
}
|
||||
|
||||
func TestUnifiedLoggerNormalization(t *testing.T) {
|
||||
type project struct {
|
||||
Server Config `json:"server"`
|
||||
}
|
||||
base := project{Server: Default()}
|
||||
base.Server.Log.File = "old.log"
|
||||
base.Server.Log.Rotation.Enabled = true
|
||||
for _, full := range []bool{false, true} {
|
||||
name := "GO_OMNI_LOG_OUTPUT"
|
||||
if full {
|
||||
name = "GO_OMNI_SERVER_LOG_OUTPUT"
|
||||
}
|
||||
cfg, err := LoadProject("", base, EnvOptions{Prefix: "GO_OMNI_", Empty: "ignore", Aliases: ServerEnvAliases("GO_OMNI_", "server"), Lookup: func(k string) (string, bool) { return "stdout", k == name }})
|
||||
if err != nil || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled || base.Server.Log.File != "old.log" {
|
||||
t.Fatal(cfg, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func FuzzConfigurationDecoders(f *testing.F) {
|
||||
f.Add(uint8(0), []byte(`{"count":1}`))
|
||||
f.Add(uint8(1), []byte("count: 1"))
|
||||
f.Add(uint8(2), []byte("<config><count>1</count></config>"))
|
||||
f.Fuzz(func(t *testing.T, format uint8, data []byte) {
|
||||
if len(data) > 8192 {
|
||||
data = data[:8192]
|
||||
}
|
||||
ds := []Decoder{JSONDecoder{}, YAMLDecoder{}, XMLDecoder{}}
|
||||
doc, err := ds[int(format)%len(ds)].Decode(data)
|
||||
if err == nil {
|
||||
_, _ = bindDocument(doc, struct {
|
||||
Count int `json:"count"`
|
||||
}{})
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func BenchmarkConfiguration(b *testing.B) {
|
||||
for name, tc := range map[string]struct {
|
||||
d Decoder
|
||||
raw string
|
||||
}{"json": {JSONDecoder{}, `{"count":9007199254740993,"timeout":"5s","values":[1,2,3]}`}, "yaml": {YAMLDecoder{}, "count: 9007199254740993\ntimeout: 5s\nvalues: [1,2,3]"}, "xml": {XMLDecoder{}, `<config><count>9007199254740993</count><timeout>5s</timeout><values type="array"><item>1</item><item>2</item><item>3</item></values></config>`}} {
|
||||
b.Run(name, func(b *testing.B) {
|
||||
b.ReportAllocs()
|
||||
raw := []byte(tc.raw)
|
||||
for i := 0; i < b.N; i++ {
|
||||
doc, err := tc.d.Decode(raw)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
_, err = bindDocument(doc, struct {
|
||||
Count int64 `json:"count"`
|
||||
Timeout time.Duration `json:"timeout"`
|
||||
Values []int `json:"values"`
|
||||
}{})
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Text-only scalars and embedded config fields use the same behavior in each format.
|
||||
type testText struct{ Value string }
|
||||
|
||||
func (v *testText) UnmarshalText(data []byte) error { v.Value = string(data); return nil }
|
||||
func TestConfigScalarAndEmbedding(t *testing.T) {
|
||||
type Fields struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
type C struct {
|
||||
Fields
|
||||
Nested Fields `json:"nested"`
|
||||
Scalar testText `json:"scalar"`
|
||||
Bytes []byte `json:"bytes"`
|
||||
}
|
||||
cases := map[string]string{
|
||||
"json": `{"name":"outer","nested":{"name":"inner"},"scalar":"text","bytes":"aGk="}`,
|
||||
"yaml": "name: outer\nnested:\n name: inner\nscalar: text\nbytes: aGk=\n",
|
||||
"xml": `<config><name>outer</name><nested><name>inner</name></nested><scalar>text</scalar><bytes>aGk=</bytes></config>`,
|
||||
}
|
||||
for ext, data := range cases {
|
||||
t.Run(ext, func(t *testing.T) {
|
||||
path := filepath.Join(t.TempDir(), "config."+ext)
|
||||
os.WriteFile(path, []byte(data), 0600)
|
||||
got, err := DecodeFile(path, &C{})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Name != "outer" || got.Nested.Name != "inner" || got.Scalar.Value != "text" || string(got.Bytes) != "hi" {
|
||||
t.Fatal(got)
|
||||
}
|
||||
})
|
||||
}
|
||||
type Tagged struct {
|
||||
Fields `json:"nested"`
|
||||
}
|
||||
doc, _ := (JSONDecoder{}).Decode([]byte(`{"name":"wrong"}`))
|
||||
if _, err := bindDocument(doc, Tagged{}); err == nil {
|
||||
t.Fatal("tagged embed incorrectly flattened")
|
||||
}
|
||||
}
|
||||
func TestConfigDefaultsIsolationAndEmptyValues(t *testing.T) {
|
||||
type Child struct {
|
||||
Name string `json:"name"`
|
||||
}
|
||||
type C struct {
|
||||
Child *Child `json:"child"`
|
||||
Items []int `json:"items"`
|
||||
Labels map[string]string `json:"labels"`
|
||||
Enabled bool `json:"enabled"`
|
||||
}
|
||||
base := C{Child: &Child{Name: "old"}, Items: []int{1}, Labels: map[string]string{"old": "yes"}, Enabled: true}
|
||||
doc, _ := (JSONDecoder{}).Decode([]byte(`{"child":{"name":"new"},"labels":{"new":"yes"},"items":[],"enabled":false}`))
|
||||
got, err := bindDocument(doc, base)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if got.Child.Name != "new" || got.Items == nil || len(got.Items) != 0 || got.Enabled {
|
||||
t.Fatal(got)
|
||||
}
|
||||
if base.Child.Name != "old" || len(base.Items) != 1 || len(base.Labels) != 1 {
|
||||
t.Fatal("mutated defaults")
|
||||
}
|
||||
bad, _ := (JSONDecoder{}).Decode([]byte(`{"child":{"name":"changed"},"items":"bad"}`))
|
||||
if _, err := bindDocument(bad, base); err == nil {
|
||||
t.Fatal("invalid input accepted")
|
||||
}
|
||||
if base.Child.Name != "old" {
|
||||
t.Fatal("failed load changed default pointer")
|
||||
}
|
||||
null, _ := (JSONDecoder{}).Decode([]byte(`{"child":null,"items":null,"labels":null,"enabled":null}`))
|
||||
got, err = bindDocument(null, base)
|
||||
if err != nil || got.Child != nil || got.Items != nil || got.Labels != nil || !got.Enabled {
|
||||
t.Fatal(got, err)
|
||||
}
|
||||
}
|
||||
func TestDefaultStageDoesNotReadEnvironment(t *testing.T) {
|
||||
reads := 0
|
||||
type C struct {
|
||||
Value int `json:"value" envDefault:"3"`
|
||||
}
|
||||
got, err := LoadProject("", C{}, EnvOptions{Lookup: func(string) (string, bool) { reads++; return "", false }})
|
||||
if err != nil || got.Value != 3 || reads != 1 {
|
||||
t.Fatal(got, err, reads)
|
||||
}
|
||||
d := locateError(Invalid("servers[0].timeout", "must be positive"), "a.json", Document{}, map[string]string{"servers": "APP_SERVERS"})
|
||||
if d.Env != "APP_SERVERS" || d.File != "" {
|
||||
t.Fatal(d)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecoderRegistryFrozen(t *testing.T) {
|
||||
l, err := NewLoader()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err = WithDecoder("extra", JSONDecoder{})(l); err == nil {
|
||||
t.Fatal("mutable registry")
|
||||
}
|
||||
}
|
||||
func BenchmarkConfigurationStartup(b *testing.B) {
|
||||
path := filepath.Join(b.TempDir(), "app.json")
|
||||
if err := os.WriteFile(path, []byte(`{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":4096}}`), 0600); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
options := EnvOptions{Prefix: "BENCH_", Lookup: func(name string) (string, bool) {
|
||||
if name == "BENCH_UPLOAD_MAX_BYTES" {
|
||||
return "8192", true
|
||||
}
|
||||
return "", false
|
||||
}}
|
||||
defaults := loaderExample{Server: Default()}
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
if _, err := LoadProject(path, defaults, options); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
func BenchmarkConfigurationLarge(b *testing.B) {
|
||||
type Entry struct {
|
||||
Timeout time.Duration `json:"timeout"`
|
||||
Count int `json:"count"`
|
||||
}
|
||||
type Large struct {
|
||||
Entries []Entry `json:"entries"`
|
||||
}
|
||||
raw := []byte(`{"entries":[` + strings.TrimSuffix(strings.Repeat(`{"timeout":"3s","count":10},`, 1000), ",") + `]}`)
|
||||
b.SetBytes(int64(len(raw)))
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
doc, err := (JSONDecoder{}).Decode(raw)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
if _, err = bindDocument(doc, Large{}); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -7,6 +7,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"math"
|
||||
"os"
|
||||
"reflect"
|
||||
@@ -34,30 +35,114 @@ func ApplyDefaults[T any](base T) (T, error) { return projectOverlay(base, EnvOp
|
||||
// LoadProject applies code/tag defaults, a strict optional file, environment,
|
||||
// then the project's Validate method, before any logger or dependency is opened.
|
||||
// envDefault is only applied to zero-valued defaults before reading the file.
|
||||
func LoadProject[T any](path string, defaults T, options EnvOptions) (T, error) {
|
||||
func LoadProject[T any](path string, defaults T, options EnvOptions, loaders ...*Loader) (T, error) {
|
||||
var zero T
|
||||
loader, err := selectLoader(loaders)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
cfg, err := projectOverlay(defaults, options, true)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
var doc Document
|
||||
if path != "" {
|
||||
cfg, err = DecodeFile(path, cfg)
|
||||
doc, err = loader.read(path)
|
||||
if err != nil {
|
||||
return zero, errors.New("config: cannot decode file; check its path, format, field names and value types")
|
||||
return zero, err
|
||||
}
|
||||
cfg, err = bindDocument(doc, cfg)
|
||||
if err != nil {
|
||||
return zero, locateError(err, path, doc, nil)
|
||||
}
|
||||
} else {
|
||||
value, e := cloneConfiguration(reflect.ValueOf(&cfg).Elem(), 0)
|
||||
if e != nil {
|
||||
return zero, &Error{Reason: "cannot clone defaults", cause: e}
|
||||
}
|
||||
cfg = value.Interface().(T)
|
||||
}
|
||||
cfg, err = OverlayEnv(cfg, options)
|
||||
origins := map[string]string{}
|
||||
cfg, err = projectOverlay(cfg, options, false, origins)
|
||||
if err != nil {
|
||||
return zero, err
|
||||
}
|
||||
normalizeProject(reflect.ValueOf(&cfg).Elem(), "", origins, 0)
|
||||
if valid, ok := any(&cfg).(interface{ Validate() error }); ok {
|
||||
if err = valid.Validate(); err != nil {
|
||||
return zero, err
|
||||
return zero, locateError(err, path, doc, origins)
|
||||
}
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
func locateError(err error, path string, doc Document, origins map[string]string) *Error {
|
||||
d := validationError(err)
|
||||
field := d.Field
|
||||
for field != "" && origins[field] == "" {
|
||||
index := strings.LastIndexAny(field, ".[")
|
||||
if index < 0 {
|
||||
field = ""
|
||||
} else {
|
||||
field = field[:index]
|
||||
}
|
||||
}
|
||||
if env := origins[field]; env != "" {
|
||||
d.Env = env
|
||||
d.File = ""
|
||||
d.Line = 0
|
||||
d.Column = 0
|
||||
return d
|
||||
}
|
||||
d.File = path
|
||||
if d.Line == 0 {
|
||||
p := doc.Positions[d.Field]
|
||||
d.Line = p.Line
|
||||
d.Column = p.Column
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
// Framework-owned normalization stays inside the common pipeline. Explicit
|
||||
// environment stream selection clears a file profile unless a file override
|
||||
// was also provided; project code never needs to repeat this rule.
|
||||
func normalizeProject(v reflect.Value, path string, origins map[string]string, depth int) {
|
||||
if depth > 32 {
|
||||
return
|
||||
}
|
||||
if v.Kind() == reflect.Pointer {
|
||||
if v.IsNil() {
|
||||
return
|
||||
}
|
||||
v = v.Elem()
|
||||
}
|
||||
if v.Kind() != reflect.Struct {
|
||||
return
|
||||
}
|
||||
if v.Type() == reflect.TypeOf(Config{}) && v.CanAddr() {
|
||||
c := v.Addr().Interface().(*Config)
|
||||
if origins[joinField(path, "log.output")] != "" && c.Log.Output != "" && (origins[joinField(path, "log.file")] == "" || c.Log.File == "") {
|
||||
c.Log.File = ""
|
||||
c.Log.Rotation = logx.RotationConfig{}
|
||||
}
|
||||
return
|
||||
}
|
||||
for i := 0; i < v.NumField(); i++ {
|
||||
f := v.Type().Field(i)
|
||||
if f.PkgPath != "" {
|
||||
continue
|
||||
}
|
||||
name := strings.Split(f.Tag.Get("json"), ",")[0]
|
||||
if name == "-" {
|
||||
continue
|
||||
}
|
||||
if name == "" {
|
||||
name = f.Name
|
||||
}
|
||||
normalizeProject(v.Field(i), joinField(path, name), origins, depth+1)
|
||||
}
|
||||
}
|
||||
|
||||
// OverlayEnv overlays the entire project struct without mutating the input.
|
||||
// Tags: env:"CUSTOM_NAME,required,secret,empty=reject"; env:"-" excludes a
|
||||
// subtree. required means nonempty string/non-nil pointer or collection; false
|
||||
@@ -70,7 +155,7 @@ func OverlayEnv[T any](base T, options EnvOptions) (T, error) {
|
||||
var textType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()
|
||||
var durationType = reflect.TypeOf(time.Duration(0))
|
||||
|
||||
func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) {
|
||||
func projectOverlay[T any](base T, options EnvOptions, defaults bool, sourceMaps ...map[string]string) (T, error) {
|
||||
var zero T
|
||||
if options.Lookup == nil {
|
||||
options.Lookup = os.LookupEnv
|
||||
@@ -88,8 +173,8 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error)
|
||||
result := reflect.New(v.Type()).Elem()
|
||||
result.Set(v)
|
||||
seen := map[string]bool{}
|
||||
var walk func(reflect.Value, string, int) error
|
||||
walk = func(value reflect.Value, prefix string, depth int) error {
|
||||
var walk func(reflect.Value, string, string, int) error
|
||||
walk = func(value reflect.Value, prefix, path string, depth int) error {
|
||||
if depth > 32 {
|
||||
return errors.New("config: nesting exceeds 32")
|
||||
}
|
||||
@@ -110,6 +195,7 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error)
|
||||
if jsonName == "" {
|
||||
jsonName = field.Name
|
||||
}
|
||||
fieldPath := joinField(path, jsonName)
|
||||
key := prefix + envName(jsonName)
|
||||
if tag[0] != "" {
|
||||
key = tag[0]
|
||||
@@ -131,7 +217,7 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error)
|
||||
}
|
||||
}
|
||||
if dst.Kind() == reflect.Struct && !dst.Addr().Type().Implements(textType) {
|
||||
if err := walk(dst, key+"_", depth+1); err != nil {
|
||||
if err := walk(dst, key+"_", fieldPath, depth+1); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
@@ -142,14 +228,14 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error)
|
||||
if !dst.IsNil() {
|
||||
child.Elem().Set(dst.Elem())
|
||||
}
|
||||
if err := walk(child.Elem(), key+"_", depth+1); err != nil {
|
||||
if err := walk(child.Elem(), key+"_", fieldPath, depth+1); err != nil {
|
||||
return err
|
||||
}
|
||||
if !dst.IsNil() || !child.Elem().IsZero() {
|
||||
dst.Set(child)
|
||||
}
|
||||
if required && !defaults && dst.IsNil() {
|
||||
return fmt.Errorf("config: %s is required", key)
|
||||
return &Error{Env: key, Field: fieldPath, Reason: "is required"}
|
||||
}
|
||||
continue
|
||||
}
|
||||
@@ -164,7 +250,11 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error)
|
||||
return fmt.Errorf("config: duplicate environment key %s", key)
|
||||
}
|
||||
seen[key] = true
|
||||
raw, present := options.Lookup(key)
|
||||
var raw string
|
||||
var present bool
|
||||
if !defaults {
|
||||
raw, present = options.Lookup(key)
|
||||
}
|
||||
selectedKey := key
|
||||
if !present && !defaults {
|
||||
for _, alias := range options.Aliases[key] {
|
||||
@@ -180,7 +270,7 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error)
|
||||
}
|
||||
if present && raw == "" && !defaults {
|
||||
if empty == "reject" {
|
||||
return fmt.Errorf("config: %s must not be empty", selectedKey)
|
||||
return &Error{Env: selectedKey, Field: fieldPath, Reason: "must not be empty"}
|
||||
}
|
||||
if empty == "ignore" {
|
||||
present = false
|
||||
@@ -189,17 +279,24 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error)
|
||||
if present {
|
||||
parsed := reflect.New(dst.Type()).Elem()
|
||||
if err := parseEnvValue(parsed, raw); err != nil {
|
||||
return fmt.Errorf("config: %s must be a valid %s", selectedKey, dst.Type())
|
||||
origin := selectedKey
|
||||
if defaults {
|
||||
origin = ""
|
||||
}
|
||||
return &Error{Env: origin, Field: fieldPath, Reason: "expected a valid " + dst.Type().String(), cause: err}
|
||||
}
|
||||
dst.Set(parsed)
|
||||
if !defaults && len(sourceMaps) > 0 {
|
||||
sourceMaps[0][fieldPath] = selectedKey
|
||||
}
|
||||
}
|
||||
if required && !defaults && emptyRequired(dst) {
|
||||
return fmt.Errorf("config: %s is required", key)
|
||||
return &Error{Env: key, Field: fieldPath, Reason: "is required"}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
if err := walk(result, options.Prefix, 0); err != nil {
|
||||
if err := walk(result, options.Prefix, "", 0); err != nil {
|
||||
return zero, err
|
||||
}
|
||||
return result.Interface().(T), nil
|
||||
|
||||
21
examples/configuration/README.md
Normal file
21
examples/configuration/README.md
Normal file
@@ -0,0 +1,21 @@
|
||||
# 统一配置加载示例
|
||||
|
||||
从框架根目录运行,无需数据库或外部服务:
|
||||
|
||||
```sh
|
||||
go run ./examples/configuration
|
||||
go run ./examples/configuration -format json
|
||||
go run ./examples/configuration -format xml
|
||||
DEMO_CLIENT_TIMEOUT=500ms DEMO_MAX_ITEMS=0 go run ./examples/configuration
|
||||
DEMO_MAX_ITEMS=wrong go run ./examples/configuration
|
||||
```
|
||||
|
||||
前三次都输出 `timeout=2s maxItems=100 allowed=[json xml]`。环境覆盖允许显式零值。
|
||||
最后一条输出定位 `maxItems` / `DEMO_MAX_ITEMS` 的安全错误,不打印输入原值。
|
||||
也可 `-config /path/to/app.yaml` 读取自己的文件。示例没有建立网络连接。
|
||||
|
||||
`Config` 定义字段,`envDefault` 定义缺省值,`Validate` 用 `config.Invalid` 返回字段与公开原因。
|
||||
`newLoader` 额外注册 `.custom`(演示复用 JSON decoder);同一个 `load` 可读取所有格式。
|
||||
JSON/YAML/XML 直接可用,无需注册。新增格式只修改解析器注册,不改业务结构或校验。
|
||||
|
||||
只打印选定非敏感字段,不输出整个配置。完整格式、错误定位和旧项目迁移见 [CONFIG.md](../../CONFIG.md)。
|
||||
5
examples/configuration/config.json
Normal file
5
examples/configuration/config.json
Normal file
@@ -0,0 +1,5 @@
|
||||
{
|
||||
"client": {"endpoint": "https://example.com", "timeout": "2s"},
|
||||
"maxItems": 100,
|
||||
"allowed": ["json", "xml"]
|
||||
}
|
||||
8
examples/configuration/config.xml
Normal file
8
examples/configuration/config.xml
Normal file
@@ -0,0 +1,8 @@
|
||||
<config>
|
||||
<client>
|
||||
<endpoint>https://example.com</endpoint>
|
||||
<timeout>2s</timeout>
|
||||
</client>
|
||||
<maxItems>100</maxItems>
|
||||
<allowed type="array"><item>json</item><item>xml</item></allowed>
|
||||
</config>
|
||||
5
examples/configuration/config.yaml
Normal file
5
examples/configuration/config.yaml
Normal file
@@ -0,0 +1,5 @@
|
||||
client:
|
||||
endpoint: https://example.com
|
||||
timeout: 2s
|
||||
maxItems: 100
|
||||
allowed: [json, xml]
|
||||
85
examples/configuration/main.go
Normal file
85
examples/configuration/main.go
Normal file
@@ -0,0 +1,85 @@
|
||||
// Run with go run ./examples/configuration -format xml, or -config /path/to/app.yaml.
|
||||
package main
|
||||
|
||||
import (
|
||||
"embed"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/config"
|
||||
)
|
||||
|
||||
//go:embed config.json config.yaml config.xml
|
||||
var examples embed.FS
|
||||
|
||||
type ClientConfig struct {
|
||||
Endpoint string `json:"endpoint"`
|
||||
Timeout time.Duration `json:"timeout" envDefault:"3s"`
|
||||
// This field is deliberately unavailable to configuration files.
|
||||
Token string `json:"-" env:"DEMO_CLIENT_TOKEN,secret"`
|
||||
}
|
||||
type Config struct {
|
||||
Client ClientConfig `json:"client"`
|
||||
MaxItems int `json:"maxItems" envDefault:"20"`
|
||||
Allowed []string `json:"allowed"`
|
||||
}
|
||||
|
||||
func (c Config) Validate() error {
|
||||
if c.Client.Timeout <= 0 {
|
||||
return config.Invalid("client.timeout", "must be positive")
|
||||
}
|
||||
if c.MaxItems < 0 {
|
||||
return config.Invalid("maxItems", "must not be negative")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func newLoader() (*config.Loader, error) {
|
||||
// Example of an instance-local extension. A TOML/etc. adapter can implement
|
||||
// Decoder and return Document using that parser instead; no global registry.
|
||||
return config.NewLoader(config.WithDecoder("custom", config.DecoderFunc(func(data []byte) (config.Document, error) {
|
||||
return (config.JSONDecoder{}).Decode(data)
|
||||
})))
|
||||
}
|
||||
func load(path string) (Config, error) {
|
||||
loader, err := newLoader()
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
return config.LoadProject(path, Config{}, config.EnvOptions{Prefix: "DEMO_", Empty: "reject"}, loader)
|
||||
}
|
||||
func run() error {
|
||||
path := flag.String("config", "", "external configuration file")
|
||||
format := flag.String("format", "yaml", "embedded example: json, yaml, xml")
|
||||
flag.Parse()
|
||||
if *path == "" {
|
||||
data, err := examples.ReadFile("config." + *format)
|
||||
if err != nil {
|
||||
return fmt.Errorf("choose json, yaml or xml")
|
||||
}
|
||||
dir, err := os.MkdirTemp("", "goOmniConfiguration-")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
*path = filepath.Join(dir, "config."+*format)
|
||||
if err = os.WriteFile(*path, data, 0600); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
cfg, err := load(*path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Print selected non-sensitive fields, never the whole configuration.
|
||||
fmt.Printf("timeout=%s maxItems=%d allowed=%v\n", cfg.Client.Timeout, cfg.MaxItems, cfg.Allowed)
|
||||
return nil
|
||||
}
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
34
examples/configuration/main_test.go
Normal file
34
examples/configuration/main_test.go
Normal file
@@ -0,0 +1,34 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestExamples(t *testing.T) {
|
||||
var previous Config
|
||||
for i, ext := range []string{"json", "yaml", "xml", "custom"} {
|
||||
source := ext
|
||||
if source == "custom" {
|
||||
source = "json"
|
||||
}
|
||||
data, err := examples.ReadFile("config." + source)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
path := filepath.Join(t.TempDir(), "app."+ext)
|
||||
if err = os.WriteFile(path, data, 0600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
got, err := load(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if i > 0 && !reflect.DeepEqual(previous, got) {
|
||||
t.Fatal("format difference")
|
||||
}
|
||||
previous = got
|
||||
}
|
||||
}
|
||||
12
examples/httpClient/README.md
Normal file
12
examples/httpClient/README.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# HTTP Client 示例
|
||||
|
||||
```sh
|
||||
go run ./examples/httpClient
|
||||
```
|
||||
|
||||
只创建本机临时 HTTP 服务,不访问业务系统、数据库或公网;结束后关闭连接和监听器。
|
||||
示例创建并复用一个 Client,显式直连,配置连接池/阶段超时,通过 JSON 选项传入 Authorization 和 Accept,并按单次请求收紧响应上限。
|
||||
|
||||
输出 `authenticated request succeeded`。示例 token 是固定演示字符串,实际项目从配置或请求上下文取得认证信息。
|
||||
|
||||
完整参数、代理规则、流式调用、生命周期、重试边界和兼容说明见 [httpc](../../httpc/README.md)。
|
||||
57
examples/httpClient/main.go
Normal file
57
examples/httpClient/main.go
Normal file
@@ -0,0 +1,57 @@
|
||||
// Run with go run ./examples/httpClient. All HTTP traffic stays on loopback.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/httpc"
|
||||
)
|
||||
|
||||
type result struct {
|
||||
Message string `json:"message"`
|
||||
}
|
||||
|
||||
func run() error {
|
||||
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Header.Get("Authorization") != "Bearer example-token" {
|
||||
w.WriteHeader(http.StatusUnauthorized)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
_ = json.NewEncoder(w).Encode(result{Message: "authenticated request succeeded"})
|
||||
}))
|
||||
defer upstream.Close()
|
||||
client, err := httpc.New(httpc.Config{
|
||||
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
|
||||
Timeout: 3 * time.Second, DialTimeout: time.Second, TLSHandshakeTimeout: time.Second,
|
||||
ResponseHeaderTimeout: time.Second, IdleConnTimeout: 30 * time.Second,
|
||||
MaxConnsPerHost: 16, MaxIdleConnsPerHost: 8, MaxIdleConns: 32,
|
||||
MaxResponseBytes: 1 << 20, DisableLogging: true,
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer client.Close()
|
||||
var output result
|
||||
if err = client.JSON(context.Background(), http.MethodGet, upstream.URL, nil, &output,
|
||||
httpc.WithHeader("Authorization", "Bearer example-token"),
|
||||
httpc.WithHeaders(http.Header{"Accept": []string{"application/json"}}),
|
||||
httpc.WithMaxResponseBytes(64<<10),
|
||||
); err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Println(output.Message)
|
||||
return nil
|
||||
}
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
9
examples/httpClient/main_test.go
Normal file
9
examples/httpClient/main_test.go
Normal file
@@ -0,0 +1,9 @@
|
||||
package main
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestExample(t *testing.T) {
|
||||
if err := run(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"generatorConfig": true,
|
||||
"templateVersion": "v0.19.0-preview",
|
||||
"templateVersion": "v0.20.0-preview",
|
||||
"profile": {
|
||||
"database": "none",
|
||||
"storage": "none",
|
||||
@@ -12,6 +12,7 @@
|
||||
"entry": "api/managed.api",
|
||||
"generated": {
|
||||
"docs/openapi.json": "0370fd6bd6a387e2ef21f6b42b009ce66749846b8bea3d481009a34f3af039a5",
|
||||
"internal/config/loadGen.go": "5d630b894abdbbd66ddfa7d7d30f7573fc4ff995020a32a9f9157cd4af3a5655",
|
||||
"internal/handler/appGen.go": "f5251b4dc33e2312b0bdb71ead70d2171977d851265e124f640458d705a2cc9d",
|
||||
"internal/handler/demo/handlersGen.go": "3c76d85dbe89eb5127d9d134dc625203ecf9719dd11d174c1f335691b99e1375",
|
||||
"internal/handler/openapiGen.go": "f24cab3abbb8557a3fc4dd93fba23cc0b093f3e0c944d3ba1bd5b0eeff2d96f3",
|
||||
|
||||
@@ -2,7 +2,7 @@ module example.com/managed
|
||||
|
||||
go 1.22.0
|
||||
|
||||
require git.yuntuo.hk/k2safe/go-omni v0.19.0-preview
|
||||
require git.yuntuo.hk/k2safe/go-omni v0.21.0-preview
|
||||
|
||||
require (
|
||||
github.com/gabriel-vasile/mimetype v1.4.3 // indirect
|
||||
|
||||
@@ -1,10 +1,9 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"git.yuntuo.hk/k2safe/go-omni/config"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"git.yuntuo.hk/k2safe/go-omni/config"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
@@ -22,42 +21,10 @@ func Default() Config {
|
||||
return Config{Server: server}
|
||||
}
|
||||
|
||||
// Load applies defaults -> file -> typed environment -> validation.
|
||||
// New fields automatically use GO_OMNI_<JSON_PATH>, or an explicit env tag.
|
||||
func Load(path string) (Config, error) {
|
||||
cfg := Default()
|
||||
var err error
|
||||
cfg, err = config.ApplyDefaults(cfg)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if path != "" {
|
||||
cfg, err = config.DecodeFile(path, cfg)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
}
|
||||
cfg, err = config.OverlayEnv(cfg, config.EnvOptions{Prefix: "GO_OMNI_", Empty: "ignore", Aliases: config.ServerEnvAliases("GO_OMNI_", "server")})
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
output, outputSet := os.LookupEnv("GO_OMNI_SERVER_LOG_OUTPUT")
|
||||
if !outputSet {
|
||||
output = os.Getenv("GO_OMNI_LOG_OUTPUT")
|
||||
}
|
||||
file, fileSet := os.LookupEnv("GO_OMNI_SERVER_LOG_FILE")
|
||||
if !fileSet {
|
||||
file = os.Getenv("GO_OMNI_LOG_FILE")
|
||||
}
|
||||
if output != "" && file == "" {
|
||||
cfg.Server.Log.File = ""
|
||||
cfg.Server.Log.Rotation = logx.RotationConfig{}
|
||||
}
|
||||
return cfg, cfg.Validate()
|
||||
}
|
||||
// Validate checks configuration before logs or dependencies open.
|
||||
func (c Config) Validate() error {
|
||||
if err := c.Server.Validate(); err != nil {
|
||||
return err
|
||||
return config.AtField("server", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
|
||||
16
examples/managedService/internal/config/loadGen.go
Normal file
16
examples/managedService/internal/config/loadGen.go
Normal file
@@ -0,0 +1,16 @@
|
||||
// Code generated by go-omni api. DO NOT EDIT.
|
||||
package config
|
||||
|
||||
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
|
||||
|
||||
// Load is maintained by the generator. Extend Config/Default/Validate in
|
||||
// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline.
|
||||
func Load(path string) (Config, error) {
|
||||
loader, err := NewLoader()
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
return omniConfig.LoadProject(path, Default(), omniConfig.EnvOptions{
|
||||
Prefix: "GO_OMNI_", Empty: "ignore", Aliases: omniConfig.ServerEnvAliases("GO_OMNI_", "server"),
|
||||
}, loader)
|
||||
}
|
||||
7
examples/managedService/internal/config/loader.go
Normal file
7
examples/managedService/internal/config/loader.go
Normal file
@@ -0,0 +1,7 @@
|
||||
package config
|
||||
|
||||
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
|
||||
|
||||
// NewLoader is project-owned. JSON/YAML/XML work without changes. Register
|
||||
// additional format decoders here only when the project needs them.
|
||||
func NewLoader() (*omniConfig.Loader, error) { return omniConfig.NewLoader() }
|
||||
153
httpc/README.md
Normal file
153
httpc/README.md
Normal file
@@ -0,0 +1,153 @@
|
||||
# httpc:复用连接的 HTTP 客户端
|
||||
|
||||
`httpc` 在标准 `net/http` 上提供独立连接池、网络阶段总超时、保守重试、请求追踪和有界 JSON 调用。普通接口可以直接带认证头调用;上传、下载、特殊协议仍使用标准 Request/Response。没有业务身份模型或全局认证信息。
|
||||
|
||||
## 创建一次,业务重复使用
|
||||
|
||||
```go
|
||||
client, err := httpc.New(httpc.Config{
|
||||
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
|
||||
Timeout: 3 * time.Second,
|
||||
DialTimeout: time.Second,
|
||||
TLSHandshakeTimeout: time.Second,
|
||||
ResponseHeaderTimeout: time.Second,
|
||||
MaxConnsPerHost: 32,
|
||||
MaxIdleConnsPerHost: 16,
|
||||
MaxIdleConns: 64,
|
||||
IdleConnTimeout: 60 * time.Second,
|
||||
MaxResponseBytes: 1 << 20,
|
||||
})
|
||||
if err != nil { return err }
|
||||
// 在 svc 初始化并持有 client,向生命周期登记 client.Close。
|
||||
// 普通独立程序也可 defer client.Close();不要每次请求重新 New。
|
||||
|
||||
err = client.JSON(ctx, http.MethodGet, serviceURL, nil, &result,
|
||||
httpc.WithHeader("Authorization", "Bearer "+token),
|
||||
httpc.WithHeaders(http.Header{"Accept": {"application/json"}}),
|
||||
)
|
||||
```
|
||||
|
||||
Client 可并发使用。每个调用的 input/output 及其读写同步由调用方负责,不要让多个请求同时写同一个 output。
|
||||
|
||||
直接运行 [examples/httpClient](../examples/httpClient/README.md),只使用本地临时服务,无需业务 Token 或外部基础设施。
|
||||
|
||||
## 代理规则
|
||||
|
||||
| 配置 | 行为 |
|
||||
| --- | --- |
|
||||
| 不填写 Proxy | 保留被克隆 DefaultTransport 的代理策略;未改动的标准库默认使用环境代理 |
|
||||
| `Mode: httpc.ProxyEnvironment` | 明确使用 HTTP_PROXY / HTTPS_PROXY / NO_PROXY(及小写形式) |
|
||||
| `Mode: httpc.ProxyDirect` | 设置 Proxy=nil,明确直连,不受环境代理影响 |
|
||||
| `Mode: httpc.ProxyURL, URL: "http://proxy.example:8080"` | 固定代理地址,不通过 NO_PROXY 排除 |
|
||||
|
||||
支持 http、https、socks5、socks5h;底层行为沿用 Go Transport。固定地址必须是代理 origin,不接受业务路径、查询、fragment 或无效端口。凭证可由 URL userinfo 提供,但配置错误不输出 URL/密码;不要打印整个 Config。环境代理规则由标准库读取并缓存,不提供运行时环境代理热更新。
|
||||
|
||||
内部服务要直连,只设置 `proxy.mode: direct`,无需创建 Transport。需要自定义路由、代理认证握手等复杂逻辑时注入自己的 Transport。
|
||||
|
||||
## 超时和连接池
|
||||
|
||||
除原有字段外,新字段都可选:**0 保留默认行为,正数覆盖,负数拒绝**。需要明确关闭某个阶段超时、关闭 TCP keepalive 或特殊 dialer 时,使用自定义 Transport。
|
||||
|
||||
| 字段 | 默认/含义 |
|
||||
| --- | --- |
|
||||
| Timeout | 10s;从 Do 开始,覆盖 DNS/建连、等待连接、重试、退避、响应头和读取响应体 |
|
||||
| DialTimeout | 不填时保留原 dialer;标准库默认 30s;配置后使用标准 net.Dialer |
|
||||
| DialKeepAlive | 标准 net.Dialer 的 TCP keepalive 探测间隔,标准默认 30s;不是 HTTP 空闲连接超时 |
|
||||
| TLSHandshakeTimeout | 沿用克隆 Transport;标准库默认 10s |
|
||||
| ResponseHeaderTimeout | 沿用 Transport,标准默认不单独限制;从完整发送请求后等响应头,总 Timeout 仍生效 |
|
||||
| ExpectContinueTimeout | 沿用 Transport;标准默认 1s,等待 `100-continue` 的时间 |
|
||||
| IdleConnTimeout | 沿用 Transport;标准默认 90s,只回收空闲连接 |
|
||||
| MaxConnsPerHost | 默认 64;每个目标的建连中、活动与空闲连接总上限 |
|
||||
| MaxIdleConnsPerHost | 默认等于 MaxConnsPerHost,可单独配置 |
|
||||
| MaxIdleConns | 默认 MaxConnsPerHost×2,可单独配置;所有目标合计的空闲连接上限 |
|
||||
| MaxResponseBytes | JSON 成功响应默认最多 1 MiB,最大允许配置 1 GiB;应按实际接口保持较小限额 |
|
||||
|
||||
每主机上限最多 65536,总空闲连接最多 131072。总空闲上限可以小于各主机上限之和。MaxConnsPerHost 不等于整个进程的连接上限,也不等于 HTTP/2 的请求并发上限。
|
||||
|
||||
配置 DialTimeout / DialKeepAlive 会创建标准 net.Dialer;未填的另一个值使用标准默认 30s。不要通过全局修改 DefaultTransport 来塞入项目的特殊拨号逻辑;请显式使用借用的 Transport。若 DefaultTransport 有自定义 TLS dialer,相关阶段覆盖会报错,避免 TLS timeout 被悄悄忽略。
|
||||
|
||||
调用者 context 的更早截止时间优先。总超时不能抢占 Go 的 JSON 编解码或不遵守 context 的自定义 RoundTripper;计时从 Do 开始,不包括调用前的 json.Marshal。TLS 校验默认保持开启,没有新增 InsecureSkipVerify 配置。
|
||||
|
||||
## JSON 的单次请求选项
|
||||
|
||||
```go
|
||||
err := client.JSON(ctx, "POST", endpoint, input, &output,
|
||||
httpc.WithHeader("Authorization", "Bearer "+token),
|
||||
httpc.WithHeader("X-Tenant", tenant),
|
||||
httpc.WithMaxResponseBytes(64<<10),
|
||||
)
|
||||
```
|
||||
|
||||
- `WithHeader` 设置一个头;`WithHeaders` 支持多值 http.Header。后出现的选项覆盖同名头,不影响其他头。
|
||||
- WithHeaders 在创建选项时快照原 map,每次调用复制值。构建后修改原 map 不影响选项,选项可以并发复用;构建时原 map 的读写同步由调用方负责。
|
||||
- 不提供 Client 全局 Authorization,避免跨用户调用共享凭证。框架不会把请求头写进日志。
|
||||
- 有 input 时默认 Content-Type 为 application/json;显式请求头可覆盖。请求头名称/值会校验,拒绝 CRLF 等控制字符和歧义的大小写重复键。
|
||||
- Host、Content-Length、Transfer-Encoding、Trailer 属于请求目标/分帧控制,请用标准 Request 配合 Do;JSON 选项中不允许伪装成普通头。
|
||||
- `WithMaxResponseBytes` 是本次 JSON 的正数限额,不修改 Client 默认值,不改变 Do 的流式语义。
|
||||
|
||||
JSON 自动关闭响应体。仅 2xx 解码;空成功响应(例如 204)保留 output 原值。非空输入使用标准 json.Unmarshal,拒绝多个 JSON 文档,类型错误保留 Go 的错误信息与行为。
|
||||
|
||||
`output == nil` 按限额流式丢弃成功响应,不把整个 body 读进内存,也不检查 JSON 语法。响应限额按实际读取的 body 计算;标准库自动 gzip 解压后仍受上限约束。input 会先 json.Marshal 到内存,大文件上传不适合用 JSON helper。
|
||||
|
||||
非 2xx 返回 `*httpc.StatusError`,只包含状态码,不包含错误响应体。为连接复用,只尝试读完**声明长度已知且不超过 min(32 KiB, 本次响应限额)** 的小错误体,最多等待 25ms(或更早总截止时间);失败仍返回原 HTTP 状态。未知长度/大错误体立即关闭,优先及时返回。重试过程的错误体另受 32 KiB 大小及整体请求截止时间约束,不无界读取。
|
||||
|
||||
## 配置文件与生成项目
|
||||
|
||||
在项目 Config 中加一个字段:
|
||||
|
||||
```go
|
||||
Outbound httpc.Config `json:"outbound"`
|
||||
```
|
||||
|
||||
项目 Validate 中调用 `c.Outbound.Validate()`;创建时 New 也会验证。svc 初始化一次并通过生命周期登记 Close。文件例子:
|
||||
|
||||
```yaml
|
||||
outbound:
|
||||
timeout: 3s
|
||||
proxy:
|
||||
mode: direct
|
||||
dialTimeout: 500ms
|
||||
tlsHandshakeTimeout: 1s
|
||||
responseHeaderTimeout: 1s
|
||||
maxConnsPerHost: 32
|
||||
maxIdleConnsPerHost: 16
|
||||
maxIdleConns: 64
|
||||
idleConnTimeout: 60s
|
||||
maxResponseBytes: 1048576
|
||||
```
|
||||
|
||||
现有统一加载入口可读取 JSON/YAML/XML,并自动覆盖 `GO_OMNI_OUTBOUND_PROXY_MODE`、`GO_OMNI_OUTBOUND_DIAL_TIMEOUT` 等环境变量。无需为每个参数手写环境解析。详见 [统一配置](../CONFIG.md)。
|
||||
|
||||
## 标准请求、特殊 TLS 与生命周期
|
||||
|
||||
```go
|
||||
req, err := http.NewRequestWithContext(ctx, "GET", downloadURL, nil)
|
||||
if err != nil { return err }
|
||||
req.Header.Set("Authorization", "Bearer "+token)
|
||||
res, err := client.Do(req)
|
||||
if err != nil { return err }
|
||||
defer res.Body.Close()
|
||||
// 自行判断状态,按项目要求限流/限制大小,流式写入目标;不要无界 ReadAll 大文件。
|
||||
_, err = io.Copy(destination, res.Body)
|
||||
```
|
||||
|
||||
Do 不自动限制 body,允许下载/流式处理,调用方必须关闭。响应体消费/关闭前网络超时持续有效。
|
||||
|
||||
`Config.Transport` 可注入任意 http.RoundTripper,适配自定义 dialer、mTLS、根证书、监控包装或测试替身。框架借用它,**不修改、不调用其 CloseIdleConnections**。代理、连接池、TLS/建连阶段配置不能同时填入,避免无效配置;总 Timeout、JSON 限额、框架重试/日志仍有效。借用实现必须遵守标准请求和 body 所有权、并发及 context 约定。
|
||||
|
||||
`Close` 幂等:拒绝新调用,关闭自有池的空闲连接,不主动打断已开始的请求。服务关闭应先排空 HTTP/后台调用,再关闭 Client;它不是后台 drain 的替代品。生成项目可通过 svc 的生命周期登记清理。
|
||||
|
||||
## 重试、日志与兼容
|
||||
|
||||
MaxRetries 默认 0,最多 5。仅可重放的 GET/HEAD/OPTIONS 对网络错误、429/502/503/504 做框架重试。POST/PUT/PATCH/DELETE 不自动做应用层重试。标准库或自定义 Transport 自己的重放规则仍可能生效,不能承诺业务“恰好执行一次”。
|
||||
|
||||
Backoff 默认 100ms,MaxBackoff 默认 2s,采用指数退避与抖动。Retry-After 要求等待超过 MaxBackoff 时返回当前响应,不提前请求;总截止时间限制全部尝试。成功或无重试的调用不计算退避。
|
||||
|
||||
不跟随重定向,避免自动向其他地址转发认证信息;3xx 由调用者判断。默认传递 context 中的 X-Request-ID,显式请求头优先。日志只包含 method、hostname、status、attempt、收到响应头的耗时等,不打印路径、查询、用户名密码、认证头或响应体;标准 net/http 返回的 URL 错误可能携带地址,项目不要把 URL 中的密钥当作安全日志字段。
|
||||
|
||||
v0.21 兼容说明:
|
||||
|
||||
- 原来的 JSON 五参数直接调用继续有效;新方法末尾增加 `...httpc.RequestOption`,自行定义的 interface/方法值类型需要同步签名。
|
||||
- 原默认代理、总超时、重试、池大小和成功响应含义保留;新字段未设置时不改变它们。
|
||||
- `Transport` 与显式池/代理/阶段参数混用现在报错;此前 MaxConnsPerHost 在自定义 Transport 下实际被忽略。迁移时把参数放到自己的 Transport,或删掉 Transport 改用内置配置。
|
||||
- 没有移除 Do、JSON、StatusError 或自定义 Transport,没有引入新第三方依赖。
|
||||
100
httpc/benchmark_test.go
Normal file
100
httpc/benchmark_test.go
Normal file
@@ -0,0 +1,100 @@
|
||||
package httpc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
type benchmarkTransport struct {
|
||||
payload string
|
||||
status int
|
||||
}
|
||||
|
||||
func (t benchmarkTransport) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
if r.Body != nil {
|
||||
_ = r.Body.Close()
|
||||
}
|
||||
return &http.Response{StatusCode: t.status, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(t.payload)), ContentLength: int64(len(t.payload)), Request: r}, nil
|
||||
}
|
||||
func BenchmarkHTTPClient(b *testing.B) {
|
||||
for _, size := range []int{128, 4096, 65536} {
|
||||
for _, discard := range []bool{false, true} {
|
||||
label := map[int]string{128: "128B", 4096: "4KiB", 65536: "64KiB"}[size]
|
||||
if discard {
|
||||
label += "/discard"
|
||||
} else {
|
||||
label += "/decode"
|
||||
}
|
||||
b.Run(label, func(b *testing.B) {
|
||||
c, err := New(Config{Transport: benchmarkTransport{`{"value":"` + strings.Repeat("x", size) + `"}`, 200}, DisableLogging: true})
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
defer c.Close()
|
||||
var output struct {
|
||||
Value string `json:"value"`
|
||||
}
|
||||
var target any = &output
|
||||
if discard {
|
||||
target = nil
|
||||
}
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
if err := c.JSON(context.Background(), "GET", "http://example.invalid/item", nil, target); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
b.Run("Do", func(b *testing.B) {
|
||||
c, _ := New(Config{Transport: benchmarkTransport{"", 200}, DisableLogging: true})
|
||||
defer c.Close()
|
||||
r, _ := http.NewRequest("GET", "http://example.invalid/item", nil)
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
res, err := c.Do(r)
|
||||
if err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
res.Body.Close()
|
||||
}
|
||||
})
|
||||
}
|
||||
func BenchmarkHTTPPool(b *testing.B) {
|
||||
for _, status := range []int{200, 401} {
|
||||
b.Run(http.StatusText(status), func(b *testing.B) {
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(status); io.WriteString(w, `{"ok":true}`) }))
|
||||
defer s.Close()
|
||||
c, _ := New(Config{DisableLogging: true})
|
||||
defer c.Close()
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
|
||||
if status == 200 && err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func BenchmarkHTTPHeaders(b *testing.B) {
|
||||
c, _ := New(Config{Transport: benchmarkTransport{`{"ok":true}`, 200}, DisableLogging: true})
|
||||
defer c.Close()
|
||||
option := WithHeaders(http.Header{"Authorization": []string{"Bearer example"}, "X-Request-Id": []string{"example-request"}})
|
||||
b.ReportAllocs()
|
||||
b.ResetTimer()
|
||||
for i := 0; i < b.N; i++ {
|
||||
if err := c.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, option); err != nil {
|
||||
b.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
156
httpc/client.go
156
httpc/client.go
@@ -3,9 +3,7 @@
|
||||
package httpc
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
@@ -22,18 +20,6 @@ import (
|
||||
var ErrClosed = errors.New("httpc: closed")
|
||||
var ErrResponseTooLarge = errors.New("httpc: response exceeds limit")
|
||||
|
||||
type Config struct {
|
||||
Timeout time.Duration `json:"timeout"`
|
||||
MaxRetries int `json:"maxRetries"`
|
||||
Backoff time.Duration `json:"backoff"`
|
||||
MaxBackoff time.Duration `json:"maxBackoff"`
|
||||
MaxConnsPerHost int `json:"maxConnsPerHost"`
|
||||
MaxResponseBytes int64 `json:"maxResponseBytes"`
|
||||
DisableRequestID bool `json:"disableRequestId"`
|
||||
DisableLogging bool `json:"disableLogging"`
|
||||
Transport http.RoundTripper `json:"-"` // Borrowed. The caller owns its lifecycle.
|
||||
}
|
||||
|
||||
type Client struct {
|
||||
config Config
|
||||
client *http.Client
|
||||
@@ -41,43 +27,19 @@ type Client struct {
|
||||
closed atomic.Bool
|
||||
}
|
||||
|
||||
// New creates a reusable client. Close it after callers and response streams drain.
|
||||
func New(cfg Config) (*Client, error) {
|
||||
if cfg.Timeout < 0 || cfg.Backoff < 0 || cfg.MaxBackoff < 0 || cfg.MaxRetries < 0 || cfg.MaxRetries > 5 || cfg.MaxConnsPerHost < 0 || cfg.MaxResponseBytes < 0 {
|
||||
return nil, errors.New("httpc: invalid configuration")
|
||||
normalized, err := normalizeConfig(cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if cfg.Timeout == 0 {
|
||||
cfg.Timeout = 10 * time.Second
|
||||
}
|
||||
if cfg.Backoff == 0 {
|
||||
cfg.Backoff = 100 * time.Millisecond
|
||||
}
|
||||
if cfg.MaxBackoff == 0 {
|
||||
cfg.MaxBackoff = 2 * time.Second
|
||||
}
|
||||
if cfg.Backoff > cfg.MaxBackoff {
|
||||
return nil, errors.New("httpc: backoff exceeds maxBackoff")
|
||||
}
|
||||
if cfg.MaxConnsPerHost == 0 {
|
||||
cfg.MaxConnsPerHost = 64
|
||||
}
|
||||
if cfg.MaxResponseBytes == 0 {
|
||||
cfg.MaxResponseBytes = 1 << 20
|
||||
}
|
||||
if cfg.MaxResponseBytes > 1<<30 || cfg.MaxConnsPerHost > 65536 {
|
||||
return nil, errors.New("httpc: response or connection limit exceeds supported capacity")
|
||||
}
|
||||
c := &Client{config: cfg}
|
||||
c := &Client{config: normalized}
|
||||
transport := cfg.Transport
|
||||
if transport == nil {
|
||||
// Own a private pool; never mutate the process-wide DefaultTransport.
|
||||
base, ok := http.DefaultTransport.(*http.Transport)
|
||||
if !ok {
|
||||
return nil, errors.New("httpc: provide Transport when DefaultTransport is replaced")
|
||||
c.owned, err = ownedTransport(normalized)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
c.owned = base.Clone()
|
||||
c.owned.MaxConnsPerHost = cfg.MaxConnsPerHost
|
||||
c.owned.MaxIdleConnsPerHost = cfg.MaxConnsPerHost
|
||||
c.owned.MaxIdleConns = cfg.MaxConnsPerHost * 2
|
||||
transport = c.owned
|
||||
}
|
||||
c.client = &http.Client{Transport: transport, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
|
||||
@@ -114,7 +76,10 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
|
||||
}
|
||||
replayable := (base.Method == "" || base.Method == "GET" || base.Method == "HEAD" || base.Method == "OPTIONS") && (base.Body == nil || base.Body == http.NoBody || base.GetBody != nil)
|
||||
for attempt := 0; ; attempt++ {
|
||||
req := base.Clone(ctx)
|
||||
req := base
|
||||
if attempt > 0 {
|
||||
req = base.Clone(ctx)
|
||||
}
|
||||
if attempt > 0 && base.Body != nil && base.Body != http.NoBody {
|
||||
body, err := base.GetBody()
|
||||
if err != nil {
|
||||
@@ -133,23 +98,9 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
|
||||
logx.InfoContext(ctx, "http.client", "method", req.Method, "host", req.URL.Hostname(), "status", status, "attempt", attempt+1, "durationMs", time.Since(started).Seconds()*1000, "failed", err != nil)
|
||||
}
|
||||
retry := replayable && attempt < c.config.MaxRetries && ctx.Err() == nil && (err != nil || status == 429 || status == 502 || status == 503 || status == 504)
|
||||
delay := c.config.Backoff
|
||||
for i := 0; i < attempt && delay < c.config.MaxBackoff; i++ {
|
||||
if delay > c.config.MaxBackoff/2 {
|
||||
delay = c.config.MaxBackoff
|
||||
} else {
|
||||
delay *= 2
|
||||
}
|
||||
}
|
||||
if response != nil && response.Header.Get("Retry-After") != "" {
|
||||
if value, ok := retryAfter(response.Header.Get("Retry-After")); ok {
|
||||
delay = value
|
||||
}
|
||||
} else if delay > 1 {
|
||||
delay = delay/2 + time.Duration(rand.Int63n(int64(delay/2)+1))
|
||||
}
|
||||
if delay > c.config.MaxBackoff {
|
||||
retry = false
|
||||
var delay time.Duration
|
||||
if retry {
|
||||
delay, retry = c.retryDelay(attempt, response)
|
||||
}
|
||||
if !retry {
|
||||
if err != nil {
|
||||
@@ -160,9 +111,7 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
|
||||
return response, nil
|
||||
}
|
||||
if response != nil {
|
||||
if response.ContentLength >= 0 && response.ContentLength <= 32<<10 {
|
||||
_, _ = io.Copy(io.Discard, response.Body)
|
||||
}
|
||||
drainSmallBody(response, 32<<10)
|
||||
_ = response.Body.Close()
|
||||
}
|
||||
timer := time.NewTimer(delay)
|
||||
@@ -204,48 +153,6 @@ type StatusError struct{ StatusCode int }
|
||||
|
||||
func (e *StatusError) Error() string { return fmt.Sprintf("httpc: HTTP status %d", e.StatusCode) }
|
||||
|
||||
// JSON bounds response memory, closes the body, and decodes only 2xx responses.
|
||||
// Error bodies, URLs, credentials and query strings are not included in logs.
|
||||
func (c *Client) JSON(ctx context.Context, method, url string, input, output any) error {
|
||||
if ctx == nil {
|
||||
return errors.New("httpc: nil context")
|
||||
}
|
||||
var body io.Reader
|
||||
if input != nil {
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
body = bytes.NewReader(data)
|
||||
}
|
||||
request, err := http.NewRequestWithContext(ctx, method, url, body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if input != nil {
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
response, err := c.Do(request)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode < 200 || response.StatusCode >= 300 {
|
||||
return &StatusError{StatusCode: response.StatusCode}
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(response.Body, c.config.MaxResponseBytes+1))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if int64(len(data)) > c.config.MaxResponseBytes {
|
||||
return ErrResponseTooLarge
|
||||
}
|
||||
if output == nil || len(bytes.TrimSpace(data)) == 0 {
|
||||
return nil
|
||||
}
|
||||
return json.Unmarshal(data, output)
|
||||
}
|
||||
|
||||
func (c *Client) Close() error {
|
||||
c.closed.Store(true)
|
||||
if c.owned != nil {
|
||||
@@ -253,3 +160,34 @@ func (c *Client) Close() error {
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// retryDelay is deliberately outside the success path: no jitter/random work
|
||||
// for calls that cannot or do not need to retry.
|
||||
func (c *Client) retryDelay(attempt int, response *http.Response) (time.Duration, bool) {
|
||||
delay := c.config.Backoff
|
||||
for i := 0; i < attempt && delay < c.config.MaxBackoff; i++ {
|
||||
if delay > c.config.MaxBackoff/2 {
|
||||
delay = c.config.MaxBackoff
|
||||
} else {
|
||||
delay *= 2
|
||||
}
|
||||
}
|
||||
if response != nil {
|
||||
if value, ok := retryAfter(response.Header.Get("Retry-After")); ok {
|
||||
return value, value <= c.config.MaxBackoff
|
||||
}
|
||||
}
|
||||
if delay > 1 {
|
||||
delay = delay/2 + time.Duration(rand.Int63n(int64(delay/2)+1))
|
||||
}
|
||||
return delay, delay <= c.config.MaxBackoff
|
||||
}
|
||||
|
||||
// Drain only a known small error body. Unknown/large bodies are closed promptly;
|
||||
// never allocate or consume an unbounded error document just to retain a pool slot.
|
||||
func drainSmallBody(response *http.Response, limit int64) {
|
||||
limit = min(limit, 32<<10)
|
||||
if response.Body != nil && response.ContentLength >= 0 && response.ContentLength <= limit {
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(response.Body, limit+1))
|
||||
}
|
||||
}
|
||||
|
||||
205
httpc/config.go
Normal file
205
httpc/config.go
Normal file
@@ -0,0 +1,205 @@
|
||||
package httpc
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"reflect"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// ProxyMode controls the owned transport's proxy selection. Empty preserves
|
||||
// DefaultTransport's policy (normally environment proxies).
|
||||
type ProxyMode string
|
||||
|
||||
const (
|
||||
ProxyEnvironment ProxyMode = "environment"
|
||||
ProxyDirect ProxyMode = "direct"
|
||||
ProxyURL ProxyMode = "url"
|
||||
)
|
||||
|
||||
type ProxyConfig struct {
|
||||
Mode ProxyMode `json:"mode"`
|
||||
URL string `json:"url"` // May contain credentials; never log the configuration.
|
||||
}
|
||||
|
||||
// Config owns one private connection pool unless Transport is supplied.
|
||||
// New duration/pool fields are optional: zero retains existing defaults, positive
|
||||
// values override them. Use a custom Transport for disabling timeouts, custom
|
||||
// dialers, mTLS and other protocol-specific settings.
|
||||
type Config struct {
|
||||
Timeout time.Duration `json:"timeout"`
|
||||
MaxRetries int `json:"maxRetries"`
|
||||
Backoff time.Duration `json:"backoff"`
|
||||
MaxBackoff time.Duration `json:"maxBackoff"`
|
||||
MaxConnsPerHost int `json:"maxConnsPerHost"`
|
||||
MaxIdleConns int `json:"maxIdleConns"`
|
||||
MaxIdleConnsPerHost int `json:"maxIdleConnsPerHost"`
|
||||
DialTimeout time.Duration `json:"dialTimeout"`
|
||||
DialKeepAlive time.Duration `json:"dialKeepAlive"`
|
||||
TLSHandshakeTimeout time.Duration `json:"tlsHandshakeTimeout"`
|
||||
ResponseHeaderTimeout time.Duration `json:"responseHeaderTimeout"`
|
||||
ExpectContinueTimeout time.Duration `json:"expectContinueTimeout"`
|
||||
IdleConnTimeout time.Duration `json:"idleConnTimeout"`
|
||||
Proxy ProxyConfig `json:"proxy"`
|
||||
MaxResponseBytes int64 `json:"maxResponseBytes"`
|
||||
DisableRequestID bool `json:"disableRequestId"`
|
||||
DisableLogging bool `json:"disableLogging"`
|
||||
Transport http.RoundTripper `json:"-"` // Borrowed; never modified or closed.
|
||||
}
|
||||
|
||||
const maxResponseBytes int64 = 1 << 30
|
||||
|
||||
// Validate checks values without creating pools or making network requests.
|
||||
func (cfg Config) Validate() error { _, err := normalizeConfig(cfg); return err }
|
||||
func normalizeConfig(cfg Config) (Config, error) {
|
||||
for _, field := range []struct {
|
||||
name string
|
||||
value time.Duration
|
||||
}{
|
||||
{"timeout", cfg.Timeout}, {"backoff", cfg.Backoff}, {"maxBackoff", cfg.MaxBackoff},
|
||||
{"dialTimeout", cfg.DialTimeout}, {"dialKeepAlive", cfg.DialKeepAlive}, {"tlsHandshakeTimeout", cfg.TLSHandshakeTimeout},
|
||||
{"responseHeaderTimeout", cfg.ResponseHeaderTimeout}, {"expectContinueTimeout", cfg.ExpectContinueTimeout}, {"idleConnTimeout", cfg.IdleConnTimeout},
|
||||
} {
|
||||
if field.value < 0 {
|
||||
return Config{}, fmt.Errorf("httpc: %s must not be negative", field.name)
|
||||
}
|
||||
}
|
||||
if cfg.MaxRetries < 0 || cfg.MaxRetries > 5 {
|
||||
return Config{}, errors.New("httpc: maxRetries must be 0..5")
|
||||
}
|
||||
if cfg.MaxConnsPerHost < 0 || cfg.MaxConnsPerHost > 65536 || cfg.MaxIdleConnsPerHost < 0 || cfg.MaxIdleConnsPerHost > 65536 || cfg.MaxIdleConns < 0 || cfg.MaxIdleConns > 131072 {
|
||||
return Config{}, errors.New("httpc: invalid connection pool limits")
|
||||
}
|
||||
if cfg.MaxResponseBytes < 0 || cfg.MaxResponseBytes > maxResponseBytes {
|
||||
return Config{}, errors.New("httpc: maxResponseBytes must be 0..1 GiB")
|
||||
}
|
||||
if _, err := proxyFunction(cfg.Proxy); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.Transport != nil {
|
||||
value := reflect.ValueOf(cfg.Transport)
|
||||
switch value.Kind() {
|
||||
case reflect.Pointer, reflect.Map, reflect.Func, reflect.Slice, reflect.Interface, reflect.Chan:
|
||||
if value.IsNil() {
|
||||
return Config{}, errors.New("httpc: nil Transport")
|
||||
}
|
||||
}
|
||||
if cfg.hasTransportOptions() {
|
||||
return Config{}, errors.New("httpc: Transport cannot be combined with proxy, dial, TLS or connection pool options; configure the borrowed transport directly")
|
||||
}
|
||||
}
|
||||
if cfg.Timeout == 0 {
|
||||
cfg.Timeout = 10 * time.Second
|
||||
}
|
||||
if cfg.Backoff == 0 {
|
||||
cfg.Backoff = 100 * time.Millisecond
|
||||
}
|
||||
if cfg.MaxBackoff == 0 {
|
||||
cfg.MaxBackoff = 2 * time.Second
|
||||
}
|
||||
if cfg.Backoff > cfg.MaxBackoff {
|
||||
return Config{}, errors.New("httpc: backoff exceeds maxBackoff")
|
||||
}
|
||||
if cfg.MaxConnsPerHost == 0 {
|
||||
cfg.MaxConnsPerHost = 64
|
||||
}
|
||||
if cfg.MaxIdleConnsPerHost == 0 {
|
||||
cfg.MaxIdleConnsPerHost = cfg.MaxConnsPerHost
|
||||
}
|
||||
if cfg.MaxIdleConns == 0 {
|
||||
cfg.MaxIdleConns = cfg.MaxConnsPerHost * 2
|
||||
}
|
||||
if cfg.MaxResponseBytes == 0 {
|
||||
cfg.MaxResponseBytes = 1 << 20
|
||||
}
|
||||
return cfg, nil
|
||||
}
|
||||
func (c Config) hasTransportOptions() bool {
|
||||
return c.Proxy.Mode != "" || c.Proxy.URL != "" || c.MaxConnsPerHost != 0 || c.MaxIdleConns != 0 || c.MaxIdleConnsPerHost != 0 || c.DialTimeout != 0 || c.DialKeepAlive != 0 || c.TLSHandshakeTimeout != 0 || c.ResponseHeaderTimeout != 0 || c.ExpectContinueTimeout != 0 || c.IdleConnTimeout != 0
|
||||
}
|
||||
func proxyFunction(c ProxyConfig) (func(*http.Request) (*url.URL, error), error) {
|
||||
if c.Mode != ProxyURL && c.URL != "" {
|
||||
return nil, errors.New("httpc: proxy URL requires mode url")
|
||||
}
|
||||
switch c.Mode {
|
||||
case "":
|
||||
return nil, nil
|
||||
case ProxyDirect:
|
||||
return nil, nil
|
||||
case ProxyEnvironment:
|
||||
return http.ProxyFromEnvironment, nil
|
||||
case ProxyURL:
|
||||
u, err := url.Parse(c.URL)
|
||||
if err != nil || u == nil || u.Hostname() == "" || u.Opaque != "" || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || (u.Path != "" && u.Path != "/") {
|
||||
return nil, errors.New("httpc: proxy URL must be an absolute proxy origin without path, query or fragment")
|
||||
}
|
||||
switch u.Scheme {
|
||||
case "http", "https", "socks5", "socks5h":
|
||||
default:
|
||||
return nil, errors.New("httpc: proxy scheme must be http, https, socks5 or socks5h")
|
||||
}
|
||||
if strings.HasSuffix(u.Host, ":") {
|
||||
return nil, errors.New("httpc: proxy port must be 1..65535")
|
||||
}
|
||||
if port := u.Port(); port != "" {
|
||||
n, e := strconv.Atoi(port)
|
||||
if e != nil || n < 1 || n > 65535 {
|
||||
return nil, errors.New("httpc: proxy port must be 1..65535")
|
||||
}
|
||||
}
|
||||
u.Path = ""
|
||||
u.RawPath = ""
|
||||
return http.ProxyURL(u), nil
|
||||
default:
|
||||
return nil, errors.New("httpc: proxy mode must be environment, direct or url")
|
||||
}
|
||||
}
|
||||
func ownedTransport(cfg Config) (*http.Transport, error) {
|
||||
base, ok := http.DefaultTransport.(*http.Transport)
|
||||
if !ok || base == nil {
|
||||
return nil, errors.New("httpc: provide Transport when DefaultTransport is replaced")
|
||||
}
|
||||
if (base.DialTLSContext != nil || base.DialTLS != nil) && (cfg.DialTimeout > 0 || cfg.DialKeepAlive > 0 || cfg.TLSHandshakeTimeout > 0) {
|
||||
return nil, errors.New("httpc: provide Transport when DefaultTransport has a custom TLS dialer")
|
||||
}
|
||||
t := base.Clone()
|
||||
t.MaxConnsPerHost = cfg.MaxConnsPerHost
|
||||
t.MaxIdleConnsPerHost = cfg.MaxIdleConnsPerHost
|
||||
t.MaxIdleConns = cfg.MaxIdleConns
|
||||
if cfg.Proxy.Mode != "" {
|
||||
proxy, err := proxyFunction(cfg.Proxy)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
t.Proxy = proxy
|
||||
}
|
||||
if cfg.DialTimeout > 0 || cfg.DialKeepAlive > 0 {
|
||||
dialer := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
|
||||
if cfg.DialTimeout > 0 {
|
||||
dialer.Timeout = cfg.DialTimeout
|
||||
}
|
||||
if cfg.DialKeepAlive > 0 {
|
||||
dialer.KeepAlive = cfg.DialKeepAlive
|
||||
}
|
||||
t.DialContext = dialer.DialContext
|
||||
t.Dial = nil
|
||||
}
|
||||
if cfg.TLSHandshakeTimeout > 0 {
|
||||
t.TLSHandshakeTimeout = cfg.TLSHandshakeTimeout
|
||||
}
|
||||
if cfg.ResponseHeaderTimeout > 0 {
|
||||
t.ResponseHeaderTimeout = cfg.ResponseHeaderTimeout
|
||||
}
|
||||
if cfg.ExpectContinueTimeout > 0 {
|
||||
t.ExpectContinueTimeout = cfg.ExpectContinueTimeout
|
||||
}
|
||||
if cfg.IdleConnTimeout > 0 {
|
||||
t.IdleConnTimeout = cfg.IdleConnTimeout
|
||||
}
|
||||
return t, nil
|
||||
}
|
||||
277
httpc/config_test.go
Normal file
277
httpc/config_test.go
Normal file
@@ -0,0 +1,277 @@
|
||||
package httpc
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"net/url"
|
||||
"os"
|
||||
"os/exec"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func testClient(t *testing.T, cfg Config) *Client {
|
||||
t.Helper()
|
||||
cfg.DisableLogging = true
|
||||
c, err := New(cfg)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
t.Cleanup(func() { c.Close() })
|
||||
return c
|
||||
}
|
||||
|
||||
func TestTransportConfigurationAndIsolation(t *testing.T) {
|
||||
base := http.DefaultTransport.(*http.Transport)
|
||||
oldConns, oldIdle, oldTimeout := base.MaxConnsPerHost, base.MaxIdleConns, base.ResponseHeaderTimeout
|
||||
cfg := Config{Proxy: ProxyConfig{Mode: ProxyDirect}, DialTimeout: time.Second, DialKeepAlive: 20 * time.Second, TLSHandshakeTimeout: 2 * time.Second, ResponseHeaderTimeout: 3 * time.Second, ExpectContinueTimeout: 4 * time.Second, IdleConnTimeout: 5 * time.Second, MaxConnsPerHost: 8, MaxIdleConns: 12, MaxIdleConnsPerHost: 3}
|
||||
c := testClient(t, cfg)
|
||||
if c.owned == base || c.owned.Proxy != nil || c.owned.DialContext == nil || c.owned.MaxConnsPerHost != 8 || c.owned.MaxIdleConns != 12 || c.owned.MaxIdleConnsPerHost != 3 || c.owned.TLSHandshakeTimeout != 2*time.Second || c.owned.ResponseHeaderTimeout != 3*time.Second || c.owned.ExpectContinueTimeout != 4*time.Second || c.owned.IdleConnTimeout != 5*time.Second {
|
||||
t.Fatal("transport options not applied")
|
||||
}
|
||||
if base.MaxConnsPerHost != oldConns || base.MaxIdleConns != oldIdle || base.ResponseHeaderTimeout != oldTimeout {
|
||||
t.Fatal("modified shared transport")
|
||||
}
|
||||
if c.owned.TLSClientConfig != nil && c.owned.TLSClientConfig.InsecureSkipVerify {
|
||||
t.Fatal("TLS verification disabled")
|
||||
}
|
||||
d := testClient(t, Config{})
|
||||
if d.owned.MaxConnsPerHost != 64 || d.owned.MaxIdleConnsPerHost != 64 || d.owned.MaxIdleConns != 128 || d.owned.IdleConnTimeout != base.IdleConnTimeout || d.owned.TLSHandshakeTimeout != base.TLSHandshakeTimeout || d.owned.ResponseHeaderTimeout != base.ResponseHeaderTimeout {
|
||||
t.Fatal("default behavior changed")
|
||||
}
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
cancel()
|
||||
if conn, err := c.owned.DialContext(ctx, "tcp", "127.0.0.1:1"); err == nil {
|
||||
conn.Close()
|
||||
t.Fatal("dial ignored context")
|
||||
}
|
||||
}
|
||||
func TestProxyValidation(t *testing.T) {
|
||||
for _, scheme := range []string{"http", "https", "socks5", "socks5h"} {
|
||||
c := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyURL, URL: scheme + "://user:private-value@proxy.invalid:8080"}})
|
||||
req, _ := http.NewRequest("GET", "https://target.invalid", nil)
|
||||
u, err := c.owned.Proxy(req)
|
||||
if err != nil || u.Host != "proxy.invalid:8080" || u.Scheme != scheme {
|
||||
t.Fatal("explicit proxy unavailable")
|
||||
}
|
||||
}
|
||||
bad := []Config{
|
||||
{Proxy: ProxyConfig{Mode: "private-value"}},
|
||||
{Proxy: ProxyConfig{URL: "http://private-value"}},
|
||||
{Proxy: ProxyConfig{Mode: ProxyDirect, URL: "http://private-value"}},
|
||||
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://user:private-value@proxy.invalid:bad"}},
|
||||
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "file:///private-value"}},
|
||||
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid/private-value"}},
|
||||
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid?token=private-value"}},
|
||||
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid:65536"}},
|
||||
{DialTimeout: -1}, {DialKeepAlive: -1}, {TLSHandshakeTimeout: -1}, {ResponseHeaderTimeout: -1}, {ExpectContinueTimeout: -1}, {IdleConnTimeout: -1}, {MaxIdleConns: -1}, {MaxIdleConnsPerHost: 65537},
|
||||
{Transport: benchmarkTransport{}, Proxy: ProxyConfig{Mode: ProxyDirect}},
|
||||
{Transport: benchmarkTransport{}, DialTimeout: time.Second}, {Transport: benchmarkTransport{}, MaxConnsPerHost: 2},
|
||||
}
|
||||
for i, cfg := range bad {
|
||||
if _, err := New(cfg); err == nil || strings.Contains(err.Error(), "private-value") {
|
||||
t.Fatalf("case %d: unsafe or missing error %v", i, err)
|
||||
}
|
||||
}
|
||||
var nilTransport *http.Transport
|
||||
if _, err := New(Config{Transport: nilTransport}); err == nil {
|
||||
t.Fatal("typed nil accepted")
|
||||
}
|
||||
}
|
||||
func TestProxyRouting(t *testing.T) {
|
||||
var proxied, target atomic.Int32
|
||||
proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
proxied.Add(1)
|
||||
if r.URL.Scheme != "http" || r.URL.Host != "target.invalid" {
|
||||
t.Error("not an absolute proxy request")
|
||||
}
|
||||
io.WriteString(w, `{"proxy":true}`)
|
||||
}))
|
||||
defer proxy.Close()
|
||||
origin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { target.Add(1); io.WriteString(w, `{}`) }))
|
||||
defer origin.Close()
|
||||
// Isolate the standard library's process-cached environment proxy lookup.
|
||||
executable, err := os.Executable()
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(ctx, executable, "-test.run=^TestProxyEnvironmentProcess$")
|
||||
for _, value := range os.Environ() {
|
||||
key := strings.SplitN(value, "=", 2)[0]
|
||||
switch strings.ToUpper(key) {
|
||||
case "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "ALL_PROXY", "GO_OMNI_HTTPC_PROXY_CHILD", "GO_OMNI_HTTPC_TARGET":
|
||||
continue
|
||||
}
|
||||
cmd.Env = append(cmd.Env, value)
|
||||
}
|
||||
cmd.Env = append(cmd.Env, "GO_OMNI_HTTPC_PROXY_CHILD=1", "GO_OMNI_HTTPC_TARGET="+origin.URL, "HTTP_PROXY="+proxy.URL, "HTTPS_PROXY="+proxy.URL, "NO_PROXY=bypass.invalid")
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
t.Fatal(err, string(out))
|
||||
}
|
||||
if proxied.Load() != 3 || target.Load() != 1 {
|
||||
t.Fatalf("proxy=%d direct=%d", proxied.Load(), target.Load())
|
||||
}
|
||||
}
|
||||
func TestProxyEnvironmentProcess(t *testing.T) {
|
||||
if os.Getenv("GO_OMNI_HTTPC_PROXY_CHILD") != "1" {
|
||||
t.Skip("subprocess only")
|
||||
}
|
||||
for _, proxy := range []ProxyConfig{{}, {Mode: ProxyEnvironment}, {Mode: ProxyURL, URL: os.Getenv("HTTP_PROXY")}} {
|
||||
c := testClient(t, Config{Proxy: proxy})
|
||||
if err := c.JSON(context.Background(), "GET", "http://target.invalid/path", nil, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
environment := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyEnvironment}})
|
||||
req, _ := http.NewRequest("GET", "http://bypass.invalid", nil)
|
||||
u, err := environment.owned.Proxy(req)
|
||||
if err != nil || u != nil {
|
||||
t.Fatal("NO_PROXY ignored")
|
||||
}
|
||||
direct := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyDirect}})
|
||||
if direct.owned.Proxy != nil {
|
||||
t.Fatal("direct still has proxy")
|
||||
}
|
||||
if err := direct.JSON(context.Background(), "GET", os.Getenv("GO_OMNI_HTTPC_TARGET"), nil, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
func TestResponseHeaderTimeoutRecovery(t *testing.T) {
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.URL.Path == "/slow" {
|
||||
<-r.Context().Done()
|
||||
return
|
||||
}
|
||||
io.WriteString(w, `{}`)
|
||||
}))
|
||||
defer s.Close()
|
||||
c := testClient(t, Config{Timeout: 2 * time.Second, ResponseHeaderTimeout: 30 * time.Millisecond})
|
||||
start := time.Now()
|
||||
err := c.JSON(context.Background(), "GET", s.URL+"/slow", nil, nil)
|
||||
var network net.Error
|
||||
if !errors.As(err, &network) || !network.Timeout() || time.Since(start) > time.Second {
|
||||
t.Fatal("header timeout", err)
|
||||
}
|
||||
if err = c.JSON(context.Background(), "GET", s.URL+"/ok", nil, nil); err != nil {
|
||||
t.Fatal("did not recover", err)
|
||||
}
|
||||
}
|
||||
func TestTLSHandshakeTimeout(t *testing.T) {
|
||||
listener, err := net.Listen("tcp", "127.0.0.1:0")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer listener.Close()
|
||||
done := make(chan struct{})
|
||||
go func() {
|
||||
defer close(done)
|
||||
conn, err := listener.Accept()
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
defer conn.Close()
|
||||
_, _ = io.Copy(io.Discard, conn)
|
||||
}()
|
||||
c := testClient(t, Config{Timeout: 2 * time.Second, Proxy: ProxyConfig{Mode: ProxyDirect}, TLSHandshakeTimeout: 30 * time.Millisecond})
|
||||
start := time.Now()
|
||||
err = c.JSON(context.Background(), "GET", "https://"+listener.Addr().String(), nil, nil)
|
||||
var network net.Error
|
||||
if !errors.As(err, &network) || !network.Timeout() || time.Since(start) > time.Second {
|
||||
t.Fatal("handshake timeout", err)
|
||||
}
|
||||
select {
|
||||
case <-done:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("TLS connection leaked")
|
||||
}
|
||||
}
|
||||
func TestPoolLimitsAndIdleExpiry(t *testing.T) {
|
||||
var connections atomic.Int32
|
||||
entered := make(chan struct{}, 12)
|
||||
release := make(chan struct{})
|
||||
closed := make(chan struct{}, 12)
|
||||
s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
entered <- struct{}{}
|
||||
select {
|
||||
case <-release:
|
||||
case <-r.Context().Done():
|
||||
return
|
||||
}
|
||||
io.WriteString(w, `{}`)
|
||||
}))
|
||||
s.Config.ConnState = func(_ net.Conn, state http.ConnState) {
|
||||
if state == http.StateNew {
|
||||
connections.Add(1)
|
||||
}
|
||||
if state == http.StateClosed {
|
||||
closed <- struct{}{}
|
||||
}
|
||||
}
|
||||
s.Start()
|
||||
defer s.Close()
|
||||
c := testClient(t, Config{Timeout: 2 * time.Second, MaxConnsPerHost: 2, MaxIdleConnsPerHost: 2, MaxIdleConns: 2, IdleConnTimeout: 40 * time.Millisecond})
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 12; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
for i := 0; i < 2; i++ {
|
||||
select {
|
||||
case <-entered:
|
||||
case <-time.After(time.Second):
|
||||
close(release)
|
||||
t.Fatal("requests did not start")
|
||||
}
|
||||
}
|
||||
if connections.Load() != 2 {
|
||||
t.Error("connection cap", connections.Load())
|
||||
}
|
||||
close(release)
|
||||
wg.Wait()
|
||||
if connections.Load() != 2 {
|
||||
t.Fatal("pool was not reused", connections.Load())
|
||||
}
|
||||
for i := 0; i < 2; i++ {
|
||||
select {
|
||||
case <-closed:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("idle pool was not reclaimed")
|
||||
}
|
||||
}
|
||||
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if connections.Load() != 3 {
|
||||
t.Fatal("new connection not established after idle timeout")
|
||||
}
|
||||
c.Close()
|
||||
select {
|
||||
case <-closed:
|
||||
case <-time.After(time.Second):
|
||||
t.Fatal("Close left idle connections open")
|
||||
}
|
||||
}
|
||||
func TestHTTPSVerificationRetained(t *testing.T) {
|
||||
s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, `{}`) }))
|
||||
defer s.Close()
|
||||
c := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyDirect}})
|
||||
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
|
||||
var urlError *url.Error
|
||||
if !errors.As(err, &urlError) {
|
||||
t.Fatal("self-signed server unexpectedly trusted", err)
|
||||
}
|
||||
}
|
||||
119
httpc/json.go
Normal file
119
httpc/json.go
Normal file
@@ -0,0 +1,119 @@
|
||||
package httpc
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"time"
|
||||
)
|
||||
|
||||
// JSON bounds response memory, closes the body, and decodes only 2xx responses.
|
||||
// Optional headers/limits belong to this call and never alter the client.
|
||||
// output=nil discards a bounded stream; it does not buffer or validate JSON.
|
||||
// Empty successful bodies leave output unchanged. StatusError contains no body.
|
||||
func (c *Client) JSON(ctx context.Context, method, url string, input, output any, options ...RequestOption) error {
|
||||
if ctx == nil {
|
||||
return errors.New("httpc: nil context")
|
||||
}
|
||||
if c.closed.Load() {
|
||||
return ErrClosed
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return err
|
||||
}
|
||||
settings := requestOptions{maxResponseBytes: c.config.MaxResponseBytes}
|
||||
if len(options) > 0 {
|
||||
var err error
|
||||
settings, err = jsonOptions(c.config.MaxResponseBytes, options)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
var body io.Reader
|
||||
if input != nil {
|
||||
data, err := json.Marshal(input)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
body = bytes.NewReader(data)
|
||||
}
|
||||
request, err := http.NewRequestWithContext(ctx, method, url, body)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if settings.headers != nil {
|
||||
request.Header = settings.headers
|
||||
}
|
||||
if input != nil {
|
||||
if _, exists := request.Header["Content-Type"]; !exists {
|
||||
request.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
}
|
||||
response, err := c.Do(request)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer response.Body.Close()
|
||||
if response.StatusCode < 200 || response.StatusCode >= 300 {
|
||||
drainStatusBody(response, settings.maxResponseBytes)
|
||||
return &StatusError{StatusCode: response.StatusCode}
|
||||
}
|
||||
if response.ContentLength > settings.maxResponseBytes {
|
||||
return ErrResponseTooLarge
|
||||
}
|
||||
limited := io.LimitReader(response.Body, settings.maxResponseBytes+1)
|
||||
if output == nil {
|
||||
count, err := io.Copy(io.Discard, limited)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if count > settings.maxResponseBytes {
|
||||
return ErrResponseTooLarge
|
||||
}
|
||||
return nil
|
||||
}
|
||||
data, err := io.ReadAll(limited)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if int64(len(data)) > settings.maxResponseBytes {
|
||||
return ErrResponseTooLarge
|
||||
}
|
||||
if len(bytes.TrimSpace(data)) == 0 {
|
||||
return nil
|
||||
}
|
||||
return json.Unmarshal(data, output)
|
||||
}
|
||||
|
||||
// A known small status body may still be slow. Spend at most a short grace
|
||||
// period trying to retain the connection, then cancel the HTTP read. The status
|
||||
// remains authoritative and the deferred Close still owns body cleanup.
|
||||
func drainStatusBody(response *http.Response, limit int64) {
|
||||
if response.ContentLength < 0 || response.ContentLength > min(limit, 32<<10) {
|
||||
return
|
||||
}
|
||||
body, ok := response.Body.(*responseBody)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
timer := time.AfterFunc(25*time.Millisecond, body.cancel)
|
||||
defer timer.Stop()
|
||||
drainSmallBody(response, limit)
|
||||
}
|
||||
|
||||
// Keep the option builder out of the allocation path of existing plain calls.
|
||||
func jsonOptions(limit int64, options []RequestOption) (requestOptions, error) {
|
||||
settings := requestOptions{maxResponseBytes: limit}
|
||||
for _, option := range options {
|
||||
if option == nil {
|
||||
return requestOptions{}, errors.New("httpc: nil request option")
|
||||
}
|
||||
if err := option(&settings); err != nil {
|
||||
return requestOptions{}, err
|
||||
}
|
||||
}
|
||||
return settings, nil
|
||||
}
|
||||
99
httpc/request.go
Normal file
99
httpc/request.go
Normal file
@@ -0,0 +1,99 @@
|
||||
package httpc
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type requestOptions struct {
|
||||
headers http.Header
|
||||
maxResponseBytes int64
|
||||
}
|
||||
|
||||
// RequestOption applies only to one JSON call. Returned options can be reused
|
||||
// concurrently; request headers are independent, with no client-wide credentials.
|
||||
type RequestOption func(*requestOptions) error
|
||||
|
||||
// WithHeader sets a header for this call; later options replace earlier values.
|
||||
// Framing and Host must be controlled through a standard Request passed to Do.
|
||||
func WithHeader(name, value string) RequestOption {
|
||||
name = http.CanonicalHeaderKey(name)
|
||||
err := checkHeader(name, []string{value})
|
||||
return func(o *requestOptions) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.headers == nil {
|
||||
o.headers = make(http.Header)
|
||||
}
|
||||
o.headers[name] = []string{value}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// WithHeaders snapshots headers immediately. Editing the original map after
|
||||
// construction does not change the option; each invocation gets separate slices.
|
||||
func WithHeaders(headers http.Header) RequestOption {
|
||||
snapshot := make(http.Header, len(headers))
|
||||
var err error
|
||||
for key, values := range headers {
|
||||
name := http.CanonicalHeaderKey(key)
|
||||
if err = checkHeader(name, values); err != nil {
|
||||
break
|
||||
}
|
||||
if _, exists := snapshot[name]; exists {
|
||||
err = errors.New("httpc: duplicate canonical header name")
|
||||
break
|
||||
}
|
||||
snapshot[name] = append([]string(nil), values...)
|
||||
}
|
||||
return func(o *requestOptions) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if o.headers == nil {
|
||||
o.headers = make(http.Header, len(snapshot))
|
||||
}
|
||||
for name, values := range snapshot {
|
||||
o.headers[name] = append([]string(nil), values...)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
|
||||
// WithMaxResponseBytes overrides JSON's bounded response read for one call.
|
||||
// The size is positive and at most 1 GiB. Do remains a caller-owned stream.
|
||||
func WithMaxResponseBytes(limit int64) RequestOption {
|
||||
return func(o *requestOptions) error {
|
||||
if limit <= 0 || limit > maxResponseBytes {
|
||||
return errors.New("httpc: response limit must be positive and at most 1 GiB")
|
||||
}
|
||||
o.maxResponseBytes = limit
|
||||
return nil
|
||||
}
|
||||
}
|
||||
func checkHeader(name string, values []string) error {
|
||||
if name == "" {
|
||||
return errors.New("httpc: header name must not be empty")
|
||||
}
|
||||
for _, c := range []byte(name) {
|
||||
if c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || strings.ContainsRune("!#$%&'*+-.^_`|~", rune(c)) {
|
||||
continue
|
||||
}
|
||||
return errors.New("httpc: invalid header name")
|
||||
}
|
||||
switch name {
|
||||
case "Host", "Content-Length", "Transfer-Encoding", "Trailer":
|
||||
return errors.New("httpc: use Do for Host or request framing controls")
|
||||
}
|
||||
for _, value := range values {
|
||||
for i := 0; i < len(value); i++ {
|
||||
c := value[i]
|
||||
if c == '\x7f' || c < ' ' && c != '\t' {
|
||||
return errors.New("httpc: invalid header value")
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
316
httpc/request_test.go
Normal file
316
httpc/request_test.go
Normal file
@@ -0,0 +1,316 @@
|
||||
package httpc
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"compress/gzip"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"git.yuntuo.hk/k2safe/go-omni/logx"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestJSONHeadersConcurrentSnapshot(t *testing.T) {
|
||||
var leaked atomic.Bool
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
if r.Header.Get("Authorization") != "Bearer "+r.URL.Query().Get("caller") {
|
||||
leaked.Store(true)
|
||||
}
|
||||
if r.Header.Get("X-Shared") != "original" || len(r.Header.Values("X-Tag")) != 2 {
|
||||
leaked.Store(true)
|
||||
}
|
||||
if r.Header.Get("Content-Type") != "application/vnd.example+json" {
|
||||
leaked.Store(true)
|
||||
}
|
||||
io.WriteString(w, `{"ok":true}`)
|
||||
}))
|
||||
defer s.Close()
|
||||
c := testClient(t, Config{})
|
||||
headers := http.Header{"X-Shared": []string{"original"}, "X-Tag": []string{"one", "two"}}
|
||||
snapshot := WithHeaders(headers)
|
||||
headers.Set("X-Shared", "changed")
|
||||
headers["X-Tag"][0] = "changed"
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < 24; i++ {
|
||||
wg.Add(1)
|
||||
go func(i int) {
|
||||
defer wg.Done()
|
||||
id := fmt.Sprint(i)
|
||||
var output struct {
|
||||
OK bool `json:"ok"`
|
||||
}
|
||||
if err := c.JSON(context.Background(), "POST", s.URL+"?caller="+id, map[string]bool{"hello": true}, &output, snapshot, WithHeader("Authorization", "Bearer "+id), WithHeader("content-type", "application/vnd.example+json")); err != nil || !output.OK {
|
||||
t.Errorf("JSON failed: %v", err)
|
||||
}
|
||||
}(i)
|
||||
}
|
||||
wg.Wait()
|
||||
if leaked.Load() {
|
||||
t.Fatal("headers leaked across requests or changed with source map")
|
||||
}
|
||||
}
|
||||
func TestRequestOptionsValidationBeforeNetwork(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
c := testClient(t, Config{Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
|
||||
calls.Add(1)
|
||||
return benchmarkTransport{`{}`, 200}.RoundTrip(r)
|
||||
})})
|
||||
invalid := []RequestOption{nil, WithHeader("bad name", "private-token"), WithHeader("Authorization", "private-token\r\nInjected: yes"), WithHeader("Host", "example.invalid"), WithHeader("Content-Length", "1"), WithHeader("Transfer-Encoding", "chunked"), WithHeader("Trailer", "X-Signature"), WithHeaders(http.Header{"X-Name": []string{"one"}, "x-name": []string{"two"}}), WithMaxResponseBytes(0), WithMaxResponseBytes(maxResponseBytes + 1)}
|
||||
for _, option := range invalid {
|
||||
err := c.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, option)
|
||||
if err == nil || strings.Contains(err.Error(), "private-token") {
|
||||
t.Fatal("missing or unsafe error", err)
|
||||
}
|
||||
}
|
||||
if calls.Load() != 0 {
|
||||
t.Fatal("invalid options reached network")
|
||||
}
|
||||
}
|
||||
func TestJSONResponseLimitsAndStreaming(t *testing.T) {
|
||||
var calls atomic.Int32
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
calls.Add(1)
|
||||
switch r.URL.Path {
|
||||
case "/empty":
|
||||
w.WriteHeader(204)
|
||||
case "/chunked":
|
||||
w.(http.Flusher).Flush()
|
||||
io.WriteString(w, strings.Repeat("x", 17))
|
||||
case "/status-stream":
|
||||
w.WriteHeader(401)
|
||||
w.(http.Flusher).Flush()
|
||||
<-r.Context().Done()
|
||||
case "/multiple":
|
||||
io.WriteString(w, `{} {}`)
|
||||
default:
|
||||
io.WriteString(w, `{"ok":true}`)
|
||||
}
|
||||
}))
|
||||
defer server.Close()
|
||||
c := testClient(t, Config{MaxResponseBytes: 8, Timeout: time.Second})
|
||||
var out struct {
|
||||
OK bool `json:"ok"`
|
||||
}
|
||||
if err := c.JSON(context.Background(), "GET", server.URL, nil, &out, WithMaxResponseBytes(32)); err != nil || !out.OK {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := c.JSON(context.Background(), "GET", server.URL, nil, nil); !errors.Is(err, ErrResponseTooLarge) {
|
||||
t.Fatal("limit changed permanently", err)
|
||||
}
|
||||
for _, target := range []any{nil, &out} {
|
||||
if err := c.JSON(context.Background(), "GET", server.URL+"/chunked", nil, target, WithMaxResponseBytes(16)); !errors.Is(err, ErrResponseTooLarge) {
|
||||
t.Fatal("unbounded chunked body", err)
|
||||
}
|
||||
}
|
||||
if err := c.JSON(context.Background(), "GET", server.URL+"/empty", nil, &out); err != nil || !out.OK {
|
||||
t.Fatal("empty success changed output", err)
|
||||
}
|
||||
if err := c.JSON(context.Background(), "GET", server.URL+"/multiple", nil, &out); err == nil {
|
||||
t.Fatal("multiple JSON documents accepted")
|
||||
}
|
||||
start := time.Now()
|
||||
err := c.JSON(context.Background(), "GET", server.URL+"/status-stream", nil, nil)
|
||||
var status *StatusError
|
||||
if !errors.As(err, &status) || status.StatusCode != 401 || time.Since(start) > 500*time.Millisecond {
|
||||
t.Fatal("waited for unknown status body", err)
|
||||
}
|
||||
if calls.Load() != 7 {
|
||||
t.Fatal(calls.Load())
|
||||
}
|
||||
}
|
||||
func TestErrorBodyPoolReuseAndDrainBound(t *testing.T) {
|
||||
var connections atomic.Int32
|
||||
s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.WriteHeader(401)
|
||||
io.WriteString(w, "private-error-body")
|
||||
}))
|
||||
s.Config.ConnState = func(_ net.Conn, state http.ConnState) {
|
||||
if state == http.StateNew {
|
||||
connections.Add(1)
|
||||
}
|
||||
}
|
||||
s.Start()
|
||||
defer s.Close()
|
||||
c := testClient(t, Config{})
|
||||
for i := 0; i < 10; i++ {
|
||||
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
|
||||
var status *StatusError
|
||||
if !errors.As(err, &status) || strings.Contains(err.Error(), "private-error-body") {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
if connections.Load() != 1 {
|
||||
t.Fatal("small status response prevented reuse", connections.Load())
|
||||
}
|
||||
body := &countedBody{Reader: strings.NewReader(strings.Repeat("x", 1<<20))}
|
||||
custom := testClient(t, Config{Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
|
||||
return &http.Response{StatusCode: 401, ContentLength: 1, Body: body, Header: make(http.Header)}, nil
|
||||
})})
|
||||
err := custom.JSON(context.Background(), "GET", "http://example.invalid", nil, nil)
|
||||
var status *StatusError
|
||||
if !errors.As(err, &status) || !body.closed || body.read > (32<<10)+1 {
|
||||
t.Fatal("unbounded/missing drain", body.read, body.closed, err)
|
||||
}
|
||||
}
|
||||
|
||||
type countedBody struct {
|
||||
io.Reader
|
||||
read int
|
||||
closed bool
|
||||
}
|
||||
|
||||
func (b *countedBody) Read(p []byte) (int, error) {
|
||||
n, err := b.Reader.Read(p)
|
||||
b.read += n
|
||||
return n, err
|
||||
}
|
||||
func (b *countedBody) Close() error { b.closed = true; return nil }
|
||||
|
||||
type roundTripperFunc func(*http.Request) (*http.Response, error)
|
||||
|
||||
func (f roundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
|
||||
|
||||
type borrowedTransport struct{ closed atomic.Bool }
|
||||
|
||||
func (t *borrowedTransport) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
return benchmarkTransport{`{}`, 200}.RoundTrip(r)
|
||||
}
|
||||
func (t *borrowedTransport) CloseIdleConnections() { t.closed.Store(true) }
|
||||
func TestBorrowedTransportAndHeaderRetry(t *testing.T) {
|
||||
owned := &borrowedTransport{}
|
||||
c := testClient(t, Config{Transport: owned})
|
||||
c.Close()
|
||||
if owned.closed.Load() {
|
||||
t.Fatal("closed caller's transport")
|
||||
}
|
||||
var attempts atomic.Int32
|
||||
retry := testClient(t, Config{MaxRetries: 1, Backoff: time.Millisecond, Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
|
||||
if r.Header.Get("Authorization") != "Bearer test" {
|
||||
t.Error("retry lost header")
|
||||
}
|
||||
status := 200
|
||||
if attempts.Add(1) == 1 {
|
||||
status = 503
|
||||
}
|
||||
return benchmarkTransport{`{}`, status}.RoundTrip(r)
|
||||
})})
|
||||
if err := retry.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, WithHeader("Authorization", "Bearer test")); err != nil || attempts.Load() != 2 {
|
||||
t.Fatal(err, attempts.Load())
|
||||
}
|
||||
}
|
||||
func FuzzRequestHeaderOptions(f *testing.F) {
|
||||
f.Add("Authorization", "Bearer example")
|
||||
f.Add("X-Name", "one\r\nX-Test: bad")
|
||||
f.Fuzz(func(t *testing.T, name, value string) {
|
||||
if len(name)+len(value) > 8192 {
|
||||
t.Skip()
|
||||
}
|
||||
option := WithHeader(name, value)
|
||||
var settings requestOptions
|
||||
if err := option(&settings); err != nil {
|
||||
return
|
||||
}
|
||||
req, err := http.NewRequest("GET", "http://example.invalid", nil)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
req.Header = settings.headers
|
||||
// The standard Request writer must be able to serialize accepted headers.
|
||||
if err = req.Write(io.Discard); err != nil {
|
||||
t.Fatal("accepted malformed header", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestHTTPLogDoesNotContainCredentials(t *testing.T) {
|
||||
var buffer bytes.Buffer
|
||||
logger, err := logx.New(logx.Config{Writer: &buffer})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer logger.Close()
|
||||
c, err := New(Config{Transport: benchmarkTransport{"private-response", 401}})
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer c.Close()
|
||||
ctx := logx.WithLogger(context.Background(), logger)
|
||||
err = c.JSON(ctx, "GET", "http://private-user:private-password@example.invalid/private-path?token=private-query", nil, nil, WithHeader("Authorization", "Bearer private-token"))
|
||||
if err == nil {
|
||||
t.Fatal("expected status error")
|
||||
}
|
||||
for _, secret := range []string{"private-user", "private-password", "private-path", "private-query", "private-token", "private-response"} {
|
||||
if strings.Contains(buffer.String(), secret) || strings.Contains(err.Error(), secret) {
|
||||
t.Fatal("sensitive field leaked")
|
||||
}
|
||||
}
|
||||
if !strings.Contains(buffer.String(), "http.client") {
|
||||
t.Fatal("missing access log")
|
||||
}
|
||||
}
|
||||
func TestCloseAllowsInflightCallToFinish(t *testing.T) {
|
||||
entered := make(chan struct{})
|
||||
release := make(chan struct{})
|
||||
var once sync.Once
|
||||
releaseNow := func() { once.Do(func() { close(release) }) }
|
||||
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { close(entered); <-release; io.WriteString(w, `{}`) }))
|
||||
defer s.Close()
|
||||
c := testClient(t, Config{Timeout: time.Second})
|
||||
done := make(chan error, 1)
|
||||
go func() { done <- c.JSON(context.Background(), "GET", s.URL, nil, nil) }()
|
||||
defer releaseNow()
|
||||
select {
|
||||
case <-entered:
|
||||
case err := <-done:
|
||||
t.Fatal("request failed before handler", err)
|
||||
case <-time.After(2 * time.Second):
|
||||
t.Fatal("handler not entered")
|
||||
}
|
||||
c.Close()
|
||||
releaseNow()
|
||||
if err := <-done; err != nil {
|
||||
t.Fatal("Close interrupted in-flight call", err)
|
||||
}
|
||||
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); !errors.Is(err, ErrClosed) {
|
||||
t.Fatal("accepted new call after Close", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSlowKnownErrorBodyHasShortDrainBudget(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Length", "100")
|
||||
w.WriteHeader(503)
|
||||
w.(http.Flusher).Flush()
|
||||
<-r.Context().Done()
|
||||
}))
|
||||
defer server.Close()
|
||||
c := testClient(t, Config{Timeout: 2 * time.Second})
|
||||
start := time.Now()
|
||||
err := c.JSON(context.Background(), "GET", server.URL, nil, nil)
|
||||
var status *StatusError
|
||||
if !errors.As(err, &status) || status.StatusCode != 503 || time.Since(start) > 500*time.Millisecond {
|
||||
t.Fatal("slow error body delayed status", err, time.Since(start))
|
||||
}
|
||||
}
|
||||
|
||||
func TestJSONLimitAppliesAfterGzipExpansion(t *testing.T) {
|
||||
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("Content-Encoding", "gzip")
|
||||
writer := gzip.NewWriter(w)
|
||||
_, _ = io.WriteString(writer, strings.Repeat("x", 4096))
|
||||
_ = writer.Close()
|
||||
}))
|
||||
defer server.Close()
|
||||
c := testClient(t, Config{MaxResponseBytes: 64})
|
||||
if err := c.JSON(context.Background(), "GET", server.URL, nil, nil); !errors.Is(err, ErrResponseTooLarge) {
|
||||
t.Fatal("gzip bypassed response limit", err)
|
||||
}
|
||||
}
|
||||
61
reports/configurationAcceptance.md
Normal file
61
reports/configurationAcceptance.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# v0.20 统一配置加载验收(2026-09-26)
|
||||
|
||||
本轮修改 go-omni 配置包、生成模板与文档。没有修改接入项目的业务配置/数据库,也没有使用线上服务器做压力测试。
|
||||
|
||||
## 回归结果
|
||||
|
||||
- 全量 `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race ./...`:47 个有测试的包通过,包含实际生成项目、生命周期失败清理、错误目录/响应继承、上传和拷贝工具回归。
|
||||
- `go vet ./...`、`go build ./...` 通过。
|
||||
- `GOTOOLCHAIN=go1.22.12 go test ./config ./examples/configuration` 通过。
|
||||
- 新生成消费者的 `go test -race ./...`、`go build .`、generate/check/dry-run 通过。
|
||||
- checked-in managedService 已显式迁移旧 Load,生成检查、race 和构建通过;三种格式示例与环境覆盖命令实际运行通过。
|
||||
|
||||
针对性验证:
|
||||
|
||||
| 场景 | 结果 |
|
||||
| --- | --- |
|
||||
| JSON / YAML / XML 的 defaults→file→env→Validate | 同值,0/false 可覆盖,envDefault 不覆盖文件 |
|
||||
| 大于 2^53 的整数、动态 json.Number | 精度保留 |
|
||||
| null、空数组、嵌套指针、map、duration、TextUnmarshaler、JSON tag / 嵌入字段 | 语义回归通过 |
|
||||
| 文件字段类型错、未知字段、重复字段、业务 Validate 失败 | 文件/字段/可用行列与安全原因 |
|
||||
| 环境类型或校验失败、集合子字段校验 | 定位环境来源,不错误归因于文件 |
|
||||
| 内部错误链和默认值隔离 | errors.Is/As 可用,失败不提交半成品/修改默认值 |
|
||||
| 24 个并发加载、自定义 Decoder、禁止事后注册 | race 通过,实例间不泄漏 |
|
||||
| 超大文件、超深输入、重复 XML 根、DTD/entity、YAML alias/merge、UTF-8 | 拒绝;解析器 panic 不公开 panic 内容 |
|
||||
| 业务新增字段与自定义格式 | 不改 main/loadGen,生成保留手写文件 |
|
||||
| 配置校验失败 | 依赖初始化之前返回;未触发测试中的 Init panic |
|
||||
| 旧手写 Load、loadGen 漂移 | generate/check/dry-run 拒绝冲突,不改业务文件 |
|
||||
|
||||
30 秒、4 worker 的 `FuzzConfigurationDecoders`:383586 次执行,无失败(总运行 31.309 秒,包含停止收尾)。模糊输入上限 8 KiB;另有确定性 1 MiB/深度上限和恶意格式回归,不把 fuzz 次数当作形式证明。
|
||||
|
||||
## 性能
|
||||
|
||||
Go 1.27.1、darwin/arm64、Apple M2 Max;无 race 的本地基准,每个场景三轮。原始数据:[configurationBenchmark.txt](configurationBenchmark.txt)。
|
||||
|
||||
| 场景 | 中位数耗时 | 典型分配 |
|
||||
| --- | ---: | ---: |
|
||||
| 小配置 JSON 解析+绑定 | 5.068 µs | 3781 B / 75 allocs |
|
||||
| 小配置 YAML 解析+绑定 | 14.855 µs | 15602 B / 181 allocs |
|
||||
| 小配置 XML 解析+绑定 | 7.137 µs | 5605 B / 122 allocs |
|
||||
| 含完整 Server 默认值:文件读取+环境覆盖+校验 | 107.660 µs | 约 67.4 KB / 1288 allocs |
|
||||
| 1000 个条目,约 28 KB JSON 解析+绑定 | 2.548 ms | 约 2.40 MB / 31585 allocs |
|
||||
|
||||
配置在启动阶段加载,不经过业务请求路径。这些是暖文件缓存、本机测试结果,不能等同冷磁盘启动耗时、请求 QPS 或跨机器性能承诺。没有为此引入全局反射缓存、解析器池或后台线程。
|
||||
|
||||
## 边界与迁移
|
||||
|
||||
- 旧手写 Load 必须先明确迁移,生成器不会猜测其解密/其他特殊行为。
|
||||
- 内置 XML 是明确的配置子集;YAML alias/merge/custom tag 拒绝;固定 Go 数组严格长度。
|
||||
- 自定义 Decoder/反序列化/Validate 是可信 Go 回调,其并发、副作用与公开 Reason 的脱敏由项目负责。
|
||||
- 第三方非结构化验证错误仅定位子树,并保留内部原因;业务用 Invalid/AtField 提供准确的公开字段与原因。
|
||||
- 未进行业务数据库/第三方真实依赖验收,未修改或替代各接入项目的业务回归。
|
||||
|
||||
## 远程发布验收
|
||||
|
||||
`v0.20.0-preview` 标签固定在 `c60841f901a3bcc3e9582eece6cfb2cad008fb9e`。空 module 缓存安装已发布 CLI;生成消费者引用相同远程 module 和 checksum,**没有本地 replace**。
|
||||
|
||||
根目录 `go run .` / 构建、nethttp/native 退出信号与端口释放、上传下载/原始回调、两服务共享错误目录,以及 8 组消费者 race 回归全部通过。新消费者验证 JSON/YAML/XML/自定义扩展、环境来源、初始化前验证、业务文件保留与 loadGen 漂移检查。
|
||||
|
||||
完整机器记录:[configurationRemote.json](configurationRemote.json),本轮耗时 50.24 秒。标签不因补充验收记录移动;记录随后提交到 main。
|
||||
|
||||
完整使用与迁移:[CONFIG.md](../CONFIG.md)。
|
||||
21
reports/configurationBenchmark.txt
Normal file
21
reports/configurationBenchmark.txt
Normal file
@@ -0,0 +1,21 @@
|
||||
goos: darwin
|
||||
goarch: arm64
|
||||
pkg: git.yuntuo.hk/k2safe/go-omni/config
|
||||
cpu: Apple M2 Max
|
||||
BenchmarkConfiguration/json-12 239197 5068 ns/op 3781 B/op 75 allocs/op
|
||||
BenchmarkConfiguration/json-12 234421 5058 ns/op 3781 B/op 75 allocs/op
|
||||
BenchmarkConfiguration/json-12 233330 5564 ns/op 3781 B/op 75 allocs/op
|
||||
BenchmarkConfiguration/yaml-12 83022 14827 ns/op 15602 B/op 181 allocs/op
|
||||
BenchmarkConfiguration/yaml-12 83197 14855 ns/op 15604 B/op 181 allocs/op
|
||||
BenchmarkConfiguration/yaml-12 74828 16070 ns/op 15602 B/op 181 allocs/op
|
||||
BenchmarkConfiguration/xml-12 167509 7144 ns/op 5606 B/op 122 allocs/op
|
||||
BenchmarkConfiguration/xml-12 171202 7086 ns/op 5605 B/op 122 allocs/op
|
||||
BenchmarkConfiguration/xml-12 167727 7137 ns/op 5605 B/op 122 allocs/op
|
||||
BenchmarkConfigurationStartup-12 10000 107660 ns/op 67429 B/op 1288 allocs/op
|
||||
BenchmarkConfigurationStartup-12 10000 107420 ns/op 67427 B/op 1288 allocs/op
|
||||
BenchmarkConfigurationStartup-12 10000 107828 ns/op 67430 B/op 1288 allocs/op
|
||||
BenchmarkConfigurationLarge-12 481 2548066 ns/op 10.99 MB/s 2403822 B/op 31585 allocs/op
|
||||
BenchmarkConfigurationLarge-12 471 2532248 ns/op 11.06 MB/s 2403851 B/op 31586 allocs/op
|
||||
BenchmarkConfigurationLarge-12 477 2641417 ns/op 10.61 MB/s 2403402 B/op 31583 allocs/op
|
||||
PASS
|
||||
ok git.yuntuo.hk/k2safe/go-omni/config 20.682s
|
||||
146
reports/configurationRemote.json
Normal file
146
reports/configurationRemote.json
Normal file
@@ -0,0 +1,146 @@
|
||||
{
|
||||
"passed": true,
|
||||
"frameworkVersion": "v0.20.0-preview",
|
||||
"frameworkSum": "h1:UoymLOIS36Uf7GJ2FhUMP5f1RLpGQXpkLjpqQUZzpv4=",
|
||||
"freshModuleCache": true,
|
||||
"noLocalReplace": true,
|
||||
"durationSeconds": 50.24,
|
||||
"checks": [
|
||||
"remoteCLIInstall",
|
||||
"freshModuleCache",
|
||||
"fixedVersionNoReplace",
|
||||
"contractChangePreview",
|
||||
"persistentLogicSignatureFailure",
|
||||
"businessSourcePreserved",
|
||||
"fixedConsumerBuild",
|
||||
"rootGoRun",
|
||||
"nethttpAndNativeSignals",
|
||||
"listenerReleased",
|
||||
"default404And405",
|
||||
"goOwnedErrorCatalog",
|
||||
"unwrappedDownloads",
|
||||
"rawCallback",
|
||||
"multipart400And413",
|
||||
"nethttpAndNativeTransfers",
|
||||
"sharedGoCatalogExport",
|
||||
"documentOnlyErrorSelection",
|
||||
"workspaceCheckNoWrites",
|
||||
"staleCatalogRejected",
|
||||
"twoIndependentServiceBuilds",
|
||||
"initializationAndListenFailureCleanup",
|
||||
"drainBeforeDependencyClose",
|
||||
"generatedConsumerRaceTests",
|
||||
"errorParamsAndInheritance",
|
||||
"manualHooksAndScalars",
|
||||
"dependencyFatalShutdown",
|
||||
"fatalProcessExitCode",
|
||||
"initializationShutdownNotification",
|
||||
"configuredBuiltinEndpoints",
|
||||
"projectHealthEnvelope",
|
||||
"endpointConflictCleanup",
|
||||
"responseArrayNullableNull",
|
||||
"primitiveAndDynamicArrays",
|
||||
"strictRequestPolicies",
|
||||
"declarativeMultipartBinding",
|
||||
"backgroundStopDrainResourceOrder",
|
||||
"backgroundInitAndListenFailureCleanup",
|
||||
"backgroundTimeoutRetainsResourcesAndExits",
|
||||
"flatFieldComposition",
|
||||
"copyxThreeEndpointConsumer",
|
||||
"explicitNullAndMoneyPolicies",
|
||||
"fieldAdditionWithoutCopyRuleChanges",
|
||||
"middlewareFactoryInjectionAndCleanup",
|
||||
"optionalRepositoryPreservesBusiness",
|
||||
"generatedUnifiedConfig",
|
||||
"jsonYamlXmlAndCustomDecoder",
|
||||
"configValidationBeforeDependencies",
|
||||
"configEnvProvenance",
|
||||
"configExtensionPreserved",
|
||||
"configLoaderDriftCheck"
|
||||
],
|
||||
"sourceHashes": {
|
||||
"lifecycle/runtime.go": "7d890150b931cdc6c5d03f3706ea09b1d220a958d91fbd6ddc6715a9ec13a5d2",
|
||||
"lifecycle/phases.go": "1436856c708d004ebb28704a867316bf7a44553de56b3e29152d41475a2f793a",
|
||||
"config/lifecycle_test.go": "8e5fccf0f7432c64d7fa8a52687002307d3ebb1fd039adb6ae09da0c9a2891bf",
|
||||
"task/task.go": "acb1a2cc0e26029331065545582c30e0c547e3b6869f7d8b16c7d25ab0077545",
|
||||
"cmd/go-omni/scaffold/templates/api-runtime.go.tmpl": "e8575c04b0923d924d96e4334731313076284fd068c4990715b7a07a2724f969",
|
||||
"cmd/go-omni/apigen/templates.go": "60bdb5d9443ff9f772c0db1ccd637e60ec9906413d3fb4c52ea162e5c4341a22",
|
||||
"config/project.go": "e3478bc08125895c0c65a61b818a7fb893b276af955865c9a088d878d1b11e51",
|
||||
"contract/bind.go": "7590c3100a9bf90236aba7fda0a021d065d9dacd8c432c8eed1066f2ed8d37c8",
|
||||
"contract/policy.go": "73a3a55fb88195b32f22b2bc066d9b13dccd7fd520e98c5a1b2979a3fd2447eb",
|
||||
"contract/multipart.go": "db0377599b1b8b39246d41129513e8187109e44c1892971244d581c22ccc8ecd",
|
||||
"contract/file.go": "20dd2de17e4854d452afc3c65f0293d504fb8fc25e91583ef45207fdab30948d",
|
||||
"cmd/go-omni/apigen/business_test.go": "3b484352282d945f5a58047682079e3d64d4b0bf6ec6bafc23151b1197bb3942",
|
||||
"cmd/go-omni/apigen/generate.go": "b0234696420ab9f9fb8b54ba1df682ced7d155bae03c65f37fe4498a6b574685",
|
||||
"lifecycle/controller.go": "1fb9393df1908afed93d87a1bf3ecea8d79d78e437a656b56bb7f1e5f6e51d31",
|
||||
"config/endpoints.go": "9f5123f962d52b71e5b73d5942316170875e895129a94e8b669ce782969ef27d",
|
||||
"health/health.go": "41f07f27a7fdcfa7973c2c554a22b73129ebc5d44e945e70e054e1cff24ab866",
|
||||
"httpx/response.go": "2cf78b8f0eee8049ee612bb409a6eecc1d39eab9ab8f3b533ea21e3cac1611cb",
|
||||
"app/app.go": "e5255ffdd36cba1c155037afc42f1c921b1f91bc79db75798a7feb0e9d51b1c0",
|
||||
"app/transport.go": "88212d67d375e4c48c250a2e4c62fbff4e7873a67f91f2deb113a60634149f73",
|
||||
"scripts/verifyManagedRelease.py": "f9775c8497d258723955e3e28966de16dc641564e9ac4edc21e1292c85f06deb",
|
||||
"scripts/verifyManagedProject.py": "b0e4b1aea4f44d61470c35c41f971e0882ae36b0eb5df2be190400d62fc60a18",
|
||||
"cmd/go-omni/apigen/managed_test.go": "e0b2b5aa6c4d6383590b467d57d7b4e00505ca35d6b6cde2475e4ad574041e94",
|
||||
"cmd/go-omni/apigen/testdata/managed/options.go": "30d48e4b5aab476c887ac53cc89b48b2b7e250c8cd25e55fecb49bb2a3589247",
|
||||
"cmd/go-omni/apigen/testdata/managed/background.go": "fb357b5deefd5187f2df406be5189fc4d57da857bd55b474f4726bf065c77167",
|
||||
"cmd/go-omni/apigen/testdata/managed/client.go": "cdd7d394ddad4bcdc10224890a884aa513ed2ffd8275ecdad8de9963dd10986a",
|
||||
"cmd/go-omni/apigen/testdata/managed/phases_test.go": "4d826d7133027ac2399f8209f03efa46ed54aa45dcd0a35274a9009314941dca",
|
||||
"cmd/go-omni/apigen/testdata/managed/shutdown_test.go": "79fbf8dd4508b8217af888a31aaf2f2a01c1fa3869b2e80420605af697c66319",
|
||||
"cmd/go-omni/apigen/testdata/managed/consumer_test.go": "9e103c9de7a1a906903bc09787e1b38905dc5bb4d891c88938028da5dee98f6f",
|
||||
"cmd/go-omni/apigen/testdata/managed/operations.go": "44cea63e6023458c5818c9fdae223631cd1721f0841e671457398bfde8bc48f1",
|
||||
"cmd/go-omni/apigen/testdata/business/accessFactory.go": "2f982cf39b5e1a4be98596ce5d0a296af0fbf8c94616543a268e67ec376f31b4",
|
||||
"cmd/go-omni/apigen/testdata/business/client.go": "d4cbaeec18dfb3a0e2821217e8eb0515000e351a745a93ec811dabca44e58c8a",
|
||||
"cmd/go-omni/apigen/testdata/business/consumer_test.go": "e4159862a729ca2c5c9699840cc16a56350fc2914c43841f08946e9a6e9c8091",
|
||||
"cmd/go-omni/apigen/testdata/business/operations.go": "11ded7921a0753f22975c525e334e3d45366143625e41d24fe500504e785c64c",
|
||||
"cmd/go-omni/apigen/responseShape.go": "5b705130b66a3d4aaed842025b75abe54e6427acac8366366ca88ae9862429e1",
|
||||
"cmd/go-omni/apigen/documentRefs.go": "7b841a668154d872493104eda0f9049340c67d9323eb99f1621a1ef6ba343891",
|
||||
"cmd/go-omni/apigen/catalogDocument.go": "94df60328b2f105c35b73d9c6362ea14abadce6cb18e4d9921193d2d46854252",
|
||||
"cmd/go-omni/apigen/workspaceCatalog.go": "c00784220b379ed80deb07c060532b756cbc47434ecb8b1ca10bc12b12159328",
|
||||
"config/env.go": "ea84bccdcd39e3f6b4958a0c8ac8e7d6a0be516b24930f93039248165361bf35",
|
||||
"cmd/go-omni/apigen/arrayResponse_test.go": "9e3e4475f76683d3346d349cdfa94e9e75cf913ac6979bd7a4ed86628466d879",
|
||||
"cmd/go-omni/apigen/testdata/arrays/values.go": "d5e7f3adf12aabb301f46c296e8c894a7b2f6f70a6b6e7e02c74da7f1fd17bc5",
|
||||
"cmd/go-omni/apigen/testdata/arrays/consumer_test.go": "435932f2952ad897943e2cda83b04eccb2950579d8d88772661541e1e2a2333d",
|
||||
"cmd/go-omni/apigen/composition.go": "7ed3761b27d9c932d8ddc43e63c2aca798bdbe38f2b40571af8916c8401ea650",
|
||||
"cmd/go-omni/apigen/generatorConfig.go": "78db0c7808cf6ec997c5b64a4a5fbea11b74049709bc5900ccfbc5d8dabebbc8",
|
||||
"cmd/go-omni/apigen/copyConsumer_test.go": "e08ae4daed5dd25ab89d8872f653caf19499984dcaddc9835c13884c410fc0c3",
|
||||
"cmd/go-omni/apigen/generatorConfig_test.go": "a6d8c8ebea5f1accf0e37dd044e57b48e422821fe75086d8d2eb3abe3acb147c",
|
||||
"cmd/go-omni/apigen/middlewareFactory.go": "cc0220edb3eff25d7f66b2c37e206455f8730a2b909613713322bee7caf72303",
|
||||
"cmd/go-omni/apigen/testdata/composition/accessFactory.go": "e1e5057ff79f806b3218e93249e8191d097b5c4844977204a8c32848443cba33",
|
||||
"cmd/go-omni/apigen/testdata/composition/domain.go": "ab27ebd7ad5e3ed5505a2aa268b9b5ceedbb01477b59011bec63bb270114078b",
|
||||
"cmd/go-omni/apigen/testdata/composition/amount.go": "f2d1834502d71d9ecf72ebe3b1077e5ef41f44fd816abd4617ac19f9bb962fdd",
|
||||
"cmd/go-omni/apigen/testdata/composition/copy.go": "32d37d6f1f0e9aa2409aeb90b052801d1cdca70e504c2dd294cdae01422c9cb7",
|
||||
"cmd/go-omni/apigen/testdata/composition/client.go": "19102e191c0471cae9e7405623906d0d804865c40b80ff1d5b240a41ceab6a4e",
|
||||
"cmd/go-omni/apigen/testdata/composition/service.api": "01d230d5409756e2f3eb602a7fe065efa7d9599f7a4089da4bf8f07a2474444e",
|
||||
"cmd/go-omni/apigen/testdata/composition/consumer_test.go": "74aaf60318cd0de67495fc9142b3f67c3b6e232bb48b46b3e134de5aadd225ad",
|
||||
"cmd/go-omni/apigen/testdata/composition/dependencies.go": "edc153204b06da00c0e45c52db43a4a22f8ffe22c4a6d3f3ebec0137b861f84d",
|
||||
"cmd/go-omni/apigen/testdata/composition/converters.go": "ecdcfe31c75ca815dab8d1a82533b12cbbd40e12b3a5c444e815097f6c4df08e",
|
||||
"copyx/options.go": "edea2f3d2043bf813e56dedf0aea319aaa52301d3b589f41fef986e33a77dc32",
|
||||
"copyx/copy.go": "a752a81acc0e847158043daaf0fdaaecec43abaf27d5561725a47b3e120ffbdc",
|
||||
"copyx/graph.go": "ae6a2a11a8175033d327b580cd570c97fe9d2ddb814bb50884cc19a90766badd",
|
||||
"copyx/benchmark_test.go": "b1bf408957cadea47b47215ffb099d140d5d1a2eb061024a6d0c1d769f469a79",
|
||||
"copyx/copy_test.go": "89b65a6281591d7cd5823cc1c7d2c384c6bd446ef06aa1763ddc3acaa84dfb4a",
|
||||
"config/decoders.go": "6d31f67c82563c2d63e37ad1ce9b9fbad6232b2b785b9b58dd025be857f0fc28",
|
||||
"config/config.go": "e51ece1fa824d8b0fc7385e027b4a4408e459ec2c89444fab99e1e6a19e8c1cf",
|
||||
"config/errorCatalog_test.go": "8dcaa73bfc0c98f30b39103f30048e22fc3190d66f5ad70f5670decedf837479",
|
||||
"config/http_test.go": "0cae3dcf213369cfaff5487bf630c2bc3ec097739ca1d40794124e69c3880db8",
|
||||
"config/response_test.go": "b4ccb111f813f80dc180a7ddb11034a409b9409cbb1b57e654c04d678465d201",
|
||||
"config/projectAdvanced_test.go": "8734a4c24a48f7f28c417e4a5e50783096d4782050b658ecc8199c257f93d1e2",
|
||||
"config/file_test.go": "1479d625aad26a1f773bc4a08c146347f163bbf7b4f11385a32f687119df5817",
|
||||
"config/aliases_test.go": "38de1dc2633c5513fc3870cb87ff42217e968487f934e8b7fec1488ba3555eb3",
|
||||
"config/errorParams_test.go": "a5a07c2f13524dfc136174d502e7cff37b45db97dc8b26a5dc8b89bafcb6b8af",
|
||||
"config/http.go": "6fbeac9c21f5dedad05471deb03a6b4465a0cb10f252d651da2f6f6cb3841998",
|
||||
"config/loader.go": "c1cf9ede71ad0dd9a16d0c3e4e0eebb8b0e0e2045b5a66dcab042692a9ca3520",
|
||||
"config/file.go": "cc24c1f647d925fd91656a12f5817660ed97b4e46ba26f36f4934c4af7c988e9",
|
||||
"config/diagnostic.go": "1e6324cdc229872c16e77a5d5eb1ce2a1d81b730a8573e518d06c02ed08127d8",
|
||||
"config/bind.go": "b757082fdc58e7eee3b8229cac6093eb31e3a9cd226806984c476ff6bfbbc724",
|
||||
"config/clone.go": "6146bf8a7d7549d975142a843db5c6f95c67e98b62cbd98e20c77842abd38040",
|
||||
"config/endpoints_test.go": "3c0117939955f41cdbffc70da7c2d92e6cb184b14124c357d4f1024cb00389d5",
|
||||
"config/config_test.go": "f85630d171d54b7a1e9a3eed6b622cabef244409027e7d6c1dce3a3b9eb9aac4",
|
||||
"config/project_test.go": "040757b9512cf0490f76566510d4038349b0312eac0623ff473307b2c610731d",
|
||||
"config/loader_test.go": "b09aeebfdfbcae0d4c207715f341fc473c9b3d199532d4e8358658460fc815cf",
|
||||
"cmd/go-omni/apigen/configLoader_test.go": "4910930bd26b84632446d7993bf98d3a88a34b392440bd6f0a78639d0fed5b47",
|
||||
"cmd/go-omni/apigen/testdata/configuration/consumer_test.go": "3f0d2230d00684d2da46f10ece33d9acdbac41d1100bc8d310cf6588ad4c5a8f",
|
||||
"cmd/go-omni/scaffold/templates/api-load.go.tmpl": "86d92dbac769d0f2748ea9c957457800939bf0038643917222e1a3455f2dce6f",
|
||||
"cmd/go-omni/scaffold/templates/api-loader.go.tmpl": "847ac407827808022dcf493823b58671e1ae50349b173a5e4500580e07c64150"
|
||||
}
|
||||
}
|
||||
@@ -68,3 +68,15 @@ go run ./cmd/go-omni api generate --dir examples/managedService --check
|
||||
- converter 是受信项目代码,必须只读输入;不能自动回滚其副作用、证明其未共享外部全局对象,或强杀无限循环。
|
||||
- 当前记录是本机库/生成器验收,不是线上服务吞吐、Linux soak、真实业务数据库回归或 OSS/R2 云账号验收。本轮没有对线上服务器施压。
|
||||
- 旧消费者仍需按 [COPY.md](../COPY.md#从旧-mapping-迁移) 迁移,不会被这次框架发布自动修改。
|
||||
|
||||
## 发布后远程验收
|
||||
|
||||
标签 `v0.19.0-preview` 固定在 `fd83235ce7fc15ce88a482d385d70e61f25d7b32`。空 module 缓存安装远程 CLI,消费者固定同版依赖、具有校验和、无本地 replace,7 组实际生成消费者 race 回归通过;包含三端 copyx 示例。
|
||||
|
||||
根目录 `go run .` / 构建、nethttp/native 的 SIGINT/SIGTERM、404/405、Go 错误目录、原始回调、上传/下载、workspace 共用错误目录及资源收尾均通过。远程单独执行 `go run git.yuntuo.hk/k2safe/go-omni/examples/copying@v0.19.0-preview` 也验证深拷贝隔离与共享 converter 拒绝。
|
||||
|
||||
```sh
|
||||
python3 scripts/verifyManagedRelease.py --version v0.19.0-preview --report reports/copyxRemote.json
|
||||
```
|
||||
|
||||
发布后验收工具同步移除了旧 mapping 用例,改为 copyx 消费者并核对实际远程依赖校验和。详细检查项、源摘要见 [copyxRemote.json](copyxRemote.json)。标签不随验收记录的后续文档提交移动。
|
||||
|
||||
117
reports/copyxRemote.json
Normal file
117
reports/copyxRemote.json
Normal file
@@ -0,0 +1,117 @@
|
||||
{
|
||||
"passed": true,
|
||||
"frameworkVersion": "v0.19.0-preview",
|
||||
"frameworkSum": "h1:pMMAyBOgKpGvU8WDxKIwvMU4jWf72cx0aKHVO+snFE4=",
|
||||
"freshModuleCache": true,
|
||||
"noLocalReplace": true,
|
||||
"durationSeconds": 62.46,
|
||||
"checks": [
|
||||
"remoteCLIInstall",
|
||||
"freshModuleCache",
|
||||
"fixedVersionNoReplace",
|
||||
"contractChangePreview",
|
||||
"persistentLogicSignatureFailure",
|
||||
"businessSourcePreserved",
|
||||
"fixedConsumerBuild",
|
||||
"rootGoRun",
|
||||
"nethttpAndNativeSignals",
|
||||
"listenerReleased",
|
||||
"default404And405",
|
||||
"goOwnedErrorCatalog",
|
||||
"unwrappedDownloads",
|
||||
"rawCallback",
|
||||
"multipart400And413",
|
||||
"nethttpAndNativeTransfers",
|
||||
"sharedGoCatalogExport",
|
||||
"documentOnlyErrorSelection",
|
||||
"workspaceCheckNoWrites",
|
||||
"staleCatalogRejected",
|
||||
"twoIndependentServiceBuilds",
|
||||
"initializationAndListenFailureCleanup",
|
||||
"drainBeforeDependencyClose",
|
||||
"generatedConsumerRaceTests",
|
||||
"errorParamsAndInheritance",
|
||||
"manualHooksAndScalars",
|
||||
"dependencyFatalShutdown",
|
||||
"fatalProcessExitCode",
|
||||
"initializationShutdownNotification",
|
||||
"configuredBuiltinEndpoints",
|
||||
"projectHealthEnvelope",
|
||||
"endpointConflictCleanup",
|
||||
"responseArrayNullableNull",
|
||||
"primitiveAndDynamicArrays",
|
||||
"strictRequestPolicies",
|
||||
"declarativeMultipartBinding",
|
||||
"backgroundStopDrainResourceOrder",
|
||||
"backgroundInitAndListenFailureCleanup",
|
||||
"backgroundTimeoutRetainsResourcesAndExits",
|
||||
"flatFieldComposition",
|
||||
"copyxThreeEndpointConsumer",
|
||||
"explicitNullAndMoneyPolicies",
|
||||
"fieldAdditionWithoutCopyRuleChanges",
|
||||
"middlewareFactoryInjectionAndCleanup",
|
||||
"optionalRepositoryPreservesBusiness"
|
||||
],
|
||||
"sourceHashes": {
|
||||
"lifecycle/runtime.go": "7d890150b931cdc6c5d03f3706ea09b1d220a958d91fbd6ddc6715a9ec13a5d2",
|
||||
"lifecycle/phases.go": "1436856c708d004ebb28704a867316bf7a44553de56b3e29152d41475a2f793a",
|
||||
"config/lifecycle_test.go": "8e5fccf0f7432c64d7fa8a52687002307d3ebb1fd039adb6ae09da0c9a2891bf",
|
||||
"task/task.go": "acb1a2cc0e26029331065545582c30e0c547e3b6869f7d8b16c7d25ab0077545",
|
||||
"cmd/go-omni/scaffold/templates/api-runtime.go.tmpl": "e8575c04b0923d924d96e4334731313076284fd068c4990715b7a07a2724f969",
|
||||
"cmd/go-omni/apigen/templates.go": "980856557a247784a3d8e63212bf7fcc3965bcbf7689119f8ee999ccd7867f26",
|
||||
"config/project.go": "0917888f4e471761b897ddd59426eb098bf586edb5c5dc856320b84b3bdb5f7f",
|
||||
"contract/bind.go": "7590c3100a9bf90236aba7fda0a021d065d9dacd8c432c8eed1066f2ed8d37c8",
|
||||
"contract/policy.go": "73a3a55fb88195b32f22b2bc066d9b13dccd7fd520e98c5a1b2979a3fd2447eb",
|
||||
"contract/multipart.go": "db0377599b1b8b39246d41129513e8187109e44c1892971244d581c22ccc8ecd",
|
||||
"contract/file.go": "20dd2de17e4854d452afc3c65f0293d504fb8fc25e91583ef45207fdab30948d",
|
||||
"cmd/go-omni/apigen/business_test.go": "3b484352282d945f5a58047682079e3d64d4b0bf6ec6bafc23151b1197bb3942",
|
||||
"cmd/go-omni/apigen/generate.go": "c4bc6652784714f7cb0fd4815094a1668f230b244827fc33cb8f4aa90bbc76fc",
|
||||
"lifecycle/controller.go": "1fb9393df1908afed93d87a1bf3ecea8d79d78e437a656b56bb7f1e5f6e51d31",
|
||||
"config/endpoints.go": "9f5123f962d52b71e5b73d5942316170875e895129a94e8b669ce782969ef27d",
|
||||
"health/health.go": "41f07f27a7fdcfa7973c2c554a22b73129ebc5d44e945e70e054e1cff24ab866",
|
||||
"httpx/response.go": "2cf78b8f0eee8049ee612bb409a6eecc1d39eab9ab8f3b533ea21e3cac1611cb",
|
||||
"app/app.go": "e5255ffdd36cba1c155037afc42f1c921b1f91bc79db75798a7feb0e9d51b1c0",
|
||||
"app/transport.go": "88212d67d375e4c48c250a2e4c62fbff4e7873a67f91f2deb113a60634149f73",
|
||||
"scripts/verifyManagedRelease.py": "fdc33d4357d501a9bf37d3612fe4fc51aa2a3a5686a475328ad62f1663eab022",
|
||||
"scripts/verifyManagedProject.py": "b0e4b1aea4f44d61470c35c41f971e0882ae36b0eb5df2be190400d62fc60a18",
|
||||
"cmd/go-omni/apigen/managed_test.go": "e0b2b5aa6c4d6383590b467d57d7b4e00505ca35d6b6cde2475e4ad574041e94",
|
||||
"cmd/go-omni/apigen/testdata/managed/options.go": "30d48e4b5aab476c887ac53cc89b48b2b7e250c8cd25e55fecb49bb2a3589247",
|
||||
"cmd/go-omni/apigen/testdata/managed/background.go": "fb357b5deefd5187f2df406be5189fc4d57da857bd55b474f4726bf065c77167",
|
||||
"cmd/go-omni/apigen/testdata/managed/client.go": "cdd7d394ddad4bcdc10224890a884aa513ed2ffd8275ecdad8de9963dd10986a",
|
||||
"cmd/go-omni/apigen/testdata/managed/phases_test.go": "4d826d7133027ac2399f8209f03efa46ed54aa45dcd0a35274a9009314941dca",
|
||||
"cmd/go-omni/apigen/testdata/managed/shutdown_test.go": "79fbf8dd4508b8217af888a31aaf2f2a01c1fa3869b2e80420605af697c66319",
|
||||
"cmd/go-omni/apigen/testdata/managed/consumer_test.go": "9e103c9de7a1a906903bc09787e1b38905dc5bb4d891c88938028da5dee98f6f",
|
||||
"cmd/go-omni/apigen/testdata/managed/operations.go": "44cea63e6023458c5818c9fdae223631cd1721f0841e671457398bfde8bc48f1",
|
||||
"cmd/go-omni/apigen/testdata/business/accessFactory.go": "2f982cf39b5e1a4be98596ce5d0a296af0fbf8c94616543a268e67ec376f31b4",
|
||||
"cmd/go-omni/apigen/testdata/business/client.go": "d4cbaeec18dfb3a0e2821217e8eb0515000e351a745a93ec811dabca44e58c8a",
|
||||
"cmd/go-omni/apigen/testdata/business/consumer_test.go": "e4159862a729ca2c5c9699840cc16a56350fc2914c43841f08946e9a6e9c8091",
|
||||
"cmd/go-omni/apigen/testdata/business/operations.go": "11ded7921a0753f22975c525e334e3d45366143625e41d24fe500504e785c64c",
|
||||
"cmd/go-omni/apigen/responseShape.go": "5b705130b66a3d4aaed842025b75abe54e6427acac8366366ca88ae9862429e1",
|
||||
"cmd/go-omni/apigen/documentRefs.go": "7b841a668154d872493104eda0f9049340c67d9323eb99f1621a1ef6ba343891",
|
||||
"cmd/go-omni/apigen/catalogDocument.go": "94df60328b2f105c35b73d9c6362ea14abadce6cb18e4d9921193d2d46854252",
|
||||
"cmd/go-omni/apigen/workspaceCatalog.go": "c00784220b379ed80deb07c060532b756cbc47434ecb8b1ca10bc12b12159328",
|
||||
"config/env.go": "ea84bccdcd39e3f6b4958a0c8ac8e7d6a0be516b24930f93039248165361bf35",
|
||||
"cmd/go-omni/apigen/arrayResponse_test.go": "9e3e4475f76683d3346d349cdfa94e9e75cf913ac6979bd7a4ed86628466d879",
|
||||
"cmd/go-omni/apigen/testdata/arrays/values.go": "d5e7f3adf12aabb301f46c296e8c894a7b2f6f70a6b6e7e02c74da7f1fd17bc5",
|
||||
"cmd/go-omni/apigen/testdata/arrays/consumer_test.go": "435932f2952ad897943e2cda83b04eccb2950579d8d88772661541e1e2a2333d",
|
||||
"cmd/go-omni/apigen/composition.go": "7ed3761b27d9c932d8ddc43e63c2aca798bdbe38f2b40571af8916c8401ea650",
|
||||
"cmd/go-omni/apigen/generatorConfig.go": "78db0c7808cf6ec997c5b64a4a5fbea11b74049709bc5900ccfbc5d8dabebbc8",
|
||||
"cmd/go-omni/apigen/copyConsumer_test.go": "e08ae4daed5dd25ab89d8872f653caf19499984dcaddc9835c13884c410fc0c3",
|
||||
"cmd/go-omni/apigen/generatorConfig_test.go": "a6d8c8ebea5f1accf0e37dd044e57b48e422821fe75086d8d2eb3abe3acb147c",
|
||||
"cmd/go-omni/apigen/middlewareFactory.go": "cc0220edb3eff25d7f66b2c37e206455f8730a2b909613713322bee7caf72303",
|
||||
"cmd/go-omni/apigen/testdata/composition/accessFactory.go": "e1e5057ff79f806b3218e93249e8191d097b5c4844977204a8c32848443cba33",
|
||||
"cmd/go-omni/apigen/testdata/composition/domain.go": "ab27ebd7ad5e3ed5505a2aa268b9b5ceedbb01477b59011bec63bb270114078b",
|
||||
"cmd/go-omni/apigen/testdata/composition/amount.go": "f2d1834502d71d9ecf72ebe3b1077e5ef41f44fd816abd4617ac19f9bb962fdd",
|
||||
"cmd/go-omni/apigen/testdata/composition/copy.go": "32d37d6f1f0e9aa2409aeb90b052801d1cdca70e504c2dd294cdae01422c9cb7",
|
||||
"cmd/go-omni/apigen/testdata/composition/client.go": "19102e191c0471cae9e7405623906d0d804865c40b80ff1d5b240a41ceab6a4e",
|
||||
"cmd/go-omni/apigen/testdata/composition/service.api": "01d230d5409756e2f3eb602a7fe065efa7d9599f7a4089da4bf8f07a2474444e",
|
||||
"cmd/go-omni/apigen/testdata/composition/consumer_test.go": "74aaf60318cd0de67495fc9142b3f67c3b6e232bb48b46b3e134de5aadd225ad",
|
||||
"cmd/go-omni/apigen/testdata/composition/dependencies.go": "edc153204b06da00c0e45c52db43a4a22f8ffe22c4a6d3f3ebec0137b861f84d",
|
||||
"cmd/go-omni/apigen/testdata/composition/converters.go": "ecdcfe31c75ca815dab8d1a82533b12cbbd40e12b3a5c444e815097f6c4df08e",
|
||||
"copyx/options.go": "edea2f3d2043bf813e56dedf0aea319aaa52301d3b589f41fef986e33a77dc32",
|
||||
"copyx/copy.go": "a752a81acc0e847158043daaf0fdaaecec43abaf27d5561725a47b3e120ffbdc",
|
||||
"copyx/graph.go": "ae6a2a11a8175033d327b580cd570c97fe9d2ddb814bb50884cc19a90766badd",
|
||||
"copyx/benchmark_test.go": "b1bf408957cadea47b47215ffb099d140d5d1a2eb061024a6d0c1d769f469a79",
|
||||
"copyx/copy_test.go": "89b65a6281591d7cd5823cc1c7d2c384c6bd446ef06aa1763ddc3acaa84dfb4a"
|
||||
}
|
||||
}
|
||||
47
reports/httpcAcceptance.md
Normal file
47
reports/httpcAcceptance.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# v0.21 httpc 验收(2026-09-26)
|
||||
|
||||
目标:让普通项目直接配置代理、连接阶段与空闲池参数,带请求头完成有界 JSON 调用,同时保留标准 Do/Transport 扩展。没有加入业务服务规则,也没有修改业务项目或在线服务器。
|
||||
|
||||
## 功能与安全验证
|
||||
|
||||
- 代理:独立子进程验证环境代理缓存、默认/显式 environment、NO_PROXY;本地代理与本地目标验证固定代理/直连。配置错误不回显凭证。
|
||||
- Transport:克隆池与进程默认池互不修改;可配置建连/keepalive/握手/响应头/100-continue/空闲参数,保留 TLS 证书验证。借用 Transport 不修改、不关闭,混合配置明确失败,typed nil 拒绝。
|
||||
- 实际 socket:慢 TLS 握手、慢响应头、超时后恢复、12 并发下最多 2 条 HTTP/1.1 连接、空闲回收与关闭后的连接释放。
|
||||
- 请求选项:24 个并发调用的认证头独立;WithHeaders 快照源 map、多值头与大小写规范化,重试不丢头,控制字符/分帧头拒绝,错误不会带 Token。
|
||||
- JSON:单次响应限额不影响其他请求,已知/未知长度、chunked、自动 gzip 解压后的限制、空成功体、多 JSON 文档、状态错误、调用后关闭。
|
||||
- 错误响应:已知小体复用连接;读取大小有上限;已知但缓慢的错误体最多花 25ms 尝试读取(标准 Transport 遵守 context),未知流直接关闭,不等完整 body。
|
||||
- 生命周期:Close 不中断已开始的调用,拒绝新调用;借用 Transport 生命周期保持项目所有。既有重试、总超时、请求 body 拒绝时关闭、重定向不跟随测试保留。
|
||||
- 日志不含 URL 路径、查询、URL 密码、Authorization 或响应体;标准返回的 URL 错误不承诺自动脱敏,项目应自行选择安全日志字段。
|
||||
|
||||
全量 `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race ./...`:48 个测试包通过,包含实际生成项目回归。另验证 Go 1.22.12 的 httpc/示例、vet、build 与 managedService 生成检查。没有改动生成模板的兼容门槛;CLI 当前版本与最低模板版本分开,v0.20 的生成能力保持兼容。
|
||||
|
||||
请求头 Fuzz:30s / 4 worker,1104189 次执行,无失败,总收尾时间 31.293s。模糊输入上限 8 KiB,验证被接受的头可以由标准 Request writer 序列化;不等同形式化安全证明。
|
||||
|
||||
## 本机性能对比
|
||||
|
||||
Go 1.27.1 / darwin arm64 / Apple M2 Max,关闭请求日志;每项 500ms,三轮取中位数。基线源代码为 `068e85132f74edccb7657629b162505d038154f7`(v0.20 发布后文档提交),使用相同的基准函数。
|
||||
|
||||
前七项使用内存 RoundTripper 测量包装/JSON 开销;Pool 项使用本地 TCP 服务;Headers 项是新增选项。原始记录:[修改前](httpcBaselineBenchmark.txt)、[修改后](httpcBenchmark.txt)。列内单位依次为 ns/op、B/op、allocs/op。
|
||||
|
||||
| 场景 | 修改前 | 修改后 |
|
||||
| --- | ---: | ---: |
|
||||
| Client/128B/decode | 1681 / 2861 / 25 | 1408 / 2348 / 22 |
|
||||
| Client/128B/discard | 1226 / 2856 / 25 | 1070 / 1836 / 21 |
|
||||
| Client/4KiB/decode | 7590 / 17599 / 34 | 7650 / 17085 / 31 |
|
||||
| Client/4KiB/discard | 2775 / 13480 / 33 | 1130 / 1836 / 21 |
|
||||
| Client/64KiB/decode | 82261 / 214437 / 41 | 83673 / 213925 / 38 |
|
||||
| Client/64KiB/discard | 17397 / 148648 / 40 | 2285 / 1836 / 21 |
|
||||
| Client/Do | 843.6 / 1808 / 20 | 712.1 / 1296 / 17 |
|
||||
| Pool/OK | 39726 / 7338 / 79 | 39598 / 6372 / 75 |
|
||||
| Pool/Unauthorized | 42912 / 7333 / 83 | 41195 / 6503 / 77 |
|
||||
| Headers | 新增能力 | 1761 / 3059 / 30 |
|
||||
|
||||
64 KiB discard 的分配从约 149 KB 降到 1.8 KB,不再为丢弃响应整体 ReadAll;普通 Do 少一次请求复制,也不在成功路径计算退避。大 JSON 解码耗时基本持平(4 KiB 约 +0.8%,64 KiB 约 +1.7%),分配略减;本轮没有更换 JSON 实现,也不宣称每个场景都更快。Headers 有独立快照/请求复制成本,见新增场景。
|
||||
|
||||
这些数字不是公网吞吐、冷 DNS、TLS 建连或高压容量承诺。连接容量与超时用确定性本地场景验证,未对线上资源施压。
|
||||
|
||||
## 兼容与使用
|
||||
|
||||
原 JSON 直接调用保持有效;自定义 interface/函数值中的 JSON 方法签名须补 variadic RequestOption。过去自定义 Transport 下被忽略的 MaxConnsPerHost,现在与其他代理/阶段选项一起明确拒绝混配;应配置借用 Transport 本身。
|
||||
|
||||
无新增第三方依赖。出站实现仍是标准 net/http,Do/Transport 保持通用扩展;HTTP 状态与业务响应码由调用项目解释。完整 [httpc 指南](../httpc/README.md)、[运行示例](../examples/httpClient/README.md)。远程固定版本安装验证在发布后追加。
|
||||
33
reports/httpcBaselineBenchmark.txt
Normal file
33
reports/httpcBaselineBenchmark.txt
Normal file
@@ -0,0 +1,33 @@
|
||||
goos: darwin
|
||||
goarch: arm64
|
||||
pkg: git.yuntuo.hk/k2safe/go-omni/httpc
|
||||
cpu: Apple M2 Max
|
||||
BenchmarkHTTPClient/128B/decode-12 386178 2005 ns/op 2861 B/op 25 allocs/op
|
||||
BenchmarkHTTPClient/128B/decode-12 382789 1681 ns/op 2861 B/op 25 allocs/op
|
||||
BenchmarkHTTPClient/128B/decode-12 374196 1561 ns/op 2861 B/op 25 allocs/op
|
||||
BenchmarkHTTPClient/128B/discard-12 453388 1243 ns/op 2856 B/op 25 allocs/op
|
||||
BenchmarkHTTPClient/128B/discard-12 512002 1226 ns/op 2856 B/op 25 allocs/op
|
||||
BenchmarkHTTPClient/128B/discard-12 511116 1221 ns/op 2856 B/op 25 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/decode-12 79958 7905 ns/op 17599 B/op 34 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/decode-12 82110 7590 ns/op 17599 B/op 34 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/decode-12 81975 7333 ns/op 17598 B/op 34 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/discard-12 234456 2796 ns/op 13480 B/op 33 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/discard-12 216720 2764 ns/op 13480 B/op 33 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/discard-12 225117 2775 ns/op 13480 B/op 33 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/decode-12 7014 82261 ns/op 214441 B/op 41 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/decode-12 7381 82513 ns/op 214437 B/op 41 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/decode-12 7546 81872 ns/op 214430 B/op 41 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/discard-12 36520 17397 ns/op 148648 B/op 40 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/discard-12 35896 17424 ns/op 148649 B/op 40 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/discard-12 33877 17195 ns/op 148648 B/op 40 allocs/op
|
||||
BenchmarkHTTPClient/Do-12 640717 846.8 ns/op 1808 B/op 20 allocs/op
|
||||
BenchmarkHTTPClient/Do-12 738248 843.6 ns/op 1808 B/op 20 allocs/op
|
||||
BenchmarkHTTPClient/Do-12 695224 830.3 ns/op 1808 B/op 20 allocs/op
|
||||
BenchmarkHTTPPool/OK-12 15036 40489 ns/op 7338 B/op 79 allocs/op
|
||||
BenchmarkHTTPPool/OK-12 15148 39087 ns/op 7337 B/op 79 allocs/op
|
||||
BenchmarkHTTPPool/OK-12 14715 39726 ns/op 7338 B/op 79 allocs/op
|
||||
BenchmarkHTTPPool/Unauthorized-12 13894 42471 ns/op 7333 B/op 83 allocs/op
|
||||
BenchmarkHTTPPool/Unauthorized-12 14020 43046 ns/op 7351 B/op 83 allocs/op
|
||||
BenchmarkHTTPPool/Unauthorized-12 13992 42912 ns/op 7328 B/op 83 allocs/op
|
||||
PASS
|
||||
ok git.yuntuo.hk/k2safe/go-omni/httpc 20.475s
|
||||
36
reports/httpcBenchmark.txt
Normal file
36
reports/httpcBenchmark.txt
Normal file
@@ -0,0 +1,36 @@
|
||||
goos: darwin
|
||||
goarch: arm64
|
||||
pkg: git.yuntuo.hk/k2safe/go-omni/httpc
|
||||
cpu: Apple M2 Max
|
||||
BenchmarkHTTPClient/128B/decode-12 403567 1408 ns/op 2348 B/op 22 allocs/op
|
||||
BenchmarkHTTPClient/128B/decode-12 462315 1400 ns/op 2348 B/op 22 allocs/op
|
||||
BenchmarkHTTPClient/128B/decode-12 416222 1427 ns/op 2348 B/op 22 allocs/op
|
||||
BenchmarkHTTPClient/128B/discard-12 595370 1041 ns/op 1837 B/op 21 allocs/op
|
||||
BenchmarkHTTPClient/128B/discard-12 583654 1070 ns/op 1836 B/op 21 allocs/op
|
||||
BenchmarkHTTPClient/128B/discard-12 550502 1086 ns/op 1836 B/op 21 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/decode-12 77176 7547 ns/op 17084 B/op 31 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/decode-12 81969 7677 ns/op 17085 B/op 31 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/decode-12 77571 7650 ns/op 17087 B/op 31 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/discard-12 544194 1132 ns/op 1836 B/op 21 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/discard-12 539278 1128 ns/op 1836 B/op 21 allocs/op
|
||||
BenchmarkHTTPClient/4KiB/discard-12 522939 1130 ns/op 1836 B/op 21 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/decode-12 6986 84111 ns/op 213930 B/op 38 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/decode-12 7156 83673 ns/op 213925 B/op 38 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/decode-12 7516 83041 ns/op 213921 B/op 38 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/discard-12 270482 2285 ns/op 1836 B/op 21 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/discard-12 253564 2259 ns/op 1835 B/op 21 allocs/op
|
||||
BenchmarkHTTPClient/64KiB/discard-12 254343 2290 ns/op 1836 B/op 21 allocs/op
|
||||
BenchmarkHTTPClient/Do-12 857305 713.4 ns/op 1296 B/op 17 allocs/op
|
||||
BenchmarkHTTPClient/Do-12 811660 712.1 ns/op 1296 B/op 17 allocs/op
|
||||
BenchmarkHTTPClient/Do-12 861867 708.3 ns/op 1296 B/op 17 allocs/op
|
||||
BenchmarkHTTPPool/OK-12 14582 40133 ns/op 6374 B/op 75 allocs/op
|
||||
BenchmarkHTTPPool/OK-12 15596 38152 ns/op 6372 B/op 75 allocs/op
|
||||
BenchmarkHTTPPool/OK-12 15513 39598 ns/op 6369 B/op 75 allocs/op
|
||||
BenchmarkHTTPPool/Unauthorized-12 14952 41483 ns/op 6503 B/op 77 allocs/op
|
||||
BenchmarkHTTPPool/Unauthorized-12 14442 40927 ns/op 6516 B/op 77 allocs/op
|
||||
BenchmarkHTTPPool/Unauthorized-12 14697 41195 ns/op 6498 B/op 77 allocs/op
|
||||
BenchmarkHTTPHeaders-12 319768 1761 ns/op 3059 B/op 30 allocs/op
|
||||
BenchmarkHTTPHeaders-12 334965 1731 ns/op 3059 B/op 30 allocs/op
|
||||
BenchmarkHTTPHeaders-12 348751 1774 ns/op 3059 B/op 30 allocs/op
|
||||
PASS
|
||||
ok git.yuntuo.hk/k2safe/go-omni/httpc 21.809s
|
||||
@@ -124,7 +124,7 @@ def main():
|
||||
checks += ["sharedGoCatalogExport", "documentOnlyErrorSelection", "workspaceCheckNoWrites", "staleCatalogRejected", "twoIndependentServiceBuilds"]
|
||||
|
||||
env.update(GO_OMNI_SCAFFOLD_INTEGRATION="1", GO_OMNI_RELEASE_CLI=str(cli), GO_OMNI_RELEASE_VERSION=args.version)
|
||||
tests = ["TestManagedGeneratedProject", "TestErrorInheritanceConsumer", "TestErrorParamsConsumer", "TestCompatibilityConsumer", "TestBusinessGeneratedProject", "TestPrimitiveArrayConsumer", "TestCompositionMappingConsumer", "TestNullableMappingConsumer"]
|
||||
tests = ["TestManagedGeneratedProject", "TestErrorInheritanceConsumer", "TestErrorParamsConsumer", "TestCompatibilityConsumer", "TestBusinessGeneratedProject", "TestPrimitiveArrayConsumer", "TestCopyxThreeEndpointConsumer", "TestGeneratedConfigurationConsumer"]
|
||||
print("Running generated consumers against the remote framework, including lifecycle failure cleanup", flush=True)
|
||||
output = run(["go", "test", "-race", "-count=1", "-json", "-timeout=240s", "./cmd/go-omni/apigen", "-run", "^(" + "|".join(tests) + ")$"], ROOT, env)
|
||||
events = [json.loads(line) for line in output.splitlines() if line.startswith("{")]
|
||||
@@ -135,19 +135,33 @@ def main():
|
||||
assert len(sums) == len(tests) and all(s == dependency["Sum"] for s in sums), "remote consumer checksum mismatch"
|
||||
checks += ["initializationAndListenFailureCleanup", "drainBeforeDependencyClose", "generatedConsumerRaceTests", "errorParamsAndInheritance", "manualHooksAndScalars", "dependencyFatalShutdown", "fatalProcessExitCode", "initializationShutdownNotification", "configuredBuiltinEndpoints", "projectHealthEnvelope", "endpointConflictCleanup", "responseArrayNullableNull", "primitiveAndDynamicArrays", "strictRequestPolicies", "declarativeMultipartBinding", "backgroundStopDrainResourceOrder", "backgroundInitAndListenFailureCleanup", "backgroundTimeoutRetainsResourcesAndExits"]
|
||||
|
||||
checks += ["flatFieldComposition", "typedStructureMapping", "explicitNullAndMoneyPolicies", "mappingFailureBeforeWrites", "mappingConsumerRaceTests", "middlewareFactoryInjectionAndCleanup", "optionalRepositoryPreservesBusiness"]
|
||||
# Run tests from the downloaded module, not the framework checkout.
|
||||
http_output = run(["go", "test", "-race", "-count=1", "-json", MODULE + "/httpc"], starter, env)
|
||||
http_events = [json.loads(line) for line in http_output.splitlines() if line.startswith("{")]
|
||||
http_passed = {e.get("Test") for e in http_events if e.get("Action") == "pass"}
|
||||
assert {"TestProxyRouting", "TestJSONHeadersConcurrentSnapshot", "TestPoolLimitsAndIdleExpiry",
|
||||
"TestErrorBodyPoolReuseAndDrainBound", "TestSlowKnownErrorBodyHasShortDrainBudget"} <= http_passed
|
||||
example_output = run(["go", "run", MODULE + "/examples/httpClient"], starter, env)
|
||||
assert "authenticated request succeeded" in example_output
|
||||
checks += ["remoteHTTPClientRaceTests", "remoteHTTPProxyAndTimeouts", "remoteHTTPHeaderIsolation",
|
||||
"remoteHTTPPoolAndDrainBudgets", "remoteHTTPExample"]
|
||||
|
||||
checks += ["nullableNestedMappings", "splitMappingFiles", "explicitNilPolicies", "pointerResultHelpers", "mappingWorkspaceRegeneration", "nestedErrorChainAndPrivateCause", "mappingPreviewNoWrites"]
|
||||
checks += ["flatFieldComposition", "copyxThreeEndpointConsumer", "explicitNullAndMoneyPolicies", "fieldAdditionWithoutCopyRuleChanges", "middlewareFactoryInjectionAndCleanup", "optionalRepositoryPreservesBusiness"]
|
||||
|
||||
checks += ["generatedUnifiedConfig", "jsonYamlXmlAndCustomDecoder", "configValidationBeforeDependencies", "configEnvProvenance", "configExtensionPreserved", "configLoaderDriftCheck"]
|
||||
|
||||
sources = ["lifecycle/runtime.go", "lifecycle/phases.go", "config/lifecycle_test.go", "task/task.go", "cmd/go-omni/scaffold/templates/api-runtime.go.tmpl", "cmd/go-omni/apigen/templates.go", "config/project.go", "contract/bind.go", "contract/policy.go", "contract/multipart.go", "contract/file.go", "cmd/go-omni/apigen/business_test.go", "cmd/go-omni/apigen/generate.go", "lifecycle/controller.go", "config/endpoints.go", "health/health.go", "httpx/response.go", "app/app.go", "app/transport.go", "scripts/verifyManagedRelease.py", "scripts/verifyManagedProject.py", "cmd/go-omni/apigen/managed_test.go"]
|
||||
sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/managed").glob("*.go")]
|
||||
sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/business").glob("*.go")]
|
||||
sources += ["cmd/go-omni/apigen/responseShape.go", "cmd/go-omni/apigen/documentRefs.go", "cmd/go-omni/apigen/catalogDocument.go", "cmd/go-omni/apigen/workspaceCatalog.go", "config/env.go", "cmd/go-omni/apigen/arrayResponse_test.go"]
|
||||
sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/arrays").glob("*.go")]
|
||||
sources += ["cmd/go-omni/apigen/" + name for name in ("composition.go", "generatorConfig.go", "mapping.go", "mappingLoad.go", "mapping_test.go", "middlewareFactory.go")]
|
||||
sources += ["cmd/go-omni/apigen/" + name for name in ("composition.go", "generatorConfig.go", "copyConsumer_test.go", "generatorConfig_test.go", "middlewareFactory.go")]
|
||||
sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/composition").iterdir() if p.is_file()]
|
||||
sources += ["cmd/go-omni/apigen/" + name for name in ("mappingPointer.go", "mappingNullable_test.go", "generatorConfig_test.go")]
|
||||
sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/nullable").iterdir() if p.is_file()]
|
||||
sources += [str(p.relative_to(ROOT)) for p in (ROOT / "copyx").glob("*.go")]
|
||||
sources += [str(p.relative_to(ROOT)) for p in (ROOT / "config").glob("*.go")]
|
||||
sources += ["cmd/go-omni/apigen/configLoader_test.go", "cmd/go-omni/apigen/testdata/configuration/consumer_test.go", "cmd/go-omni/scaffold/templates/api-load.go.tmpl", "cmd/go-omni/scaffold/templates/api-loader.go.tmpl"]
|
||||
sources += [str(p.relative_to(ROOT)) for p in (ROOT / "httpc").glob("*.go")]
|
||||
sources += [str(p.relative_to(ROOT)) for p in (ROOT / "examples/httpClient").glob("*.go")]
|
||||
report = {"passed": True, "frameworkVersion": args.version, "frameworkSum": dependency["Sum"],
|
||||
"freshModuleCache": True, "noLocalReplace": True,
|
||||
"durationSeconds": round(time.monotonic() - started, 2), "checks": checks,
|
||||
|
||||
Reference in New Issue
Block a user