diff --git a/CHANGELOG.md b/CHANGELOG.md index b928528..5c55223 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,14 @@ # Changelog +## v0.20.0-preview（2026-09-26） + +- 生成项目配置加载入口持续由生成器维护；保留业务字段、默认值、校验与 NewLoader 扩展文件。 +- config.LoadProject 统一 JSON/YAML/XML、环境覆盖和校验；提供实例级 Decoder 注册。 +- 文件/字段/环境/行列的安全诊断，Invalid/AtField 与原始错误链。 +- 增加格式等价、精度、null/空数组、并发、资源限制、默认值隔离、真实生成项目与性能验收。 +- 旧手写 Load 必须显式迁移；低层 API 保留。固定数组严格长度、YAML alias/merge/custom tag 拒绝。用法及限制见 CONFIG.md。 + + ## v0.19.0-preview（2026-09-25） - 新增通用可选 copyx：构造器、浅/深拷贝、精确类型转换、字段规则、nil/零值策略、并发复用。 diff --git a/CONFIG.md b/CONFIG.md new file mode 100644 index 0000000..634148d --- /dev/null +++ b/CONFIG.md @@ -0,0 +1,178 @@ +# 统一配置加载 + +v0.20 起，新生成服务的 `main.go` 调用生成器维护的 `config.Load`。项目只需要定义业务字段、默认值和校验；日志、HTTP、数据库等配置使用同一流程，配置加载时不打开数据库、监听端口或创建日志文件。 + +```text +代码默认值 → envDefault（仅填零值）→ 可选配置文件 +→ 完整结构体环境覆盖（含别名）→ 框架规范化 → 项目 Validate +→ 初始化日志及项目依赖 → 启动服务 +``` + +## 生成项目要维护什么 + +| 文件 | 维护方 | 用途 | +| --- | --- | --- | +| `main.go` | 生成器 | 启动、信号、请求排空、依赖关闭 | +| `internal/config/loadGen.go` | 生成器 | 调用统一加载入口；纳入 `generate --check` | +| `internal/config/config.go` | 项目 | `Config`、`Default()`、`Validate()` | +| `internal/config/loader.go` | 项目 | `NewLoader()`，只有增加文件格式时才需修改 | +| `etc/config.yaml` | 项目 | 部署配置；可改用 JSON/XML 并传 `-config` | +| `internal/svc` | 项目与生成器分文件 | 初始化依赖、登记清理、AppOptions | + +比如增加业务的批次数量：在 Config 增加一个字段，在 Validate 加范围校验；需要部署值时改配置文件或环境变量。不修改 main、Load 或 Handler。 + +```go +// 添加到项目 Config。 +BatchSize int `json:"batchSize" envDefault:"100"` + +// 添加到项目 Validate。 +if c.BatchSize < 1 || c.BatchSize > 1000 { + return omniConfig.Invalid("batchSize", "must be between 1 and 1000") +} +``` + +自动支持 `GO_OMNI_BATCH_SIZE`。生成服务的空环境值策略为 `ignore`，0 和 false 是有效覆盖；范围由 Validate 决定。 +重新生成保留 config.go、loader.go 和 svc 的手写扩展。 + +## 手写项目用法 + +```go +package main + +import ( + "time" + "git.yuntuo.hk/k2safe/go-omni/config" +) + +type Config struct { + Timeout time.Duration `json:"timeout" envDefault:"3s"` + Limit int `json:"limit" envDefault:"100"` + Token string `json:"-" env:"APP_TOKEN,secret"` +} +func (c Config) Validate() error { + if c.Timeout <= 0 { + return config.Invalid("timeout", "must be positive") + } + return nil +} + +// cfg, err := config.LoadProject("etc/config.yaml", Config{}, +// config.EnvOptions{Prefix: "APP_", Empty: "reject"}) +``` + +`LoadProject` 接收结构体默认值，可实现值接收器或指针接收器的 `Validate() error`。空路径表示不读文件。 +`DecodeFile` 只做文件覆盖，不做 envDefault、环境覆盖或业务 Validate；旧的两参数调用仍有效，可用结构体或结构体指针。低层 `ApplyDefaults`、`OverlayEnv`、`LoadFile` 继续存在。 + +嵌套校验用 `config.AtField("client", err)` 加路径前缀。公共原因通过 `config.Invalid` 提供，原因中不要包含密码、DSN、Token 或输入原值。 + +## JSON / YAML / XML + +按文件扩展名选择解析器：`.json`、`.yaml`、`.yml`、`.xml`，大小写不敏感。所有格式使用 Go 结构体的 **json 字段名**，无需为 XML 复制一份结构或标签。 +以下配置等价： + +```json +{"client":{"timeout":"2s"},"enabled":false,"ports":[8080,8081],"optional":null} +``` + +```yaml +client: + timeout: 2s +enabled: false +ports: [8080, 8081] +optional: null +``` + +```xml + + 2s + false + 80808081 + + +``` + +XML 必须使用一个 `` 根；普通叶节点按目标 Go 类型转换。数组显式 `type="array"` 加 ``，不根据重复元素猜测类型。`type="array"` 空元素表示空数组，`type="object"` 表示空对象，`type="null"` 表示 null。`any` 中的 XML 叶节点默认是字符串，需要动态数字/布尔时标注 `type="number"` / `type="bool"`；也可用 `type="string"`。 + +类型和覆盖规则： + +- 缺失字段保留默认值；显式 0、false、空字符串正常覆盖。 +- null 清空指针、map、slice、interface；普通值字段保持默认值，沿用 Go JSON 解码语义。需要区分 null 时用指针。 +- 数组整体替换；空 slice 保持非 nil；固定长度 Go 数组要求长度一致，避免丢弃多余配置。 +- map 合并键，已提供的键整体覆盖对应值；map 中未提供的键保留。 +- duration 支持 `250ms`、`3s`；整数保持精度，动态 JSON 数字使用 `json.Number`，不先转 float64。 +- 自定义 `json.Unmarshaler` / `encoding.TextUnmarshaler` 仍可接入。这些回调是可信项目代码，必须自己保证并发和副作用安全；不要修改共享外部状态。 +- JSON `,string` 字段、嵌入字段遵循 JSON 字段选择；未知字段、重复绑定、冲突均拒绝。 + +内置解析器限制：单文件最多 1 MiB，解析最多 64 层、100000 个节点；绑定和默认值复制也有深度限制。只接受普通文件，不读取设备或命名管道。不支持多文档、重复字段、无效 UTF-8；YAML 拒绝 alias/merge/custom tag，XML 拒绝 DTD/外部实体、命名空间、业务属性和混合内容。XML 仅支持配置所需的子集，不是通用 XML 文档映射器。 + +这些限制针对启动配置，避免意外超大文件、别名扩展和模糊覆盖。自定义值解码器的内部字段校验由该解码器负责；例如现有错误目录使用其专有解码规则。 + +## 环境覆盖 + +统一入口的 EnvOptions 控制前缀、空值策略、别名和 Lookup。完整 JSON 路径自动转换为 UPPER_SNAKE_CASE：`client.timeout` → `APP_CLIENT_TIMEOUT`。 + +```go +config.EnvOptions{ + Prefix: "GO_OMNI_", Empty: "ignore", + Aliases: config.ServerEnvAliases("GO_OMNI_", "server"), +} +``` + +生成项目同时接受完整的 `GO_OMNI_SERVER_HTTP_READ_TIMEOUT` 与短别名 `GO_OMNI_HTTP_READ_TIMEOUT`。完整名存在时优先，即便其空值被 ignore，也不回退别名。不会再二次调用 ApplyEnv 覆盖业务字段。 + +显式 `env:"CUSTOM_NAME"` 是绝对名称；`env:"-"` 跳过子树。`json:"-"` 只在声明显式 env 名时参与，可让密钥只能从环境加载。`secret` 标签表达用途，解析错误对所有字段都不打印值。slice/map 的环境值采用 JSON。`required` 检查字符串非空、指针/集合非 nil；复杂范围由 Validate 负责。详见 [完整业务环境覆盖](DEVELOPMENT.md#完整业务配置的环境覆盖)。 + +显式环境选择日志 stdout/stderr 会清除文件 profile 的 File/Rotation，除非同时提供非空 File 覆盖。该兼容规则由框架处理。 + +## 扩展解析器 + +内置格式直接可用。新增格式只在 `internal/config/loader.go` 的 `NewLoader` 注册： + +```go +func NewLoader() (*omniConfig.Loader, error) { + return omniConfig.NewLoader( + omniConfig.WithDecoder("custom", omniConfig.DecoderFunc( + func(data []byte) (omniConfig.Document, error) { + // 演示注册扩展名；实际可在这里调用项目选定的 TOML 等解析库。 + return (omniConfig.JSONDecoder{}).Decode(data) + }, + )), + ) +} +``` + +Decoder 返回 Document：Value 是 map[string]any 根，子值为 string/bool/json.Number/nil/[]any/map[string]any；无自身类型的叶节点可返回 config.Text。Positions 可选，用 `client.timeout` 或 `servers[0].timeout` 标记 1-based 行列。不提供位置时仍有文件、字段、原因。 + +自己的语法错误可返回 `&config.Error{Field: "client.timeout", Line: 3, Column: 5, Reason: "expected a duration"}`。Reason 必须安全；普通第三方错误仅公开 `invalid configuration syntax`，内部 cause 通过 errors.Is/As 保留。 + +注册表属于 Loader 实例，无全局 init 注册；不能覆盖已有扩展名。Loader 可共享，但 Decoder 必须支持并发且不保留/修改输入。解析插件是可信 Go 代码，框架不是插件沙箱。新格式依赖由选择它的项目显式引入，核心不强制引入 TOML 等库。 + +## 定位配置错误 + +```text +config "etc/config.yaml":4:12 field "client.timeout": expected a duration such as 5s +config env "APP_CLIENT_TIMEOUT" field "client.timeout": expected a valid time.Duration +``` + +`errors.As(err, &diagnostic)` 可获取 `*config.Error` 的 File/Field/Env/Line/Column/Reason；原 `DecodeError` 名称保留为别名。`errors.Is` 可判断文件不存在等 I/O 原因。 + +解析失败显示文件和可获取的行列；Validate 使用 Invalid/AtField 才能定位业务字段。环境覆盖导致的错误优先标环境变量，不错误地归因到文件；数组元素验证也会追溯到覆盖它的集合环境变量。默认值未出现在文件里时没有行列。YAML 语法错误通常只有行号，列显示 1。 + +不打印底层错误原文或配置值；任意普通 `errors.New(...)` 校验失败的公开原因是 `validation failed`，内部原因仍在链中。项目应改用 Invalid 给出安全的原因。第三方子配置校验若没有字段元数据，只定位到对应子树，框架不会从任意错误文案猜字段。 + +不要把 Unwrap 后的原始错误、整个配置对象写入公开日志/HTTP；自定义解析器与校验器传入的 Reason 由项目负责脱敏。 + +## v0.19 及更早项目迁移 + +先提交现有代码，再升级匹配的 CLI 与框架。旧项目的 `internal/config/config.go` 含手写 Load，新生成器会明确阻止 generate/check/dry-run，不会直接覆盖它。 + +1. 保留 Config、Default、Validate、业务配置结构及辅助函数。 +2. 删除旧 Load。普通的 defaults/file/env/Validate 流程已由 LoadProject 接管；移除由此不再使用的 import。将自定义默认值和校验分别迁入 Default / Validate，自定义文件格式迁入 NewLoader。 +3. 框架只在 `loader.go` 不存在时创建 NewLoader；已有项目文件同名时需要保证它提供同签名函数。无需手写 loadGen.go。 +4. 重新生成，执行 `generate --check`、`go test ./...`、`go build .`，再用真实配置验证启动。 + +保留业务需要的显式 env 标签/规则；如果原 Load 有解密或其他项目特殊步骤，不要直接删除，应先明确其归属，必要时在自定义 Decoder 中处理。生成器不自动搬动或猜测手写行为。`api upgrade` 仍可预览较早版本的基础设施差异，但已修改的配置必须按上述步骤迁移。 + +不会移除低层配置 API；本次移除的是**新项目里重复手写的加载流程**，不删除旧项目的业务文件。 + +完整可运行例子：[examples/configuration](examples/configuration/README.md)。 diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index f4419e4..fcc2b27 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -1,4 +1,4 @@ -# 通用业务开发指南（v0.19.0-preview） +# 通用业务开发指南（v0.20.0-preview） 从 [README](README.md) 创建服务后，业务主要维护 `.api`、配置、`svc.Init`、Logic、Repository 和项目中间件。本页说明通用接入能力；字段、错误码、权限、表结构和事务边界都由项目定义。 @@ -102,7 +102,9 @@ defer reader.Close() ## 完整业务配置的环境覆盖 -生成项目默认执行：代码与 `envDefault` 默认值 → JSON/YAML 文件 → 全结构体环境覆盖 → 既有框架环境别名 → `Validate`。增加业务字段不需要另外写 `os.Getenv`。 +生成器维护 `internal/config/loadGen.go`；项目维护 Config/Default/Validate 和可选 NewLoader。三种格式、可扩展解析器、字段错误定位及旧 Load 迁移见 [统一配置指南](CONFIG.md)。 + +生成项目默认执行：代码与 `envDefault` 默认值 → JSON/YAML/XML 文件 → 全结构体环境覆盖（含别名）→ 规范化 → `Validate`。增加业务字段不需要另外写 `os.Getenv`。 ```go type ClientConfig struct { @@ -278,7 +280,7 @@ go-omni workspace check --file omni.workspace.json ## v0.13 项目升级 -先使用与目标框架匹配的 CLI；CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.19.0-preview（或显式引用当前源码）。 +先使用与目标框架匹配的 CLI；CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.20.0-preview（或显式引用当前源码）。 ```sh go-omni api upgrade --dir . --dry-run diff --git a/PROJECT.md b/PROJECT.md index b8958b8..342dc85 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -1,6 +1,6 @@ # 生成项目与文件维护边界 -v0.19.0-preview 提供受管理的生成项目。安装同版远程 CLI，生成的项目直接引用该版本，无需本地 replace。根目录入口支持 `go run .` 和 `go build .`；框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。 +v0.20.0-preview 提供受管理的生成项目。安装同版远程 CLI，生成的项目直接引用该版本，无需本地 replace。根目录入口支持 `go run .` 和 `go build .`；框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。 ## 创建和日常开发 @@ -38,7 +38,7 @@ ERROR [logicSignature] internal/logic/greet/greet.go:12 (GET /api/greet/:name): 这是轻量源码分析，不运行项目 init、不执行 Go 命令、不下载依赖。项目自定义类型别名、嵌入方法或 build tags 等无法可靠判断时，会给 REVIEW 并交给 Go 编译器验证；不把这些情况强行判为签名错误。它不检查方法体是否引用已删除字段，也不证明业务逻辑正确。CI 中应同时执行 `go-omni api generate --check` 和 `go test ./...`，按项目实际构建参数测试。当前版本首次接手缺少快照的 manifest v2 时，会明确提示建立基线，不猜测旧契约；Logic 签名检查仍会执行。 -本版新增生成选项、公共字段组合和映射，详见 [生成能力指南](GENERATION.md)。 +生成选项、公共字段组合和可选 Go 拷贝工具，详见 [生成能力指南](GENERATION.md)。 ## 哪些文件由用户维护 @@ -46,7 +46,8 @@ ERROR [logicSignature] internal/logic/greet/greet.go:12 (GET /api/greet/:name): | --- | --- | | `api/*.api` | 路由、DTO、具名中间件、响应模板、请求/响应模式 | | `etc/config.yaml` | 运行参数；密钥通过环境变量提供 | -| `internal/config/config.go` | 自定义配置字段、加载、默认值与校验 | +| `internal/config/config.go` | 自定义配置字段、默认值与校验 | +| `internal/config/loader.go` | 需要额外文件格式时注册 Decoder | | `internal/svc/serviceContext.go` | 依赖字段和初始化，持有加载后的 Config | | `internal/svc/options.go` | AppOptions、GlobalMiddleware；注册 Go 错误目录和全局扩展 | | `internal/svc/databaseOptions.go` | 可选 MySQL 驱动与项目启动检查扩展 | @@ -62,7 +63,7 @@ ERROR [logicSignature] internal/logic/greet/greet.go:12 (GET /api/greet/:name): 以上骨架只补缺失文件，既有内容按字节保留。删除已废弃接口后，对应业务文件也不会被自动删除；由开发者整理编译器指出的失效类型或签名。 -生成器持续管理：`internal/svc/runtimeGen.go`、`main.go`、`internal/handler/*Gen.go`、分组的 `handlersGen.go`、`internal/types/*Gen.go`、中间件 `registryGen.go`、`docs/openapi.json`、`.goOmni/apiManifest.json`。如果在 `.api` 里选择声明错误，还会管理 `internal/apierrors/errorsGen.go`。文件标注 DO NOT EDIT；手改会被拒绝覆盖，需先移走自定义内容。 +生成器持续管理：`internal/config/loadGen.go`、`internal/svc/runtimeGen.go`、`main.go`、`internal/handler/*Gen.go`、分组的 `handlersGen.go`、`internal/types/*Gen.go`、中间件 `registryGen.go`、`docs/openapi.json`、`.goOmni/apiManifest.json`。如果在 `.api` 里选择声明错误，还会管理 `internal/apierrors/errorsGen.go`。文件标注 DO NOT EDIT；手改会被拒绝覆盖，需先移走自定义内容。 Logic 实例在注册时创建并由请求并发复用；请求数据用方法参数和局部变量传递，不存入共享 Logic/svc 字段。 @@ -114,6 +115,8 @@ c.RequestShutdown(err) `Lifecycle *lifecycle.Controller` 是 svc 持有的通用控制器，初始化时创建，生成入口使用它的 Context。该 Context 表示服务正在停止，不代表数据库已经关闭。不能把它当作在途请求的 Context，也不要在它取消时立刻关闭请求仍依赖的资源。发号器失去租约后拒绝发号等业务安全规则仍由项目实现。 +配置加载入口 `internal/config/loadGen.go` 也由生成器维护。业务扩展 Config/Default/Validate；只有新格式才需要修改 NewLoader。JSON/YAML/XML、错误定位和旧 Load 迁移见 [配置指南](CONFIG.md)。 + ## 配置内置接口 生成项目在 `etc/config.yaml` 明确列出以下默认配置；可以独立关闭、修改路径，或让健康接口返回原始 JSON： @@ -212,7 +215,7 @@ get /download (DownloadReq) ## 已有服务接入步骤 -每个独立服务生成一个项目，module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.19.0-preview： +每个独立服务生成一个项目，module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.20.0-preview： 1. 先保留旧服务，整理实际路由及现有响应、鉴权和超时要求。在新目录执行 `go-omni new --api /path/to/service.api --module example.com/service ./service`；不对旧入口强制覆盖。 2. 在 `.api` 中选择 `@app(defaultResponse: ...)`，显式保留原项目的字段、成功码、错误码和 `dataMode`。需要独立回执的路由单独声明 raw/produces。普通路由、404/405 和提前返回的中间件一起验收。 diff --git a/README.md b/README.md index 4f67f4c..0f84260 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ GoOmni 是面向多个业务项目复用的 Go 后端框架：用 `.api` 定义接口、请求和返回类型，生成项目目录、HTTP 接入代码及 OpenAPI，业务代码负责 Logic 和 Repository。也可以在已有项目中直接使用 `omni.New()`，或单独引入日志、存储等模块。 -**从这份 README 开始接入。** 当前版本 **`v0.19.0-preview`**，新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入；文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md)，版本变化与验收见 [发布说明](RELEASE.md)。 +**从这份 README 开始接入。** 当前版本 **`v0.20.0-preview`**，新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入；文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md)，版本变化与验收见 [发布说明](RELEASE.md)。 **v0.19 改动**：新增可选 `copyx` 工具，用普通 Go 调用完成浅拷贝、深拷贝和显式类型转换；深拷贝会拒绝自定义转换中残留的源对象可变引用。**移除了 `mappings` / `mappingFiles` 和映射代码生成**，已有使用者须按 [迁移步骤](COPY.md#从旧-mapping-迁移) 调整，不能直接删除仍在调用的旧文件。 @@ -38,7 +38,7 @@ Go **1.22+**。框架不依赖 Gin，默认使用标准库网络层，提供可 | 错误目录与动态提示 | `WithErrors(catalog)` 注册一次；`ByCode(...).WithParams(...).WithCause(err)` | [错误目录](errors/README.md)、[模板参数示例](examples/contracts/errorParams.api) | | OpenAPI 与错误公共引用 | `.api` 自动生成；手写接口可使用 `openapi` 注册器；Go 错误目录显式导出 | [契约工具](DEVELOPMENT.md)、[注册示例](examples/platform/main.go) | | 结构化业务日志 | `logx.Info/Error`、`logx.FromContext(ctx)`、`logx.New`；支持轮转、脱敏、慢请求和级别更新 | [下方日志示例](#日志)、[完整配置](logx/README.md) | -| 文件配置与环境覆盖 | 默认值→JSON/YAML→环境；完整业务结构用 `config.LoadProject[T]` / 环境别名 | [配置详解与例子](DEVELOPMENT.md)、[动态级别](ADVANCED.md#配置与日志热更新) | +| 文件配置与环境覆盖 | 默认值→JSON/YAML/XML→环境→校验；完整业务结构用 `config.LoadProject[T]` / 环境别名 | [统一配置指南](CONFIG.md)、[完整可运行示例](examples/configuration/README.md)、[动态级别](ADVANCED.md#配置与日志热更新) | | 生命周期与健康检查 | svc 登记 OnClose/OnStop/OnDrain，RequestShutdown 通知故障；健康路径/开关可配 | [生命周期用法](lifecycle/README.md)、[健康检查](health/README.md)、[生成项目扩展](PROJECT.md) | | 数据库、连接池与事务 | `database.Open` / `OpenConnector`；事务与检查由项目显式调用 | [通用数据库](database/README.md)、[MySQL TLS/Logger](database/mysql/README.md) | | 可选 GORM 与迁移 | gormx 复用已有连接池；`go-omni migrate` 预览，显式 apply | [GORM 示例](database/gormx/README.md)、[迁移用法](ADVANCED.md#迁移审计与分页) | @@ -54,6 +54,29 @@ Go **1.22+**。框架不依赖 Gin，默认使用标准库网络层，提供可 | 可选结构转换与深拷贝 | `copyx.Copy` / `Clone` / `New`；Go converter 处理差异 | [下方示例](#可选结构转换与深拷贝)、[完整 copyx 指南](COPY.md)、[可运行代码](examples/copying) | | 自定义适配器验收 | `transporttest.Run`、`storagetest.Run` / `RunUploads` 验证扩展契约 | [网络契约测试入口](transport/transporttest/router.go)、[存储契约测试入口](storage/storagetest/contract.go) | +## 统一配置加载 + +新生成项目的加载流程由框架管理。业务在 `internal/config/config.go` 增加字段、默认值和 `Validate`，无需改 main 或复制环境变量解析。JSON、YAML、XML 共用字段和校验；扩展格式在 `loader.go` 注册。 + +```go +// 项目 Config 中增加字段，自动支持 GO_OMNI_BATCH_SIZE： +BatchSize int `json:"batchSize" envDefault:"100"` + +// 项目 Validate 中增加业务范围检查： +if c.BatchSize < 1 { + return omniConfig.Invalid("batchSize", "must be positive") +} +``` + +错误包含文件、字段、可用的行列或环境变量名；输入原值不自动写入错误。 + +```sh +go run ./examples/configuration -format xml +DEMO_CLIENT_TIMEOUT=500ms go run ./examples/configuration +``` + +详见 [统一加载、解析器、迁移](CONFIG.md) 和 [完整示例](examples/configuration/README.md)。**旧项目须先迁移手写 Load，再重新生成**；生成器会保留业务配置并拒绝冲突，不静默覆盖。 + ## 选择接入方式 | 你的项目 | 从哪里开始 | @@ -74,7 +97,7 @@ module 地址是 `git.yuntuo.hk/k2safe/go-omni`。先确保本机 Go 和 Git 可 ```sh export GOPRIVATE="$(go env GOPRIVATE),git.yuntuo.hk" -go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.19.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.20.0-preview export PATH="$(go env GOPATH)/bin:$PATH" go-omni --help ``` @@ -231,7 +254,7 @@ go-omni new --api catalog.api \ 在业务项目的 **go.mod 所在目录**执行： ```sh -go get git.yuntuo.hk/k2safe/go-omni@v0.19.0-preview +go get git.yuntuo.hk/k2safe/go-omni@v0.20.0-preview ``` 已有项目可直接复用现有目录、Service、Repository 和数据库连接，先选择一个接口接入。也可以只用 `logx`、`storage` 等包，暂时保留原来的 HTTP 框架。希望入口和 HTTP 接入都交给生成器的项目，按 [已有服务接入步骤](PROJECT.md#已有服务接入步骤) 在新目录生成后迁入业务代码。 @@ -533,7 +556,7 @@ go test ./... `main.go`、`Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理，手动修改会报告冲突。Logic、Repository、中间件实现、配置和依赖装配属于项目，重复生成保留手写内容。契约变化会提示受影响的 Logic 文件和 DTO 字段，明确的签名错误会给出原签名与期望签名；生成后 `--check` 仍能检出未修复的问题。将 `.api`、`omni.gen.json`、生成文件和 `.goOmni/apiManifest.json` 一起提交，CI 同时运行 `generate --check` 与 `go test ./...`。详见 [契约变更提示与检查边界](PROJECT.md#契约变更提示)。 -CLI 与业务 module 固定使用 `v0.19.0-preview`。v0.14.x 项目升级依赖与 CLI 后重新生成，框架维护的 main/runtimeGen 自动更新；v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理，不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移)，旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 +CLI 与业务 module 固定使用 `v0.20.0-preview`。v0.14–v0.19 项目先按 [配置入口迁移](CONFIG.md#v019-及更早项目迁移) 整理旧 Load，再升级依赖与 CLI 并重新生成，框架维护的 main/runtimeGen 自动更新；v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理，不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移)，旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 ## 按需深入 diff --git a/RELEASE.md b/RELEASE.md index fb60975..e826058 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,4 +1,28 @@ -# v0.19.0-preview +# v0.20.0-preview + +统一配置加载，减少生成项目重复维护的启动代码。 + +- 生成器维护 `internal/config/loadGen.go`，调用 `config.LoadProject`；项目只扩展 Config/Default/Validate，可选在 loader.go/NewLoader 注册新格式。入口、环境覆盖、日志输出规范化无需业务重复实现。 +- JSON/YAML/XML 使用同一个加载与绑定流程；Decoder 通过实例注册，无全局 init 或额外强制依赖。内置格式、字段、文件大小与层数均有明确限制。 +- `config.Error` 携带文件、字段、环境变量、可用行列及安全原因；`Invalid` / `AtField` 支持业务验证。保留内部错误链，不自动输出原值；大整数、空集合、null、默认值隔离和自定义反序列化有回归覆盖。 +- 配置解析在日志、依赖和监听初始化之前；生成/check/dry-run 使用一致入口和文件所有权规则。 +- **迁移要求：旧项目先整理手写 Load，再重新生成。** 不静默覆盖业务文件。低层配置 API 保留；新项目不再复制手写加载流程。固定 Go 数组严格检查长度，YAML alias/merge/custom tag 不再接受，避免模糊覆盖和扩展行为。 + +```sh +go get git.yuntuo.hk/k2safe/go-omni@v0.20.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.20.0-preview +# 先按 CONFIG.md 迁移旧 Load，再执行： +go-omni api generate +go-omni api generate --check +go test ./... +go build . +``` + +完整用法、格式、脱敏边界和迁移见 [统一配置指南](CONFIG.md)，可运行例子见 [examples/configuration](examples/configuration/README.md)。回归、模糊测试与启动加载性能见 [验收记录](reports/configurationAcceptance.md)。仍为 preview；项目的实际配置和业务依赖应自行验收。 + +--- + +# v0.19.0-preview（历史版本） 将可选结构转换改为直接 Go 调用，减少业务项目需要维护的配置。普通接口继续使用 `.api` 生成 DTO，不要求增加独立模型或任何拷贝规则。 diff --git a/cmd/go-omni/README.md b/cmd/go-omni/README.md index ea60944..2cfeaaf 100644 --- a/cmd/go-omni/README.md +++ b/cmd/go-omni/README.md @@ -14,7 +14,7 @@ v0.14 新增 `api format [--check]`、`api upgrade [--dry-run|--preview <空目 ```sh # 仓库读取权限与 PATH 配置见根目录 README -go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.19.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.20.0-preview go-omni new --module example.com/hello ./hello # 从自己的契约生成；service.api 及其相对 import 文件需要在本地存在 @@ -49,14 +49,14 @@ main、Gen.go、OpenAPI 和 manifest 由生成器维护；业务文件按字节 ## 版本与迁移 -CLI 与框架依赖固定使用 v0.19.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod，不添加本地 replace；`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换： +CLI 与框架依赖固定使用 v0.20.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod，不添加本地 replace；`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换： ```sh # 在 go-omni 源码仓库中执行 go run ./cmd/go-omni new --module example.com/hello --framework-dir . ../hello ``` -当前模板与依赖以 v0.19.0-preview 为基线，继续使用 manifest v2；GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级)：原始模板可自动替换，含业务修改的基础文件按预览合并；通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务，不使用旧 `add` 命令的手动路由装配。 +当前模板与依赖以 v0.20.0-preview 为基线，继续使用 manifest v2；GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级)：原始模板可自动替换，含业务修改的基础文件按预览合并；通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务，不使用旧 `add` 命令的手动路由装配。 ## 数据库迁移命令 @@ -69,3 +69,7 @@ go-omni migrate --dir migrations --apply ``` SQL 命名示例 `0001_createUsers.up.sql`；预览只显示版本和 SHA-256，不连接数据库。失败保留 dirty 状态，不自动 down/force。细节见 [扩展模块](../../ADVANCED.md)。 + +### v0.20 配置入口 + +新生成 `internal/config/loadGen.go` 调用统一加载，纳入生成检查；项目 config.go 只维护字段、默认值和校验，loader.go 用于注册额外格式。JSON/YAML/XML 内置。旧手写 Load 会阻止生成，请先按 [迁移步骤](../../CONFIG.md#v019-及更早项目迁移) 整理，不会自动覆盖业务代码。 diff --git a/cmd/go-omni/apigen/configLoader_test.go b/cmd/go-omni/apigen/configLoader_test.go new file mode 100644 index 0000000..dc8e467 --- /dev/null +++ b/cmd/go-omni/apigen/configLoader_test.go @@ -0,0 +1,113 @@ +package apigen + +import ( + "context" + "encoding/json" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestGeneratedConfigurationConsumer(t *testing.T) { + if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") != "1" { + t.Skip("set GO_OMNI_SCAFFOLD_INTEGRATION=1") + } + dir := filepath.Join(t.TempDir(), "app") + api := filepath.Join(t.TempDir(), "service.api") + writeFile(t, api, "service configuration {\n@handler ping\nget /ping\n}\n") + o := Options{API: api, Dir: dir, Module: "example.com/configuration", FrameworkDir: frameworkRoot(t)} + if err := generateManagedTestProject(t, o); err != nil { + t.Fatal(err) + } + main := readFile(t, filepath.Join(dir, "main.go")) + // The business only changes fields/defaults/validation, not the loading flow. + writeFile(t, filepath.Join(dir, "internal/config/config.go"), `package config + import("git.yuntuo.hk/k2safe/go-omni/config";"git.yuntuo.hk/k2safe/go-omni/logx") + type Config struct { Server config.Config `+"`json:\"server\"`"+`;Limit int `+"`json:\"limit\" envDefault:\"10\"`"+` } + func Default()Config{ c:=config.Default();c.Log.File="./var/log/app.log";c.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:10};return Config{Server:c} } + func(c Config)Validate()error{if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)};if c.Limit<0{return config.Invalid("limit","must not be negative")};return nil} + `) + writeFile(t, filepath.Join(dir, "internal/config/loader.go"), `package config + import omniConfig "git.yuntuo.hk/k2safe/go-omni/config" + func NewLoader()(*omniConfig.Loader,error){return omniConfig.NewLoader(omniConfig.WithDecoder("custom",omniConfig.JSONDecoder{}))} + `) + svc := filepath.Join(dir, "internal/svc/serviceContext.go") + writeFile(t, svc, strings.Replace(readFile(t, svc), "return nil", "panic(\"dependencies must not initialize on invalid config\")", 1)) + writeFile(t, filepath.Join(dir, "consumer_test.go"), readFile(t, "testdata/configuration/consumer_test.go")) + runConsumerGo(t, dir, "mod", "tidy") + runConsumerGo(t, dir, "test", "-race", "-count=1", "./...") + runConsumerGo(t, dir, "build", ".") + o.API = filepath.Join(dir, "api/service.api") + before := snapshot(t, dir) + for _, mode := range []string{"dry", "generate", "check"} { + o.DryRun = mode == "dry" + o.Check = mode == "check" + if err := generateManagedTestProject(t, o); err != nil { + t.Fatal(err) + } + } + if !sameFiles(before, snapshot(t, dir)) { + t.Fatal("business files overwritten") + } + if main != readFile(t, filepath.Join(dir, "main.go")) { + t.Fatal("main changed") + } + load := filepath.Join(dir, "internal/config/loadGen.go") + original := readFile(t, load) + writeFile(t, load, original+"\n// drift\n") + if err := generateManagedTestProject(t, o); err == nil { + t.Fatal("loadGen drift accepted") + } + writeFile(t, load, original) + if os.Getenv("GO_OMNI_RELEASE_CLI") != "" { + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, "go", "list", "-m", "-json", frameworkModule) + cmd.Dir = dir + cmd.Env = append(os.Environ(), "GOWORK=off") + data, err := cmd.CombinedOutput() + if err != nil { + t.Fatal(err, string(data)) + } + var dependency struct { + Version, Sum string + Replace any + } + if err = json.Unmarshal(data, &dependency); err != nil { + t.Fatal(err) + } + if dependency.Version != os.Getenv("GO_OMNI_RELEASE_VERSION") || dependency.Replace != nil || dependency.Sum == "" { + t.Fatal("not a remote consumer") + } + t.Logf("remote framework %s checksum %s; no local replace", dependency.Version, dependency.Sum) + } +} +func TestConfigLoadMigrationGuard(t *testing.T) { + dir := filepath.Join(t.TempDir(), "app") + o := Options{Dir: dir, Module: "example.com/upgrade", FrameworkDir: frameworkRoot(t)} + if _, err := New(o); err != nil { + t.Fatal(err) + } + o.API = filepath.Join(dir, "api/service.api") + matches, err := filepath.Glob(filepath.Join(dir, "api/*.api")) + if err != nil || len(matches) != 1 { + t.Fatal(matches, err) + } + o.API = matches[0] + writeFile(t, filepath.Join(dir, "internal/config/legacy.go"), "package config\nfunc Load(path string)(Config,error){return Default(),nil}\n") + before := snapshot(t, dir) + for _, mode := range []string{"dry", "generate", "check"} { + o.DryRun = mode == "dry" + o.Check = mode == "check" + _, err := Generate(o) + if err == nil || !strings.Contains(err.Error(), "CONFIG.md") { + t.Fatal("missing migration diagnostic", err) + } + if !sameFiles(before, snapshot(t, dir)) { + t.Fatal("migration guard changed files") + } + } +} diff --git a/cmd/go-omni/apigen/generate.go b/cmd/go-omni/apigen/generate.go index 8509872..d154939 100644 --- a/cmd/go-omni/apigen/generate.go +++ b/cmd/go-omni/apigen/generate.go @@ -19,7 +19,7 @@ import ( ) const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" -const minimumVersion = "v0.19.0-preview" +const minimumVersion = "v0.20.0-preview" const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n" type Options struct { @@ -106,6 +106,9 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) { if err = checkRuntimeExtension(directory); err != nil { return Output{}, err } + if err = checkConfigLoadExtension(directory); err != nil { + return Output{}, err + } for _, r := range c.Spec.Routes { if !r.Manual { path := filepath.Join(directory, "internal/handler", r.Group, lower(r.Handler)+"Manual.go") @@ -152,7 +155,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) { if !filepath.IsAbs(path) { path = filepath.Join(directory, path) } - if source, err := os.ReadFile(filepath.Join(path, "config", "env.go")); err == nil && strings.Contains(string(source), "func ServerEnvAliases(") { + if source, err := os.ReadFile(filepath.Join(path, "config", "loader.go")); err == nil && strings.Contains(string(source), "func NewLoader(") { localCurrent = true } } diff --git a/cmd/go-omni/apigen/templates.go b/cmd/go-omni/apigen/templates.go index 1c53758..c047f19 100644 --- a/cmd/go-omni/apigen/templates.go +++ b/cmd/go-omni/apigen/templates.go @@ -58,6 +58,8 @@ Unimplemented/nil factories fail startup and clean registered dependencies. Existing projects retain middleware: handlers until explicitly migrated. Ordinary Logic uses the generated Request/Response types directly. Where an independent business type is needed, use ordinary Go or the optional copyx package. +Configuration loading is generator-managed in internal/config/loadGen.go. Extend fields/defaults/Validate in config.go; additional decoders belong in loader.go/NewLoader. JSON/YAML/XML use one pipeline. See the framework CONFIG.md. + There are no mappings/mappingFiles settings or generated mapping files in v0.19. See COPY.md in the framework repository for copy policies and migration steps, and GENERATION.md for field composition and middleware factories. diff --git a/cmd/go-omni/apigen/testdata/configuration/consumer_test.go b/cmd/go-omni/apigen/testdata/configuration/consumer_test.go new file mode 100644 index 0000000..ebfd12a --- /dev/null +++ b/cmd/go-omni/apigen/testdata/configuration/consumer_test.go @@ -0,0 +1,59 @@ +package main + +import ( + "context" + "errors" + "example.com/configuration/internal/config" + omniConfig "git.yuntuo.hk/k2safe/go-omni/config" + "os" + "path/filepath" + "strings" + "testing" +) + +func TestGeneratedConfiguration(t *testing.T) { + for ext, source := range map[string]string{ + "json": `{"limit":7}`, "yaml": "limit: 7\n", "xml": "7", "custom": `{"limit":7}`, + } { + t.Run(ext, func(t *testing.T) { + path := filepath.Join(t.TempDir(), "app."+ext) + if err := os.WriteFile(path, []byte(source), 0600); err != nil { + t.Fatal(err) + } + cfg, err := config.Load(path) + if err != nil { + t.Fatal(err) + } + if cfg.Limit != 7 { + t.Fatal(cfg.Limit) + } + t.Setenv("GO_OMNI_LIMIT", "0") + cfg, err = config.Load(path) + if err != nil || cfg.Limit != 0 { + t.Fatal(cfg.Limit, err) + } + }) + } + t.Setenv("GO_OMNI_LOG_OUTPUT", "stderr") + cfg, err := config.Load("") + if err != nil { + t.Fatal(err) + } + if cfg.Limit != 10 || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled { + t.Fatal("defaults/log normalization", cfg, err) + } + path := filepath.Join(t.TempDir(), "bad.yaml") + os.WriteFile(path, []byte("limit: -1\n"), 0600) + // Validation must run before svc.Initialize; the project initializer panics + // below if reached, so this also detects accidentally starting dependencies. + err = run(context.Background(), path) + var diagnostic *omniConfig.Error + if !errors.As(err, &diagnostic) || diagnostic.Field != "limit" || diagnostic.File != path || diagnostic.Line != 1 { + t.Fatal(err) + } + t.Setenv("GO_OMNI_LIMIT", "private-not-a-number") + _, err = config.Load(path) + if !errors.As(err, &diagnostic) || diagnostic.Env != "GO_OMNI_LIMIT" || strings.Contains(err.Error(), "private-not-a-number") { + t.Fatal(err) + } +} diff --git a/cmd/go-omni/apigen/upgrade.go b/cmd/go-omni/apigen/upgrade.go index b97588d..6c81c1e 100644 --- a/cmd/go-omni/apigen/upgrade.go +++ b/cmd/go-omni/apigen/upgrade.go @@ -30,6 +30,39 @@ type UpgradePlan struct { Module, Entry string } +// Handwritten old loaders must be migrated explicitly, never silently replaced +// or left competing with the generated entry point. +func checkConfigLoadExtension(root string) error { + dir := filepath.Join(root, "internal/config") + entries, err := os.ReadDir(dir) + if errors.Is(err, os.ErrNotExist) { + return nil + } + if err != nil { + return err + } + for _, entry := range entries { + name := entry.Name() + if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") || name == "loadGen.go" { + continue + } + rel := "internal/config/" + name + if err := safePath(root, rel); err != nil { + return err + } + f, err := goparser.ParseFile(gotoken.NewFileSet(), filepath.Join(root, rel), nil, 0) + if err != nil { + return fmt.Errorf("parse %s: %w", rel, err) + } + for _, decl := range f.Decls { + if fn, ok := decl.(*ast.FuncDecl); ok && fn.Recv == nil && fn.Name.Name == "Load" { + return fmt.Errorf("%s: migrate the project-owned Load function before generating internal/config/loadGen.go; retain Config/Default/Validate and move decoder options to NewLoader; see CONFIG.md", rel) + } + } + } + return nil +} + // PlanUpgrade compares infrastructure against the exact previous release seeds. // A custom file is never classified as safe to replace. Logic, middleware, // options and repositories outside the changed templates are left untouched. diff --git a/cmd/go-omni/apigen/write.go b/cmd/go-omni/apigen/write.go index 1d80e7d..0ecc9bb 100644 --- a/cmd/go-omni/apigen/write.go +++ b/cmd/go-omni/apigen/write.go @@ -284,6 +284,9 @@ func apply(root, module, entry string, generated, seeds map[string][]byte, check } func digest(data []byte) string { sum := sha256.Sum256(data); return hex.EncodeToString(sum[:]) } func ownedPath(name string) bool { + if name == "internal/config/loadGen.go" { + return true + } if strings.HasPrefix(name, "internal/types/dto") && strings.HasSuffix(name, "Gen.go") && goName.MatchString(strings.TrimSuffix(filepath.Base(name), ".go")) { return true } diff --git a/cmd/go-omni/main_test.go b/cmd/go-omni/main_test.go index 89848b9..ab0ff58 100644 --- a/cmd/go-omni/main_test.go +++ b/cmd/go-omni/main_test.go @@ -8,7 +8,7 @@ import ( ) func TestNewWithRemoteFrameworkVersion(t *testing.T) { - for _, flag := range [][]string{{"--framework-version", "v0.19.0-preview"}, {"--framework-version=v0.19.0-preview"}} { + for _, flag := range [][]string{{"--framework-version", "v0.20.0-preview"}, {"--framework-version=v0.20.0-preview"}} { project := filepath.Join(t.TempDir(), "billing") args := append([]string{"--module", "example.com/billing"}, flag...) if err := runNew(append(args, project)); err != nil { @@ -18,7 +18,7 @@ func TestNewWithRemoteFrameworkVersion(t *testing.T) { if err != nil { t.Fatal(err) } - if !strings.Contains(string(content), "go-omni v0.19.0-preview") || strings.Contains(string(content), "replace ") { + if !strings.Contains(string(content), "go-omni v0.20.0-preview") || strings.Contains(string(content), "replace ") { t.Fatalf("CLI did not pin remote dependency: %s", content) } } diff --git a/cmd/go-omni/scaffold/apiInfrastructure.go b/cmd/go-omni/scaffold/apiInfrastructure.go index 95af63c..339d215 100644 --- a/cmd/go-omni/scaffold/apiInfrastructure.go +++ b/cmd/go-omni/scaffold/apiInfrastructure.go @@ -21,6 +21,8 @@ func apiInfrastructure(module, database, storage, logOutput, logFormat string, l "internal/svc/runtimeGen.go": "templates/api-runtime.go.tmpl", "internal/svc/options.go": "templates/api-options.go.tmpl", "internal/config/config.go": "templates/api-config.go.tmpl", + "internal/config/loadGen.go": "templates/api-load.go.tmpl", + "internal/config/loader.go": "templates/api-loader.go.tmpl", "etc/config.yaml": "templates/modules-config.yaml.tmpl", "internal/svc/serviceContext.go": "templates/api-context.go.tmpl", } @@ -31,6 +33,8 @@ func apiInfrastructure(module, database, storage, logOutput, logFormat string, l files["internal/svc/storage.go"] = "templates/modules-storage.go.tmpl" } if legacy { + delete(files, "internal/config/loadGen.go") + delete(files, "internal/config/loader.go") delete(files, "internal/svc/runtimeGen.go") delete(files, "internal/svc/databaseOptions.go") files["internal/svc/close.go"] = "templates/api-close.go.tmpl" diff --git a/cmd/go-omni/scaffold/templates/api-config.go.tmpl b/cmd/go-omni/scaffold/templates/api-config.go.tmpl index 8471d4f..8b47982 100644 --- a/cmd/go-omni/scaffold/templates/api-config.go.tmpl +++ b/cmd/go-omni/scaffold/templates/api-config.go.tmpl @@ -1,10 +1,9 @@ package settings import ( - "os" {{if and (not .API) (ne .Storage "none")}}"strings"{{end}} "time" "git.yuntuo.hk/k2safe/go-omni/config" - "git.yuntuo.hk/k2safe/go-omni/logx" + {{if eq .LogOutput "file"}}"git.yuntuo.hk/k2safe/go-omni/logx"{{end}} {{if eq .Database "mysql"}}mysqladapter "git.yuntuo.hk/k2safe/go-omni/database/mysql"{{end}} {{if ne .Storage "none"}}"errors" {{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}} @@ -35,22 +34,10 @@ func Default() Config { }{{end}}} } -// Load applies defaults -> file -> typed environment -> validation. -// New fields automatically use GO_OMNI_, or an explicit env tag. -func Load(path string)(Config,error){ - cfg:=Default();var err error - cfg,err=config.ApplyDefaults(cfg);if err!=nil{return Config{},err} - if path!="" {cfg,err=config.DecodeFile(path,cfg);if err!=nil{return Config{},err}} - cfg,err=config.OverlayEnv(cfg,config.EnvOptions{Prefix:"GO_OMNI_",Empty:"ignore",Aliases:config.ServerEnvAliases("GO_OMNI_","server")});if err!=nil{return Config{},err} - output,outputSet:=os.LookupEnv("GO_OMNI_SERVER_LOG_OUTPUT");if !outputSet {output=os.Getenv("GO_OMNI_LOG_OUTPUT")} - file,fileSet:=os.LookupEnv("GO_OMNI_SERVER_LOG_FILE");if !fileSet {file=os.Getenv("GO_OMNI_LOG_FILE")} - if output!="" && file=="" {cfg.Server.Log.File="";cfg.Server.Log.Rotation=logx.RotationConfig{}} - return cfg,cfg.Validate() -} func(c Config)Validate()error{ - if err:=c.Server.Validate();err!=nil{return err} - {{if eq .Database "mysql"}}if err:=c.Database.Validate();err!=nil{return err}{{end}} - {{if ne .Storage "none"}}if err:=c.Storage.Validate();err!=nil{return err}{{end}} + if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)} + {{if eq .Database "mysql"}}if err:=c.Database.Validate();err!=nil{return config.AtField("database",err)}{{end}} + {{if ne .Storage "none"}}if err:=c.Storage.Validate();err!=nil{return config.AtField("storage",err)}{{end}} return nil } {{if ne .Storage "none"}} diff --git a/cmd/go-omni/scaffold/templates/api-load.go.tmpl b/cmd/go-omni/scaffold/templates/api-load.go.tmpl new file mode 100644 index 0000000..51b82fe --- /dev/null +++ b/cmd/go-omni/scaffold/templates/api-load.go.tmpl @@ -0,0 +1,13 @@ +package config + +import omniConfig "git.yuntuo.hk/k2safe/go-omni/config" + +// Load is maintained by the generator. Extend Config/Default/Validate in +// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline. +func Load(path string) (Config, error) { + loader,err:=NewLoader() + if err!=nil{return Config{},err} + return omniConfig.LoadProject(path,Default(),omniConfig.EnvOptions{ + Prefix:"GO_OMNI_",Empty:"ignore",Aliases:omniConfig.ServerEnvAliases("GO_OMNI_","server"), + },loader) +} diff --git a/cmd/go-omni/scaffold/templates/api-loader.go.tmpl b/cmd/go-omni/scaffold/templates/api-loader.go.tmpl new file mode 100644 index 0000000..b5bbfad --- /dev/null +++ b/cmd/go-omni/scaffold/templates/api-loader.go.tmpl @@ -0,0 +1,7 @@ +package config + +import omniConfig "git.yuntuo.hk/k2safe/go-omni/config" + +// NewLoader is project-owned. JSON/YAML/XML work without changes. Register +// additional format decoders here only when the project needs them. +func NewLoader() (*omniConfig.Loader,error) { return omniConfig.NewLoader() } diff --git a/cmd/go-omni/scaffold/version.go b/cmd/go-omni/scaffold/version.go index 45b5802..599f4f7 100644 --- a/cmd/go-omni/scaffold/version.go +++ b/cmd/go-omni/scaffold/version.go @@ -10,7 +10,7 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" // Source builds can carry an unpublished VCS version (including +dirty). // Keep this fallback in sync with the next release; installed CLIs use their // downloaded module version, which also has a module checksum. -const defaultFrameworkVersion = "v0.19.0-preview" +const defaultFrameworkVersion = "v0.20.0-preview" // This target is not a claim that the release already exists. Source users // select a local framework checkout until the module templates are published. diff --git a/config/bind.go b/config/bind.go new file mode 100644 index 0000000..a0aa212 --- /dev/null +++ b/config/bind.go @@ -0,0 +1,320 @@ +package config + +import ( + "encoding/json" + "fmt" + "reflect" + "sort" + "strings" +) + +var jsonUnmarshalerType = reflect.TypeOf((*json.Unmarshaler)(nil)).Elem() + +type documentBinder struct { + doc Document + nodes int +} + +func (b *documentBinder) step(path string, depth int) error { + b.nodes++ + if depth > maxConfigDepth || b.nodes > maxConfigNodes { + return b.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil) + } + return nil +} + +func bindDocument[T any](doc Document, defaults T) (T, error) { + var zero T + base := reflect.ValueOf(&defaults).Elem() + if base.Kind() != reflect.Struct && (base.Kind() != reflect.Pointer || base.Type().Elem().Kind() != reflect.Struct) { + return zero, &Error{Reason: "project configuration must be a struct"} + } + result, err := cloneConfiguration(base, 0) + if err != nil { + return zero, &Error{Reason: "cannot clone configuration defaults", cause: err} + } + b := documentBinder{doc: doc} + if err = b.bind(doc.Value, result, "", 0, false); err != nil { + return zero, err + } + return result.Interface().(T), nil +} + +func (b *documentBinder) decode(v any, dst reflect.Value, path string) (err error) { + defer func() { + if recover() != nil { + err = b.doc.diagnostic(path, "custom value decoder panicked", nil) + } + }() + raw, err := json.Marshal(v) + if err != nil { + return b.doc.diagnostic(path, "unsupported configuration value", err) + } + if err = json.Unmarshal(raw, dst.Addr().Interface()); err != nil { + return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err) + } + return nil +} +func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int, quoted bool) error { + if err := b.step(path, depth); err != nil { + return err + } + if v == nil { + return b.decode(nil, dst, path) + } // Preserve encoding/json null semantics. + if dst.Kind() == reflect.Pointer { + if dst.IsNil() { + dst.Set(reflect.New(dst.Type().Elem())) + } + return b.bind(v, dst.Elem(), path, depth+1, quoted) + } + if dst.Type() == durationType { + var raw string + switch x := v.(type) { + case string: + raw = x + case Text: + raw = string(x) + default: + return b.decode(v, dst, path) + } + if err := parseEnvValue(dst, strings.TrimSpace(raw)); err != nil { + return b.doc.diagnostic(path, "expected a duration such as 5s", err) + } + return nil + } + if dst.CanAddr() && dst.Addr().Type().Implements(jsonUnmarshalerType) { + normalized, err := b.dynamic(v, path, depth+1) + if err != nil { + return err + } + return b.decode(normalized, dst, path) + } + if dst.CanAddr() && dst.Addr().Type().Implements(textType) { + if text, ok := v.(Text); ok { + v = string(text) + } + return b.decode(v, dst, path) + } + if text, ok := v.(Text); ok { + if dst.Kind() == reflect.String || dst.Kind() == reflect.Interface || (dst.Kind() == reflect.Slice && dst.Type().Elem().Kind() == reflect.Uint8) { + return b.decode(string(text), dst, path) + } + switch dst.Kind() { + case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64: + if err := parseEnvValue(dst, strings.TrimSpace(string(text))); err != nil { + return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err) + } + return nil + default: + return b.doc.diagnostic(path, "expected an object or array, not scalar text", nil) + } + } + if quoted { + s, ok := v.(string) + if !ok { + return b.doc.diagnostic(path, "expected a JSON-encoded string", nil) + } + if err := json.Unmarshal([]byte(s), dst.Addr().Interface()); err != nil { + return b.doc.diagnostic(path, "invalid string-encoded value", err) + } + return nil + } + switch dst.Kind() { + case reflect.Struct: + m, ok := v.(map[string]any) + if !ok { + return b.doc.diagnostic(path, "expected an object", nil) + } + fields := configurationFields(dst.Type()) + used := map[string]bool{} + for _, name := range sortedKeys(m) { + child := joinField(path, name) + f, ok := fields[name] + canonical := name + if !ok { + for n, field := range fields { + if strings.EqualFold(n, name) { + if ok { + return b.doc.diagnostic(child, "ambiguous field name", nil) + } + f, ok, canonical = field, true, n + } + } + } + if !ok { + return b.doc.diagnostic(child, "unknown field", nil) + } + if used[canonical] { + return b.doc.diagnostic(child, "duplicate field binding", nil) + } + used[canonical] = true + target := dst + for _, index := range f.Index { + if target.Kind() == reflect.Pointer { + if target.IsNil() { + if !target.CanSet() { + return b.doc.diagnostic(child, "field cannot be assigned", nil) + } + target.Set(reflect.New(target.Type().Elem())) + } + target = target.Elem() + } + target = target.Field(index) + } + if !target.CanSet() { + return b.doc.diagnostic(child, "field cannot be assigned", nil) + } + if err := b.bind(m[name], target, child, depth+1, strings.Contains(","+f.Tag.Get("json")+",", ",string,")); err != nil { + return err + } + } + return nil + case reflect.Map: + m, ok := v.(map[string]any) + if !ok || dst.Type().Key().Kind() != reflect.String { + return b.doc.diagnostic(path, "expected an object with string keys", nil) + } + if dst.IsNil() { + dst.Set(reflect.MakeMap(dst.Type())) + } + for _, key := range sortedKeys(m) { + value := reflect.New(dst.Type().Elem()).Elem() + if err := b.bind(m[key], value, joinField(path, key), depth+1, false); err != nil { + return err + } + k := reflect.New(dst.Type().Key()).Elem() + k.SetString(key) + dst.SetMapIndex(k, value) + } + return nil + case reflect.Slice, reflect.Array: + if _, ok := v.(string); ok && dst.Kind() == reflect.Slice && dst.Type().Elem().Kind() == reflect.Uint8 { + return b.decode(v, dst, path) + } + items, ok := v.([]any) + if !ok { + return b.doc.diagnostic(path, "expected an array", nil) + } + if dst.Kind() == reflect.Array && len(items) != dst.Len() { + return b.doc.diagnostic(path, "array length does not match its declared type", nil) + } + if dst.Kind() == reflect.Slice { + dst.Set(reflect.MakeSlice(dst.Type(), len(items), len(items))) + } + for i, value := range items { + if err := b.bind(value, dst.Index(i), fmt.Sprintf("%s[%d]", path, i), depth+1, false); err != nil { + return err + } + } + return nil + case reflect.Interface: + v, err := b.dynamic(v, path, depth+1) + if err != nil { + return err + } + if dst.Type().NumMethod() == 0 { + dst.Set(reflect.ValueOf(v)) + return nil + } + return b.decode(v, dst, path) + default: + return b.decode(v, dst, path) + } +} +func sortedKeys(m map[string]any) []string { + keys := make([]string, 0, len(m)) + for k := range m { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} +func (b *documentBinder) dynamic(v any, path string, depth int) (any, error) { + if err := b.step(path, depth); err != nil { + return nil, err + } + switch x := v.(type) { + case Text: + return string(x), nil + case map[string]any: + out := make(map[string]any, len(x)) + for _, k := range sortedKeys(x) { + c, err := b.dynamic(x[k], joinField(path, k), depth+1) + if err != nil { + return nil, err + } + out[k] = c + } + return out, nil + case []any: + out := make([]any, len(x)) + for i, v := range x { + c, err := b.dynamic(v, fmt.Sprintf("%s[%d]", path, i), depth+1) + if err != nil { + return nil, err + } + out[i] = c + } + return out, nil + default: + return v, nil + } +} + +// Match encoding/json's field promotion: the shallowest declaration wins; +// explicit tags win ties, and ambiguous fields remain unavailable. +func configurationFields(t reflect.Type) map[string]reflect.StructField { + type candidate struct { + field reflect.StructField + tagged bool + } + candidates := map[string][]candidate{} + var visit func(reflect.Type, []int, map[reflect.Type]bool) + visit = func(current reflect.Type, prefix []int, visiting map[reflect.Type]bool) { + if visiting[current] { + return + } + visiting[current] = true + defer delete(visiting, current) + for i := 0; i < current.NumField(); i++ { + f := current.Field(i) + name := strings.Split(f.Tag.Get("json"), ",")[0] + if name == "-" { + continue + } + child := f.Type + if child.Kind() == reflect.Pointer { + child = child.Elem() + } + if f.PkgPath != "" && (!f.Anonymous || child.Kind() != reflect.Struct) { + continue + } + f.Index = append(append([]int(nil), prefix...), i) + if name == "" && f.Anonymous && child.Kind() == reflect.Struct { + visit(child, f.Index, visiting) + continue + } + tagged := name != "" + if name == "" { + name = f.Name + } + candidates[name] = append(candidates[name], candidate{f, tagged}) + } + } + visit(t, nil, map[reflect.Type]bool{}) + fields := map[string]reflect.StructField{} + for name, list := range candidates { + sort.Slice(list, func(i, j int) bool { + if len(list[i].field.Index) != len(list[j].field.Index) { + return len(list[i].field.Index) < len(list[j].field.Index) + } + return list[i].tagged && !list[j].tagged + }) + if len(list) > 1 && len(list[0].field.Index) == len(list[1].field.Index) && list[0].tagged == list[1].tagged { + continue + } + fields[name] = list[0].field + } + return fields +} diff --git a/config/config.go b/config/config.go index 960348e..839e96d 100644 --- a/config/config.go +++ b/config/config.go @@ -62,19 +62,19 @@ func Default() Config { // Validate checks configuration values before an application starts. func (c Config) Validate() error { if err := c.Lifecycle.Validate(); err != nil { - return err + return AtField("lifecycle", err) } if err := c.Request.Validate(); err != nil { - return err + return AtField("request", err) } if err := c.Log.Validate(); err != nil { - return err + return AtField("log", err) } if err := c.Endpoints.Validate(); err != nil { return err } if err := c.Errors.Validate(); err != nil { - return err + return AtField("errors", err) } encoder := c.ResponseEncoder if encoder == nil { @@ -90,7 +90,7 @@ func (c Config) Validate() error { return err } if err := c.Response.Validate(); err != nil { - return err + return AtField("response", err) } if c.SlowRequestThreshold < 0 { return &ConfigError{Field: "slowRequestThreshold", Message: "must be nonnegative"} @@ -99,13 +99,13 @@ func (c Config) Validate() error { return &ConfigError{Field: "address", Message: "is required"} } if strings.TrimSpace(c.RequestIDHeader) == "" { - return &ConfigError{Field: "request_id_header", Message: "is required"} + return &ConfigError{Field: "requestIdHeader", Message: "is required"} } if c.ShutdownTimeout <= 0 { - return &ConfigError{Field: "shutdown_timeout", Message: "must be positive"} + return &ConfigError{Field: "shutdownTimeout", Message: "must be positive"} } if _, err := httpx.NewClientIPResolver(c.TrustedProxies...); err != nil { - return &ConfigError{Field: "trusted_proxies", Message: err.Error()} + return &Error{Field: "trustedProxies", Reason: "expected IP addresses or CIDRs", cause: err} } return c.ValidateTransport() } diff --git a/config/config_test.go b/config/config_test.go index c7a2e85..c09e6a1 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -112,7 +112,7 @@ func TestLoadFromEnvRejectsInvalidValues(t *testing.T) { t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "1s") t.Setenv("GO_OMNI_TRUSTED_PROXIES", "proxy.example.com") - if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "trusted_proxies") { + if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "trustedProxies") { t.Fatalf("expected invalid trusted proxy error, got %v", err) } t.Setenv("GO_OMNI_TRUSTED_PROXIES", "") @@ -123,7 +123,7 @@ func TestLoadFromEnvRejectsInvalidValues(t *testing.T) { } cfg = Default() cfg.RequestIDHeader = " " - if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "request_id_header") { + if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "requestIdHeader") { t.Fatalf("expected invalid request header error, got %v", err) } } diff --git a/config/decoders.go b/config/decoders.go new file mode 100644 index 0000000..44c19b8 --- /dev/null +++ b/config/decoders.go @@ -0,0 +1,398 @@ +package config + +import ( + "bytes" + "encoding/json" + "encoding/xml" + "errors" + "fmt" + "io" + "regexp" + "sort" + "strconv" + "strings" + "unicode/utf8" + + "gopkg.in/yaml.v3" +) + +type JSONDecoder struct{} +type YAMLDecoder struct{} +type XMLDecoder struct{} + +type parseState struct { + doc Document + nodes int +} + +func newParseState() *parseState { return &parseState{doc: Document{Positions: map[string]Position{}}} } +func (p *parseState) enter(path string, pos Position, depth int) error { + p.nodes++ + p.doc.Positions[path] = pos + if depth > maxConfigDepth || p.nodes > maxConfigNodes { + return p.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil) + } + return nil +} + +type sourcePositions struct{ starts []int } + +func newSourcePositions(data []byte) sourcePositions { + p := sourcePositions{starts: []int{0}} + for i, c := range data { + if c == '\n' { + p.starts = append(p.starts, i+1) + } + } + return p +} +func (p sourcePositions) at(offset int64) Position { + if offset < 0 { + offset = 0 + } + i := sort.Search(len(p.starts), func(i int) bool { return int64(p.starts[i]) > offset }) - 1 + return Position{Line: i + 1, Column: int(offset) - p.starts[i] + 1} +} + +var yamlLine = regexp.MustCompile(`line ([0-9]+)`) + +func yamlSyntax(err error) *Error { + d := &Error{Reason: "invalid YAML syntax", cause: err} + if m := yamlLine.FindStringSubmatch(err.Error()); len(m) == 2 { + d.Line, _ = strconv.Atoi(m[1]) + d.Column = 1 + } + return d +} +func (JSONDecoder) Decode(data []byte) (Document, error) { + p := newParseState() + if len(data) > maxConfigBytes { + return p.doc, &Error{Reason: "file exceeds 1 MiB"} + } + if !utf8.Valid(data) { + return p.doc, &Error{Reason: "configuration must be UTF-8"} + } + positions := newSourcePositions(data) + d := json.NewDecoder(bytes.NewReader(data)) + d.UseNumber() + var parse func(string, int) (any, error) + parse = func(path string, depth int) (any, error) { + offset := d.InputOffset() + for offset < int64(len(data)) && strings.ContainsRune(" \t\r\n:,", rune(data[offset])) { + offset++ + } + if err := p.enter(path, positions.at(offset), depth); err != nil { + return nil, err + } + token, err := d.Token() + if err != nil { + pos := positions.at(d.InputOffset()) + var syntax *json.SyntaxError + if errors.As(err, &syntax) { + pos = positions.at(syntax.Offset - 1) + } + return nil, &Error{Field: path, Line: pos.Line, Column: pos.Column, Reason: "invalid JSON syntax", cause: err} + } + if delim, ok := token.(json.Delim); ok { + switch delim { + case '{': + out := map[string]any{} + for d.More() { + k, e := d.Token() + if e != nil { + return nil, p.doc.diagnostic(path, "invalid object key", e) + } + key, ok := k.(string) + if !ok { + return nil, p.doc.diagnostic(path, "object key must be a string", nil) + } + child := joinField(path, key) + if _, exists := out[key]; exists { + return nil, p.doc.diagnostic(child, "duplicate field", nil) + } + out[key], e = parse(child, depth+1) + if e != nil { + return nil, e + } + } + if _, err = d.Token(); err != nil { + return nil, p.doc.diagnostic(path, "unterminated object", err) + } + return out, nil + case '[': + out := []any{} + for d.More() { + v, e := parse(fmt.Sprintf("%s[%d]", path, len(out)), depth+1) + if e != nil { + return nil, e + } + out = append(out, v) + } + if _, err = d.Token(); err != nil { + return nil, p.doc.diagnostic(path, "unterminated array", err) + } + return out, nil + default: + return nil, p.doc.diagnostic(path, "unexpected JSON delimiter", nil) + } + } + return token, nil + } + var err error + p.doc.Value, err = parse("", 0) + if err == nil { + if _, e := d.Token(); e != io.EOF { + err = p.doc.diagnostic("", "exactly one JSON document required", e) + } + } + return p.doc, err +} + +func (YAMLDecoder) Decode(data []byte) (Document, error) { + p := newParseState() + if len(data) > maxConfigBytes { + return p.doc, &Error{Reason: "file exceeds 1 MiB"} + } + if !utf8.Valid(data) { + return p.doc, &Error{Reason: "configuration must be UTF-8"} + } + d := yaml.NewDecoder(bytes.NewReader(data)) + var root yaml.Node + if err := d.Decode(&root); err != nil { + return p.doc, yamlSyntax(err) + } + var extra yaml.Node + if err := d.Decode(&extra); err != io.EOF { + return p.doc, &Error{Reason: "exactly one YAML document required", cause: err} + } + var walk func(*yaml.Node, string, int) (any, error) + walk = func(n *yaml.Node, path string, depth int) (any, error) { + if err := p.enter(path, Position{n.Line, n.Column}, depth); err != nil { + return nil, err + } + switch n.Kind { + case yaml.MappingNode: + out := map[string]any{} + for i := 0; i < len(n.Content); i += 2 { + k, v := n.Content[i], n.Content[i+1] + if k.Tag != "!!str" { + return nil, p.doc.diagnostic(path, "mapping keys must be strings; merge keys are not supported", nil) + } + child := joinField(path, k.Value) + if _, ok := out[k.Value]; ok { + return nil, &Error{Field: child, Line: k.Line, Column: k.Column, Reason: "duplicate field"} + } + x, err := walk(v, child, depth+1) + if err != nil { + return nil, err + } + out[k.Value] = x + } + return out, nil + case yaml.SequenceNode: + out := make([]any, len(n.Content)) + for i, c := range n.Content { + var err error + out[i], err = walk(c, fmt.Sprintf("%s[%d]", path, i), depth+1) + if err != nil { + return nil, err + } + } + return out, nil + case yaml.AliasNode: + return nil, p.doc.diagnostic(path, "YAML aliases are not supported", nil) + case yaml.ScalarNode: + switch n.Tag { + case "!!str", "!!timestamp": + return n.Value, nil + case "!!null": + return nil, nil + case "!!bool", "!!int", "!!float": + var v any + if err := n.Decode(&v); err != nil { + return nil, p.doc.diagnostic(path, "invalid YAML scalar", err) + } + raw, err := json.Marshal(v) + if err != nil { + return nil, p.doc.diagnostic(path, "number must be finite", err) + } + var out any + dec := json.NewDecoder(bytes.NewReader(raw)) + dec.UseNumber() + if err = dec.Decode(&out); err != nil { + return nil, p.doc.diagnostic(path, "invalid YAML scalar", err) + } + return out, nil + default: + return nil, p.doc.diagnostic(path, "unsupported YAML tag", nil) + } + } + return nil, p.doc.diagnostic(path, "unsupported YAML node", nil) + } + if len(root.Content) != 1 { + return p.doc, &Error{Reason: "one nonempty YAML document required"} + } + var err error + p.doc.Value, err = walk(root.Content[0], "", 0) + return p.doc, err +} + +// XML uses a config root, element field names, type="array" with item children, +// type="object" for empty objects, and type="null" for explicit null. Scalar +// text is bound using the destination type; optional number/bool/string types +// let dynamic any values preserve their intended JSON type. +func (XMLDecoder) Decode(data []byte) (Document, error) { + p := newParseState() + if len(data) > maxConfigBytes { + return p.doc, &Error{Reason: "file exceeds 1 MiB"} + } + if !utf8.Valid(data) { + return p.doc, &Error{Reason: "configuration must be UTF-8"} + } + d := xml.NewDecoder(bytes.NewReader(data)) + var parse func(xml.StartElement, string, int, Position) (any, error) + parse = func(start xml.StartElement, path string, depth int, pos Position) (any, error) { + if err := p.enter(path, pos, depth); err != nil { + return nil, err + } + if start.Name.Space != "" { + return nil, p.doc.diagnostic(path, "XML namespaces are not supported", nil) + } + kind := "" + attrSeen := false + for _, a := range start.Attr { + if a.Name.Space != "" || a.Name.Local != "type" || attrSeen { + return nil, p.doc.diagnostic(path, "only one XML type attribute is supported", nil) + } + kind = a.Value + attrSeen = true + } + if depth == 0 && kind == "" { + kind = "object" + } + switch kind { + case "", "object", "array", "null", "string", "number", "bool": + default: + return nil, p.doc.diagnostic(path, "unsupported XML type", nil) + } + children := map[string]any{} + array := []any{} + var text strings.Builder + for { + line, column := d.InputPos() + token, err := d.Token() + if err != nil { + return nil, &Error{Field: path, Line: line, Column: column, Reason: "invalid XML syntax", cause: err} + } + switch x := token.(type) { + case xml.StartElement: + if kind != "" && kind != "object" && kind != "array" { + return nil, p.doc.diagnostic(path, "scalar XML element cannot contain children", nil) + } + child := joinField(path, x.Name.Local) + if kind == "array" { + if x.Name.Local != "item" { + return nil, p.doc.diagnostic(path, "XML array children must be item elements", nil) + } + child = fmt.Sprintf("%s[%d]", path, len(array)) + } else if _, ok := children[x.Name.Local]; ok { + return nil, p.doc.diagnostic(child, "duplicate field; use type=\"array\" for lists", nil) + } + v, e := parse(x, child, depth+1, Position{line, column}) + if e != nil { + return nil, e + } + if kind == "array" { + array = append(array, v) + } else { + children[x.Name.Local] = v + } + case xml.CharData: + text.Write([]byte(x)) + case xml.Comment: + case xml.EndElement: + raw := text.String() + if len(children) > 0 || kind == "array" || kind == "object" { + if strings.TrimSpace(raw) != "" { + return nil, p.doc.diagnostic(path, "mixed XML text and elements are not supported", nil) + } + if kind == "array" { + return array, nil + } + return children, nil + } + switch kind { + case "null": + if strings.TrimSpace(raw) != "" { + return nil, p.doc.diagnostic(path, "null element must be empty", nil) + } + return nil, nil + case "string": + return raw, nil + case "number": + var n any + dec := json.NewDecoder(strings.NewReader(strings.TrimSpace(raw))) + dec.UseNumber() + if err := dec.Decode(&n); err != nil { + return nil, p.doc.diagnostic(path, "invalid number", err) + } + if _, ok := n.(json.Number); !ok { + return nil, p.doc.diagnostic(path, "expected number", nil) + } + if _, e := dec.Token(); e != io.EOF { + return nil, p.doc.diagnostic(path, "invalid number", e) + } + return n, nil + case "bool": + b, e := strconv.ParseBool(strings.TrimSpace(raw)) + if e != nil { + return nil, p.doc.diagnostic(path, "expected boolean", e) + } + return b, nil + } + return Text(raw), nil + default: + return nil, p.doc.diagnostic(path, "XML directives and processing instructions are not supported", nil) + } + } + } + found := false + prolog := false + for { + line, column := d.InputPos() + token, err := d.Token() + if err == io.EOF { + break + } + if err != nil { + return p.doc, &Error{Line: line, Column: column, Reason: "invalid XML syntax", cause: err} + } + switch x := token.(type) { + case xml.StartElement: + if found || x.Name.Local != "config" { + return p.doc, &Error{Line: line, Column: column, Reason: "exactly one config root required"} + } + found = true + p.doc.Value, err = parse(x, "", 0, Position{line, column}) + if err != nil { + return p.doc, err + } + case xml.CharData: + if strings.TrimSpace(string(x)) != "" { + return p.doc, &Error{Reason: "unexpected text outside XML root"} + } + case xml.Comment: + case xml.ProcInst: + if found || prolog || x.Target != "xml" { + return p.doc, &Error{Reason: "XML processing instructions are not supported"} + } + prolog = true + default: + return p.doc, &Error{Reason: "XML directives are not supported"} + } + } + if !found { + return p.doc, &Error{Reason: "config root required"} + } + return p.doc, nil +} diff --git a/config/diagnostic.go b/config/diagnostic.go new file mode 100644 index 0000000..6028ccd --- /dev/null +++ b/config/diagnostic.go @@ -0,0 +1,94 @@ +package config + +import ( + "errors" + "fmt" + "strings" +) + +// Error is a value-free diagnostic. Cause may contain private parser details; +// errors.Is/As remain available, but Error never formats Cause automatically. +type Error struct { + File, Field, Env, Reason string + Line, Column int + cause error +} + +func (e *Error) Error() string { + where := "config" + if e.File != "" { + where = fmt.Sprintf("config %q", e.File) + } + if e.Line > 0 { + where += fmt.Sprintf(":%d:%d", e.Line, e.Column) + } + if e.Env != "" { + where += fmt.Sprintf(" env %q", e.Env) + } + if e.Field != "" { + where += fmt.Sprintf(" field %q", e.Field) + } + return where + ": " + e.Reason +} +func (e *Error) Unwrap() error { return e.cause } + +// DecodeError is retained for callers of errors.As from earlier releases. +type DecodeError = Error + +// Invalid reports a public, value-free validation reason for a configuration field. +func Invalid(field, reason string) error { return &ConfigError{Field: field, Message: reason} } + +// AtField prefixes a nested validation error without losing its original cause. +// Unstructured validation errors are deliberately not echoed; use Invalid when +// a custom validator needs a public reason rather than a private error message. +func AtField(field string, err error) error { + if err == nil { + return nil + } + d := validationError(err) + d.Field = joinField(field, d.Field) + return d +} +func validationError(err error) *Error { + var d *Error + if errors.As(err, &d) { + out := *d + out.cause = err + return &out + } + var c *ConfigError + if errors.As(err, &c) { + return &Error{Field: c.Field, Reason: c.Message, cause: err} + } + return &Error{Reason: "validation failed", cause: err} +} +func joinField(parent, child string) string { + if parent == "" { + return child + } + if child == "" { + return parent + } + if strings.HasPrefix(child, "[") { + return parent + child + } + return parent + "." + child +} + +// Position uses one-based source line/column numbers. Zero means unavailable. +type Position struct{ Line, Column int } + +// Document is the common decoder result. Values use JSON-compatible data: +// string, bool, json.Number, nil, []any, map[string]any. Text is available for +// formats such as XML whose scalar tokens have no inherent JSON type. +type Document struct { + Value any + Positions map[string]Position +} + +type Text string + +func (d Document) diagnostic(field, reason string, cause error) *Error { + p := d.Positions[field] + return &Error{Field: field, Reason: reason, Line: p.Line, Column: p.Column, cause: cause} +} diff --git a/config/file.go b/config/file.go index 160e4e6..d5c4836 100644 --- a/config/file.go +++ b/config/file.go @@ -1,100 +1,16 @@ package config import ( - "bytes" "context" - "encoding/json" "errors" - "io" - "os" - "path/filepath" "reflect" "strings" "sync" "time" - apierrors "git.yuntuo.hk/k2safe/go-omni/errors" "git.yuntuo.hk/k2safe/go-omni/logx" - "gopkg.in/yaml.v3" ) -// DecodeFile merges a bounded JSON/YAML document into defaults. It uses JSON -// field names for both formats, rejects unknown fields and accepts durations -// such as "3s". Application-specific validation remains with the caller. -func DecodeFile[T any](path string, defaults T) (T, error) { - result, err := decodeFile(path, defaults) - if err != nil { - var zero T - return zero, &DecodeError{cause: err} - } - return result, nil -} - -// DecodeError keeps input values out of public diagnostics while preserving -// errors.Is/As for explicitly handled underlying I/O and syntax errors. -type DecodeError struct{ cause error } - -func (e *DecodeError) Error() string { - return "config: cannot decode file; check its path, format, field names and value types" -} -func (e *DecodeError) Unwrap() error { return e.cause } -func decodeFile[T any](path string, defaults T) (T, error) { - var zero T - f, err := os.Open(path) - if err != nil { - return zero, err - } - defer f.Close() - data, err := io.ReadAll(io.LimitReader(f, (1<<20)+1)) - if err != nil { - return zero, err - } - if len(data) > 1<<20 { - return zero, errors.New("config: file exceeds 1 MiB") - } - var value any - switch strings.ToLower(filepath.Ext(path)) { - case ".yaml", ".yml": - dec := yaml.NewDecoder(bytes.NewReader(data)) - if err := dec.Decode(&value); err != nil { - return zero, err - } - var extra any - if err := dec.Decode(&extra); err != io.EOF { - return zero, errors.New("config: one YAML document required") - } - case ".json": - if err := apierrors.DecodeCatalogJSON(data, &value); err != nil { - return zero, err - } - default: - return zero, errors.New("config: expected .json, .yaml or .yml") - } - if _, ok := value.(map[string]any); !ok { - return zero, errors.New("config: root must be an object") - } - value, err = durations(value, reflect.TypeOf(defaults)) - if err != nil { - return zero, err - } - encoded, err := json.Marshal(value) - if err != nil { - return zero, err - } - // Clone configuration without serializing borrowed json:"-" capabilities. - var result T - original, err := cloneConfiguration(reflect.ValueOf(&defaults).Elem(), 0) - if err != nil { - return zero, err - } - reflect.ValueOf(&result).Elem().Set(original) - dec := json.NewDecoder(bytes.NewReader(encoded)) - dec.DisallowUnknownFields() - if err = dec.Decode(&result); err != nil { - return zero, err - } - return result, nil -} func durations(v any, t reflect.Type) (any, error) { if t == nil { return v, nil diff --git a/config/loader.go b/config/loader.go new file mode 100644 index 0000000..0675be0 --- /dev/null +++ b/config/loader.go @@ -0,0 +1,175 @@ +package config + +import ( + "errors" + "io" + "os" + "path/filepath" + "reflect" + "strings" +) + +const maxConfigBytes = 1 << 20 +const maxConfigDepth = 64 +const maxConfigNodes = 100000 + +// Decoder must not retain or mutate input and must support concurrent calls +// when its Loader is shared. It is trusted application code, not a sandbox. +type Decoder interface { + Decode([]byte) (Document, error) +} +type DecoderFunc func([]byte) (Document, error) + +func (f DecoderFunc) Decode(data []byte) (Document, error) { return f(data) } + +// Loader owns an immutable per-instance decoder registry. No global registration. +type Loader struct { + decoders map[string]Decoder + sealed bool +} +type LoaderOption func(*Loader) error + +// WithDecoder adds a new extension; builtin/duplicate extensions cannot be replaced. +func WithDecoder(extension string, decoder Decoder) LoaderOption { + return func(l *Loader) error { + if l == nil || l.decoders == nil || l.sealed { + return errors.New("config: decoder registry cannot be changed after construction") + } + ext := strings.ToLower(strings.TrimPrefix(extension, ".")) + if ext == "" || strings.ContainsAny(ext, "./\\ \t\r\n") { + return errors.New("config: invalid decoder extension") + } + if decoder == nil { + return errors.New("config: nil decoder") + } + v := reflect.ValueOf(decoder) + if (v.Kind() == reflect.Pointer || v.Kind() == reflect.Func || v.Kind() == reflect.Map || v.Kind() == reflect.Slice || v.Kind() == reflect.Interface) && v.IsNil() { + return errors.New("config: nil decoder") + } + if _, exists := l.decoders[ext]; exists { + return errors.New("config: decoder extension already registered") + } + l.decoders[ext] = decoder + return nil + } +} + +func NewLoader(options ...LoaderOption) (*Loader, error) { + l := &Loader{decoders: map[string]Decoder{"json": JSONDecoder{}, "yaml": YAMLDecoder{}, "yml": YAMLDecoder{}, "xml": XMLDecoder{}}} + for _, o := range options { + if o == nil { + return nil, errors.New("config: nil loader option") + } + if err := o(l); err != nil { + return nil, err + } + } + l.sealed = true + return l, nil +} + +var defaultLoader, _ = NewLoader() + +func selectLoader(loaders []*Loader) (*Loader, error) { + if len(loaders) == 0 { + return defaultLoader, nil + } + if len(loaders) != 1 || loaders[0] == nil { + return nil, errors.New("config: expected one non-nil loader") + } + return loaders[0], nil +} +func (l *Loader) read(path string) (doc Document, err error) { + defer func() { + if err != nil { + var d *Error + if errors.As(err, &d) { + c := *d + c.File = path + err = &c + } else { + reason := "cannot read configuration file" + if errors.Is(err, os.ErrNotExist) { + reason = "configuration file does not exist" + } else if errors.Is(err, os.ErrPermission) { + reason = "permission denied reading configuration file" + } + err = &Error{File: path, Reason: reason, cause: err} + } + } + }() + decoder := l.decoders[strings.TrimPrefix(strings.ToLower(filepath.Ext(path)), ".")] + if decoder == nil { + return doc, &Error{Reason: "unsupported configuration format"} + } + info, e := os.Stat(path) + if e != nil { + return doc, e + } + if !info.Mode().IsRegular() { + return doc, &Error{Reason: "configuration must be a regular file"} + } + f, e := os.Open(path) + if e != nil { + return doc, e + } + defer f.Close() + info, e = f.Stat() + if e != nil { + return doc, e + } + if !info.Mode().IsRegular() { + return doc, &Error{Reason: "configuration must be a regular file"} + } + data, e := io.ReadAll(io.LimitReader(f, maxConfigBytes+1)) + if e != nil { + return doc, e + } + if len(data) > maxConfigBytes { + return doc, &Error{Reason: "file exceeds 1 MiB"} + } + doc, err = decodeSafely(decoder, data) + if err != nil { + return doc, err + } + if _, ok := doc.Value.(map[string]any); !ok { + return doc, doc.diagnostic("", "root must be an object", nil) + } + return doc, nil +} +func decodeSafely(decoder Decoder, data []byte) (doc Document, err error) { + defer func() { + if recover() != nil { + err = &Error{Reason: "decoder panicked"} + } + }() + doc, err = decoder.Decode(data) + if err != nil { + var d *Error + if !errors.As(err, &d) { + err = &Error{Reason: "invalid configuration syntax", cause: err} + } + } + return +} + +// DecodeFile overlays a bounded document onto independent defaults. With no +// loader argument it accepts JSON/YAML/XML. It never opens dependencies. +func DecodeFile[T any](path string, defaults T, loaders ...*Loader) (T, error) { + var zero T + l, err := selectLoader(loaders) + if err != nil { + return zero, err + } + doc, err := l.read(path) + if err != nil { + return zero, err + } + result, err := bindDocument(doc, defaults) + if err != nil { + d := validationError(err) + d.File = path + return zero, d + } + return result, nil +} diff --git a/config/loader_test.go b/config/loader_test.go new file mode 100644 index 0000000..32519c2 --- /dev/null +++ b/config/loader_test.go @@ -0,0 +1,397 @@ +package config + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "reflect" + "strings" + "sync" + "testing" + "time" +) + +type loaderExample struct { + Server Config `json:"server"` + Upload struct { + MaxBytes int64 `json:"maxBytes" envDefault:"1024"` + } `json:"upload"` + Values []int64 `json:"values"` + Optional *string `json:"optional"` + Enabled bool `json:"enabled"` + Dynamic any `json:"dynamic"` +} + +func (c loaderExample) Validate() error { + if err := c.Server.Validate(); err != nil { + return AtField("server", err) + } + if c.Upload.MaxBytes < 0 { + return Invalid("upload.maxBytes", "must be nonnegative") + } + return nil +} +func noEnvironment(string) (string, bool) { return "", false } +func configFile(t *testing.T, ext, raw string) string { + t.Helper() + path := filepath.Join(t.TempDir(), "service."+ext) + if err := os.WriteFile(path, []byte(raw), 0600); err != nil { + t.Fatal(err) + } + return path +} + +func TestUnifiedFormatAndPrecedence(t *testing.T) { + formats := map[string]string{ + "json": `{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":7},"values":[9007199254740993],"optional":null,"enabled":true,"dynamic":{"id":9007199254740993}}`, + "yaml": "server:\n http:\n readTimeout: 2s\nupload:\n maxBytes: 7\nvalues: [9007199254740993]\noptional: null\nenabled: true\ndynamic:\n id: 9007199254740993\n", + "xml": `2s79007199254740993true9007199254740993`, + } + var reference loaderExample + for _, ext := range []string{"json", "yaml", "xml"} { + t.Run(ext, func(t *testing.T) { + old := "old" + base := loaderExample{Server: Default(), Optional: &old} + cfg, err := LoadProject(configFile(t, ext, formats[ext]), base, EnvOptions{Prefix: "APP_", Lookup: func(key string) (string, bool) { + v, ok := map[string]string{"APP_UPLOAD_MAX_BYTES": "0", "APP_ENABLED": "false"}[key] + return v, ok + }}) + if err != nil { + t.Fatal(err) + } + if cfg.Upload.MaxBytes != 0 || cfg.Enabled || cfg.Optional != nil || cfg.Values[0] != 9007199254740993 || cfg.Server.HTTP.ReadTimeout != 2*time.Second { + t.Fatal("values changed", cfg) + } + if cfg.Dynamic.(map[string]any)["id"] != json.Number("9007199254740993") { + t.Fatal("dynamic integer precision lost") + } + if ext == "json" { + reference = cfg + } else if !reflect.DeepEqual(reference, cfg) { + t.Fatal("format-dependent semantics") + } + if base.Optional == nil || *base.Optional != "old" { + t.Fatal("defaults mutated") + } + }) + } + cfg, err := LoadProject("", loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}) + if err != nil || cfg.Upload.MaxBytes != 1024 { + t.Fatal(cfg, err) + } +} + +func TestConfigDiagnosticsAndCauses(t *testing.T) { + for _, tc := range []struct { + ext, body, field string + line bool + }{ + {"json", "{\n\"upload\": {\"maxBytes\":\"private-value\"}}", "upload.maxBytes", true}, + {"yaml", "server:\n http:\n readTimeout: private-value", "server.http.readTimeout", true}, + {"xml", "\nprivate-value", "upload.maxBytes", true}, + {"json", `{"upload":{"maxBytes":-1}}`, "upload.maxBytes", true}, + {"yaml", "server:\n http:\n readTimeout: -1s", "server.http.readTimeout", true}, + {"json", `{"upload":{"typo":3}}`, "upload.typo", true}, + } { + t.Run(tc.ext+tc.field, func(t *testing.T) { + path := configFile(t, tc.ext, tc.body) + _, err := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}) + var d *Error + if !errors.As(err, &d) || d.File != path || d.Field != tc.field || d.Reason == "" || tc.line && d.Line < 1 { + t.Fatalf("missing location: %#v %v", d, err) + } + if strings.Contains(err.Error(), "private-value") { + t.Fatal("secret leaked", err) + } + }) + } + path := filepath.Join(t.TempDir(), "missing.json") + _, err := LoadProject(path, loaderExample{}, EnvOptions{}) + if !errors.Is(err, os.ErrNotExist) { + t.Fatal("I/O cause discarded", err) + } + _, err = DecodeFile(configFile(t, "json", `{"count":"private-value"}`), struct { + Count int `json:"count"` + }{}) + var typed *json.UnmarshalTypeError + if !errors.As(err, &typed) { + t.Fatal("type cause discarded", err) + } + _, err = LoadProject(configFile(t, "yaml", "upload:\n maxBytes: 7"), loaderExample{Server: Default()}, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { return "-1", k == "APP_UPLOAD_MAX_BYTES" }}) + var d *Error + if !errors.As(err, &d) || d.Env != "APP_UPLOAD_MAX_BYTES" || d.Field != "upload.maxBytes" || d.File != "" || d.Line != 0 { + t.Fatal("environment validation wrongly blamed file", err) + } +} + +func TestMalformedFormatsAndLimits(t *testing.T) { + for ext, bodies := range map[string][]string{ + "json": {`{"enabled":true,"enabled":false}`, `{} {}`, `{"upload":`, `{"enabled":true,"Enabled":false}`, strings.Repeat(`{"a":`, 70) + `0` + strings.Repeat(`}`, 70)}, + "yaml": {"enabled: true\nenabled: false", "---\n{}\n---\n{}", "a: &x [1]\nb: *x", "a:\n <<: {}", "upload: [\n"}, + "xml": {``, `&private-value;`, ``, ``, ``, `truefalse`, `texttrue`, `1`}, + } { + for i, raw := range bodies { + t.Run(fmt.Sprintf("%s%d", ext, i), func(t *testing.T) { + _, err := LoadProject(configFile(t, ext, raw), loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}) + if err == nil || strings.Contains(err.Error(), "private-value") { + t.Fatal(err) + } + }) + } + } + if _, err := DecodeFile(configFile(t, "json", strings.Repeat(" ", maxConfigBytes+1)), loaderExample{}); err == nil { + t.Fatal("unbounded file") + } + if _, err := DecodeFile(configFile(t, "json", string([]byte{'{', '"', 0xff, '"', ':', '0', '}'})), loaderExample{}); err == nil { + t.Fatal("invalid UTF-8") + } +} + +func TestDecoderRegistryAndConcurrentUse(t *testing.T) { + decoder := DecoderFunc(func([]byte) (Document, error) { + return Document{Value: map[string]any{"upload": map[string]any{"maxBytes": json.Number("42")}}}, nil + }) + l, err := NewLoader(WithDecoder("custom", decoder)) + if err != nil { + t.Fatal(err) + } + path := configFile(t, "custom", "ignored") + var wg sync.WaitGroup + for i := 0; i < 24; i++ { + wg.Add(1) + go func() { + defer wg.Done() + cfg, e := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}, l) + if e != nil || cfg.Upload.MaxBytes != 42 { + t.Error(e) + } + }() + } + wg.Wait() + if _, err := DecodeFile(path, loaderExample{}); err == nil { + t.Fatal("decoder registration leaked globally") + } + for _, options := range [][]LoaderOption{{WithDecoder("json", decoder)}, {WithDecoder("x", decoder), WithDecoder("x", decoder)}, {WithDecoder("x", DecoderFunc(nil))}} { + if _, err = NewLoader(options...); err == nil { + t.Fatal("invalid registration") + } + } + l, _ = NewLoader(WithDecoder("panic", DecoderFunc(func([]byte) (Document, error) { panic("private-value") }))) + _, err = DecodeFile(configFile(t, "panic", ""), loaderExample{}, l) + if err == nil || strings.Contains(err.Error(), "private-value") { + t.Fatal(err) + } + cycle := map[string]any{} + cycle["self"] = cycle + l, _ = NewLoader(WithDecoder("cycle", DecoderFunc(func([]byte) (Document, error) { return Document{Value: map[string]any{"dynamic": cycle}}, nil }))) + if _, err = DecodeFile(configFile(t, "cycle", ""), loaderExample{}, l); err == nil { + t.Fatal("decoder bypassed traversal limit") + } +} + +func TestUnifiedLoggerNormalization(t *testing.T) { + type project struct { + Server Config `json:"server"` + } + base := project{Server: Default()} + base.Server.Log.File = "old.log" + base.Server.Log.Rotation.Enabled = true + for _, full := range []bool{false, true} { + name := "GO_OMNI_LOG_OUTPUT" + if full { + name = "GO_OMNI_SERVER_LOG_OUTPUT" + } + cfg, err := LoadProject("", base, EnvOptions{Prefix: "GO_OMNI_", Empty: "ignore", Aliases: ServerEnvAliases("GO_OMNI_", "server"), Lookup: func(k string) (string, bool) { return "stdout", k == name }}) + if err != nil || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled || base.Server.Log.File != "old.log" { + t.Fatal(cfg, err) + } + } +} + +func FuzzConfigurationDecoders(f *testing.F) { + f.Add(uint8(0), []byte(`{"count":1}`)) + f.Add(uint8(1), []byte("count: 1")) + f.Add(uint8(2), []byte("1")) + f.Fuzz(func(t *testing.T, format uint8, data []byte) { + if len(data) > 8192 { + data = data[:8192] + } + ds := []Decoder{JSONDecoder{}, YAMLDecoder{}, XMLDecoder{}} + doc, err := ds[int(format)%len(ds)].Decode(data) + if err == nil { + _, _ = bindDocument(doc, struct { + Count int `json:"count"` + }{}) + } + }) +} + +func BenchmarkConfiguration(b *testing.B) { + for name, tc := range map[string]struct { + d Decoder + raw string + }{"json": {JSONDecoder{}, `{"count":9007199254740993,"timeout":"5s","values":[1,2,3]}`}, "yaml": {YAMLDecoder{}, "count: 9007199254740993\ntimeout: 5s\nvalues: [1,2,3]"}, "xml": {XMLDecoder{}, `90071992547409935s123`}} { + b.Run(name, func(b *testing.B) { + b.ReportAllocs() + raw := []byte(tc.raw) + for i := 0; i < b.N; i++ { + doc, err := tc.d.Decode(raw) + if err != nil { + b.Fatal(err) + } + _, err = bindDocument(doc, struct { + Count int64 `json:"count"` + Timeout time.Duration `json:"timeout"` + Values []int `json:"values"` + }{}) + if err != nil { + b.Fatal(err) + } + } + }) + } +} + +// Text-only scalars and embedded config fields use the same behavior in each format. +type testText struct{ Value string } + +func (v *testText) UnmarshalText(data []byte) error { v.Value = string(data); return nil } +func TestConfigScalarAndEmbedding(t *testing.T) { + type Fields struct { + Name string `json:"name"` + } + type C struct { + Fields + Nested Fields `json:"nested"` + Scalar testText `json:"scalar"` + Bytes []byte `json:"bytes"` + } + cases := map[string]string{ + "json": `{"name":"outer","nested":{"name":"inner"},"scalar":"text","bytes":"aGk="}`, + "yaml": "name: outer\nnested:\n name: inner\nscalar: text\nbytes: aGk=\n", + "xml": `outerinnertextaGk=`, + } + for ext, data := range cases { + t.Run(ext, func(t *testing.T) { + path := filepath.Join(t.TempDir(), "config."+ext) + os.WriteFile(path, []byte(data), 0600) + got, err := DecodeFile(path, &C{}) + if err != nil { + t.Fatal(err) + } + if got.Name != "outer" || got.Nested.Name != "inner" || got.Scalar.Value != "text" || string(got.Bytes) != "hi" { + t.Fatal(got) + } + }) + } + type Tagged struct { + Fields `json:"nested"` + } + doc, _ := (JSONDecoder{}).Decode([]byte(`{"name":"wrong"}`)) + if _, err := bindDocument(doc, Tagged{}); err == nil { + t.Fatal("tagged embed incorrectly flattened") + } +} +func TestConfigDefaultsIsolationAndEmptyValues(t *testing.T) { + type Child struct { + Name string `json:"name"` + } + type C struct { + Child *Child `json:"child"` + Items []int `json:"items"` + Labels map[string]string `json:"labels"` + Enabled bool `json:"enabled"` + } + base := C{Child: &Child{Name: "old"}, Items: []int{1}, Labels: map[string]string{"old": "yes"}, Enabled: true} + doc, _ := (JSONDecoder{}).Decode([]byte(`{"child":{"name":"new"},"labels":{"new":"yes"},"items":[],"enabled":false}`)) + got, err := bindDocument(doc, base) + if err != nil { + t.Fatal(err) + } + if got.Child.Name != "new" || got.Items == nil || len(got.Items) != 0 || got.Enabled { + t.Fatal(got) + } + if base.Child.Name != "old" || len(base.Items) != 1 || len(base.Labels) != 1 { + t.Fatal("mutated defaults") + } + bad, _ := (JSONDecoder{}).Decode([]byte(`{"child":{"name":"changed"},"items":"bad"}`)) + if _, err := bindDocument(bad, base); err == nil { + t.Fatal("invalid input accepted") + } + if base.Child.Name != "old" { + t.Fatal("failed load changed default pointer") + } + null, _ := (JSONDecoder{}).Decode([]byte(`{"child":null,"items":null,"labels":null,"enabled":null}`)) + got, err = bindDocument(null, base) + if err != nil || got.Child != nil || got.Items != nil || got.Labels != nil || !got.Enabled { + t.Fatal(got, err) + } +} +func TestDefaultStageDoesNotReadEnvironment(t *testing.T) { + reads := 0 + type C struct { + Value int `json:"value" envDefault:"3"` + } + got, err := LoadProject("", C{}, EnvOptions{Lookup: func(string) (string, bool) { reads++; return "", false }}) + if err != nil || got.Value != 3 || reads != 1 { + t.Fatal(got, err, reads) + } + d := locateError(Invalid("servers[0].timeout", "must be positive"), "a.json", Document{}, map[string]string{"servers": "APP_SERVERS"}) + if d.Env != "APP_SERVERS" || d.File != "" { + t.Fatal(d) + } +} + +func TestDecoderRegistryFrozen(t *testing.T) { + l, err := NewLoader() + if err != nil { + t.Fatal(err) + } + if err = WithDecoder("extra", JSONDecoder{})(l); err == nil { + t.Fatal("mutable registry") + } +} +func BenchmarkConfigurationStartup(b *testing.B) { + path := filepath.Join(b.TempDir(), "app.json") + if err := os.WriteFile(path, []byte(`{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":4096}}`), 0600); err != nil { + b.Fatal(err) + } + options := EnvOptions{Prefix: "BENCH_", Lookup: func(name string) (string, bool) { + if name == "BENCH_UPLOAD_MAX_BYTES" { + return "8192", true + } + return "", false + }} + defaults := loaderExample{Server: Default()} + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + if _, err := LoadProject(path, defaults, options); err != nil { + b.Fatal(err) + } + } +} +func BenchmarkConfigurationLarge(b *testing.B) { + type Entry struct { + Timeout time.Duration `json:"timeout"` + Count int `json:"count"` + } + type Large struct { + Entries []Entry `json:"entries"` + } + raw := []byte(`{"entries":[` + strings.TrimSuffix(strings.Repeat(`{"timeout":"3s","count":10},`, 1000), ",") + `]}`) + b.SetBytes(int64(len(raw))) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + doc, err := (JSONDecoder{}).Decode(raw) + if err != nil { + b.Fatal(err) + } + if _, err = bindDocument(doc, Large{}); err != nil { + b.Fatal(err) + } + } +} diff --git a/config/project.go b/config/project.go index 2a4fbdd..85ec00e 100644 --- a/config/project.go +++ b/config/project.go @@ -7,6 +7,7 @@ import ( "errors" "fmt" apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "git.yuntuo.hk/k2safe/go-omni/logx" "math" "os" "reflect" @@ -34,30 +35,114 @@ func ApplyDefaults[T any](base T) (T, error) { return projectOverlay(base, EnvOp // LoadProject applies code/tag defaults, a strict optional file, environment, // then the project's Validate method, before any logger or dependency is opened. // envDefault is only applied to zero-valued defaults before reading the file. -func LoadProject[T any](path string, defaults T, options EnvOptions) (T, error) { +func LoadProject[T any](path string, defaults T, options EnvOptions, loaders ...*Loader) (T, error) { var zero T + loader, err := selectLoader(loaders) + if err != nil { + return zero, err + } cfg, err := projectOverlay(defaults, options, true) if err != nil { return zero, err } + var doc Document if path != "" { - cfg, err = DecodeFile(path, cfg) + doc, err = loader.read(path) if err != nil { - return zero, errors.New("config: cannot decode file; check its path, format, field names and value types") + return zero, err } + cfg, err = bindDocument(doc, cfg) + if err != nil { + return zero, locateError(err, path, doc, nil) + } + } else { + value, e := cloneConfiguration(reflect.ValueOf(&cfg).Elem(), 0) + if e != nil { + return zero, &Error{Reason: "cannot clone defaults", cause: e} + } + cfg = value.Interface().(T) } - cfg, err = OverlayEnv(cfg, options) + origins := map[string]string{} + cfg, err = projectOverlay(cfg, options, false, origins) if err != nil { return zero, err } + normalizeProject(reflect.ValueOf(&cfg).Elem(), "", origins, 0) if valid, ok := any(&cfg).(interface{ Validate() error }); ok { if err = valid.Validate(); err != nil { - return zero, err + return zero, locateError(err, path, doc, origins) } } return cfg, nil } +func locateError(err error, path string, doc Document, origins map[string]string) *Error { + d := validationError(err) + field := d.Field + for field != "" && origins[field] == "" { + index := strings.LastIndexAny(field, ".[") + if index < 0 { + field = "" + } else { + field = field[:index] + } + } + if env := origins[field]; env != "" { + d.Env = env + d.File = "" + d.Line = 0 + d.Column = 0 + return d + } + d.File = path + if d.Line == 0 { + p := doc.Positions[d.Field] + d.Line = p.Line + d.Column = p.Column + } + return d +} + +// Framework-owned normalization stays inside the common pipeline. Explicit +// environment stream selection clears a file profile unless a file override +// was also provided; project code never needs to repeat this rule. +func normalizeProject(v reflect.Value, path string, origins map[string]string, depth int) { + if depth > 32 { + return + } + if v.Kind() == reflect.Pointer { + if v.IsNil() { + return + } + v = v.Elem() + } + if v.Kind() != reflect.Struct { + return + } + if v.Type() == reflect.TypeOf(Config{}) && v.CanAddr() { + c := v.Addr().Interface().(*Config) + if origins[joinField(path, "log.output")] != "" && c.Log.Output != "" && (origins[joinField(path, "log.file")] == "" || c.Log.File == "") { + c.Log.File = "" + c.Log.Rotation = logx.RotationConfig{} + } + return + } + for i := 0; i < v.NumField(); i++ { + f := v.Type().Field(i) + if f.PkgPath != "" { + continue + } + name := strings.Split(f.Tag.Get("json"), ",")[0] + if name == "-" { + continue + } + if name == "" { + name = f.Name + } + normalizeProject(v.Field(i), joinField(path, name), origins, depth+1) + } +} + // OverlayEnv overlays the entire project struct without mutating the input. // Tags: env:"CUSTOM_NAME,required,secret,empty=reject"; env:"-" excludes a // subtree. required means nonempty string/non-nil pointer or collection; false @@ -70,7 +155,7 @@ func OverlayEnv[T any](base T, options EnvOptions) (T, error) { var textType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem() var durationType = reflect.TypeOf(time.Duration(0)) -func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) { +func projectOverlay[T any](base T, options EnvOptions, defaults bool, sourceMaps ...map[string]string) (T, error) { var zero T if options.Lookup == nil { options.Lookup = os.LookupEnv @@ -88,8 +173,8 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) result := reflect.New(v.Type()).Elem() result.Set(v) seen := map[string]bool{} - var walk func(reflect.Value, string, int) error - walk = func(value reflect.Value, prefix string, depth int) error { + var walk func(reflect.Value, string, string, int) error + walk = func(value reflect.Value, prefix, path string, depth int) error { if depth > 32 { return errors.New("config: nesting exceeds 32") } @@ -110,6 +195,7 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) if jsonName == "" { jsonName = field.Name } + fieldPath := joinField(path, jsonName) key := prefix + envName(jsonName) if tag[0] != "" { key = tag[0] @@ -131,7 +217,7 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) } } if dst.Kind() == reflect.Struct && !dst.Addr().Type().Implements(textType) { - if err := walk(dst, key+"_", depth+1); err != nil { + if err := walk(dst, key+"_", fieldPath, depth+1); err != nil { return err } continue @@ -142,14 +228,14 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) if !dst.IsNil() { child.Elem().Set(dst.Elem()) } - if err := walk(child.Elem(), key+"_", depth+1); err != nil { + if err := walk(child.Elem(), key+"_", fieldPath, depth+1); err != nil { return err } if !dst.IsNil() || !child.Elem().IsZero() { dst.Set(child) } if required && !defaults && dst.IsNil() { - return fmt.Errorf("config: %s is required", key) + return &Error{Env: key, Field: fieldPath, Reason: "is required"} } continue } @@ -164,7 +250,11 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) return fmt.Errorf("config: duplicate environment key %s", key) } seen[key] = true - raw, present := options.Lookup(key) + var raw string + var present bool + if !defaults { + raw, present = options.Lookup(key) + } selectedKey := key if !present && !defaults { for _, alias := range options.Aliases[key] { @@ -180,7 +270,7 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) } if present && raw == "" && !defaults { if empty == "reject" { - return fmt.Errorf("config: %s must not be empty", selectedKey) + return &Error{Env: selectedKey, Field: fieldPath, Reason: "must not be empty"} } if empty == "ignore" { present = false @@ -189,17 +279,24 @@ func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) if present { parsed := reflect.New(dst.Type()).Elem() if err := parseEnvValue(parsed, raw); err != nil { - return fmt.Errorf("config: %s must be a valid %s", selectedKey, dst.Type()) + origin := selectedKey + if defaults { + origin = "" + } + return &Error{Env: origin, Field: fieldPath, Reason: "expected a valid " + dst.Type().String(), cause: err} } dst.Set(parsed) + if !defaults && len(sourceMaps) > 0 { + sourceMaps[0][fieldPath] = selectedKey + } } if required && !defaults && emptyRequired(dst) { - return fmt.Errorf("config: %s is required", key) + return &Error{Env: key, Field: fieldPath, Reason: "is required"} } } return nil } - if err := walk(result, options.Prefix, 0); err != nil { + if err := walk(result, options.Prefix, "", 0); err != nil { return zero, err } return result.Interface().(T), nil diff --git a/examples/configuration/README.md b/examples/configuration/README.md new file mode 100644 index 0000000..093d223 --- /dev/null +++ b/examples/configuration/README.md @@ -0,0 +1,21 @@ +# 统一配置加载示例 + +从框架根目录运行，无需数据库或外部服务： + +```sh +go run ./examples/configuration +go run ./examples/configuration -format json +go run ./examples/configuration -format xml +DEMO_CLIENT_TIMEOUT=500ms DEMO_MAX_ITEMS=0 go run ./examples/configuration +DEMO_MAX_ITEMS=wrong go run ./examples/configuration +``` + +前三次都输出 `timeout=2s maxItems=100 allowed=[json xml]`。环境覆盖允许显式零值。 +最后一条输出定位 `maxItems` / `DEMO_MAX_ITEMS` 的安全错误，不打印输入原值。 +也可 `-config /path/to/app.yaml` 读取自己的文件。示例没有建立网络连接。 + +`Config` 定义字段，`envDefault` 定义缺省值，`Validate` 用 `config.Invalid` 返回字段与公开原因。 +`newLoader` 额外注册 `.custom`（演示复用 JSON decoder）；同一个 `load` 可读取所有格式。 +JSON/YAML/XML 直接可用，无需注册。新增格式只修改解析器注册，不改业务结构或校验。 + +只打印选定非敏感字段，不输出整个配置。完整格式、错误定位和旧项目迁移见 [CONFIG.md](../../CONFIG.md)。 diff --git a/examples/configuration/config.json b/examples/configuration/config.json new file mode 100644 index 0000000..9cf71b9 --- /dev/null +++ b/examples/configuration/config.json @@ -0,0 +1,5 @@ +{ + "client": {"endpoint": "https://example.com", "timeout": "2s"}, + "maxItems": 100, + "allowed": ["json", "xml"] +} diff --git a/examples/configuration/config.xml b/examples/configuration/config.xml new file mode 100644 index 0000000..ea4492d --- /dev/null +++ b/examples/configuration/config.xml @@ -0,0 +1,8 @@ + + + https://example.com + 2s + + 100 + jsonxml + diff --git a/examples/configuration/config.yaml b/examples/configuration/config.yaml new file mode 100644 index 0000000..41a32e3 --- /dev/null +++ b/examples/configuration/config.yaml @@ -0,0 +1,5 @@ +client: + endpoint: https://example.com + timeout: 2s +maxItems: 100 +allowed: [json, xml] diff --git a/examples/configuration/main.go b/examples/configuration/main.go new file mode 100644 index 0000000..785ca4b --- /dev/null +++ b/examples/configuration/main.go @@ -0,0 +1,85 @@ +// Run with go run ./examples/configuration -format xml, or -config /path/to/app.yaml. +package main + +import ( + "embed" + "flag" + "fmt" + "os" + "path/filepath" + "time" + + "git.yuntuo.hk/k2safe/go-omni/config" +) + +//go:embed config.json config.yaml config.xml +var examples embed.FS + +type ClientConfig struct { + Endpoint string `json:"endpoint"` + Timeout time.Duration `json:"timeout" envDefault:"3s"` + // This field is deliberately unavailable to configuration files. + Token string `json:"-" env:"DEMO_CLIENT_TOKEN,secret"` +} +type Config struct { + Client ClientConfig `json:"client"` + MaxItems int `json:"maxItems" envDefault:"20"` + Allowed []string `json:"allowed"` +} + +func (c Config) Validate() error { + if c.Client.Timeout <= 0 { + return config.Invalid("client.timeout", "must be positive") + } + if c.MaxItems < 0 { + return config.Invalid("maxItems", "must not be negative") + } + return nil +} +func newLoader() (*config.Loader, error) { + // Example of an instance-local extension. A TOML/etc. adapter can implement + // Decoder and return Document using that parser instead; no global registry. + return config.NewLoader(config.WithDecoder("custom", config.DecoderFunc(func(data []byte) (config.Document, error) { + return (config.JSONDecoder{}).Decode(data) + }))) +} +func load(path string) (Config, error) { + loader, err := newLoader() + if err != nil { + return Config{}, err + } + return config.LoadProject(path, Config{}, config.EnvOptions{Prefix: "DEMO_", Empty: "reject"}, loader) +} +func run() error { + path := flag.String("config", "", "external configuration file") + format := flag.String("format", "yaml", "embedded example: json, yaml, xml") + flag.Parse() + if *path == "" { + data, err := examples.ReadFile("config." + *format) + if err != nil { + return fmt.Errorf("choose json, yaml or xml") + } + dir, err := os.MkdirTemp("", "goOmniConfiguration-") + if err != nil { + return err + } + defer os.RemoveAll(dir) + *path = filepath.Join(dir, "config."+*format) + if err = os.WriteFile(*path, data, 0600); err != nil { + return err + } + } + cfg, err := load(*path) + if err != nil { + return err + } + // Print selected non-sensitive fields, never the whole configuration. + fmt.Printf("timeout=%s maxItems=%d allowed=%v\n", cfg.Client.Timeout, cfg.MaxItems, cfg.Allowed) + return nil +} +func main() { + if err := run(); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } +} diff --git a/examples/configuration/main_test.go b/examples/configuration/main_test.go new file mode 100644 index 0000000..62de65f --- /dev/null +++ b/examples/configuration/main_test.go @@ -0,0 +1,34 @@ +package main + +import ( + "os" + "path/filepath" + "reflect" + "testing" +) + +func TestExamples(t *testing.T) { + var previous Config + for i, ext := range []string{"json", "yaml", "xml", "custom"} { + source := ext + if source == "custom" { + source = "json" + } + data, err := examples.ReadFile("config." + source) + if err != nil { + t.Fatal(err) + } + path := filepath.Join(t.TempDir(), "app."+ext) + if err = os.WriteFile(path, data, 0600); err != nil { + t.Fatal(err) + } + got, err := load(path) + if err != nil { + t.Fatal(err) + } + if i > 0 && !reflect.DeepEqual(previous, got) { + t.Fatal("format difference") + } + previous = got + } +} diff --git a/examples/managedService/.goOmni/apiManifest.json b/examples/managedService/.goOmni/apiManifest.json index 9812898..319ab01 100644 --- a/examples/managedService/.goOmni/apiManifest.json +++ b/examples/managedService/.goOmni/apiManifest.json @@ -1,6 +1,6 @@ { "generatorConfig": true, - "templateVersion": "v0.19.0-preview", + "templateVersion": "v0.20.0-preview", "profile": { "database": "none", "storage": "none", @@ -12,6 +12,7 @@ "entry": "api/managed.api", "generated": { "docs/openapi.json": "0370fd6bd6a387e2ef21f6b42b009ce66749846b8bea3d481009a34f3af039a5", + "internal/config/loadGen.go": "5d630b894abdbbd66ddfa7d7d30f7573fc4ff995020a32a9f9157cd4af3a5655", "internal/handler/appGen.go": "f5251b4dc33e2312b0bdb71ead70d2171977d851265e124f640458d705a2cc9d", "internal/handler/demo/handlersGen.go": "3c76d85dbe89eb5127d9d134dc625203ecf9719dd11d174c1f335691b99e1375", "internal/handler/openapiGen.go": "f24cab3abbb8557a3fc4dd93fba23cc0b093f3e0c944d3ba1bd5b0eeff2d96f3", diff --git a/examples/managedService/go.mod b/examples/managedService/go.mod index edc3717..e875e86 100644 --- a/examples/managedService/go.mod +++ b/examples/managedService/go.mod @@ -2,7 +2,7 @@ module example.com/managed go 1.22.0 -require git.yuntuo.hk/k2safe/go-omni v0.19.0-preview +require git.yuntuo.hk/k2safe/go-omni v0.20.0-preview require ( github.com/gabriel-vasile/mimetype v1.4.3 // indirect diff --git a/examples/managedService/internal/config/config.go b/examples/managedService/internal/config/config.go index 9d37b18..0062d72 100644 --- a/examples/managedService/internal/config/config.go +++ b/examples/managedService/internal/config/config.go @@ -1,10 +1,9 @@ package config import ( - "git.yuntuo.hk/k2safe/go-omni/config" - "git.yuntuo.hk/k2safe/go-omni/logx" - "os" "time" + + "git.yuntuo.hk/k2safe/go-omni/config" ) type Config struct { @@ -22,42 +21,10 @@ func Default() Config { return Config{Server: server} } -// Load applies defaults -> file -> typed environment -> validation. -// New fields automatically use GO_OMNI_, or an explicit env tag. -func Load(path string) (Config, error) { - cfg := Default() - var err error - cfg, err = config.ApplyDefaults(cfg) - if err != nil { - return Config{}, err - } - if path != "" { - cfg, err = config.DecodeFile(path, cfg) - if err != nil { - return Config{}, err - } - } - cfg, err = config.OverlayEnv(cfg, config.EnvOptions{Prefix: "GO_OMNI_", Empty: "ignore", Aliases: config.ServerEnvAliases("GO_OMNI_", "server")}) - if err != nil { - return Config{}, err - } - output, outputSet := os.LookupEnv("GO_OMNI_SERVER_LOG_OUTPUT") - if !outputSet { - output = os.Getenv("GO_OMNI_LOG_OUTPUT") - } - file, fileSet := os.LookupEnv("GO_OMNI_SERVER_LOG_FILE") - if !fileSet { - file = os.Getenv("GO_OMNI_LOG_FILE") - } - if output != "" && file == "" { - cfg.Server.Log.File = "" - cfg.Server.Log.Rotation = logx.RotationConfig{} - } - return cfg, cfg.Validate() -} +// Validate checks configuration before logs or dependencies open. func (c Config) Validate() error { if err := c.Server.Validate(); err != nil { - return err + return config.AtField("server", err) } return nil diff --git a/examples/managedService/internal/config/loadGen.go b/examples/managedService/internal/config/loadGen.go new file mode 100644 index 0000000..eb7f776 --- /dev/null +++ b/examples/managedService/internal/config/loadGen.go @@ -0,0 +1,16 @@ +// Code generated by go-omni api. DO NOT EDIT. +package config + +import omniConfig "git.yuntuo.hk/k2safe/go-omni/config" + +// Load is maintained by the generator. Extend Config/Default/Validate in +// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline. +func Load(path string) (Config, error) { + loader, err := NewLoader() + if err != nil { + return Config{}, err + } + return omniConfig.LoadProject(path, Default(), omniConfig.EnvOptions{ + Prefix: "GO_OMNI_", Empty: "ignore", Aliases: omniConfig.ServerEnvAliases("GO_OMNI_", "server"), + }, loader) +} diff --git a/examples/managedService/internal/config/loader.go b/examples/managedService/internal/config/loader.go new file mode 100644 index 0000000..64ec950 --- /dev/null +++ b/examples/managedService/internal/config/loader.go @@ -0,0 +1,7 @@ +package config + +import omniConfig "git.yuntuo.hk/k2safe/go-omni/config" + +// NewLoader is project-owned. JSON/YAML/XML work without changes. Register +// additional format decoders here only when the project needs them. +func NewLoader() (*omniConfig.Loader, error) { return omniConfig.NewLoader() } diff --git a/reports/configurationAcceptance.md b/reports/configurationAcceptance.md new file mode 100644 index 0000000..82e9ac5 --- /dev/null +++ b/reports/configurationAcceptance.md @@ -0,0 +1,53 @@ +# v0.20 统一配置加载验收（2026-09-26） + +本轮修改 go-omni 配置包、生成模板与文档。没有修改接入项目的业务配置/数据库，也没有使用线上服务器做压力测试。 + +## 回归结果 + +- 全量 `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race ./...`：47 个有测试的包通过，包含实际生成项目、生命周期失败清理、错误目录/响应继承、上传和拷贝工具回归。 +- `go vet ./...`、`go build ./...` 通过。 +- `GOTOOLCHAIN=go1.22.12 go test ./config ./examples/configuration` 通过。 +- 新生成消费者的 `go test -race ./...`、`go build .`、generate/check/dry-run 通过。 +- checked-in managedService 已显式迁移旧 Load，生成检查、race 和构建通过；三种格式示例与环境覆盖命令实际运行通过。 + +针对性验证： + +| 场景 | 结果 | +| --- | --- | +| JSON / YAML / XML 的 defaults→file→env→Validate | 同值，0/false 可覆盖，envDefault 不覆盖文件 | +| 大于 2^53 的整数、动态 json.Number | 精度保留 | +| null、空数组、嵌套指针、map、duration、TextUnmarshaler、JSON tag / 嵌入字段 | 语义回归通过 | +| 文件字段类型错、未知字段、重复字段、业务 Validate 失败 | 文件/字段/可用行列与安全原因 | +| 环境类型或校验失败、集合子字段校验 | 定位环境来源，不错误归因于文件 | +| 内部错误链和默认值隔离 | errors.Is/As 可用，失败不提交半成品/修改默认值 | +| 24 个并发加载、自定义 Decoder、禁止事后注册 | race 通过，实例间不泄漏 | +| 超大文件、超深输入、重复 XML 根、DTD/entity、YAML alias/merge、UTF-8 | 拒绝；解析器 panic 不公开 panic 内容 | +| 业务新增字段与自定义格式 | 不改 main/loadGen，生成保留手写文件 | +| 配置校验失败 | 依赖初始化之前返回；未触发测试中的 Init panic | +| 旧手写 Load、loadGen 漂移 | generate/check/dry-run 拒绝冲突，不改业务文件 | + +30 秒、4 worker 的 `FuzzConfigurationDecoders`：383586 次执行，无失败（总运行 31.309 秒，包含停止收尾）。模糊输入上限 8 KiB；另有确定性 1 MiB/深度上限和恶意格式回归，不把 fuzz 次数当作形式证明。 + +## 性能 + +Go 1.27.1、darwin/arm64、Apple M2 Max；无 race 的本地基准，每个场景三轮。原始数据：[configurationBenchmark.txt](configurationBenchmark.txt)。 + +| 场景 | 中位数耗时 | 典型分配 | +| --- | ---: | ---: | +| 小配置 JSON 解析＋绑定 | 5.068 µs | 3781 B / 75 allocs | +| 小配置 YAML 解析＋绑定 | 14.855 µs | 15602 B / 181 allocs | +| 小配置 XML 解析＋绑定 | 7.137 µs | 5605 B / 122 allocs | +| 含完整 Server 默认值：文件读取＋环境覆盖＋校验 | 107.660 µs | 约 67.4 KB / 1288 allocs | +| 1000 个条目，约 28 KB JSON 解析＋绑定 | 2.548 ms | 约 2.40 MB / 31585 allocs | + +配置在启动阶段加载，不经过业务请求路径。这些是暖文件缓存、本机测试结果，不能等同冷磁盘启动耗时、请求 QPS 或跨机器性能承诺。没有为此引入全局反射缓存、解析器池或后台线程。 + +## 边界与迁移 + +- 旧手写 Load 必须先明确迁移，生成器不会猜测其解密/其他特殊行为。 +- 内置 XML 是明确的配置子集；YAML alias/merge/custom tag 拒绝；固定 Go 数组严格长度。 +- 自定义 Decoder/反序列化/Validate 是可信 Go 回调，其并发、副作用与公开 Reason 的脱敏由项目负责。 +- 第三方非结构化验证错误仅定位子树，并保留内部原因；业务用 Invalid/AtField 提供准确的公开字段与原因。 +- 未进行业务数据库/第三方真实依赖验收，未修改或替代各接入项目的业务回归。 + +完整使用与迁移：[CONFIG.md](../CONFIG.md)。远程固定版本安装验收在发布后追加。 diff --git a/reports/configurationBenchmark.txt b/reports/configurationBenchmark.txt new file mode 100644 index 0000000..a5cb34c --- /dev/null +++ b/reports/configurationBenchmark.txt @@ -0,0 +1,21 @@ +goos: darwin +goarch: arm64 +pkg: git.yuntuo.hk/k2safe/go-omni/config +cpu: Apple M2 Max +BenchmarkConfiguration/json-12 239197 5068 ns/op 3781 B/op 75 allocs/op +BenchmarkConfiguration/json-12 234421 5058 ns/op 3781 B/op 75 allocs/op +BenchmarkConfiguration/json-12 233330 5564 ns/op 3781 B/op 75 allocs/op +BenchmarkConfiguration/yaml-12 83022 14827 ns/op 15602 B/op 181 allocs/op +BenchmarkConfiguration/yaml-12 83197 14855 ns/op 15604 B/op 181 allocs/op +BenchmarkConfiguration/yaml-12 74828 16070 ns/op 15602 B/op 181 allocs/op +BenchmarkConfiguration/xml-12 167509 7144 ns/op 5606 B/op 122 allocs/op +BenchmarkConfiguration/xml-12 171202 7086 ns/op 5605 B/op 122 allocs/op +BenchmarkConfiguration/xml-12 167727 7137 ns/op 5605 B/op 122 allocs/op +BenchmarkConfigurationStartup-12 10000 107660 ns/op 67429 B/op 1288 allocs/op +BenchmarkConfigurationStartup-12 10000 107420 ns/op 67427 B/op 1288 allocs/op +BenchmarkConfigurationStartup-12 10000 107828 ns/op 67430 B/op 1288 allocs/op +BenchmarkConfigurationLarge-12 481 2548066 ns/op 10.99 MB/s 2403822 B/op 31585 allocs/op +BenchmarkConfigurationLarge-12 471 2532248 ns/op 11.06 MB/s 2403851 B/op 31586 allocs/op +BenchmarkConfigurationLarge-12 477 2641417 ns/op 10.61 MB/s 2403402 B/op 31583 allocs/op +PASS +ok git.yuntuo.hk/k2safe/go-omni/config 20.682s diff --git a/scripts/verifyManagedRelease.py b/scripts/verifyManagedRelease.py index 658954b..2f3e413 100644 --- a/scripts/verifyManagedRelease.py +++ b/scripts/verifyManagedRelease.py @@ -124,7 +124,7 @@ def main(): checks += ["sharedGoCatalogExport", "documentOnlyErrorSelection", "workspaceCheckNoWrites", "staleCatalogRejected", "twoIndependentServiceBuilds"] env.update(GO_OMNI_SCAFFOLD_INTEGRATION="1", GO_OMNI_RELEASE_CLI=str(cli), GO_OMNI_RELEASE_VERSION=args.version) - tests = ["TestManagedGeneratedProject", "TestErrorInheritanceConsumer", "TestErrorParamsConsumer", "TestCompatibilityConsumer", "TestBusinessGeneratedProject", "TestPrimitiveArrayConsumer", "TestCopyxThreeEndpointConsumer"] + tests = ["TestManagedGeneratedProject", "TestErrorInheritanceConsumer", "TestErrorParamsConsumer", "TestCompatibilityConsumer", "TestBusinessGeneratedProject", "TestPrimitiveArrayConsumer", "TestCopyxThreeEndpointConsumer", "TestGeneratedConfigurationConsumer"] print("Running generated consumers against the remote framework, including lifecycle failure cleanup", flush=True) output = run(["go", "test", "-race", "-count=1", "-json", "-timeout=240s", "./cmd/go-omni/apigen", "-run", "^(" + "|".join(tests) + ")$"], ROOT, env) events = [json.loads(line) for line in output.splitlines() if line.startswith("{")] @@ -137,6 +137,8 @@ def main(): checks += ["flatFieldComposition", "copyxThreeEndpointConsumer", "explicitNullAndMoneyPolicies", "fieldAdditionWithoutCopyRuleChanges", "middlewareFactoryInjectionAndCleanup", "optionalRepositoryPreservesBusiness"] + checks += ["generatedUnifiedConfig", "jsonYamlXmlAndCustomDecoder", "configValidationBeforeDependencies", "configEnvProvenance", "configExtensionPreserved", "configLoaderDriftCheck"] + sources = ["lifecycle/runtime.go", "lifecycle/phases.go", "config/lifecycle_test.go", "task/task.go", "cmd/go-omni/scaffold/templates/api-runtime.go.tmpl", "cmd/go-omni/apigen/templates.go", "config/project.go", "contract/bind.go", "contract/policy.go", "contract/multipart.go", "contract/file.go", "cmd/go-omni/apigen/business_test.go", "cmd/go-omni/apigen/generate.go", "lifecycle/controller.go", "config/endpoints.go", "health/health.go", "httpx/response.go", "app/app.go", "app/transport.go", "scripts/verifyManagedRelease.py", "scripts/verifyManagedProject.py", "cmd/go-omni/apigen/managed_test.go"] sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/managed").glob("*.go")] sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/business").glob("*.go")] @@ -145,6 +147,8 @@ def main(): sources += ["cmd/go-omni/apigen/" + name for name in ("composition.go", "generatorConfig.go", "copyConsumer_test.go", "generatorConfig_test.go", "middlewareFactory.go")] sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/composition").iterdir() if p.is_file()] sources += [str(p.relative_to(ROOT)) for p in (ROOT / "copyx").glob("*.go")] + sources += [str(p.relative_to(ROOT)) for p in (ROOT / "config").glob("*.go")] + sources += ["cmd/go-omni/apigen/configLoader_test.go", "cmd/go-omni/apigen/testdata/configuration/consumer_test.go", "cmd/go-omni/scaffold/templates/api-load.go.tmpl", "cmd/go-omni/scaffold/templates/api-loader.go.tmpl"] report = {"passed": True, "frameworkVersion": args.version, "frameworkSum": dependency["Sum"], "freshModuleCache": True, "noLocalReplace": True, "durationSeconds": round(time.monotonic() - started, 2), "checks": checks,