diff --git a/ADVANCED.md b/ADVANCED.md index 00babfb..1dc8415 100644 --- a/ADVANCED.md +++ b/ADVANCED.md @@ -11,7 +11,7 @@ | `openapi` | 路由与 OpenAPI 3.0.3 同时注册、DTO schema、响应信封 | 显式声明请求/响应；不自动推断任意 Handler 行为，不附带 Swagger UI | | `auth` | RS256/EdDSA JWT 验证与公钥轮换、可撤销 Session、Policy 接口 | 登录验证和业务授权规则由接入方提供；不自动获取 token 指定的密钥 | | `telemetry` | HTTP 入站/出站、Redis、SQL spans，HTTP 指标和 traceId 日志关联 | 注入 OTel exporter/reader 后才外发；SQL Query span 不覆盖后续遍历；不接管全局 provider | -| `config` / `logx` | JSON/YAML 文件、环境变量覆盖、动态日志级别、轮转/脱敏/堆栈/慢请求 | 热更新只开放日志级别；连接池、监听、认证等静态变更要求重启 | +| `config` / `logx` | JSON/YAML/XML 文件、显式环境引用、动态日志级别、轮转/脱敏/堆栈/慢请求 | 热更新只开放日志级别；连接池、监听、认证等静态变更要求重启 | | `resilience` | 失败阈值、熔断冷却、单探针恢复、上游并发隔离 | 一个实例对应一个依赖；默认内存状态，不做跨进程全局熔断 | | `middleware/redis` | 跨副本限流、幂等 claim/回放、容量限制 | 专用 Redis namespace；TTL 窗口内保护不等于永久业务去重 | | `queue` / `queue/redis` | 持久消息接口、Streams 接管、有限重试、失败消息管理 | 至少一次交付，处理器必须幂等；持久性取决于 Redis 部署 | @@ -102,7 +102,7 @@ err = consumer.Run(serviceContext, func(ctx context.Context, item queue.Delivery ## 配置与日志热更新 -`config.LoadFile(path, prefix)` 按默认值→文件→非空环境变量加载并验证。两种文件格式都使用 JSON tag 的 lowerCamelCase，duration 可以写 `"5s"`；文件上限 1 MiB，拒绝未知字段、多文档和非法值。自定义 prefix 要包含期望的尾部下划线，例如 `BILLING_`。空环境变量沿用文件值，不清空已配置字段。 +`config.LoadFile(path)` 使用默认值→文件字段显式引用→类型绑定→校验；没有自动环境覆盖。用 `${NAME}` 或 `${NAME:-fallback}` 指定来源，完整规则与迁移见 [CONFIG.md](CONFIG.md)。 ```yaml address: '127.0.0.1:8080' @@ -113,7 +113,7 @@ log: format: json ``` -先按配置创建 `logx.New(cfg.Log)`，再 `config.NewReloader(path, prefix, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`；后者阻塞，生命周期由应用负责。只有 `log.level` 允许变化，校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。 +先按配置创建 `logx.New(cfg.Log)`，再 `config.NewReloader(path, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`；后者阻塞，生命周期由应用负责。只有 `log.level` 允许变化，校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。 ## 熔断与观测 diff --git a/AGENTS.md b/AGENTS.md index c905727..94c8a15 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -6,3 +6,5 @@ - examples 使用通用场景，避免依赖业务项目、私有数据库或线上服务。必要的外部基础设施配置应明确说明。 - 2026-09-25 用户决定：移除 mappings/mappingFiles 及结构映射代码生成机制，改用可选 copyx 工具包。不得恢复另一套 JSON 或 Go 映射声明语言。 - 深拷贝的自定义转换结果不得与输入共享可变引用；发现共享应报错，不能降级成浅拷贝或静默接受。 + +- 2026-09-26 用户决定：配置文件显式声明固定值或 `${NAME}` / `${NAME:-fallback}`，旧自动环境覆盖、前缀/别名及 env/envDefault 标签模式移除，不得重新引入隐式二次覆盖。项目维护 Config/Default/Validate/文件，生成器维护 Load/main。 diff --git a/API.md b/API.md index d3dc213..776d461 100644 --- a/API.md +++ b/API.md @@ -48,7 +48,7 @@ go run ./cmd/go-omni new --api examples/contracts/user.api \ --log-output file --log-format json ../userService ``` -`api generate` 在新目录中也会生成完整项目。默认只监听 localhost；`GO_OMNI_HTTP_ADDR` 修改地址，`GO_OMNI_TRANSPORT=native` 切换网络层。配置顺序为默认值 → `etc/config.yaml` → 环境变量。文件日志支持现有轮转配置。远程试用使用与 CLI 一致的固定版本，见 [发布说明](RELEASE.md)。 +`api generate` 在新目录中也会生成完整项目。默认只监听 localhost；`GO_OMNI_HTTP_ADDR` 修改地址，`GO_OMNI_TRANSPORT=native` 切换网络层。配置顺序为 Default() → `etc/config.yaml` 中的显式环境引用 → 类型绑定 → Validate()；固定值不再被隐式覆盖。文件日志支持现有轮转配置。远程试用使用与 CLI 一致的固定版本，见 [发布说明](RELEASE.md)。 API 项目现在支持组合数据库、存储和日志： @@ -63,7 +63,7 @@ go-omni new --api user.api --module example.com/userService \ 初次选项记录在 `.goOmni/apiManifest.json`，以后直接 `api generate` 会继承。已有项目通过参数切换选项会报错，因为初始化代码属于业务方；可在新目录生成对照后手动迁移。`--force` 不允许覆盖 API 项目。new 与 api new 统一生成契约项目，不再使用旧式 CRUD 目录。 -MySQL 使用官方 `database/mysql` 结构化适配器，密码从 `GO_OMNI_DB_PASSWORD` 覆盖，非秘密池配置位于 YAML，环境变量优先。存储使用原有 `GO_OMNI_STORAGE_*`；API 项目不会额外生成公开的签名发放接口，鉴权和对象权限由业务实现。初始化失败会清理已打开资源，退出清理可重复调用。 +MySQL 使用官方 `database/mysql` 结构化适配器，生成文件以 `password: "${GO_OMNI_DB_PASSWORD}"` 显式读取密码；其他字段可写固定值或引用，缺失项保留 Default()。存储配置也只读取文件明确引用的变量；API 项目不会额外生成公开的签名发放接口，鉴权和对象权限由业务实现。初始化失败会清理已打开资源，退出清理可重复调用。 HTTP 请求类型不能推导数据库表结构；迁移和 Repository 由业务自己维护。当前目录的完整示例见 [managedService](examples/managedService/README.md)。MySQL、事务审计和鉴权实现可参考 [旧目录用户管理示例](examples/userService/README.md)，接入时按 [项目指南](PROJECT.md) 装配到 config/svc/logic。 diff --git a/BUSINESS_DEVELOPMENT.md b/BUSINESS_DEVELOPMENT.md index 4ba7ec1..7fc6f35 100644 --- a/BUSINESS_DEVELOPMENT.md +++ b/BUSINESS_DEVELOPMENT.md @@ -5,7 +5,7 @@ ## 实施与验收清单 - [x] 运行期：资源登记、逆序释放、幂等关闭、失败回滚、故障原因保留归框架；生成入口与项目依赖初始化边界明确。多 App 日志隔离、标准 log.Logger 适配、可信请求元数据与依赖中间件示例。 -- [x] 配置：项目完整结构体环境覆盖；默认值→文件→环境；嵌套、duration、slice、显式零值、空值策略、必填与脱敏；日志无副作用完整校验。 +- [x] 配置：Default()→文件显式环境引用→类型绑定→Validate；嵌套、duration、slice、显式零值、必填/备用值、转义与安全诊断；日志无副作用完整校验。 - [x] 数据库：官方 MySQL 结构化配置/TLS/驱动日志适配，继续复用 OpenConnector 和唯一连接池；独立启动检查、失败清理、生成器接入；不内置版本/表结构/业务错误策略。 - [x] 响应契约：数组、可空对象、成功 null、独立 HTTP 204；Logic/Handler/OpenAPI/漂移检查一致。 - [x] 请求契约：未知 query、JSON UTF-8/charset、无 Body/必填空对象/可选 Body，App 默认与路由覆盖；scalar/any/数组/大小限制一致，普通 JSON 只绑定一次。 diff --git a/CHANGELOG.md b/CHANGELOG.md index d18ce29..9ce9a2f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,12 @@ # Changelog +## v0.22.0-preview（2026-09-26） + +- 文件显式环境引用与备用值，字符串组合/转义，安全有界展开和类型绑定；变量值原样插入、不递归处理。 +- 移除隐式环境覆盖、EnvOptions/前缀/别名及 env/envDefault；数据库统一使用文件加载，不再单独扫描环境。 +- 生成 main/Load 新增 -check-config，精简配置文件，保留开发者扩展；错误字段建议和迁移检查。 +- README、CONFIG、数据库/HTTP 客户端指南及示例同步新规则；配置中的响应模板需要 $$ 转义，.api 不变。 + ## v0.21.0-preview（2026-09-26） - httpc 可配置环境/直连/固定代理、常用连接阶段超时和独立空闲池参数。 diff --git a/CONFIG.md b/CONFIG.md index 634148d..e07ca71 100644 --- a/CONFIG.md +++ b/CONFIG.md @@ -1,178 +1,177 @@ -# 统一配置加载 +# 配置：文件明确声明值的来源 -v0.20 起，新生成服务的 `main.go` 调用生成器维护的 `config.Load`。项目只需要定义业务字段、默认值和校验；日志、HTTP、数据库等配置使用同一流程，配置加载时不打开数据库、监听端口或创建日志文件。 +v0.22 起只使用这一条流程： ```text -代码默认值 → envDefault（仅填零值）→ 可选配置文件 -→ 完整结构体环境覆盖（含别名）→ 框架规范化 → 项目 Validate -→ 初始化日志及项目依赖 → 启动服务 +项目 Default() → 解析选中的一个文件 → 解析字段值中的环境引用 +→ 按 Go 类型绑定已提供的字段 → 项目 Validate() → 初始化日志和依赖 ``` -## 生成项目要维护什么 +没有环境前缀、别名表或隐式二次覆盖。业务只维护 `Config`、`Default()`、`Validate()` 和配置文件。项目 Validate 中继续调用嵌套框架配置的 Validate；生成模板已包含这些调用。验证函数只检查值，不应打开资源。 -| 文件 | 维护方 | 用途 | -| --- | --- | --- | -| `main.go` | 生成器 | 启动、信号、请求排空、依赖关闭 | -| `internal/config/loadGen.go` | 生成器 | 调用统一加载入口；纳入 `generate --check` | -| `internal/config/config.go` | 项目 | `Config`、`Default()`、`Validate()` | -| `internal/config/loader.go` | 项目 | `NewLoader()`，只有增加文件格式时才需修改 | -| `etc/config.yaml` | 项目 | 部署配置；可改用 JSON/XML 并传 `-config` | -| `internal/svc` | 项目与生成器分文件 | 初始化依赖、登记清理、AppOptions | - -比如增加业务的批次数量：在 Config 增加一个字段，在 Validate 加范围校验；需要部署值时改配置文件或环境变量。不修改 main、Load 或 Handler。 +## 直接使用 ```go -// 添加到项目 Config。 -BatchSize int `json:"batchSize" envDefault:"100"` - -// 添加到项目 Validate。 -if c.BatchSize < 1 || c.BatchSize > 1000 { - return omniConfig.Invalid("batchSize", "must be between 1 and 1000") -} -``` - -自动支持 `GO_OMNI_BATCH_SIZE`。生成服务的空环境值策略为 `ignore`，0 和 false 是有效覆盖；范围由 Validate 决定。 -重新生成保留 config.go、loader.go 和 svc 的手写扩展。 - -## 手写项目用法 - -```go -package main - -import ( - "time" - "git.yuntuo.hk/k2safe/go-omni/config" -) - type Config struct { - Timeout time.Duration `json:"timeout" envDefault:"3s"` - Limit int `json:"limit" envDefault:"100"` - Token string `json:"-" env:"APP_TOKEN,secret"` + Server omniConfig.Config `json:"server"` + BatchSize int `json:"batchSize"` + Enabled bool `json:"enabled"` + Password string `json:"password"` + Timeout time.Duration `json:"timeout"` +} +func Default() Config { + return Config{Server: omniConfig.Default(), BatchSize: 100, Enabled: true, + Timeout: 3*time.Second} } func (c Config) Validate() error { - if c.Timeout <= 0 { - return config.Invalid("timeout", "must be positive") + if err := c.Server.Validate(); err != nil { + return omniConfig.AtField("server", err) } + if c.BatchSize < 0 { return omniConfig.Invalid("batchSize", "must not be negative") } return nil } - -// cfg, err := config.LoadProject("etc/config.yaml", Config{}, -// config.EnvOptions{Prefix: "APP_", Empty: "reject"}) -``` - -`LoadProject` 接收结构体默认值，可实现值接收器或指针接收器的 `Validate() error`。空路径表示不读文件。 -`DecodeFile` 只做文件覆盖，不做 envDefault、环境覆盖或业务 Validate；旧的两参数调用仍有效，可用结构体或结构体指针。低层 `ApplyDefaults`、`OverlayEnv`、`LoadFile` 继续存在。 - -嵌套校验用 `config.AtField("client", err)` 加路径前缀。公共原因通过 `config.Invalid` 提供，原因中不要包含密码、DSN、Token 或输入原值。 - -## JSON / YAML / XML - -按文件扩展名选择解析器：`.json`、`.yaml`、`.yml`、`.xml`，大小写不敏感。所有格式使用 Go 结构体的 **json 字段名**，无需为 XML 复制一份结构或标签。 -以下配置等价： - -```json -{"client":{"timeout":"2s"},"enabled":false,"ports":[8080,8081],"optional":null} +// 手写项目；生成项目的 Load 已自动调用此入口。 +cfg, err := omniConfig.LoadProject("etc/config.yaml", Default()) ``` ```yaml -client: - timeout: 2s +server: + address: "${HTTP_ADDR:-127.0.0.1:8080}" +password: "${DB_PASSWORD}" +timeout: "${REQUEST_TIMEOUT:-3s}" enabled: false -ports: [8080, 8081] -optional: null +``` + +文件未写 batchSize，保留 100；enabled 明确写 false，就使用 false。变量名字完全由文件决定，不需要对应 Go 字段名。配置文件可只列出部署需要变动的字段；不会自动导出/打印包含凭据的完整配置。 + +## 引用语法 + +| 文件里的值 | 含义 | +| --- | --- | +| `host: 127.0.0.1` | 固定值，不读取任何环境变量 | +| `password: "${DB_PASSWORD}"` | 未设置或空字符串报错，不回退到 Default() | +| `port: "${DB_PORT:-3306}"` | 未设置或为空时使用 3306 | +| `sessionToken: "${SESSION_TOKEN:-}"` | 未设置或为空时使用空字符串，是否合法由目标类型和 Validate 决定 | +| `url: "https://${HOST}:${PORT:-443}/v1"` | 普通字符串中可出现多个引用 | +| `literal: "$${DB_PASSWORD}"` | 得到字面量 `${DB_PASSWORD}`，不读取变量 | +| `literal: "$$cost"` | `$$` 转义一个 `$`，得到 `$cost` | + +变量名遵循 `[A-Za-z_][A-Za-z0-9_]*`。只支持 `${NAME}`、`${NAME:-fallback}` 和 `$$` 转义，不支持 `${NAME-default}`、嵌套引用或 shell 表达式。普通单独的 `$` 保留。fallback 到第一个 `}` 结束，可包含冒号、`:-`、引号等文件格式允许的字符；需要包含 `}` 的值可使用固定值或环境变量。 + +解析只扫描文件的**已解码字段值**，不替换 YAML/JSON/XML 源文本，不扩展字段名，也不扫描 Default() 的字符串。插入的环境值和 fallback 不再展开。密码 `DDSFLKJlsf:saf:-123`、引号、换行、`${OTHER}` 均原样保留，不破坏文档结构。字符串不统一 TrimSpace；`"0"`、`"false"` 和全空格字符串均属于已提供。 + +同一变量每次加载最多读取一次；下一次加载会重新读取。没有全局变量值缓存。并发加载各自维护结果，不能把调用者正在修改的默认配置当作共享可变输入。 + +## 类型与覆盖 + +目标 Go 类型决定引用结果的转换：整数、浮点数、布尔值、`time.Duration`、字符串、指针及已有自定义 JSON/Text 反序列化。数值范围按目标位宽检查，整数不经过 float64。自定义反序列化仍由自定义类型解释文本，不猜测其业务含义。 + +- 文件缺失字段保留初始值；显式 0、false、空字符串正常覆盖，不用零值判断字段是否存在。 +- `${MISSING}` 失败就报错；需要备用值必须写 `${MISSING:-value}`。 +- 字符串编号 `"001"` 保持 `001`；绑定到 int 才转整数。环境里的 `null` 不自动变成空指针，`[1,2]` 不自动变成数组。 +- 集合直接在文件里声明，可在每个标量元素中使用引用，例如 `ports: ["${PORT_A:-8080}", "${PORT_B:-8081}"]`。 +- 数组整体替换；空 slice 保持非 nil；固定长度 Go 数组要求长度一致。map 合并键，已提供键的值整体替换。 +- 固定文件 null 沿用 JSON 语义：清空指针/map/slice/interface，普通值字段保持初始值。需要区分 null 时使用指针。 +- `any` 的引用结果保持字符串，不猜测动态类型。固定 JSON/YAML 数字使用 json.Number 保留精度。 +- 引用成功后的值不会再覆盖一次。旧 `GO_OMNI_*` 名称只有出现在文件引用中才有作用；不隐式重置日志 file/rotation。 + +普通固定字符串仍按原有严格绑定规则处理；不会为了支持变量，把所有 `"123"` 都自动转换成数字。 + +## YAML、JSON、XML 一致 + +使用 Go 的 `json` 字段名，无需维护三份 tag。以下三份文件等价： + +```yaml +port: "${PORT:-3306}" +password: "${PASSWORD}" +``` + +```json +{"port":"${PORT:-3306}","password":"${PASSWORD}"} ``` ```xml - - 2s - false - 80808081 - - +${PORT:-3306}${PASSWORD} ``` -XML 必须使用一个 `` 根；普通叶节点按目标 Go 类型转换。数组显式 `type="array"` 加 ``，不根据重复元素猜测类型。`type="array"` 空元素表示空数组，`type="object"` 表示空对象，`type="null"` 表示 null。`any` 中的 XML 叶节点默认是字符串，需要动态数字/布尔时标注 `type="number"` / `type="bool"`；也可用 `type="string"`。 +XML 使用 `` 根，标量占位符写在普通叶元素或 `type="string"` 元素中，由 Go 字段决定类型；显式 `type="number"` / `type="bool"` 表示文件里已经有合法数字/布尔常量。XML 数组用 `type="array"` 加 ``；空对象 `type="object"`、null 用 `type="null"`。不从重复元素猜测数组。 -类型和覆盖规则： +`.api` 不经过配置加载器，响应模板 `${code}` 不变。**如果把响应模板写在 YAML/JSON/XML 配置文件里，须转义：** -- 缺失字段保留默认值；显式 0、false、空字符串正常覆盖。 -- null 清空指针、map、slice、interface；普通值字段保持默认值，沿用 Go JSON 解码语义。需要区分 null 时用指针。 -- 数组整体替换；空 slice 保持非 nil；固定长度 Go 数组要求长度一致，避免丢弃多余配置。 -- map 合并键，已提供的键整体覆盖对应值；map 中未提供的键保留。 -- duration 支持 `250ms`、`3s`；整数保持精度，动态 JSON 数字使用 `json.Number`，不先转 float64。 -- 自定义 `json.Unmarshaler` / `encoding.TextUnmarshaler` 仍可接入。这些回调是可信项目代码，必须自己保证并发和副作用安全；不要修改共享外部状态。 -- JSON `,string` 字段、嵌入字段遵循 JSON 字段选择；未知字段、重复绑定、冲突均拒绝。 - -内置解析器限制：单文件最多 1 MiB，解析最多 64 层、100000 个节点；绑定和默认值复制也有深度限制。只接受普通文件，不读取设备或命名管道。不支持多文档、重复字段、无效 UTF-8；YAML 拒绝 alias/merge/custom tag，XML 拒绝 DTD/外部实体、命名空间、业务属性和混合内容。XML 仅支持配置所需的子集，不是通用 XML 文档映射器。 - -这些限制针对启动配置，避免意外超大文件、别名扩展和模糊覆盖。自定义值解码器的内部字段校验由该解码器负责；例如现有错误目录使用其专有解码规则。 - -## 环境覆盖 - -统一入口的 EnvOptions 控制前缀、空值策略、别名和 Lookup。完整 JSON 路径自动转换为 UPPER_SNAKE_CASE：`client.timeout` → `APP_CLIENT_TIMEOUT`。 - -```go -config.EnvOptions{ - Prefix: "GO_OMNI_", Empty: "ignore", - Aliases: config.ServerEnvAliases("GO_OMNI_", "server"), -} +```yaml +server: + response: + body: + code: "$${code}" + msg: "$${message}" + data: "$${data}" ``` -生成项目同时接受完整的 `GO_OMNI_SERVER_HTTP_READ_TIMEOUT` 与短别名 `GO_OMNI_HTTP_READ_TIMEOUT`。完整名存在时优先，即便其空值被 ignore，也不回退别名。不会再二次调用 ApplyEnv 覆盖业务字段。 +加载后恢复为 `${code}` 等响应占位符，仍由响应编码器处理。 -显式 `env:"CUSTOM_NAME"` 是绝对名称；`env:"-"` 跳过子树。`json:"-"` 只在声明显式 env 名时参与，可让密钥只能从环境加载。`secret` 标签表达用途，解析错误对所有字段都不打印值。slice/map 的环境值采用 JSON。`required` 检查字符串非空、指针/集合非 nil；复杂范围由 Validate 负责。详见 [完整业务环境覆盖](DEVELOPMENT.md#完整业务配置的环境覆盖)。 +## 生成项目与检查 -显式环境选择日志 stdout/stderr 会清除文件 profile 的 File/Rotation，除非同时提供非空 File 覆盖。该兼容规则由框架处理。 +| 文件 | 维护方 | +| --- | --- | +| `main.go`、`internal/config/loadGen.go` | 生成器维护，纳入 `generate --check` | +| `internal/config/config.go` | 项目维护 Config、Default、Validate | +| `etc/config.yaml` 等选定文件 | 项目维护，重新生成不覆盖 | +| `internal/config/loader.go` | 初次生成即可使用，仅添加自定义格式时才修改 | -## 扩展解析器 +```sh +go run . -config etc/config.test.yaml -check-config +go run . -config etc/config.local.yaml +go build . +go-omni api generate --check +``` -内置格式直接可用。新增格式只在 `internal/config/loader.go` 的 `NewLoader` 注册： +`-check-config` 使用与启动完全相同的 Load，通过时退出 0、失败时非零；不打印配置，不创建日志文件、数据库、后台组件或 HTTP 监听。它只校验配置，不检测真实数据库连接或证书握手。 + +`-config` 只选一个文件，不自动叠加 local/test/default。指定文件不存在时报错；`-config ''` 只使用 Default() 后校验，不读取任何环境变量，也不寻找其他文件。示例：[examples/configuration](examples/configuration/README.md)。 + +## 安全诊断与限制 + +```text +config "etc/config.yaml":7:9 env "DB_PORT" field "database.port": expected a valid int +``` + +`*config.Error` 包含 File/Field/Env/Line/Column/Reason。引用错误同时保留文件与环境变量名；组合引用失败可列出多个名字。行列定位原文件字段，不把环境值伪装成源文件文本。无法取得位置时仍提供可用的文件/字段信息。 + +用 `config.Invalid(field, publicReason)` 或 `config.AtField(prefix, err)` 报业务校验错误。未知字段给出唯一且接近的已声明字段建议，不自动修改。错误默认不打印原值；自定义 Reason 不应拼入秘密值。原始 cause 仍支持 errors.Is/As，不要把 Unwrap 后的错误或整个配置写到日志。 + +文件最多 1 MiB；单个环境值/展开字段最多 1 MiB，展开字符串合计/变量缓存分别最多 4 MiB；解析/展开限制 64 层、100000 个节点与引用计数。拒绝无效 UTF-8、多文档、重复字段、未知字段、非普通文件；YAML 不接受 alias/merge/custom tag，XML 不接受 DTD/外部实体。不执行 shell，不自动加载 `.env`。 + +## 可选解析器与测试来源 + +内置格式无需项目手写加载器。需要额外格式时在已有 NewLoader 注册 Decoder，后续仍自动经过同一引用/绑定/校验流程： ```go func NewLoader() (*omniConfig.Loader, error) { return omniConfig.NewLoader( - omniConfig.WithDecoder("custom", omniConfig.DecoderFunc( - func(data []byte) (omniConfig.Document, error) { - // 演示注册扩展名；实际可在这里调用项目选定的 TOML 等解析库。 - return (omniConfig.JSONDecoder{}).Decode(data) - }, - )), + omniConfig.WithDecoder("custom", omniConfig.JSONDecoder{}), ) } ``` -Decoder 返回 Document：Value 是 map[string]any 根，子值为 string/bool/json.Number/nil/[]any/map[string]any；无自身类型的叶节点可返回 config.Text。Positions 可选，用 `client.timeout` 或 `servers[0].timeout` 标记 1-based 行列。不提供位置时仍有文件、字段、原因。 +Decoder 返回 Document（map/string/bool/json.Number/nil/[]any，XML 类标量可用 Text），可提供 Positions。实例注册无全局 init；插件必须可并发调用、不修改共享输入，并且是可信 Go 代码。 -自己的语法错误可返回 `&config.Error{Field: "client.timeout", Line: 3, Column: 5, Reason: "expected a duration"}`。Reason 必须安全；普通第三方错误仅公开 `invalid configuration syntax`，内部 cause 通过 errors.Is/As 保留。 +测试或嵌入宿主可以用 `WithLookupEnv(func(string)(string,bool))` 替换引用查找函数。它仍然只读取文件明确引用的名称，不启用任何自动覆盖。普通业务不需要这个选项。 -注册表属于 Loader 实例，无全局 init 注册；不能覆盖已有扩展名。Loader 可共享，但 Decoder 必须支持并发且不保留/修改输入。解析插件是可信 Go 代码，框架不是插件沙箱。新格式依赖由选择它的项目显式引入，核心不强制引入 TOML 等库。 +## v0.22 迁移：旧模式已移除 -## 定位配置错误 +这是不兼容调整，不保留两套加载逻辑： -```text -config "etc/config.yaml":4:12 field "client.timeout": expected a duration such as 5s -config env "APP_CLIENT_TIMEOUT" field "client.timeout": expected a valid time.Duration -``` +- 删除 `EnvOptions`、`LoadFromEnv`、`ApplyEnv`、`OverlayEnv`、`ApplyDefaults`、`ServerEnvAliases` 和 `env`/`envDefault` 标签处理；检出遗留标签会提示迁移。 +- 删除 `database.ConfigFromEnv` / `database.ApplyEnv`；数据库配置作为项目 Config 的字段绑定。DSN 字段可在文件里使用 `dsn: "${DB_DSN}"`，序列化 database.Config 不输出 DSN 原值。 +- `LoadProject(path, defaults[, loader])` 不再接收 EnvOptions；`LoadFile(path)`、`NewReloader(path, logger)` 不再接收前缀。 +- `omni.LoadConfig(path)` 的参数现在是文件路径；空路径只用框架默认值。 -`errors.As(err, &diagnostic)` 可获取 `*config.Error` 的 File/Field/Env/Line/Column/Reason；原 `DecodeError` 名称保留为别名。`errors.Is` 可判断文件不存在等 I/O 原因。 +迁移顺序： -解析失败显示文件和可获取的行列；Validate 使用 Invalid/AtField 才能定位业务字段。环境覆盖导致的错误优先标环境变量，不错误地归因到文件；数组元素验证也会追溯到覆盖它的集合环境变量。默认值未出现在文件里时没有行列。YAML 语法错误通常只有行号，列显示 1。 +1. 将 envDefault 默认值搬入 Default()，移除 env/envDefault 标签。需要文件绑定的旧 `json:"-"` 密钥字段改成明确 json 名称；不打印配置对象。 +2. 按实际部署需要，在选定文件写环境引用。固定值保持固定，不按原前缀猜测。原先依赖环境切换日志输出的项目，应在文件明确配置 file/output/rotation。 +3. 配置文件中的其他 `${...}` 模板改成 `$${...}` 字面量；`.api` 无需修改。 +4. 手写项目改为 LoadProject。生成项目保留 Config/Default/Validate/NewLoader，删除旧手写 Load 后重新生成；main/loadGen 由生成器更新，业务配置文件不会被静默改写。 +5. 升级匹配的 CLI 与框架，执行 generate、generate --check、go test、go build 和 `-check-config`，再使用真实部署配置启动。 -不打印底层错误原文或配置值；任意普通 `errors.New(...)` 校验失败的公开原因是 `validation failed`，内部原因仍在链中。项目应改用 Invalid 给出安全的原因。第三方子配置校验若没有字段元数据，只定位到对应子树，框架不会从任意错误文案猜字段。 - -不要把 Unwrap 后的原始错误、整个配置对象写入公开日志/HTTP；自定义解析器与校验器传入的 Reason 由项目负责脱敏。 - -## v0.19 及更早项目迁移 - -先提交现有代码，再升级匹配的 CLI 与框架。旧项目的 `internal/config/config.go` 含手写 Load，新生成器会明确阻止 generate/check/dry-run，不会直接覆盖它。 - -1. 保留 Config、Default、Validate、业务配置结构及辅助函数。 -2. 删除旧 Load。普通的 defaults/file/env/Validate 流程已由 LoadProject 接管；移除由此不再使用的 import。将自定义默认值和校验分别迁入 Default / Validate，自定义文件格式迁入 NewLoader。 -3. 框架只在 `loader.go` 不存在时创建 NewLoader；已有项目文件同名时需要保证它提供同签名函数。无需手写 loadGen.go。 -4. 重新生成，执行 `generate --check`、`go test ./...`、`go build .`，再用真实配置验证启动。 - -保留业务需要的显式 env 标签/规则；如果原 Load 有解密或其他项目特殊步骤，不要直接删除，应先明确其归属，必要时在自定义 Decoder 中处理。生成器不自动搬动或猜测手写行为。`api upgrade` 仍可预览较早版本的基础设施差异，但已修改的配置必须按上述步骤迁移。 - -不会移除低层配置 API；本次移除的是**新项目里重复手写的加载流程**，不删除旧项目的业务文件。 - -完整可运行例子：[examples/configuration](examples/configuration/README.md)。 +低层 `DecodeFile(path, defaults[, loader])` 也解析显式引用，但不调用 Validate；默认值和原始 Document 都不会因失败而被部分修改。日志 Reloader 仍只允许 log.level 变化，其他静态字段变更要求重启。 diff --git a/DEVELOPMENT.md b/DEVELOPMENT.md index b8fb66f..04fbbb6 100644 --- a/DEVELOPMENT.md +++ b/DEVELOPMENT.md @@ -1,4 +1,4 @@ -# 通用业务开发指南（v0.21.0-preview） +# 通用业务开发指南（v0.22.0-preview） 从 [README](README.md) 创建服务后，业务主要维护 `.api`、配置、`svc.Init`、Logic、Repository 和项目中间件。本页说明通用接入能力；字段、错误码、权限、表结构和事务边界都由项目定义。 @@ -100,44 +100,13 @@ defer reader.Close() 不声明 multipart 字段时，保留原来的 `body *httpx.Body` 流式入口，不创建临时文件。该方式适合特殊协议；项目应在不可撤销副作用之前完成校验并调用 `body.Complete()`。下载和原始回调继续使用 `@produces` / `@body(mode: raw)`。 -## 完整业务配置的环境覆盖 +## 文件声明配置来源 -生成器维护 `internal/config/loadGen.go`；项目维护 Config/Default/Validate 和可选 NewLoader。三种格式、可扩展解析器、字段错误定位及旧 Load 迁移见 [统一配置指南](CONFIG.md)。 +项目维护 Config、Default()、Validate() 和配置文件；生成器维护 Load/main。流程为默认值→选定文件→显式 `${NAME}` / `${NAME:-fallback}` 引用→类型绑定→校验。没有前缀、别名或 env/envDefault 隐式覆盖。 -生成项目默认执行：代码与 `envDefault` 默认值 → JSON/YAML/XML 文件 → 全结构体环境覆盖（含别名）→ 规范化 → `Validate`。增加业务字段不需要另外写 `os.Getenv`。 +固定值保持固定，缺失字段保留 Default()，0/false/空字符串正常覆盖；引用失败不回退初始化默认值。`$$` 转义 `$`，字符串凭据保留空格与特殊字符。 -```go -type ClientConfig struct { - Endpoint string `json:"endpoint" env:"CLIENT_ENDPOINT,required"` - Token string `json:"-" env:"CLIENT_TOKEN,required,secret,empty=reject"` - Timeout time.Duration `json:"timeout" envDefault:"3s"` - Retries []int `json:"retries"` - Enabled bool `json:"enabled"` -} -``` - -把字段加入项目 `Config` 后，默认环境名取 `GO_OMNI_` 加 JSON 路径的 UPPER_SNAKE_CASE，如 `GO_OMNI_CLIENT_TIMEOUT`。显式 env 名是完整名称，可以使用自己的业务前缀。`env:"-"` 排除整个子树；`json:"-"` 字段只有显式 env 标签才参与。 - -duration 使用 `250ms`、`3s`；slice/map 使用 JSON，如 `[1,3,5]`、`{"retry":"2s"}`。支持嵌套、零/false、空数组、自定义 TextUnmarshaler。空环境值默认允许；可选择 `empty=ignore` 或 `empty=reject`。required 要求字符串非空、指针/集合非 nil；0 和 false 可以是合法业务值，范围由项目 Validate 决定。 - -手写项目可使用 `config.LoadProject(path, defaults, config.EnvOptions{Prefix:"MY_APP_"})`，或分别调用 `ApplyDefaults/DecodeFile/OverlayEnv`。读取失败不修改输入默认值；诊断不会打印输入值。不要主动打印含密钥的整个配置，或把底层解析错误展开到公开响应。Validate 不打开日志文件、数据库或存储连接；证书的可读性和握手信任在依赖初始化阶段验证。 - -### 一次覆盖完整配置与 HTTP/日志短名称 - -```go -cfg, err = config.OverlayEnv(cfg, config.EnvOptions{ - Prefix: "MY_APP_", - Empty: "reject", // allow / ignore / reject，项目自行选择 - Aliases: config.ServerEnvAliases("MY_APP_", "server"), - // Lookup 可注入测试或项目配置源；默认 os.LookupEnv。 -}) -``` - -不要再紧接着调用 `ApplyEnv(cfg.Server, ...)`。上面的调用同时处理完整配置和 HTTP_ADDR、SERVICE、LOG_LEVEL 等框架短名称。例如 `MY_APP_SERVER_ADDRESS` 与 `MY_APP_HTTP_ADDR` 指向同一字段，**完整名称优先**；未设置时才读取短名称。完整名称显式设空仍优先：allow 交给该字段解析（string 可清空；数字等仍要求合法值），ignore 保留文件/默认值，reject 报错。不会因 ignore 再退回别名。字段 `env` 标签的 empty 规则优先于全局规则。 - -自定义别名用 `EnvOptions.Aliases`，键是完整环境名、值是按优先级排列的别名数组。两种名称都使用同一 Lookup、空值策略和类型转换；布尔 false、数值 0 与未设置不同。集合始终使用 JSON，包括 `MY_APP_TRUSTED_PROXIES='["127.0.0.1"]'`。 - -新项目模板一次调用 OverlayEnv，默认 `Empty: "ignore"`。旧 `ApplyEnv` 保留原有忽略空值、修剪空白及 TRUSTED_PROXIES 逗号列表行为。已有项目的 `internal/config` 属于开发者文件，升级不覆盖它；按上例合并原来的两次覆盖调用后，保留项目自己的 Validate。日志输出的显式完整名称也优先于短名称。 +`go run . -config etc/config.test.yaml -check-config` 只校验，不打开日志、数据库或端口。生成/check/dry-run 保留业务文件；新的模板要求 v0.22。完整格式、错误定位、解析器与移除清单见 [CONFIG.md](CONFIG.md)。 ## 依赖、日志与中间件 @@ -280,7 +249,7 @@ go-omni workspace check --file omni.workspace.json ## v0.13 项目升级 -先使用与目标框架匹配的 CLI；CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.20.0-preview（或显式引用当前源码）。 +先使用与目标框架匹配的 CLI；CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.22.0-preview（或显式引用当前源码）。 ```sh go-omni api upgrade --dir . --dry-run diff --git a/PROJECT.md b/PROJECT.md index 46fd2ef..8e2fd32 100644 --- a/PROJECT.md +++ b/PROJECT.md @@ -1,6 +1,6 @@ # 生成项目与文件维护边界 -v0.21.0-preview 提供受管理的生成项目。安装同版远程 CLI，生成的项目直接引用该版本，无需本地 replace。根目录入口支持 `go run .` 和 `go build .`；框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。 +v0.22.0-preview 提供受管理的生成项目。安装同版远程 CLI，生成的项目直接引用该版本，无需本地 replace。根目录入口支持 `go run .` 和 `go build .`；框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。 ## 创建和日常开发 @@ -143,7 +143,7 @@ server: `responseMode: raw` 返回 `{"status":"ready"}` 等健康数据，有依赖时包含检查名称、结果与耗时；不包装公共体。存活检查不执行依赖探针；服务开始退出后，就绪检查返回 503，存活仍可返回 200，直到监听关闭。OpenAPI 始终输出规范文档本身，不使用业务响应包装；关闭 HTTP 文档端点不影响生成 `docs/openapi.json`。 -环境变量覆盖配置文件，例如 `GO_OMNI_ENDPOINTS_READINESS_PATH=/readyz`、`GO_OMNI_ENDPOINTS_READINESS_RESPONSE_MODE=raw`、`GO_OMNI_ENDPOINTS_OPENAPI_ENABLED=false`；liveness/readiness/openapi 都支持对应的 ENABLED 和 PATH。Config/AppOptions 的最终有效配置决定是否注册。 +内置路由配置由项目文件明确设置，也可以使用 `${NAME:-fallback}` 引用；不再按字段名自动读取环境变量。Config/AppOptions 的最终有效配置决定是否注册。 关闭或改名后，旧路径不再由框架保留，项目可以在 `.api` 自己声明。启用的内置 GET 路由与业务路由相同或被业务参数/通配路径覆盖时，在创建 App、开始监听之前报错；不会覆盖业务路由。内置接口仍经过全局中间件；项目在鉴权中间件中明确决定这些路径的访问规则，框架不自动绕过鉴权。 @@ -215,7 +215,7 @@ get /download (DownloadReq) ## 已有服务接入步骤 -每个独立服务生成一个项目，module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.21.0-preview： +每个独立服务生成一个项目，module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.22.0-preview： 1. 先保留旧服务，整理实际路由及现有响应、鉴权和超时要求。在新目录执行 `go-omni new --api /path/to/service.api --module example.com/service ./service`；不对旧入口强制覆盖。 2. 在 `.api` 中选择 `@app(defaultResponse: ...)`，显式保留原项目的字段、成功码、错误码和 `dataMode`。需要独立回执的路由单独声明 raw/produces。普通路由、404/405 和提前返回的中间件一起验收。 diff --git a/README.md b/README.md index 01e450f..1d23267 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ GoOmni 是面向多个业务项目复用的 Go 后端框架：用 `.api` 定义接口、请求和返回类型，生成项目目录、HTTP 接入代码及 OpenAPI，业务代码负责 Logic 和 Repository。也可以在已有项目中直接使用 `omni.New()`，或单独引入日志、存储等模块。 -**从这份 README 开始接入。** 当前版本 **`v0.21.0-preview`**，新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入；文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md)，版本变化与验收见 [发布说明](RELEASE.md)。 +**从这份 README 开始接入。** 当前版本 **`v0.22.0-preview`**，新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入；文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md)，版本变化与验收见 [发布说明](RELEASE.md)。 **v0.19 改动**：新增可选 `copyx` 工具，用普通 Go 调用完成浅拷贝、深拷贝和显式类型转换；深拷贝会拒绝自定义转换中残留的源对象可变引用。**移除了 `mappings` / `mappingFiles` 和映射代码生成**，已有使用者须按 [迁移步骤](COPY.md#从旧-mapping-迁移) 调整，不能直接删除仍在调用的旧文件。 @@ -12,9 +12,9 @@ Go **1.22+**。框架不依赖 Gin，默认使用标准库网络层，提供可 响应字段、业务码、登录方式、权限规则和数据库表结构由接入项目决定。文中的 `code/msg/data`、`200/9999` 都是示例配置，可以替换。 -契约支持 `returns([]string/[]int64/[]bool/[]any)`、OpenAPI 公共引用、workspace 一次导出共享 Go 错误目录，以及统一环境变量别名覆盖。详细配置与升级步骤见 [业务开发指南](DEVELOPMENT.md)。 +契约支持 `returns([]string/[]int64/[]bool/[]any)`、OpenAPI 公共引用、workspace 一次导出共享 Go 错误目录，以及文件显式环境引用与配置检查。详细配置与升级步骤见 [业务开发指南](DEVELOPMENT.md)。 -已有通用开发能力见 [业务开发指南](DEVELOPMENT.md)：数组/null、严格请求策略、声明式上传、完整配置环境覆盖、资源生命周期、共享契约和升级工具。MySQL 直接使用 [官方适配器](database/mysql/README.md)。 +已有通用开发能力见 [业务开发指南](DEVELOPMENT.md)：数组/null、严格请求策略、声明式上传、文件显式配置引用、资源生命周期、共享契约和升级工具。MySQL 直接使用 [官方适配器](database/mysql/README.md)。 后台组件可在 svc 登记 [分阶段关闭](lifecycle/README.md#后台组件分阶段关闭)，先停止接收工作并等待收尾，再关闭共享资源；无需绑定队列或 ORM。 @@ -38,7 +38,7 @@ Go **1.22+**。框架不依赖 Gin，默认使用标准库网络层，提供可 | 错误目录与动态提示 | `WithErrors(catalog)` 注册一次；`ByCode(...).WithParams(...).WithCause(err)` | [错误目录](errors/README.md)、[模板参数示例](examples/contracts/errorParams.api) | | OpenAPI 与错误公共引用 | `.api` 自动生成；手写接口可使用 `openapi` 注册器；Go 错误目录显式导出 | [契约工具](DEVELOPMENT.md)、[注册示例](examples/platform/main.go) | | 结构化业务日志 | `logx.Info/Error`、`logx.FromContext(ctx)`、`logx.New`；支持轮转、脱敏、慢请求和级别更新 | [下方日志示例](#日志)、[完整配置](logx/README.md) | -| 文件配置与环境覆盖 | 默认值→JSON/YAML/XML→环境→校验；完整业务结构用 `config.LoadProject[T]` / 环境别名 | [统一配置指南](CONFIG.md)、[完整可运行示例](examples/configuration/README.md)、[动态级别](ADVANCED.md#配置与日志热更新) | +| 文件配置与环境引用 | Default()→JSON/YAML/XML 字段引用→类型绑定→校验；无隐式覆盖，支持 `-check-config` | [统一配置指南](CONFIG.md)、[完整可运行示例](examples/configuration/README.md)、[动态级别](ADVANCED.md#配置与日志热更新) | | 生命周期与健康检查 | svc 登记 OnClose/OnStop/OnDrain，RequestShutdown 通知故障；健康路径/开关可配 | [生命周期用法](lifecycle/README.md)、[健康检查](health/README.md)、[生成项目扩展](PROJECT.md) | | 数据库、连接池与事务 | `database.Open` / `OpenConnector`；事务与检查由项目显式调用 | [通用数据库](database/README.md)、[MySQL TLS/Logger](database/mysql/README.md) | | 可选 GORM 与迁移 | gormx 复用已有连接池；`go-omni migrate` 预览，显式 apply | [GORM 示例](database/gormx/README.md)、[迁移用法](ADVANCED.md#迁移审计与分页) | @@ -81,26 +81,30 @@ go run ./examples/httpClient ## 统一配置加载 -新生成项目的加载流程由框架管理。业务在 `internal/config/config.go` 增加字段、默认值和 `Validate`，无需改 main 或复制环境变量解析。JSON、YAML、XML 共用字段和校验；扩展格式在 `loader.go` 注册。 +业务只维护 Config、Default()、Validate() 和配置文件。生成器维护 Load/main；文件缺失字段保留默认值，显式 0/false/空字符串正常覆盖，固定值不会被同名环境变量再次覆盖。 ```go -// 项目 Config 中增加字段，自动支持 GO_OMNI_BATCH_SIZE： -BatchSize int `json:"batchSize" envDefault:"100"` - -// 项目 Validate 中增加业务范围检查： -if c.BatchSize < 1 { - return omniConfig.Invalid("batchSize", "must be positive") -} +BatchSize int `json:"batchSize"` // 加入 Config；在 Default() 中设置 BatchSize: 100。 ``` -错误包含文件、字段、可用的行列或环境变量名；输入原值不自动写入错误。 +```yaml +server: + address: "${HTTP_ADDR:-127.0.0.1:8080}" +batchSize: "${BATCH_SIZE:-100}" +password: "${DB_PASSWORD}" +sessionToken: "${SESSION_TOKEN:-}" +``` + +`${NAME}` 未设置或为空就报错；`${NAME:-value}` 明确指定备用值。`$$` 转义 `$`；密码中的冒号、换行、引号和 `${...}` 原样插入，不二次解释。JSON/YAML/XML 共用解析后的字段处理和 Go 类型绑定，错误同时定位文件、字段、变量及可用行列，不打印原值。 ```sh go run ./examples/configuration -format xml DEMO_CLIENT_TIMEOUT=500ms go run ./examples/configuration +# 在生成项目中，仅检查选中的文件，不初始化依赖： +go run . -config etc/config.test.yaml -check-config ``` -详见 [统一加载、解析器、迁移](CONFIG.md) 和 [完整示例](examples/configuration/README.md)。**旧项目须先迁移手写 Load，再重新生成**；生成器会保留业务配置并拒绝冲突，不静默覆盖。 +完整 [语义与迁移](CONFIG.md)、[可运行示例](examples/configuration/README.md)。**v0.22 移除旧自动环境覆盖、前缀/别名及 env/envDefault 标签**，生成项目先迁移开发者配置再重新生成；文件中的响应模板 `${code}` 写成 `$${code}`，`.api` 保持原样。 ## 选择接入方式 @@ -122,7 +126,7 @@ module 地址是 `git.yuntuo.hk/k2safe/go-omni`。先确保本机 Go 和 Git 可 ```sh export GOPRIVATE="$(go env GOPRIVATE),git.yuntuo.hk" -go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.21.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview export PATH="$(go env GOPATH)/bin:$PATH" go-omni --help ``` @@ -279,7 +283,7 @@ go-omni new --api catalog.api \ 在业务项目的 **go.mod 所在目录**执行： ```sh -go get git.yuntuo.hk/k2safe/go-omni@v0.21.0-preview +go get git.yuntuo.hk/k2safe/go-omni@v0.22.0-preview ``` 已有项目可直接复用现有目录、Service、Repository 和数据库连接，先选择一个接口接入。也可以只用 `logx`、`storage` 等包，暂时保留原来的 HTTP 框架。希望入口和 HTTP 接入都交给生成器的项目，按 [已有服务接入步骤](PROJECT.md#已有服务接入步骤) 在新目录生成后迁入业务代码。 @@ -523,20 +527,22 @@ API 项目的数据库迁移需要单独维护并显式执行；启动不会自 ## 配置与运行 -生成项目从 `etc/config.yaml` 读取配置，优先级为 **默认值 → 配置文件 → 环境变量**。密钥通过环境变量提供。手写入口需显式调用配置加载函数并传给 `omni.New(omni.WithConfig(settings))`；单独调用 `omni.New()` 不会自动读取环境变量。 +生成项目使用 **Default() → 选定文件的显式引用 → 类型绑定 → Validate()**。固定值不会被环境变量覆盖；配置规则、转义及迁移见 [配置指南](CONFIG.md)。手写入口显式加载配置，再传入 `omni.New(omni.WithConfig(settings))`。 -| 环境变量 | 用途 | +| 文件字段 | 用途 | | --- | --- | -| `GO_OMNI_HTTP_ADDR` | 监听地址；生成项目默认 `127.0.0.1:8080` | -| `GO_OMNI_TRANSPORT` | `nethttp`（默认）、`native`、`http2` | -| `GO_OMNI_TLS_CERT_FILE` / `GO_OMNI_TLS_KEY_FILE` | 成对设置 TLS 证书和私钥路径 | -| `GO_OMNI_HTTP_READ_HEADER_TIMEOUT` / `GO_OMNI_HTTP_READ_TIMEOUT` | 请求头 / 请求读取超时，如 `5s` / `30s` | -| `GO_OMNI_HTTP_WRITE_TIMEOUT` / `GO_OMNI_HTTP_IDLE_TIMEOUT` | 响应写入 / Keep-Alive 空闲超时 | -| `GO_OMNI_HTTP_MAX_HEADER_BYTES` | 请求头大小限制，字节数 | -| `GO_OMNI_SHUTDOWN_TIMEOUT` | 优雅退出等待时间，默认 `10s` | -| `GO_OMNI_ENABLE_LOGGER` / `GO_OMNI_LOG_LEVEL` | 访问日志开关、日志级别 | -| `GO_OMNI_TRUSTED_PROXIES` | 可解析转发地址头的可信代理 IP/CIDR，逗号分隔 | -| `GO_OMNI_DB_HOST/USER/PASSWORD/DATABASE` | 官方 MySQL 适配器的结构化连接配置 | +| `server.address` | 监听地址，生成项目默认 127.0.0.1:8080 | +| `server.transport` | nethttp（默认）、native、http2 | +| `server.tls.certFile/keyFile` | 成对配置 TLS 证书和私钥路径 | +| `server.http.readHeaderTimeout/readTimeout` | 请求头 / 请求读取超时 | +| `server.http.writeTimeout/idleTimeout` | 响应写入 / Keep-Alive 空闲超时 | +| `server.http.maxHeaderBytes` | 请求头大小限制，字节数 | +| `server.shutdownTimeout` | 优雅退出等待时间，默认 10s | +| `server.enableLogger` / `server.log.level` | 访问日志开关、日志级别 | +| `server.trustedProxies` | 可信代理 IP/CIDR，文件数组 | +| `database.host/user/password/database` | 官方 MySQL 结构化连接配置 | + +初始生成文件已为地址、网络层和选中模块的必要配置声明环境引用；其他字段按需要添加固定值或引用。 例如在生成项目中切换 native： @@ -581,7 +587,7 @@ go test ./... `main.go`、`Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理，手动修改会报告冲突。Logic、Repository、中间件实现、配置和依赖装配属于项目，重复生成保留手写内容。契约变化会提示受影响的 Logic 文件和 DTO 字段，明确的签名错误会给出原签名与期望签名；生成后 `--check` 仍能检出未修复的问题。将 `.api`、`omni.gen.json`、生成文件和 `.goOmni/apiManifest.json` 一起提交，CI 同时运行 `generate --check` 与 `go test ./...`。详见 [契约变更提示与检查边界](PROJECT.md#契约变更提示)。 -CLI 与业务 module 固定使用 `v0.21.0-preview`。v0.14–v0.19 项目先按 [配置入口迁移](CONFIG.md#v019-及更早项目迁移) 整理旧 Load，再升级依赖与 CLI 并重新生成，框架维护的 main/runtimeGen 自动更新；v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理，不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移)，旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 +CLI 与业务 module 固定使用 `v0.22.0-preview`。v0.21 及更早项目先按 [配置迁移](CONFIG.md#v022-迁移旧模式已移除) 整理旧标签、文件与 Load，再升级依赖与 CLI 并重新生成，框架维护的 main/runtimeGen 自动更新；v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理，不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移)，旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 ## 按需深入 diff --git a/RELEASE.md b/RELEASE.md index 9294463..e6f8a8c 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -1,4 +1,27 @@ -# v0.21.0-preview +# v0.22.0-preview + +配置文件明确决定每个值的来源，项目只维护 Config、Default()、Validate() 和文件。 + +- `${NAME}` 必填，`${NAME:-fallback}` 在未设置/空字符串时使用备用值，`$$` 转义 `$`。支持字符串组合；环境值不二次展开，凭据空格和特殊字符原样保留。 +- 在 JSON/YAML/XML 已解析的字段值上展开，再按目标 Go 类型绑定；缺失字段保留默认值，0/false/空字符串可显式覆盖。字段错误定位文件、环境名和可用行列，不回显值。 +- 生成 Load/main 自动接入；新增 `-check-config`，选定一个文件检查，成功退出，不启动日志/依赖/监听。精简生成配置，重新生成保留业务 Config、Default、Validate 和文件。 +- **不兼容变更：旧自动环境覆盖、前缀/别名、env/envDefault、database.ConfigFromEnv/ApplyEnv 已移除。** LoadFile/NewReloader 不再接收 prefix，omni.LoadConfig 参数改为文件路径。发现旧标签提示迁移。 +- 配置文件中的响应模板用 `$${code}` 等转义；`.api` 中的 `${code}` 不变。日志 file/output/rotation 均由显式文件配置决定，不再隐式清除。 + +```sh +go get git.yuntuo.hk/k2safe/go-omni@v0.22.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview +# 按 CONFIG.md 迁移开发者配置后： +go-omni api generate +go-omni api generate --check +go run . -config etc/config.yaml -check-config +``` + +[完整规则与迁移](CONFIG.md)、[可运行示例](examples/configuration/README.md)、[场景与性能验收](reports/explicitConfigurationAcceptance.md) 已同步。48 个测试包全量 race、实际生成消费者、Go 1.22、vet/build 和约 28 万次引用 fuzz 通过；小配置固定值/带引用加载约 25 µs，属于本机启动阶段基准。当前仍为 preview。 + +--- + +# v0.21.0-preview（历史版本） 让带认证的 HTTP 调用和内部直连可以直接使用 httpc，减少手写 Transport 与 JSON 响应处理。 @@ -139,7 +162,7 @@ go-omni api generate --check go test ./... ``` -旧项目 config/Logic/svc 继续由开发者维护，不会被重新生成覆盖。已有环境配置需按 [一次覆盖完整配置](DEVELOPMENT.md#一次覆盖完整配置与-http日志短名称) 合并 OverlayEnv 与 ApplyEnv；共享错误目录接入见 [workspace 配置](DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。已有手写权限 Handler 切换自动生成前，先迁出实现并显式移除 Manual 文件，生成器不会静默删除。 +旧项目 config/Logic/svc 继续由开发者维护，不会被重新生成覆盖。已有环境配置需按 本版当时的统一覆盖方式合并 OverlayEnv 与 ApplyEnv（该模式已在 v0.22 移除，当前升级见 [配置迁移](CONFIG.md#v022-迁移旧模式已移除)）；共享错误目录接入见 [workspace 配置](DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。已有手写权限 Handler 切换自动生成前，先迁出实现并显式移除 Manual 文件，生成器不会静默删除。 全量 race（含实际生成消费者）、vet、build、根入口/退出/再生成验证通过。eTax 只读契约验收：API 文档 20,751,149 → 847,171 字节，ID 文档 1,008,210 → 392,481 字节；70 个操作展开引用后与原文档完全一致。空缓存远程 CLI / 固定版本消费者验收也已通过，没有本地 replace；标签固定在 `64e3bd6cfced225476d58e95037249cd0a4fe695`。详见 [验收记录](reports/contractFollowupAcceptance.md)。这不代表已完成业务项目迁移。 diff --git a/cmd/go-omni/README.md b/cmd/go-omni/README.md index a2a05aa..6635063 100644 --- a/cmd/go-omni/README.md +++ b/cmd/go-omni/README.md @@ -14,7 +14,7 @@ v0.14 新增 `api format [--check]`、`api upgrade [--dry-run|--preview <空目 ```sh # 仓库读取权限与 PATH 配置见根目录 README -go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.21.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview go-omni new --module example.com/hello ./hello # 从自己的契约生成；service.api 及其相对 import 文件需要在本地存在 @@ -49,14 +49,14 @@ main、Gen.go、OpenAPI 和 manifest 由生成器维护；业务文件按字节 ## 版本与迁移 -CLI 与框架依赖固定使用 v0.20.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod，不添加本地 replace；`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换： +CLI 与框架依赖固定使用 v0.22.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod，不添加本地 replace；`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换： ```sh # 在 go-omni 源码仓库中执行 go run ./cmd/go-omni new --module example.com/hello --framework-dir . ../hello ``` -当前模板与依赖以 v0.20.0-preview 为基线，继续使用 manifest v2；GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级)：原始模板可自动替换，含业务修改的基础文件按预览合并；通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务，不使用旧 `add` 命令的手动路由装配。 +当前模板与依赖以 v0.22.0-preview 为基线，继续使用 manifest v2；GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级)：原始模板可自动替换，含业务修改的基础文件按预览合并；通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务，不使用旧 `add` 命令的手动路由装配。 ## 数据库迁移命令 @@ -70,6 +70,8 @@ go-omni migrate --dir migrations --apply SQL 命名示例 `0001_createUsers.up.sql`；预览只显示版本和 SHA-256，不连接数据库。失败保留 dirty 状态，不自动 down/force。细节见 [扩展模块](../../ADVANCED.md)。 -### v0.20 配置入口 +### v0.22 文件显式配置 -新生成 `internal/config/loadGen.go` 调用统一加载，纳入生成检查；项目 config.go 只维护字段、默认值和校验，loader.go 用于注册额外格式。JSON/YAML/XML 内置。旧手写 Load 会阻止生成，请先按 [迁移步骤](../../CONFIG.md#v019-及更早项目迁移) 整理，不会自动覆盖业务代码。 +新生成 `internal/config/loadGen.go` 调用统一加载，纳入生成检查；项目 config.go 只维护字段、默认值和校验，loader.go 用于注册额外格式。JSON/YAML/XML 内置。旧手写 Load 会阻止生成，请先按 [迁移步骤](../../CONFIG.md#v022-迁移旧模式已移除) 整理，不会自动覆盖业务代码。 + +`go run . -config etc/config.test.yaml -check-config` 只做配置解析和校验，不初始化日志、数据库或监听。文件中的固定值不受环境自动覆盖；`${NAME}` 为必填引用，`${NAME:-fallback}` 提供备用值。 diff --git a/cmd/go-omni/apigen/configLoader_test.go b/cmd/go-omni/apigen/configLoader_test.go index dc8e467..5550989 100644 --- a/cmd/go-omni/apigen/configLoader_test.go +++ b/cmd/go-omni/apigen/configLoader_test.go @@ -26,8 +26,8 @@ func TestGeneratedConfigurationConsumer(t *testing.T) { // The business only changes fields/defaults/validation, not the loading flow. writeFile(t, filepath.Join(dir, "internal/config/config.go"), `package config import("git.yuntuo.hk/k2safe/go-omni/config";"git.yuntuo.hk/k2safe/go-omni/logx") - type Config struct { Server config.Config `+"`json:\"server\"`"+`;Limit int `+"`json:\"limit\" envDefault:\"10\"`"+` } - func Default()Config{ c:=config.Default();c.Log.File="./var/log/app.log";c.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:10};return Config{Server:c} } + type Config struct { Server config.Config `+"`json:\"server\"`"+`;Limit int `+"`json:\"limit\"`"+` } + func Default()Config{ c:=config.Default();c.Log.File="./var/log/app.log";c.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:10};return Config{Server:c,Limit:10} } func(c Config)Validate()error{if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)};if c.Limit<0{return config.Invalid("limit","must not be negative")};return nil} `) writeFile(t, filepath.Join(dir, "internal/config/loader.go"), `package config @@ -111,3 +111,26 @@ func TestConfigLoadMigrationGuard(t *testing.T) { } } } + +func TestRemovedEnvironmentTagsRejectBeforeGeneration(t *testing.T) { + dir := filepath.Join(t.TempDir(), "app") + o := Options{Dir: dir, Module: "example.com/migrate", FrameworkDir: frameworkRoot(t)} + if _, err := New(o); err != nil { + t.Fatal(err) + } + o.API = filepath.Join(dir, "api/service.api") + path := filepath.Join(dir, "internal/config/config.go") + source := readFile(t, path) + writeFile(t, path, strings.Replace(source, "json:\"server\"", "json:\"server\" env:\"OLD_SERVER\"", 1)) + before := snapshot(t, dir) + for _, mode := range []string{"dry", "generate", "check"} { + o.DryRun = mode == "dry" + o.Check = mode == "check" + if _, err := Generate(o); err == nil || !strings.Contains(err.Error(), "env/envDefault") { + t.Fatal(err) + } + if !sameFiles(before, snapshot(t, dir)) { + t.Fatal("migration rejection wrote files") + } + } +} diff --git a/cmd/go-omni/apigen/generate.go b/cmd/go-omni/apigen/generate.go index cf7d2c0..b4bf05e 100644 --- a/cmd/go-omni/apigen/generate.go +++ b/cmd/go-omni/apigen/generate.go @@ -21,8 +21,8 @@ import ( const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" // Runtime-only releases need not raise the managed template compatibility floor. -const minimumVersion = "v0.20.0-preview" -const currentVersion = "v0.21.0-preview" +const minimumVersion = "v0.22.0-preview" +const currentVersion = "v0.22.0-preview" const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n" type Options struct { @@ -158,7 +158,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) { if !filepath.IsAbs(path) { path = filepath.Join(directory, path) } - if source, err := os.ReadFile(filepath.Join(path, "config", "loader.go")); err == nil && strings.Contains(string(source), "func NewLoader(") { + if source, err := os.ReadFile(filepath.Join(path, "config", "references.go")); err == nil && strings.Contains(string(source), "func WithLookupEnv(") { localCurrent = true } } diff --git a/cmd/go-omni/apigen/profile_test.go b/cmd/go-omni/apigen/profile_test.go index 571733b..d9eef0c 100644 --- a/cmd/go-omni/apigen/profile_test.go +++ b/cmd/go-omni/apigen/profile_test.go @@ -63,18 +63,23 @@ func TestAPIInfrastructureConsumers(t *testing.T) { t.Fatal(err) } test := `package config -import("testing";"os") +import("testing";"os";"strings") func TestConfiguration(t *testing.T){ - t.Setenv("GO_OMNI_DB_DSN","test:test@tcp(127.0.0.1:1)/test") + t.Setenv("GO_OMNI_DB_DATABASE","test");t.Setenv("GO_OMNI_DB_USER","test");t.Setenv("GO_OMNI_DB_PASSWORD","test") t.Setenv("GO_OMNI_STORAGE_SIGNING_KEY","01234567890123456789012345678901") t.Setenv("GO_OMNI_STORAGE_ACCESS_KEY","test");t.Setenv("GO_OMNI_STORAGE_SECRET_KEY","test") t.Setenv("GO_OMNI_STORAGE_BUCKET","test");t.Setenv("GO_OMNI_STORAGE_ENDPOINT","ENDPOINT") t.Setenv("GO_OMNI_STORAGE_ACCOUNT_ID","01234567890123456789012345678901") cfg,err:=Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)} if cfg.Server.Log.File==""{t.Fatal("file logger absent")} - t.Setenv("GO_OMNI_LOG_OUTPUT","stdout");cfg,err=Load("");if err!=nil||cfg.Server.Log.File!=""{t.Fatalf("override: %v",err)} - t.Setenv("GO_OMNI_SERVER_LOG_FILE","./explicit.log");cfg,err=Load("");if err!=nil||cfg.Server.Log.File!="./explicit.log"{t.Fatalf("canonical log file: %v",err)} - t.Setenv("GO_OMNI_SERVER_LOG_OUTPUT","stderr");cfg,err=Load("");if err!=nil||cfg.Server.Log.Output!="stderr"{t.Fatalf("canonical output: %v",err)} + t.Setenv("GO_OMNI_LOG_OUTPUT","stdout");t.Setenv("GO_OMNI_SERVER_LOG_FILE","./implicit.log") + cfg,err=Load("../../etc/config.yaml");if err!=nil||cfg.Server.Log.File!=Default().Server.Log.File{t.Fatalf("implicit environment override: %v",err)} + raw,err:=os.ReadFile("../../etc/config.yaml");if err!=nil{t.Fatal(err)} + // Add explicit fields while keeping storage and DB requirements. + raw=[]byte(strings.Replace(string(raw),"server:\n","server:\n log:\n output: '${OUTPUT:-stdout}'\n file: ''\n rotation:\n enabled: false\n",1)) + path:="explicit.yaml";if err=os.WriteFile(path,raw,0600);err!=nil{t.Fatal(err)};defer os.Remove(path) + cfg,err=Load(path);if err!=nil||cfg.Server.Log.Output!="stdout"||cfg.Server.Log.File!=""||cfg.Server.Log.Rotation.Enabled{t.Fatalf("explicit file override: %v",err)} + _=os.Getenv("unused") }` endpoint := "" diff --git a/cmd/go-omni/apigen/templates.go b/cmd/go-omni/apigen/templates.go index c047f19..2d868eb 100644 --- a/cmd/go-omni/apigen/templates.go +++ b/cmd/go-omni/apigen/templates.go @@ -19,8 +19,10 @@ import ( "MODULE/internal/svc" ) func main(){ - path:=flag.String("config","etc/config.yaml","configuration file; empty uses defaults and environment") + path:=flag.String("config","etc/config.yaml","configuration file; empty uses only Default()") + check:=flag.Bool("check-config",false,"validate configuration and exit without starting dependencies") flag.Parse() + if *check {if _,err:=config.Load(*path);err!=nil{log.Fatal(err)};return} ctx,stop:=signal.NotifyContext(context.Background(),os.Interrupt,syscall.SIGTERM) defer stop() if err:=run(ctx,*path);err!=nil{log.Fatal(err)} @@ -58,7 +60,7 @@ Unimplemented/nil factories fail startup and clean registered dependencies. Existing projects retain middleware: handlers until explicitly migrated. Ordinary Logic uses the generated Request/Response types directly. Where an independent business type is needed, use ordinary Go or the optional copyx package. -Configuration loading is generator-managed in internal/config/loadGen.go. Extend fields/defaults/Validate in config.go; additional decoders belong in loader.go/NewLoader. JSON/YAML/XML use one pipeline. See the framework CONFIG.md. +Configuration loading is generator-managed in internal/config/loadGen.go. Extend fields/defaults/Validate in config.go; additional decoders belong in loader.go/NewLoader. JSON/YAML/XML use explicit ${NAME} / ${NAME:-fallback} references; fixed values are never overridden by inferred environment names. Use -check-config to validate without opening dependencies. See the framework CONFIG.md. There are no mappings/mappingFiles settings or generated mapping files in v0.19. See COPY.md in the framework repository for copy policies and migration steps, @@ -77,7 +79,7 @@ Configure server.endpoints to disable or move each built-in route. Health responseMode is app by default (project format), or raw for plain health data. Probe HTTP statuses remain 200/503 even when business errors use HTTP 200. OpenAPI remains an unwrapped document; disabled endpoints reserve no paths. -Set GO_OMNI_TRANSPORT=native to select the native transport without changing services. +The default file explicitly references GO_OMNI_TRANSPORT; set it to native or edit server.transport. The greeting starter works immediately; other new operation stubs return 501 until their business logic is implemented. Authentication hooks also fail closed with 501. @@ -139,7 +141,7 @@ Full reference: https://git.yuntuo.hk/k2safe/go-omni/src/branch/main/DEVELOPMENT api/ # editable contract and imported types main.go # GENERATED startup, lifecycle and route registration etc/config.yaml # lowerCamelCase configuration -internal/config/ # defaults -> file -> environment +internal/config/ # defaults -> file references -> typed binding -> validation internal/svc/ # dependency initialization and cleanup internal/types/*Gen.go # DTOs split by source API, one types package internal/svc/runtimeGen.go # GENERATED lifecycle and infrastructure startup @@ -240,13 +242,16 @@ func infrastructureREADME(o Options) string { Selected database: %s; storage: %s; logging: %s/%s. MySQL uses database/sql through deps.DB, including InTx and a readiness probe. -Use structured database/mysql configuration and GO_OMNI_DB_PASSWORD. Other connection/pool/TLS settings live in etc/config.yaml and GO_OMNI_DB_*. +Use structured database/mysql configuration. The generated etc/config.yaml explicitly +references GO_OMNI_DB_PASSWORD and other required connection values; add fixed +values or explicit references for pool/TLS fields. No implicit environment overlay. No tables are invented or migrated automatically from request/response DTOs. The none/memory selection opens no database; implement repositories per business. With storage enabled, services receive deps.Files, Uploads, Downloads and UploadLinks capability interfaces. Local signed GET/HEAD/PUT uses /storage/object; -set GO_OMNI_STORAGE_SIGNING_KEY (32+ bytes) and GO_OMNI_STORAGE_PUBLIC_URL. +the generated file references GO_OMNI_STORAGE_SIGNING_KEY (32+ bytes) and +GO_OMNI_STORAGE_PUBLIC_URL. File values may instead be fixed explicitly. Cloud credentials use GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY; set bucket and endpoint or R2 accountId as appropriate. Signed URLs grant temporary access: authenticate and authorize before issuing them. No public upload-link route is added. diff --git a/cmd/go-omni/apigen/testdata/configuration/consumer_test.go b/cmd/go-omni/apigen/testdata/configuration/consumer_test.go index ebfd12a..12eab95 100644 --- a/cmd/go-omni/apigen/testdata/configuration/consumer_test.go +++ b/cmd/go-omni/apigen/testdata/configuration/consumer_test.go @@ -6,28 +6,26 @@ import ( "example.com/configuration/internal/config" omniConfig "git.yuntuo.hk/k2safe/go-omni/config" "os" + "os/exec" "path/filepath" "strings" "testing" ) func TestGeneratedConfiguration(t *testing.T) { - for ext, source := range map[string]string{ - "json": `{"limit":7}`, "yaml": "limit: 7\n", "xml": "7", "custom": `{"limit":7}`, - } { + for ext, source := range map[string]string{"json": `{"limit":"${LIMIT:-7}"}`, "yaml": "limit: '${LIMIT:-7}'\n", "xml": "${LIMIT:-7}", "custom": `{"limit":"${LIMIT:-7}"}`} { t.Run(ext, func(t *testing.T) { + t.Setenv("LIMIT", "") + t.Setenv("GO_OMNI_LIMIT", "999") path := filepath.Join(t.TempDir(), "app."+ext) if err := os.WriteFile(path, []byte(source), 0600); err != nil { t.Fatal(err) } cfg, err := config.Load(path) - if err != nil { - t.Fatal(err) + if err != nil || cfg.Limit != 7 { + t.Fatal(cfg.Limit, err) } - if cfg.Limit != 7 { - t.Fatal(cfg.Limit) - } - t.Setenv("GO_OMNI_LIMIT", "0") + t.Setenv("LIMIT", "0") cfg, err = config.Load(path) if err != nil || cfg.Limit != 0 { t.Fatal(cfg.Limit, err) @@ -36,24 +34,60 @@ func TestGeneratedConfiguration(t *testing.T) { } t.Setenv("GO_OMNI_LOG_OUTPUT", "stderr") cfg, err := config.Load("") - if err != nil { - t.Fatal(err) - } - if cfg.Limit != 10 || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled { - t.Fatal("defaults/log normalization", cfg, err) + if err != nil || cfg.Limit != 10 || cfg.Server.Log.File != "./var/log/app.log" || !cfg.Server.Log.Rotation.Enabled { + t.Fatal("implicit overlay", err) } path := filepath.Join(t.TempDir(), "bad.yaml") - os.WriteFile(path, []byte("limit: -1\n"), 0600) - // Validation must run before svc.Initialize; the project initializer panics - // below if reached, so this also detects accidentally starting dependencies. + os.WriteFile(path, []byte("limit: '${LIMIT}'\n"), 0600) + t.Setenv("LIMIT", "-1") + // svc.Init panics if reached. Both invalid startup and successful check must + // stop before opening the configured file logger or project dependencies. err = run(context.Background(), path) - var diagnostic *omniConfig.Error - if !errors.As(err, &diagnostic) || diagnostic.Field != "limit" || diagnostic.File != path || diagnostic.Line != 1 { + var d *omniConfig.Error + if !errors.As(err, &d) || d.Field != "limit" || d.File != path || d.Line != 1 || d.Env != "LIMIT" { t.Fatal(err) } - t.Setenv("GO_OMNI_LIMIT", "private-not-a-number") + t.Setenv("LIMIT", "private-invalid") _, err = config.Load(path) - if !errors.As(err, &diagnostic) || diagnostic.Env != "GO_OMNI_LIMIT" || strings.Contains(err.Error(), "private-not-a-number") { + if !errors.As(err, &d) || d.Env != "LIMIT" || strings.Contains(err.Error(), "private-invalid") { t.Fatal(err) } } +func TestConfigurationCheckProcess(t *testing.T) { + exe := filepath.Join(t.TempDir(), "server") + cmd := exec.Command("go", "build", "-o", exe, ".") + if out, err := cmd.CombinedOutput(); err != nil { + t.Fatal(err, string(out)) + } + dir := t.TempDir() + path := filepath.Join(dir, "selected.yaml") + if err := os.WriteFile(path, []byte("limit: '${LIMIT:-0}'\nserver:\n address: 127.0.0.1:1\n"), 0600); err != nil { + t.Fatal(err) + } + // A broken neighboring default/local/test file must never be layered in. + for _, name := range []string{"config.yaml", "config.local.yaml", "config.test.yaml"} { + os.WriteFile(filepath.Join(dir, name), []byte("broken: ["), 0600) + } + cmd = exec.Command(exe, "-config", path, "-check-config") + cmd.Dir = dir + if out, err := cmd.CombinedOutput(); err != nil || len(out) != 0 { + t.Fatal(err, string(out)) + } + if _, err := os.Stat(filepath.Join(dir, "var")); !os.IsNotExist(err) { + t.Fatal("check opened logger/dependencies") + } + os.WriteFile(path, []byte("limit: '${MISSING_REQUIRED}'"), 0600) + cmd = exec.Command(exe, "-config", path, "-check-config") + cmd.Dir = dir + cmd.Env = append(os.Environ(), "MISSING_REQUIRED=") + out, err := cmd.CombinedOutput() + if err == nil || !strings.Contains(string(out), "MISSING_REQUIRED") || strings.Contains(string(out), "panic") { + t.Fatal(err, string(out)) + } + cmd = exec.Command(exe, "-config", path) + cmd.Dir = dir + cmd.Env = append(os.Environ(), "MISSING_REQUIRED=") + if out, err := cmd.CombinedOutput(); err == nil || strings.Contains(string(out), "panic") { + t.Fatal(err, string(out)) + } +} diff --git a/cmd/go-omni/apigen/testdata/managed/consumer_test.go b/cmd/go-omni/apigen/testdata/managed/consumer_test.go index c54b32c..d7bd97c 100644 --- a/cmd/go-omni/apigen/testdata/managed/consumer_test.go +++ b/cmd/go-omni/apigen/testdata/managed/consumer_test.go @@ -123,7 +123,7 @@ func TestGeneratedTransfers(t *testing.T) { func configFile(t *testing.T, address string, fail bool) string { t.Helper() path := filepath.Join(t.TempDir(), "config.json") - data, _ := json.Marshal(map[string]any{"server": map[string]any{"address": address}, "failInit": fail}) + data, _ := json.Marshal(map[string]any{"server": map[string]any{"address": address, "transport": "${GO_OMNI_TRANSPORT:-nethttp}"}, "failInit": fail}) if err := os.WriteFile(path, data, 0600); err != nil { t.Fatal(err) } diff --git a/cmd/go-omni/apigen/testdata/managed/phases_test.go b/cmd/go-omni/apigen/testdata/managed/phases_test.go index e53776e..0b5ae34 100644 --- a/cmd/go-omni/apigen/testdata/managed/phases_test.go +++ b/cmd/go-omni/apigen/testdata/managed/phases_test.go @@ -24,7 +24,7 @@ func phasedConfig(t *testing.T, address, mode string, fail bool) string { if mode == "stuck" { drainTimeout = "50ms" } - data, _ := json.Marshal(map[string]any{"backgroundMode": mode, "failInit": fail, "server": map[string]any{"address": address, "lifecycle": map[string]string{"stopTimeout": "1s", "drainTimeout": drainTimeout, "cancelTimeout": "50ms"}}}) + data, _ := json.Marshal(map[string]any{"backgroundMode": mode, "failInit": fail, "server": map[string]any{"address": address, "transport": "${GO_OMNI_TRANSPORT:-nethttp}", "lifecycle": map[string]string{"stopTimeout": "1s", "drainTimeout": drainTimeout, "cancelTimeout": "50ms"}}}) if err := os.WriteFile(path, data, 0600); err != nil { t.Fatal(err) } diff --git a/cmd/go-omni/apigen/upgrade.go b/cmd/go-omni/apigen/upgrade.go index 6c81c1e..4ff44f5 100644 --- a/cmd/go-omni/apigen/upgrade.go +++ b/cmd/go-omni/apigen/upgrade.go @@ -14,7 +14,9 @@ import ( "golang.org/x/mod/semver" "os" "path/filepath" + "reflect" "sort" + "strconv" "strings" ) @@ -54,6 +56,22 @@ func checkConfigLoadExtension(root string) error { if err != nil { return fmt.Errorf("parse %s: %w", rel, err) } + var removed bool + ast.Inspect(f, func(n ast.Node) bool { + field, ok := n.(*ast.Field) + if ok && field.Tag != nil { + if raw, e := strconv.Unquote(field.Tag.Value); e == nil { + tag := reflect.StructTag(raw) + _, a := tag.Lookup("env") + _, b := tag.Lookup("envDefault") + removed = removed || a || b + } + } + return true + }) + if removed { + return fmt.Errorf("%s: env/envDefault tags were removed; migrate defaults and explicit file references before generating; see CONFIG.md", rel) + } for _, decl := range f.Decls { if fn, ok := decl.(*ast.FuncDecl); ok && fn.Recv == nil && fn.Name.Name == "Load" { return fmt.Errorf("%s: migrate the project-owned Load function before generating internal/config/loadGen.go; retain Config/Default/Validate and move decoder options to NewLoader; see CONFIG.md", rel) diff --git a/cmd/go-omni/main_test.go b/cmd/go-omni/main_test.go index bfed258..dd30284 100644 --- a/cmd/go-omni/main_test.go +++ b/cmd/go-omni/main_test.go @@ -8,7 +8,7 @@ import ( ) func TestNewWithRemoteFrameworkVersion(t *testing.T) { - for _, flag := range [][]string{{"--framework-version", "v0.21.0-preview"}, {"--framework-version=v0.21.0-preview"}} { + for _, flag := range [][]string{{"--framework-version", "v0.22.0-preview"}, {"--framework-version=v0.22.0-preview"}} { project := filepath.Join(t.TempDir(), "billing") args := append([]string{"--module", "example.com/billing"}, flag...) if err := runNew(append(args, project)); err != nil { @@ -18,7 +18,7 @@ func TestNewWithRemoteFrameworkVersion(t *testing.T) { if err != nil { t.Fatal(err) } - if !strings.Contains(string(content), "go-omni v0.21.0-preview") || strings.Contains(string(content), "replace ") { + if !strings.Contains(string(content), "go-omni v0.22.0-preview") || strings.Contains(string(content), "replace ") { t.Fatalf("CLI did not pin remote dependency: %s", content) } } diff --git a/cmd/go-omni/scaffold/moduleProfiles_test.go b/cmd/go-omni/scaffold/moduleProfiles_test.go index a1a47eb..7610fb7 100644 --- a/cmd/go-omni/scaffold/moduleProfiles_test.go +++ b/cmd/go-omni/scaffold/moduleProfiles_test.go @@ -66,9 +66,13 @@ func TestGenerateModuleProfiles(t *testing.T) { t.Fatal("replacement does not resolve", err) } cfg, _ := os.ReadFile(filepath.Join(project, "etc/config.yaml")) - if strings.Contains(string(cfg), "secretKey:") || strings.Contains(string(cfg), "filesToken:") { - t.Fatal("secret defaults emitted") + for _, line := range strings.Split(string(cfg), "\n") { + key, value, ok := strings.Cut(strings.TrimSpace(line), ":") + if ok && (key == "secretKey" || key == "filesToken" || key == "signingKey") && !strings.Contains(value, "${GO_OMNI_") { + t.Fatal("secret must be an explicit reference") + } } + }) } } @@ -76,6 +80,7 @@ func TestModuleOptionsRejectBeforeWriting(t *testing.T) { for _, options := range []Options{ {LogOutput: "invalid"}, {LogFormat: "xml"}, {Storage: "ftp"}, {Storage: "local", FrameworkVersion: "v0.6.0-preview"}, + {FrameworkVersion: "v0.21.0-preview"}, {Module: "example.com/x\nrequire malicious v1.0.0"}, } { options.Dir = filepath.Join(t.TempDir(), "project") diff --git a/cmd/go-omni/scaffold/scaffold.go b/cmd/go-omni/scaffold/scaffold.go index a538f58..6a09f2b 100644 --- a/cmd/go-omni/scaffold/scaffold.go +++ b/cmd/go-omni/scaffold/scaffold.go @@ -34,9 +34,13 @@ type Options struct { } var files = map[string]string{ + "internal/settings/config.go": "templates/modules-settings.go.tmpl", + "internal/settings/config_test.go": "templates/modules-settings_test.go.tmpl", + "etc/config.yaml": "templates/modules-config.yaml.tmpl", + "go.mod": "templates/go.mod.tmpl", "README.md": "templates/README.md.tmpl", - "cmd/server/main.go": "templates/cmd-server-main.go.tmpl", + "cmd/server/main.go": "templates/modules-main.go.tmpl", "internal/model/user.go": "templates/internal-model-user.go.tmpl", "internal/model/errors.go": "templates/internal-model-errors.go.tmpl", "internal/repository/users.go": "templates/internal-repository-users.go.tmpl", @@ -109,7 +113,7 @@ func Generate(options Options) error { if options.FrameworkDir != "" || options.FrameworkVersion == "" { replace = frameworkReplace(directory, options.FrameworkDir) } - if modules && replace == "" && semver.Compare(version, modulesFrameworkVersion) < 0 && version != installedFrameworkVersion() { + if replace == "" && semver.Compare(version, modulesFrameworkVersion) < 0 && version != installedFrameworkVersion() { return fmt.Errorf("module templates require %s or newer; use --framework-dir for unpublished source", modulesFrameworkVersion) } data := templateData{ diff --git a/cmd/go-omni/scaffold/scaffold_test.go b/cmd/go-omni/scaffold/scaffold_test.go index d7e1f42..05fdfcc 100644 --- a/cmd/go-omni/scaffold/scaffold_test.go +++ b/cmd/go-omni/scaffold/scaffold_test.go @@ -39,7 +39,7 @@ func TestGenerateCreatesServiceLayout(t *testing.T) { if err != nil { t.Fatalf("read generated server: %v", err) } - for _, expected := range []string{"config.LoadFromEnv(\"\")", "logx.SetDefault(logger)", "omni.WithConfig(settings)", "servicecontext.New(ctx, logger)", "services.Health.ReadinessHandler()", "application.RunContext(ctx, settings.Address)"} { + for _, expected := range []string{"settings.Load(path)", "logx.SetDefault(logger)", "omni.WithConfig(cfg.Server)", "servicecontext.New(ctx, logger, cfg)", "services.Health.ReadinessHandler()", "application.RunContext(ctx, cfg.Server.Address)"} { if !strings.Contains(string(mainContent), expected) { t.Fatalf("generated server missing %q: %s", expected, mainContent) } @@ -110,7 +110,7 @@ func TestFrameworkReplaceUsesCanonicalPaths(t *testing.T) { } func TestGenerateRemoteProjectPinsVersion(t *testing.T) { - for _, version := range []string{"v0.1.0", "v1.2.3-rc.1", "v0.0.0-20260920140624-8b19aa79eea0"} { + for _, version := range []string{"v0.22.0-preview", "v1.2.3-rc.1", "v0.22.1-0.20260926000000-8b19aa79eea0"} { t.Run(version, func(t *testing.T) { project := filepath.Join(t.TempDir(), "billing") if err := Generate(Options{Dir: project, Module: "example.com/billing", FrameworkVersion: version}); err != nil { diff --git a/cmd/go-omni/scaffold/templates/README.md.tmpl b/cmd/go-omni/scaffold/templates/README.md.tmpl index 5984dba..32c083b 100644 --- a/cmd/go-omni/scaffold/templates/README.md.tmpl +++ b/cmd/go-omni/scaffold/templates/README.md.tmpl @@ -32,46 +32,44 @@ go mod tidy go run ./cmd/server ``` -运行时配置从 `GO_OMNI_*` 环境变量加载，例如： +配置入口是 `internal/settings.Config`、`Default()`、`Validate()` 和 `etc/config.yaml`。 +只有文件里的 `${NAME}` / `${NAME:-fallback}` 会读取环境变量，固定值不受进程环境覆盖。生成的文件已声明地址、网络层，以及所选数据库/存储的必要引用；其他参数直接写进文件。 -```sh -GO_OMNI_ENABLE_LOGGER=true GO_OMNI_LOG_LEVEL=debug go run ./cmd/server +```yaml +server: + address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}" + transport: "${GO_OMNI_TRANSPORT:-nethttp}" + log: + level: debug + shutdownTimeout: 10s ``` -`GO_OMNI_HTTP_ADDR` 可以修改监听地址。生成项目默认提供 `/health/live` 和 -`/health/ready`，日志、请求 ID 和优雅退出等基础能力由 GoOmni 统一初始化；业务依赖集中放在项目自己的 `internal/servicecontext`。 - -服务使用 `signal.NotifyContext` 处理 `SIGINT` / `SIGTERM`。部署时构建并运行服务二进制， -使信号直接到达服务进程： +用 `-config` 选择一个 YAML/JSON/XML 文件；`-config ''` 只使用 Default()；`-check-config` 仅加载和校验，不创建依赖或监听： ```sh +go run ./cmd/server -config etc/config.yaml -check-config go build -o server ./cmd/server -./server +./server -config etc/config.yaml ``` -`GO_OMNI_SHUTDOWN_TIMEOUT` 控制等待请求完成的时间，默认 `10s`。 +服务通过 `signal.NotifyContext` 处理 `SIGINT` / `SIGTERM`，先排空 HTTP 请求，再关闭依赖和日志。生成项目默认提供 `/health/live` 和 `/health/ready`，业务依赖集中在 `internal/servicecontext`。 -使用支持统一网络配置的框架版本时，设置 `GO_OMNI_TRANSPORT=native` 即可切换为自研 HTTP/1.1； -默认 `nethttp`。`GO_OMNI_TLS_CERT_FILE` 和 `GO_OMNI_TLS_KEY_FILE` 配对启用 HTTPS。 -实验版 HTTP/2 使用 `GO_OMNI_TRANSPORT=http2`，必须同时配置 TLS 证书和私钥。 -业务 Handler / Service / Repository 不需要修改；三种适配器共用上述退出等待时间。 +通过文件中的引用设置 `GO_OMNI_TRANSPORT=native` 可切换自研 HTTP/1.1，默认 nethttp。HTTPS 在文件的 `server.tls.certFile` / `server.tls.keyFile` 成对配置；实验 HTTP/2 选择 `http2` 且必须配置 TLS。不会自动读取未在文件声明的 TLS 环境变量。 {{if eq .Database "mysql"}} -`/health/ready` 已注册数据库探针：连接失败返回 `503`，数据库恢复后重新返回 `200`。 -`/health/live` 始终只检查服务存活。`GO_OMNI_DB_PING_TIMEOUT` 限制初始化检查和数据库探针的等待时间。 -退出时先排空 HTTP 请求，再关闭数据库连接池，最后关闭日志。 +`/health/ready` 已注册数据库探针，连接失败返回 503，恢复后返回 200；`/health/live` 只检查进程存活。连接池参数直接写入 `database`，缺失字段保留 Default()： -数据库连接池参数： - -```text -GO_OMNI_DB_MAX_OPEN_CONNS=10 -GO_OMNI_DB_MAX_IDLE_CONNS=5 -GO_OMNI_DB_CONN_MAX_LIFETIME=30m -GO_OMNI_DB_CONN_MAX_IDLE_TIME=5m -GO_OMNI_DB_PING_TIMEOUT=5s +```yaml +database: + driver: mysql + dsn: "${GO_OMNI_DB_DSN}" + maxOpenConns: 10 + maxIdleConns: 5 + connMaxLifetime: 30m + connMaxIdleTime: 5m + pingTimeout: 5s ``` -`MAX_IDLE_CONNS=0` 禁用空闲连接，`MAX_OPEN_CONNS=0` 表示不限制连接数。 -项目会启用 MySQL `parseTime` 并按 UTC 读取 DATETIME 字段，JSON 字段保持 lowerCamelCase。 +`maxIdleConns: 0` 禁用空闲连接；此项目要求 maxOpenConns 为正数。项目启用 MySQL parseTime，并按 UTC 读取 DATETIME；JSON 字段保持 lowerCamelCase。 事务内复用 Repository（在业务代码中导入 `database/sql` 和项目的 `internal/repository`）： diff --git a/cmd/go-omni/scaffold/templates/api-config.go.tmpl b/cmd/go-omni/scaffold/templates/api-config.go.tmpl index 8b47982..289afc8 100644 --- a/cmd/go-omni/scaffold/templates/api-config.go.tmpl +++ b/cmd/go-omni/scaffold/templates/api-config.go.tmpl @@ -11,8 +11,8 @@ import ( ) type Config struct { Server config.Config `json:"server"` - {{if eq .Database "mysql"}}Database mysqladapter.Config `json:"database" env:"GO_OMNI_DB"`{{end}} - {{if ne .Storage "none"}}Storage StorageConfig `json:"storage" env:"GO_OMNI_STORAGE"`{{end}} + {{if eq .Database "mysql"}}Database mysqladapter.Config `json:"database"`{{end}} + {{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}} } func Default() Config { server:=config.Default() @@ -54,12 +54,12 @@ type StorageConfig struct { ContentTypes []string `json:"contentTypes"` LinkTTL time.Duration `json:"linkTtl"` RequestTimeout time.Duration `json:"requestTimeout"` - // Secrets are environment-only and cannot be populated by config files. - FilesToken string `json:"-" env:"GO_OMNI_FILES_TOKEN,secret"` - SigningKey string `json:"-" env:"GO_OMNI_STORAGE_SIGNING_KEY,secret"` - AccessKey string `json:"-" env:"GO_OMNI_STORAGE_ACCESS_KEY,secret"` - SecretKey string `json:"-" env:"GO_OMNI_STORAGE_SECRET_KEY,secret"` - SessionToken string `json:"-" env:"GO_OMNI_STORAGE_SESSION_TOKEN,secret"` + // File values may explicitly reference environment variables. Never log this struct. + FilesToken string `json:"filesToken"` + SigningKey string `json:"signingKey"` + AccessKey string `json:"accessKey"` + SecretKey string `json:"secretKey"` + SessionToken string `json:"sessionToken"` } func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}} func (s StorageConfig) Validate()error { @@ -67,14 +67,14 @@ func (s StorageConfig) Validate()error { if err:=s.Policy().Validate();err!=nil{return err} if s.LinkTTLtime.Hour{return errors.New("storage.linkTtl must be 1s..1h")} if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")} - {{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("set GO_OMNI_FILES_TOKEN to a token of at least 32 bytes without whitespace")}{{end}} + {{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("storage.filesToken requires at least 32 bytes without whitespace")}{{end}} {{if eq .Storage "local"}} if s.Root=="" {return errors.New("storage.root is required")} u,err:=url.Parse(s.PublicURL) if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")} - if len(s.SigningKey)<32 {return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")} + if len(s.SigningKey)<32 {return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")} {{else}} - if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region and GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY are required")} + if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region, accessKey and secretKey are required")} {{if eq .Storage "r2"}} if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")} if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")} diff --git a/cmd/go-omni/scaffold/templates/api-load.go.tmpl b/cmd/go-omni/scaffold/templates/api-load.go.tmpl index 51b82fe..6d676fe 100644 --- a/cmd/go-omni/scaffold/templates/api-load.go.tmpl +++ b/cmd/go-omni/scaffold/templates/api-load.go.tmpl @@ -1,13 +1,8 @@ package config - import omniConfig "git.yuntuo.hk/k2safe/go-omni/config" - -// Load is maintained by the generator. Extend Config/Default/Validate in -// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline. -func Load(path string) (Config, error) { - loader,err:=NewLoader() - if err!=nil{return Config{},err} - return omniConfig.LoadProject(path,Default(),omniConfig.EnvOptions{ - Prefix:"GO_OMNI_",Empty:"ignore",Aliases:omniConfig.ServerEnvAliases("GO_OMNI_","server"), - },loader) +// Load resolves explicit file references, binds onto Default and validates. +// Ordinary projects only edit Config, Default, Validate and the selected file. +func Load(path string)(Config,error){ + loader,err:=NewLoader();if err!=nil{return Config{},err} + return omniConfig.LoadProject(path,Default(),loader) } diff --git a/cmd/go-omni/scaffold/templates/cmd-server-main.go.tmpl b/cmd/go-omni/scaffold/templates/cmd-server-main.go.tmpl deleted file mode 100644 index e2481f6..0000000 --- a/cmd/go-omni/scaffold/templates/cmd-server-main.go.tmpl +++ /dev/null @@ -1,57 +0,0 @@ -package main - -import ( - "context" - "log" - "os" - "os/signal" - "strings" - "syscall" - - omni "git.yuntuo.hk/k2safe/go-omni" - "git.yuntuo.hk/k2safe/go-omni/config" - "git.yuntuo.hk/k2safe/go-omni/logx" - "{{.Module}}/internal/handler" - "{{.Module}}/internal/service" - "{{.Module}}/internal/servicecontext" -) - -func main() { - if err := run(); err != nil { - log.Fatal(err) - } -} - -func run() error { - ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) - defer stop() - - settings, err := config.LoadFromEnv("") - if err != nil { - return err - } - if strings.TrimSpace(settings.Log.Service) == "" { - settings.Log.Service = "{{.Module}}" - } - logger, err := logx.New(settings.Log) - if err != nil { - return err - } - defer logger.Close() - // Access logs and business logs share the same configured sink. - logx.SetDefault(logger) - - services, err := servicecontext.New(ctx, logger) - if err != nil { return err } - defer services.Close() - userService := service.NewUsers(services) - userHandler := handler.NewUsers(userService) - - application := omni.New(omni.WithConfig(settings)) - application.OnShutdown(logger.Close) - application.OnShutdown(services.Close) - application.GET("/health/live", services.Health.LivenessHandler()) - application.GET("/health/ready", services.Health.ReadinessHandler()) - userHandler.RegisterRoutes(application) - return application.RunContext(ctx, settings.Address) -} diff --git a/cmd/go-omni/scaffold/templates/databaseIntegration_test.go.tmpl b/cmd/go-omni/scaffold/templates/databaseIntegration_test.go.tmpl index 1c2ff64..d5aaa02 100644 --- a/cmd/go-omni/scaffold/templates/databaseIntegration_test.go.tmpl +++ b/cmd/go-omni/scaffold/templates/databaseIntegration_test.go.tmpl @@ -13,6 +13,7 @@ import ( "git.yuntuo.hk/k2safe/go-omni/database" "git.yuntuo.hk/k2safe/go-omni/logx" "{{.Module}}/internal/model" + "{{.Module}}/internal/settings" "{{.Module}}/internal/repository" "{{.Module}}/internal/servicecontext" ) @@ -23,15 +24,12 @@ func databaseContext(t *testing.T) *servicecontext.Context { t.Helper() dsn := os.Getenv("GO_OMNI_TEST_MYSQL_DSN") if dsn == "" { t.Skip("set GO_OMNI_TEST_MYSQL_DSN to a dedicated test database") } - for _, key := range []string{"MAX_OPEN_CONNS", "MAX_IDLE_CONNS", "CONN_MAX_LIFETIME", "CONN_MAX_IDLE_TIME", "PING_TIMEOUT"} { - t.Setenv("GO_OMNI_DB_"+key, "") - } t.Setenv("GO_OMNI_DB_DSN", dsn) - t.Setenv("GO_OMNI_DB_DRIVER", "mysql") logger, err := logx.New(logx.Config{Writer: io.Discard}) if err != nil { t.Fatal(err) } t.Cleanup(func() { _ = logger.Close() }) - services, err := servicecontext.New(context.Background(), logger) + cfg,err:=settings.Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)} + services, err := servicecontext.New(context.Background(), logger,cfg) if err != nil { t.Fatal(err) } t.Cleanup(func() { _ = services.Close() }) return services diff --git a/cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl b/cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl index bde7108..6ab7ac2 100644 --- a/cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl +++ b/cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl @@ -2,6 +2,7 @@ package servicecontext import ( "context" + "{{.Module}}/internal/settings" {{if and .Modules (ne .Storage "none")}}"errors" "net/http" "sync" @@ -38,16 +39,16 @@ type Context struct { // New initializes the project's dependencies before the HTTP listener opens. // The caller owns Logger; Close releases only resources created here. -func New(ctx context.Context, logger *logx.Logger) (*Context, error) { +func New(ctx context.Context, logger *logx.Logger, project settings.Config) (*Context, error) { if ctx == nil { ctx = context.Background() } if err := ctx.Err(); err != nil { return nil, err } checker, err := health.New() if err != nil { return nil, err } services := &Context{Logger: logger, Health: checker} {{if eq .Database "mysql"}} - cfg, err := database.ConfigFromEnv("") - if err != nil { return nil, err } - if cfg.Driver != "mysql" { return nil, fmt.Errorf("this project requires GO_OMNI_DB_DRIVER=mysql") } + cfg:=project.Database + if err:=cfg.Validate();err!=nil{return nil,err} + if cfg.Driver != "mysql" { return nil, fmt.Errorf("this project requires database.driver=mysql") } dsn, err := mysql.ParseDSN(cfg.DSN) if err != nil { return nil, fmt.Errorf("parse mysql connection settings: %w", err) } if dsn.DBName == "" { return nil, fmt.Errorf("mysql connection settings require a database name") } diff --git a/cmd/go-omni/scaffold/templates/modules-README.md.tmpl b/cmd/go-omni/scaffold/templates/modules-README.md.tmpl index da7e9fb..32736b0 100644 --- a/cmd/go-omni/scaffold/templates/modules-README.md.tmpl +++ b/cmd/go-omni/scaffold/templates/modules-README.md.tmpl @@ -2,7 +2,7 @@ 本项目选用日志输出 `{{.LogOutput}}`、格式 `{{.LogFormat}}`、对象存储 `{{.Storage}}`。数据库选择 `{{.Database}}` 独立于对象存储。 -配置入口为 `etc/config.yaml`，通过 `go run ./cmd/server -config /path/to/config.yaml` 指定其他 JSON/YAML 文件；`-config ''` 只用代码默认值和环境变量。优先级：默认值 → 文件 → 非空环境变量。未知字段、无效参数会在监听前拒绝。 +配置入口为 `etc/config.yaml`，通过 `go run ./cmd/server -config /path/to/config.yaml` 指定其他 JSON/YAML/XML 文件；`-config ''` 只用代码默认值。文件用 `${NAME}` 或 `${NAME:-fallback}` 显式引用变量，固定值不受隐式环境覆盖；`-check-config` 只检查、不初始化依赖。未知字段、无效参数会在监听前拒绝。 ```sh go mod tidy @@ -24,7 +24,7 @@ go run ./cmd/server {{if eq .Database "mysql"}}还需按 README 配置独立 MySQL 数据库并应用 schema.sql；应用不自动迁移。{{end}} -默认仅监听 `127.0.0.1:8080`。`GO_OMNI_HTTP_ADDR` 覆盖地址；本地签名 URL 的外部地址通过 `GO_OMNI_STORAGE_PUBLIC_URL` 单独配置，始终以 `/storage/object` 结尾，不使用请求 Host 自动拼接可信链接。 +默认仅监听 `127.0.0.1:8080`。文件中的 `${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}` 设置地址；本地签名 URL 的外部地址通过 `GO_OMNI_STORAGE_PUBLIC_URL` 单独配置，始终以 `/storage/object` 结尾，不使用请求 Host 自动拼接可信链接。 ## 日志 @@ -35,7 +35,7 @@ curl -H 'Content-Type: application/json' -H 'X-Request-ID: example-request' \ -d '{"name":"Example","email":"example@example.com"}' http://127.0.0.1:8080/users ``` -`GO_OMNI_LOG_LEVEL`、`GO_OMNI_LOG_FORMAT`、`GO_OMNI_LOG_FILE` 覆盖对应项；显式设置 `GO_OMNI_LOG_OUTPUT=stdout` 且未设置 LOG_FILE 时，切换到控制台并关闭文件轮转。慢请求阈值为 500ms，可通过 `GO_OMNI_SLOW_REQUEST_THRESHOLD` 调整。字段脱敏不扫描 msg/error 中的自由文本；不要把凭据或签名 URL 拼到日志中。 +日志参数直接写在 `server.log`，例如 `level: debug`；也可显式写 `level: "${LOG_LEVEL:-info}"`。切换文件日志为控制台时，明确设置 `file: ""`、`output: stdout` 和 `rotation.enabled: false`，不会隐式重置其他字段。慢请求阈值使用 `server.slowRequestThreshold`，默认 500ms。字段脱敏不扫描 msg/error 自由文本；不要把凭据或签名 URL 拼到日志中。 {{if ne .Storage "none"}}## 文件示例 @@ -66,18 +66,18 @@ curl -H "Authorization: Bearer $GO_OMNI_FILES_TOKEN" -H 'Content-Type: applicati 响应包括随机 `data.key` 和 `data.request`（method/url/headers/expiresAt）。按返回的 method 和 headers 用原始文件字节上传到 URL；不能额外包 FormData。浏览器自动填写 Content-Length，其余签名 Header 原样发送；云桶需配置允许业务域名的 CORS。云端直传只绑定声明类型，实际内容需上传后验证；本地和 API 中转上传会检查允许类型及前缀嗅探结果。未完成内容验证前不要把暂存文件当作可信文件公开。 -| 参数 | 默认值 / 环境变量 | +| 文件字段 | 默认值 / 含义 | | --- | --- | -| 大小 | 32 MiB / GO_OMNI_STORAGE_MAX_BYTES | -| 类型 | text/plain、image/png、image/jpeg、application/pdf / GO_OMNI_STORAGE_CONTENT_TYPES（逗号分隔） | -| 链接有效期 | 5m / GO_OMNI_STORAGE_LINK_TTL（1s..1h） | -| 请求超时 | 30s / GO_OMNI_STORAGE_REQUEST_TIMEOUT（>0 且 <=5m） | -| 本地目录 | var/files / GO_OMNI_STORAGE_ROOT | -| 本地外部 URL | http://127.0.0.1:8080/storage/object / GO_OMNI_STORAGE_PUBLIC_URL | -| 云配置 | GO_OMNI_STORAGE_ENDPOINT、REGION、BUCKET、ACCOUNT_ID、SECURE | -| 环境密钥 | GO_OMNI_FILES_TOKEN、GO_OMNI_STORAGE_SIGNING_KEY、GO_OMNI_STORAGE_ACCESS_KEY、SECRET_KEY、SESSION_TOKEN | +| `storage.maxBytes` | 32 MiB | +| `storage.contentTypes` | text/plain、image/png、image/jpeg、application/pdf；文件数组 | +| `storage.linkTtl` | 5m，允许 1s..1h | +| `storage.requestTimeout` | 30s，大于 0 且不超过 5m | +| `storage.root` | var/files | +| `storage.publicUrl` | http://127.0.0.1:8080/storage/object | +| `storage.endpoint/region/bucket/accountId/secure` | 对应平台连接配置 | +| `storage.filesToken/signingKey/accessKey/secretKey/sessionToken` | 文件显式引用环境变量提供凭据 | -全部 JSON/YAML 字段使用 lowerCamelCase。密钥字段仅允许环境变量，配置文件中写入会报未知字段；不要把运行环境文件提交到仓库。S3 的 endpoint 为 host[:port]，OSS 的可选 endpoint 为 HTTPS origin；R2 从 accountId 派生官方 endpoint，region 为 auto。仅 S3/MinIO 的本地测试可设置 SECURE=false。 +全部 JSON/YAML/XML 字段使用 lowerCamelCase。文件可使用固定值或显式环境引用，不按字段名自动推导环境变量；凭据建议使用 `${NAME}`，不把含秘密的配置提交到仓库。S3 endpoint 为 host[:port]，OSS 可选 endpoint 为 HTTPS origin；R2 从 accountId 派生官方 endpoint，region 为 auto。仅 S3/MinIO 本地测试可设置 `storage.secure: false`。 错误响应：未授权 401、无 Content-Length 411、超限 413、类型不允许 415、长度不符/非法 key 400、对象不存在 404、存储超时 504，其余内部错误统一隐藏细节。底层签名入口遵循存储协议返回 201/403 等状态，流式传输且访问日志保留实际状态。 diff --git a/cmd/go-omni/scaffold/templates/modules-config.yaml.tmpl b/cmd/go-omni/scaffold/templates/modules-config.yaml.tmpl index aa25620..5acd801 100644 --- a/cmd/go-omni/scaffold/templates/modules-config.yaml.tmpl +++ b/cmd/go-omni/scaffold/templates/modules-config.yaml.tmpl @@ -1,68 +1,29 @@ -# Non-secret settings only. Environment variables override these values. +# Only fields listed here override Default(). Environment names are explicit. server: - address: '127.0.0.1:8080' - enableLogger: true - slowRequestThreshold: 500ms - shutdownTimeout: 10s -{{if .API}} lifecycle: - stopTimeout: 5s - drainTimeout: 10s - cancelTimeout: 5s - endpoints: - liveness: - enabled: true - path: /health/live - responseMode: app - readiness: - enabled: true - path: /health/ready - responseMode: app - openapi: - enabled: true - path: /openapi.json -{{end}} - log: - service: {{printf "%q" .Module}} - level: info - format: {{.LogFormat}} -{{if eq .LogOutput "file"}} file: './var/log/server.log' - rotation: - enabled: true - maxSizeMb: 100 - maxBackups: 10 - maxAgeDays: 7 - compress: true -{{else}} output: {{.LogOutput}} -{{end}} redactKeys: [] + address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}" + transport: "${GO_OMNI_TRANSPORT:-nethttp}" {{if ne .Storage "none"}}storage: - provider: {{.Storage}} - maxBytes: 33554432 - contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf'] - linkTtl: 5m - requestTimeout: 30s -{{if eq .Storage "local"}} root: './var/files' - publicUrl: 'http://127.0.0.1:8080/storage/object' -{{else}} bucket: '' # set GO_OMNI_STORAGE_BUCKET - secure: true -{{if eq .Storage "r2"}} accountId: '' # set GO_OMNI_STORAGE_ACCOUNT_ID - region: auto -{{else if eq .Storage "s3"}} endpoint: '' # set GO_OMNI_STORAGE_ENDPOINT (host[:port]) - region: us-east-1 -{{else}} region: cn-hangzhou - endpoint: '' # empty uses the official regional endpoint +{{if not .API}} filesToken: "${GO_OMNI_FILES_TOKEN}" +{{end}}{{if eq .Storage "local"}} root: "${GO_OMNI_STORAGE_ROOT:-var/files}" + publicUrl: "${GO_OMNI_STORAGE_PUBLIC_URL:-http://127.0.0.1:8080/storage/object}" + signingKey: "${GO_OMNI_STORAGE_SIGNING_KEY}" +{{else}} bucket: "${GO_OMNI_STORAGE_BUCKET}" + accessKey: "${GO_OMNI_STORAGE_ACCESS_KEY}" + secretKey: "${GO_OMNI_STORAGE_SECRET_KEY}" + sessionToken: "${GO_OMNI_STORAGE_SESSION_TOKEN:-}" +{{if eq .Storage "r2"}} accountId: "${GO_OMNI_STORAGE_ACCOUNT_ID}" +{{else if eq .Storage "s3"}} endpoint: "${GO_OMNI_STORAGE_ENDPOINT}" + region: "${GO_OMNI_STORAGE_REGION:-us-east-1}" +{{else}} region: "${GO_OMNI_STORAGE_REGION:-cn-hangzhou}" {{end}}{{end}}{{end}} - {{if and .API (eq .Database "mysql")}}database: - host: 127.0.0.1 - port: 3306 - database: '' # GO_OMNI_DB_DATABASE - user: '' # GO_OMNI_DB_USER; password via GO_OMNI_DB_PASSWORD - tls: - mode: required - pool: - maxOpenConns: 10 - maxIdleConns: 5 - connMaxLifetime: 30m - connMaxIdleTime: 5m - pingTimeout: 5s + host: "${GO_OMNI_DB_HOST:-127.0.0.1}" + port: "${GO_OMNI_DB_PORT:-3306}" + database: "${GO_OMNI_DB_DATABASE}" + user: "${GO_OMNI_DB_USER}" + password: "${GO_OMNI_DB_PASSWORD}" +{{end}} +{{if and (not .API) (eq .Database "mysql")}}database: + driver: mysql + dsn: "${GO_OMNI_DB_DSN}" {{end}} diff --git a/cmd/go-omni/scaffold/templates/modules-main.go.tmpl b/cmd/go-omni/scaffold/templates/modules-main.go.tmpl index a762f12..275b469 100644 --- a/cmd/go-omni/scaffold/templates/modules-main.go.tmpl +++ b/cmd/go-omni/scaffold/templates/modules-main.go.tmpl @@ -18,8 +18,10 @@ import ( ) func main(){ - path:=flag.String("config","etc/config.yaml","JSON/YAML settings; empty uses defaults and environment") + path:=flag.String("config","etc/config.yaml","configuration file; empty uses only Default()") + check:=flag.Bool("check-config",false,"validate without starting dependencies") flag.Parse() + if *check {if _,err:=settings.Load(*path);err!=nil{log.Fatal(err)};return} if err:=run(*path);err!=nil{log.Fatal(err)} } func run(path string)(runErr error){ @@ -31,7 +33,7 @@ func run(path string)(runErr error){ if err!=nil{return err} defer func(){runErr=errors.Join(runErr,logger.Close())}() logx.SetDefault(logger) - services,err:=servicecontext.New(ctx,logger) + services,err:=servicecontext.New(ctx,logger,cfg) if err!=nil{return err} defer func(){runErr=errors.Join(runErr,services.Close())}() {{if ne .Storage "none"}}if err=services.InitStorage(ctx,cfg.Storage);err!=nil{return err}{{end}} diff --git a/cmd/go-omni/scaffold/templates/modules-settings.go.tmpl b/cmd/go-omni/scaffold/templates/modules-settings.go.tmpl index 41b8742..425fbaa 100644 --- a/cmd/go-omni/scaffold/templates/modules-settings.go.tmpl +++ b/cmd/go-omni/scaffold/templates/modules-settings.go.tmpl @@ -3,21 +3,19 @@ package settings import ( "errors" - "os" "strings" "time" - {{if ne .Storage "none"}}"fmt" + {{if ne .Storage "none"}} {{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}} - "strconv" "git.yuntuo.hk/k2safe/go-omni/storage"{{end}} - {{if and .API (eq .Database "mysql")}}"git.yuntuo.hk/k2safe/go-omni/database"{{end}} + {{if eq .Database "mysql"}}"git.yuntuo.hk/k2safe/go-omni/database"{{end}} "git.yuntuo.hk/k2safe/go-omni/config" - "git.yuntuo.hk/k2safe/go-omni/logx" + {{if eq .LogOutput "file"}}"git.yuntuo.hk/k2safe/go-omni/logx"{{end}} ) type Config struct { Server config.Config `json:"server"` - {{if and .API (eq .Database "mysql")}}Database database.Config `json:"database"`{{end}} + {{if eq .Database "mysql"}}Database database.Config `json:"database"`{{end}} {{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}} } @@ -31,7 +29,8 @@ func Default() Config { {{if eq .LogOutput "file"}}server.Log.File="./var/log/server.log" server.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:100,MaxBackups:10,MaxAgeDays:7,Compress:true} {{else}}server.Log.Output={{printf "%q" .LogOutput}}{{end}} - return Config{Server:server{{if and .API (eq .Database "mysql")}},Database:database.DefaultConfig(){{end}}{{if ne .Storage "none"}},Storage:StorageConfig{ + {{if eq .Database "mysql"}}db:=database.DefaultConfig();db.Driver="mysql"{{end}} + return Config{Server:server{{if eq .Database "mysql"}},Database:db{{end}}{{if ne .Storage "none"}},Storage:StorageConfig{ Provider:{{printf "%q" .Storage}}, Root:"./var/files", PublicURL:"http://127.0.0.1:8080/storage/object", Secure:true, MaxBytes:32<<20, ContentTypes:[]string{"text/plain","image/png","image/jpeg","application/pdf"}, LinkTTL:5*time.Minute, RequestTimeout:30*time.Second, @@ -41,27 +40,10 @@ func Default() Config { }{{end}}} } -// Load uses defaults -> strict YAML/JSON -> nonempty environment variables. -// Pass an empty path to use only defaults and environment. -func Load(path string)(Config,error) { - cfg:=Default() - var err error - if path!="" { cfg,err=config.DecodeFile(path,cfg);if err!=nil{return Config{},err} } - cfg.Server,err=config.ApplyEnv(cfg.Server,"") - if err!=nil{return Config{},err} - // A stream explicitly selected by environment overrides a file profile. - if output:=strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT"));output!="" && strings.TrimSpace(os.Getenv("GO_OMNI_LOG_FILE"))=="" { - cfg.Server.Log.File="";cfg.Server.Log.Rotation=logx.RotationConfig{} - } - {{if ne .Storage "none"}} - if err=cfg.Storage.applyEnv();err!=nil{return Config{},err} - {{end}} - {{if and .API (eq .Database "mysql")}}cfg.Database,err=database.ApplyEnv(cfg.Database,"");if err!=nil{return Config{},err}{{end}} - if err=cfg.Validate();err!=nil{return Config{},err} - return cfg,nil -} +// Load uses Default(), the selected file's explicit references, and Validate. +func Load(path string)(Config,error){return config.LoadProject(path,Default())} func (c Config) Validate()error { - {{if and .API (eq .Database "mysql")}} + {{if eq .Database "mysql"}} if err:=c.Database.Validate();err!=nil{return err} if c.Database.Driver!="mysql" || c.Database.MaxOpenConns<1 {return errors.New("database requires driver mysql and a positive maxOpenConns")} {{end}} @@ -85,45 +67,27 @@ type StorageConfig struct { ContentTypes []string `json:"contentTypes"` LinkTTL time.Duration `json:"linkTtl"` RequestTimeout time.Duration `json:"requestTimeout"` - // Secrets are environment-only and cannot be populated by config files. - FilesToken string `json:"-"` - SigningKey string `json:"-"` - AccessKey string `json:"-"` - SecretKey string `json:"-"` - SessionToken string `json:"-"` + // Use explicit environment references in the configuration file for secrets. + FilesToken string `json:"filesToken"` + SigningKey string `json:"signingKey"` + AccessKey string `json:"accessKey"` + SecretKey string `json:"secretKey"` + SessionToken string `json:"sessionToken"` } func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}} -func (s *StorageConfig) applyEnv()error { - values:=map[string]*string{ - "GO_OMNI_STORAGE_ROOT":&s.Root,"GO_OMNI_STORAGE_PUBLIC_URL":&s.PublicURL, - "GO_OMNI_STORAGE_ENDPOINT":&s.Endpoint,"GO_OMNI_STORAGE_REGION":&s.Region, - "GO_OMNI_STORAGE_BUCKET":&s.Bucket,"GO_OMNI_STORAGE_ACCOUNT_ID":&s.AccountID, - "GO_OMNI_STORAGE_SIGNING_KEY":&s.SigningKey,"GO_OMNI_FILES_TOKEN":&s.FilesToken, - "GO_OMNI_STORAGE_ACCESS_KEY":&s.AccessKey,"GO_OMNI_STORAGE_SECRET_KEY":&s.SecretKey, - "GO_OMNI_STORAGE_SESSION_TOKEN":&s.SessionToken, - } - for name,target:=range values {if value:=strings.TrimSpace(os.Getenv(name));value!=""{*target=value}} - for name,target:=range map[string]*time.Duration{"GO_OMNI_STORAGE_LINK_TTL":&s.LinkTTL,"GO_OMNI_STORAGE_REQUEST_TIMEOUT":&s.RequestTimeout} { - if value:=strings.TrimSpace(os.Getenv(name));value!="" {v,err:=time.ParseDuration(value);if err!=nil{return fmt.Errorf("%s must be a duration",name)};*target=v} - } - if value:=os.Getenv("GO_OMNI_STORAGE_MAX_BYTES");value!=""{v,err:=strconv.ParseInt(value,10,64);if err!=nil{return errors.New("GO_OMNI_STORAGE_MAX_BYTES must be an integer")};s.MaxBytes=v} - if value:=os.Getenv("GO_OMNI_STORAGE_SECURE");value!=""{v,err:=strconv.ParseBool(value);if err!=nil{return errors.New("GO_OMNI_STORAGE_SECURE must be a boolean")};s.Secure=v} - if value:=strings.TrimSpace(os.Getenv("GO_OMNI_STORAGE_CONTENT_TYPES"));value!="" {s.ContentTypes=strings.Split(value,",");for i:=range s.ContentTypes{s.ContentTypes[i]=strings.TrimSpace(s.ContentTypes[i])}} - return nil -} func (s StorageConfig) Validate()error { if s.Provider!={{printf "%q" .Storage}} {return errors.New("storage.provider must match the adapter selected at generation")} if err:=s.Policy().Validate();err!=nil{return err} if s.LinkTTLtime.Hour{return errors.New("storage.linkTtl must be 1s..1h")} if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")} - {{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("set GO_OMNI_FILES_TOKEN to a token of at least 32 bytes without whitespace")}{{end}} + {{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("storage.filesToken requires at least 32 bytes without whitespace")}{{end}} {{if eq .Storage "local"}} if s.Root=="" {return errors.New("storage.root is required")} u,err:=url.Parse(s.PublicURL) if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")} - if len(s.SigningKey)<32 {return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")} + if len(s.SigningKey)<32 {return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")} {{else}} - if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region and GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY are required")} + if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region, accessKey and secretKey are required")} {{if eq .Storage "r2"}} if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")} if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")} diff --git a/cmd/go-omni/scaffold/templates/modules-settings_test.go.tmpl b/cmd/go-omni/scaffold/templates/modules-settings_test.go.tmpl index 3689990..4c86755 100644 --- a/cmd/go-omni/scaffold/templates/modules-settings_test.go.tmpl +++ b/cmd/go-omni/scaffold/templates/modules-settings_test.go.tmpl @@ -21,18 +21,18 @@ func environment(t *testing.T){ } func TestSettingsPrecedenceAndStrictness(t *testing.T){ environment(t) + {{if eq .Database "mysql"}}t.Setenv("GO_OMNI_DB_DSN","test:secret@tcp(localhost)/demo"){{end}} + source,err:=os.ReadFile("../../etc/config.yaml");if err!=nil{t.Fatal(err)} path:=filepath.Join(t.TempDir(),"config.yaml") - if err:=os.WriteFile(path,[]byte("server:\n address: '127.0.0.1:8123'\n log:\n level: debug\n"),0600);err!=nil{t.Fatal(err)} + source=[]byte(strings.Replace(string(source),"${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}","127.0.0.1:8123",1)) + if err=os.WriteFile(path,source,0600);err!=nil{t.Fatal(err)} t.Setenv("GO_OMNI_HTTP_ADDR","127.0.0.1:9123") t.Setenv("GO_OMNI_LOG_OUTPUT","stdout") cfg,err:=Load(path);if err!=nil{t.Fatal(err)} - if cfg.Server.Address!="127.0.0.1:9123" || cfg.Server.Log.Level!="debug" || cfg.Server.Log.Output!="stdout" || cfg.Server.Log.File!="" || cfg.Server.Log.Rotation.Enabled {t.Fatal("configuration precedence failed")} - for _,body:=range []string{"unexpected: true","server:\n slowRequestThreshold: -1s","server:\n log:\n format: invalid"{{if ne .Storage "none"}},"storage:\n filesToken: forbidden","storage:\n maxBytes: -1","storage:\n provider: unsupported"{{end}}}{ - if err:=os.WriteFile(path,[]byte(body),0600);err!=nil{t.Fatal(err)} - if _,err:=Load(path);err==nil{t.Fatal("invalid configuration accepted")} + initial:=Default() + if cfg.Server.Address!="127.0.0.1:8123"||cfg.Server.Log.Output!=initial.Server.Log.Output||cfg.Server.Log.File!=initial.Server.Log.File {t.Fatal("implicit overwrite")} + for _,body:=range []string{"unexpected: true","server:\n slowRequestThreshold: -1s","server:\n log:\n format: invalid"}{ + if err=os.WriteFile(path,[]byte(body),0600);err!=nil{t.Fatal(err)} + if _,err=Load(path);err==nil{t.Fatal("invalid configuration accepted")} } - {{if ne .Storage "none"}} - t.Setenv("GO_OMNI_FILES_TOKEN","") - if _,err:=Load("");err==nil{t.Fatal("missing file token accepted")} - {{end}} } diff --git a/cmd/go-omni/scaffold/version.go b/cmd/go-omni/scaffold/version.go index 1fd3ae6..643f3a2 100644 --- a/cmd/go-omni/scaffold/version.go +++ b/cmd/go-omni/scaffold/version.go @@ -10,11 +10,11 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" // Source builds can carry an unpublished VCS version (including +dirty). // Keep this fallback in sync with the next release; installed CLIs use their // downloaded module version, which also has a module checksum. -const defaultFrameworkVersion = "v0.21.0-preview" +const defaultFrameworkVersion = "v0.22.0-preview" // This target is not a claim that the release already exists. Source users // select a local framework checkout until the module templates are published. -const modulesFrameworkVersion = "v0.7.0-preview" +const modulesFrameworkVersion = "v0.22.0-preview" func installedFrameworkVersion() string { info, _ := debug.ReadBuildInfo() diff --git a/config/aliases_test.go b/config/aliases_test.go deleted file mode 100644 index 5b9ac6c..0000000 --- a/config/aliases_test.go +++ /dev/null @@ -1,66 +0,0 @@ -package config - -import ( - "strings" - "testing" - "time" -) - -func TestUnifiedEnvAliases(t *testing.T) { - type project struct { - Server Config `json:"server"` - Label string `json:"label"` - } - base := project{Server: Default(), Label: "base"} - for _, policy := range []string{"allow", "ignore", "reject"} { - t.Run(policy, func(t *testing.T) { - env := map[string]string{"TEST_HTTP_ADDR": "127.0.0.1:8099", "TEST_LOG_SERVICE": "", "TEST_SERVICE": "", "TEST_LABEL": ""} - opts := EnvOptions{Prefix: "TEST_", Empty: policy, Aliases: ServerEnvAliases("TEST_", "server"), Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }} - got, err := OverlayEnv(base, opts) - if policy == "reject" { - if err == nil || !strings.Contains(err.Error(), "empty") { - t.Fatal(err) - } - return - } - if err != nil { - t.Fatal(err) - } - if got.Server.Address != "127.0.0.1:8099" { - t.Fatal("alias ignored") - } - if policy == "allow" && got.Label != "" || policy == "ignore" && got.Label != "base" { - t.Fatal("empty policy differs") - } - env["TEST_SERVER_ADDRESS"] = "127.0.0.1:9001" - env["TEST_HTTP_READ_TIMEOUT"] = "3s" - env["TEST_ENABLE_LOGGER"] = "false" - env["TEST_TRUSTED_PROXIES"] = `["127.0.0.1"]` - got, err = OverlayEnv(base, opts) - if err != nil || got.Server.Address != "127.0.0.1:9001" || got.Server.HTTP.ReadTimeout != 3*time.Second || got.Server.EnableLogger || len(got.Server.TrustedProxies) != 1 { - t.Fatal(got, err) - } - env["TEST_SERVER_ADDRESS"] = "" - got, err = OverlayEnv(base, opts) - if err != nil { - t.Fatal(err) - } - expected := "" - if policy == "ignore" { - expected = base.Server.Address - } - if got.Server.Address != expected { - t.Fatal("empty canonical unexpectedly fell through to alias") - } - }) - } - if base.Server.Address != ":8080" || base.Label != "base" { - t.Fatal("defaults mutated") - } -} -func TestAliasTypeErrorsDoNotExposeValue(t *testing.T) { - opts := EnvOptions{Prefix: "X_", Empty: "reject", Aliases: ServerEnvAliases("X_", ""), Lookup: func(k string) (string, bool) { return "private-invalid", k == "X_HTTP_READ_TIMEOUT" }} - if _, err := OverlayEnv(Default(), opts); err == nil || strings.Contains(err.Error(), "private-invalid") { - t.Fatal(err) - } -} diff --git a/config/bind.go b/config/bind.go index a0aa212..ba8c79e 100644 --- a/config/bind.go +++ b/config/bind.go @@ -59,6 +59,20 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int, if err := b.step(path, depth); err != nil { return err } + if text, ok := v.(resolvedText); ok { + if dst.Kind() != reflect.Pointer && dst.Type() != durationType && dst.CanAddr() && !dst.Addr().Type().Implements(jsonUnmarshalerType) && !dst.Addr().Type().Implements(textType) { + switch dst.Kind() { + case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64: + if err := parseTextValue(dst, string(text)); err != nil { + return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err) + } + return nil + } + } + if dst.Kind() != reflect.Pointer { + v = string(text) + } + } if v == nil { return b.decode(nil, dst, path) } // Preserve encoding/json null semantics. @@ -78,7 +92,7 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int, default: return b.decode(v, dst, path) } - if err := parseEnvValue(dst, strings.TrimSpace(raw)); err != nil { + if err := parseTextValue(dst, strings.TrimSpace(raw)); err != nil { return b.doc.diagnostic(path, "expected a duration such as 5s", err) } return nil @@ -102,7 +116,7 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int, } switch dst.Kind() { case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64: - if err := parseEnvValue(dst, strings.TrimSpace(string(text))); err != nil { + if err := parseTextValue(dst, strings.TrimSpace(string(text))); err != nil { return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err) } return nil @@ -143,7 +157,7 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int, } } if !ok { - return b.doc.diagnostic(child, "unknown field", nil) + return b.doc.diagnostic(child, unknownFieldReason(name, fields), nil) } if used[canonical] { return b.doc.diagnostic(child, "duplicate field binding", nil) @@ -237,6 +251,8 @@ func (b *documentBinder) dynamic(v any, path string, depth int) (any, error) { switch x := v.(type) { case Text: return string(x), nil + case resolvedText: + return string(x), nil case map[string]any: out := make(map[string]any, len(x)) for _, k := range sortedKeys(x) { diff --git a/config/config_test.go b/config/config_test.go index c09e6a1..8906247 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -1,132 +1,6 @@ package config -import ( - "strings" - "testing" - "time" -) - -func TestLoadFromEnvUsesDefaults(t *testing.T) { - for _, key := range []string{ - "GO_OMNI_HTTP_ADDR", "GO_OMNI_REQUEST_ID_HEADER", "GO_OMNI_ENABLE_LOGGER", - "GO_OMNI_TRUSTED_PROXIES", - "GO_OMNI_TRANSPORT", "GO_OMNI_TLS_CERT_FILE", "GO_OMNI_TLS_KEY_FILE", - "GO_OMNI_LOG_LEVEL", "GO_OMNI_LOG_FORMAT", "GO_OMNI_LOG_OUTPUT", - "GO_OMNI_LOG_FILE", "GO_OMNI_SERVICE", "GO_OMNI_ENVIRONMENT", - "GO_OMNI_LOG_ADD_SOURCE", "GO_OMNI_SHUTDOWN_TIMEOUT", - "GO_OMNI_SLOW_REQUEST_THRESHOLD", - } { - t.Setenv(key, "") - } - - cfg, err := LoadFromEnv("") - if err != nil { - t.Fatal(err) - } - if cfg.Address != ":8080" || cfg.Transport != TransportNetHTTP || cfg.TLS != (TLSConfig{}) || cfg.RequestIDHeader != "X-Request-ID" || cfg.ShutdownTimeout != 10*time.Second { - t.Fatalf("unexpected defaults: %+v", cfg) - } - if cfg.EnableLogger || cfg.Log.Level != "info" || cfg.Log.Format != "json" || cfg.Log.Output != "stderr" { - t.Fatalf("unexpected logger defaults: %+v", cfg.Log) - } -} - -func TestTransportEnvironment(t *testing.T) { - for _, test := range []struct { - name, transport, cert, key, errorField string - }{ - {"default", "", "", "", ""}, - {"native", "native", "", "", ""}, - {"nativeTLS", "native", "server.crt", "server.key", ""}, - {"http2", "http2", "server.crt", "server.key", ""}, - {"unknown", "typo", "", "", "transport"}, - {"certOnly", "nethttp", "server.crt", "", "tls"}, - {"keyOnly", "native", "", "server.key", "tls"}, - {"http2WithoutTLS", "http2", "", "", "tls"}, - } { - t.Run(test.name, func(t *testing.T) { - t.Setenv("TRANSPORT_TEST_TRANSPORT", test.transport) - t.Setenv("TRANSPORT_TEST_TLS_CERT_FILE", test.cert) - t.Setenv("TRANSPORT_TEST_TLS_KEY_FILE", test.key) - cfg, err := LoadFromEnv("TRANSPORT_TEST_") - if test.errorField != "" { - if err == nil || !strings.Contains(err.Error(), test.errorField) { - t.Fatalf("unexpected error: %v", err) - } - return - } - if err != nil { - t.Fatal(err) - } - want := test.transport - if want == "" { - want = TransportNetHTTP - } - if cfg.Transport != want || cfg.TLS.CertFile != test.cert || cfg.TLS.KeyFile != test.key { - t.Fatalf("settings: %+v", cfg) - } - }) - } -} - -func TestLoadFromEnvOverridesSettings(t *testing.T) { - t.Setenv("GO_OMNI_HTTP_ADDR", "127.0.0.1:9090") - t.Setenv("GO_OMNI_REQUEST_ID_HEADER", "X-Correlation-ID") - t.Setenv("GO_OMNI_TRUSTED_PROXIES", "127.0.0.1, 10.0.0.0/8") - t.Setenv("GO_OMNI_ENABLE_LOGGER", "true") - t.Setenv("GO_OMNI_LOG_LEVEL", "debug") - t.Setenv("GO_OMNI_LOG_FORMAT", "text") - t.Setenv("GO_OMNI_LOG_OUTPUT", "stdout") - t.Setenv("GO_OMNI_SERVICE", "billing") - t.Setenv("GO_OMNI_ENVIRONMENT", "test") - t.Setenv("GO_OMNI_LOG_ADD_SOURCE", "true") - t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "3s") - t.Setenv("GO_OMNI_SLOW_REQUEST_THRESHOLD", "500ms") - - cfg, err := LoadFromEnv("") - if err != nil { - t.Fatal(err) - } - if cfg.Address != "127.0.0.1:9090" || cfg.RequestIDHeader != "X-Correlation-ID" || !cfg.EnableLogger || len(cfg.TrustedProxies) != 2 || cfg.TrustedProxies[1] != "10.0.0.0/8" { - t.Fatalf("unexpected application settings: %+v", cfg) - } - if cfg.ShutdownTimeout != 3*time.Second || cfg.Log.Level != "debug" || cfg.Log.Format != "text" || cfg.Log.Output != "stdout" || cfg.Log.Service != "billing" || cfg.Log.Environment != "test" || !cfg.Log.AddSource { - t.Fatalf("unexpected logger settings: %+v", cfg.Log) - } - if cfg.SlowRequestThreshold != 500*time.Millisecond { - t.Fatal(cfg.SlowRequestThreshold) - } -} - -func TestLoadFromEnvRejectsInvalidValues(t *testing.T) { - t.Setenv("GO_OMNI_ENABLE_LOGGER", "sometimes") - if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "ENABLE_LOGGER") { - t.Fatalf("expected invalid logger flag error, got %v", err) - } - - t.Setenv("GO_OMNI_ENABLE_LOGGER", "") - t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "later") - if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "SHUTDOWN_TIMEOUT") { - t.Fatalf("expected invalid timeout error, got %v", err) - } - - t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "1s") - t.Setenv("GO_OMNI_TRUSTED_PROXIES", "proxy.example.com") - if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "trustedProxies") { - t.Fatalf("expected invalid trusted proxy error, got %v", err) - } - t.Setenv("GO_OMNI_TRUSTED_PROXIES", "") - cfg := Default() - cfg.Address = "" - if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "address") { - t.Fatalf("expected invalid address error, got %v", err) - } - cfg = Default() - cfg.RequestIDHeader = " " - if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "requestIdHeader") { - t.Fatalf("expected invalid request header error, got %v", err) - } -} +import "testing" func TestWithConfigAndWithLogConfig(t *testing.T) { settings := Default() @@ -147,23 +21,3 @@ func TestWithConfigAndWithLogConfig(t *testing.T) { t.Fatalf("log config was not replaced: %+v", cfg.Log) } } - -func TestApplyEnvPreservesProjectDefaults(t *testing.T) { - cfg := Default() - cfg.Address = "127.0.0.1:8123" - cfg.Log.RedactKeys = []string{"bankAccount"} - cfg.TrustedProxies = []string{"127.0.0.1"} - t.Setenv("OVERLAY_TEST_LOG_LEVEL", "debug") - got, err := ApplyEnv(cfg, "OVERLAY_TEST_") - if err != nil { - t.Fatal(err) - } - if got.Address != cfg.Address || got.Log.Level != "debug" { - t.Fatal(got) - } - got.Log.RedactKeys[0] = "changed" - got.TrustedProxies[0] = "192.0.2.1" - if cfg.Log.RedactKeys[0] != "bankAccount" || cfg.TrustedProxies[0] != "127.0.0.1" { - t.Fatal("caller-owned defaults mutated") - } -} diff --git a/config/diagnostic.go b/config/diagnostic.go index 6028ccd..f4484d5 100644 --- a/config/diagnostic.go +++ b/config/diagnostic.go @@ -82,8 +82,9 @@ type Position struct{ Line, Column int } // string, bool, json.Number, nil, []any, map[string]any. Text is available for // formats such as XML whose scalar tokens have no inherent JSON type. type Document struct { - Value any - Positions map[string]Position + Value any + Positions map[string]Position + environments map[string][]string } type Text string diff --git a/config/endpoints_test.go b/config/endpoints_test.go index bbd4faa..52a0d37 100644 --- a/config/endpoints_test.go +++ b/config/endpoints_test.go @@ -18,14 +18,15 @@ func TestEndpointConfiguration(t *testing.T) { if cfg.Endpoints.Liveness.Enabled || cfg.Endpoints.OpenAPI.Enabled || !cfg.Endpoints.Readiness.Enabled || cfg.Endpoints.Readiness.ResponseMode != "raw" { t.Fatal(cfg.Endpoints) } + refs := configFile(t, "yaml", "endpoints:\n readiness:\n path: ${TEST_ENDPOINTS_READINESS_PATH}\n openapi:\n enabled: ${TEST_ENDPOINTS_OPENAPI_ENABLED}\n") t.Setenv("TEST_ENDPOINTS_READINESS_PATH", "/internal/ready") t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "true") - cfg, err = ApplyEnv(cfg, "TEST_") + cfg, err = LoadProject(refs, cfg) if err != nil || cfg.Endpoints.Readiness.Path != "/internal/ready" || !cfg.Endpoints.OpenAPI.Enabled { t.Fatal(cfg.Endpoints, err) } t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "invalid") - if _, err = ApplyEnv(cfg, "TEST_"); err == nil { + if _, err = LoadProject(refs, cfg); err == nil { t.Fatal("invalid bool accepted") } } diff --git a/config/env.go b/config/env.go deleted file mode 100644 index f0cce01..0000000 --- a/config/env.go +++ /dev/null @@ -1,83 +0,0 @@ -package config - -import ( - "encoding/json" - "os" - "strings" -) - -const defaultEnvPrefix = "GO_OMNI_" - -// ServerEnvAliases returns the documented short names for Config fields. -// field is the JSON path of Config in a project (usually "server"), or "" -// for Config itself. Canonical names win over aliases. Both use the same -// EnvOptions.Empty/Lookup and typed parser; collections use JSON syntax. -func ServerEnvAliases(prefix, field string) map[string][]string { - root := prefix - if field != "" { - for _, part := range strings.Split(field, ".") { - root += envName(part) + "_" - } - } - names := map[string]string{ - "ADDRESS": "HTTP_ADDR", "TRANSPORT": "TRANSPORT", "TLS_CERT_FILE": "TLS_CERT_FILE", "TLS_KEY_FILE": "TLS_KEY_FILE", - "REQUEST_ID_HEADER": "REQUEST_ID_HEADER", "ENABLE_LOGGER": "ENABLE_LOGGER", "SLOW_REQUEST_THRESHOLD": "SLOW_REQUEST_THRESHOLD", - "TRUSTED_PROXIES": "TRUSTED_PROXIES", "SHUTDOWN_TIMEOUT": "SHUTDOWN_TIMEOUT", - "LOG_LEVEL": "LOG_LEVEL", "LOG_FORMAT": "LOG_FORMAT", "LOG_OUTPUT": "LOG_OUTPUT", "LOG_FILE": "LOG_FILE", "LOG_SERVICE": "SERVICE", "LOG_ENVIRONMENT": "ENVIRONMENT", "LOG_ADD_SOURCE": "LOG_ADD_SOURCE", - "HTTP_READ_HEADER_TIMEOUT": "HTTP_READ_HEADER_TIMEOUT", "HTTP_READ_TIMEOUT": "HTTP_READ_TIMEOUT", "HTTP_WRITE_TIMEOUT": "HTTP_WRITE_TIMEOUT", "HTTP_IDLE_TIMEOUT": "HTTP_IDLE_TIMEOUT", "HTTP_MAX_HEADER_BYTES": "HTTP_MAX_HEADER_BYTES", - "LIFECYCLE_STOP_TIMEOUT": "LIFECYCLE_STOP_TIMEOUT", "LIFECYCLE_DRAIN_TIMEOUT": "LIFECYCLE_DRAIN_TIMEOUT", "LIFECYCLE_CANCEL_TIMEOUT": "LIFECYCLE_CANCEL_TIMEOUT", - } - for _, endpoint := range []string{"LIVENESS", "READINESS", "OPENAPI"} { - for _, part := range []string{"ENABLED", "PATH", "RESPONSE_MODE"} { - if endpoint == "OPENAPI" && part == "RESPONSE_MODE" { - continue - } - name := "ENDPOINTS_" + endpoint + "_" + part - names[name] = name - } - } - aliases := map[string][]string{} - for key, alias := range names { - if root+key != prefix+alias { - aliases[root+key] = []string{prefix + alias} - } - } - return aliases -} - -// LoadFromEnv loads defaults using the legacy short-name conventions. -func LoadFromEnv(prefix string) (Config, error) { return ApplyEnv(Default(), prefix) } - -// ApplyEnv retains legacy whitespace trimming, ignored empty variables and -// comma-separated TRUSTED_PROXIES. New projects should use OverlayEnv with -// ServerEnvAliases for one explicit empty-value policy across the whole config. -func ApplyEnv(defaults Config, prefix string) (Config, error) { - prefix = strings.TrimSpace(prefix) - if prefix == "" { - prefix = defaultEnvPrefix - } - lookup := func(key string) (string, bool) { - raw, ok := os.LookupEnv(key) - raw = strings.TrimSpace(raw) - if ok && raw != "" && key == prefix+"TRUSTED_PROXIES" && !strings.HasPrefix(raw, "[") { - values := strings.Split(raw, ",") - for i := range values { - values[i] = strings.TrimSpace(values[i]) - } - data, _ := json.Marshal(values) - raw = string(data) - } - return raw, ok - } - defaults.Errors = defaults.Errors.Clone() - defaults.TrustedProxies = append([]string(nil), defaults.TrustedProxies...) - defaults.Log.RedactKeys = append([]string(nil), defaults.Log.RedactKeys...) - cfg, err := OverlayEnv(defaults, EnvOptions{Prefix: prefix, Empty: "ignore", Lookup: lookup, Aliases: ServerEnvAliases(prefix, "")}) - if err != nil { - return Config{}, err - } - if err = cfg.Validate(); err != nil { - return Config{}, err - } - return cfg, nil -} diff --git a/config/errorCatalog_test.go b/config/errorCatalog_test.go index 92c6489..e2caf9e 100644 --- a/config/errorCatalog_test.go +++ b/config/errorCatalog_test.go @@ -21,7 +21,7 @@ func TestCatalogConfigFiles(t *testing.T) { if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { t.Fatal(err) } - cfg, err := config.LoadFile(path, "CATALOG_FIXTURE_") + cfg, err := config.LoadFile(path) if err != nil { t.Fatal(err) } @@ -40,7 +40,7 @@ func TestCatalogConfigFiles(t *testing.T) { if err := os.WriteFile(path, []byte(body), 0600); err != nil { t.Fatal(err) } - if _, err := config.LoadFile(path, "CATALOG_FIXTURE_"); err == nil { + if _, err := config.LoadFile(path); err == nil { t.Fatal("accepted invalid catalog", body) } } @@ -57,7 +57,7 @@ func TestCatalogReloadSnapshotIsolation(t *testing.T) { t.Fatal(err) } defer logger.Close() - reloader, err := config.NewReloader(path, "CATALOG_RELOAD_", logger) + reloader, err := config.NewReloader(path, logger) if err != nil { t.Fatal(err) } diff --git a/config/errorParams_test.go b/config/errorParams_test.go index e4986f5..085efe1 100644 --- a/config/errorParams_test.go +++ b/config/errorParams_test.go @@ -20,7 +20,7 @@ func TestTemplateConfigFiles(t *testing.T) { if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { t.Fatal(err) } - cfg, err := config.LoadFile(path, "TEMPLATE_TEST_") + cfg, err := config.LoadFile(path) if err != nil { t.Fatal(err) } diff --git a/config/file.go b/config/file.go index d5c4836..91a6aaa 100644 --- a/config/file.go +++ b/config/file.go @@ -4,103 +4,38 @@ import ( "context" "errors" "reflect" - "strings" "sync" "time" "git.yuntuo.hk/k2safe/go-omni/logx" ) -func durations(v any, t reflect.Type) (any, error) { - if t == nil { - return v, nil - } - for t.Kind() == reflect.Pointer { - t = t.Elem() - } - if t == reflect.TypeOf(time.Duration(0)) { - if s, ok := v.(string); ok { - d, err := time.ParseDuration(s) - return int64(d), err - } - return v, nil - } - if t.Kind() == reflect.Struct { - if m, ok := v.(map[string]any); ok { - for n := 0; n < t.NumField(); n++ { - field := t.Field(n) - name := strings.Split(field.Tag.Get("json"), ",")[0] - if name == "-" { - continue - } - if name == "" { - name = field.Name - } - if item, exists := m[name]; exists { - converted, err := durations(item, field.Type) - if err != nil { - return nil, err - } - m[name] = converted - } - } - } - } - if t.Kind() == reflect.Map && t.Key().Kind() == reflect.String { - if values, ok := v.(map[string]any); ok { - for key, value := range values { - converted, err := durations(value, t.Elem()) - if err != nil { - return nil, err - } - values[key] = converted - } - } - } - if t.Kind() == reflect.Slice { - if items, ok := v.([]any); ok { - for n, item := range items { - converted, err := durations(item, t.Elem()) - if err != nil { - return nil, err - } - items[n] = converted - } - } - } - return v, nil -} -func LoadFile(path, prefix string) (Config, error) { - cfg, err := DecodeFile(path, Default()) - if err != nil { - return Config{}, err - } - return ApplyEnv(cfg, prefix) -} +// LoadFile loads and validates framework configuration using explicit file references. +func LoadFile(path string) (Config, error) { return LoadProject(path, Default()) } // Reloader only accepts log.level changes. Static changes and invalid files // leave the current snapshot and logger untouched. Watch does not own a goroutine. type Reloader struct { - mu sync.Mutex - path, prefix string - current Config - logger *logx.Logger + mu sync.Mutex + path string + current Config + logger *logx.Logger } -func NewReloader(path, prefix string, logger *logx.Logger) (*Reloader, error) { - cfg, err := LoadFile(path, prefix) +func NewReloader(path string, logger *logx.Logger) (*Reloader, error) { + cfg, err := LoadFile(path) if err != nil { return nil, err } if err = logger.SetLevel(cfg.Log.Level); err != nil { return nil, err } - return &Reloader{path: path, prefix: prefix, current: cfg, logger: logger}, nil + return &Reloader{path: path, current: cfg, logger: logger}, nil } func (r *Reloader) Reload() error { r.mu.Lock() defer r.mu.Unlock() - next, err := LoadFile(r.path, r.prefix) + next, err := LoadFile(r.path) if err != nil { return err } diff --git a/config/file_test.go b/config/file_test.go index 2f43c98..e73b329 100644 --- a/config/file_test.go +++ b/config/file_test.go @@ -20,8 +20,8 @@ func TestFileMergeAndSafeReload(t *testing.T) { } write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: info\n") t.Setenv("TEST_MODULE_HTTP_ADDR", ":9123") - cfg, err := LoadFile(path, "TEST_MODULE_") - if err != nil || cfg.Address != ":9123" || cfg.ShutdownTimeout != 7*time.Second { + cfg, err := LoadFile(path) + if err != nil || cfg.Address != ":8123" || cfg.ShutdownTimeout != 7*time.Second { t.Fatal(cfg, err) } var output bytes.Buffer @@ -33,7 +33,7 @@ func TestFileMergeAndSafeReload(t *testing.T) { } defer l.Close() child := l.With("scope", "test") - r, err := NewReloader(path, "TEST_MODULE_", l) + r, err := NewReloader(path, l) if err != nil { t.Fatal(err) } diff --git a/config/http_test.go b/config/http_test.go index b848cf2..82041f0 100644 --- a/config/http_test.go +++ b/config/http_test.go @@ -22,7 +22,7 @@ func TestHTTPFileAndEnvironment(t *testing.T) { if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { t.Fatal(err) } - cfg, err := config.LoadFile(path, "HTTP_TEST_") + cfg, err := config.LoadFile(path) if err != nil { t.Fatal(err) } @@ -35,7 +35,10 @@ func TestHTTPFileAndEnvironment(t *testing.T) { t.Setenv("HTTP_TEST_HTTP_WRITE_TIMEOUT", "8s") t.Setenv("HTTP_TEST_HTTP_IDLE_TIMEOUT", "9s") t.Setenv("HTTP_TEST_HTTP_MAX_HEADER_BYTES", "16384") - got, err := config.ApplyEnv(cfg, "HTTP_TEST_") + if err := os.WriteFile(path, []byte(`{"http":{"readHeaderTimeout":"${HTTP_TEST_HTTP_READ_HEADER_TIMEOUT}","readTimeout":"${HTTP_TEST_HTTP_READ_TIMEOUT}","writeTimeout":"${HTTP_TEST_HTTP_WRITE_TIMEOUT}","idleTimeout":"${HTTP_TEST_HTTP_IDLE_TIMEOUT}","maxHeaderBytes":"${HTTP_TEST_HTTP_MAX_HEADER_BYTES}"}}`), 0600); err != nil { + t.Fatal(err) + } + got, err := config.LoadProject(path, cfg) if err != nil { t.Fatal(err) } @@ -55,7 +58,12 @@ func TestHTTPInvalidAndUnsupportedSettings(t *testing.T) { value = "-1s" } t.Setenv("HTTP_INVALID_HTTP_"+key, value) - if _, err := config.LoadFromEnv("HTTP_INVALID_"); err == nil { + names := map[string]string{"READ_HEADER_TIMEOUT": "readHeaderTimeout", "READ_TIMEOUT": "readTimeout", "WRITE_TIMEOUT": "writeTimeout", "IDLE_TIMEOUT": "idleTimeout", "MAX_HEADER_BYTES": "maxHeaderBytes"} + path := filepath.Join(t.TempDir(), "config.json") + if err := os.WriteFile(path, []byte(`{"http":{"`+names[key]+`":"${HTTP_INVALID_HTTP_`+key+`}"}}`), 0600); err != nil { + t.Fatal(err) + } + if _, err := config.LoadFile(path); err == nil { t.Fatal("invalid HTTP value accepted") } }) diff --git a/config/lifecycle_test.go b/config/lifecycle_test.go index f45140b..f5da2f6 100644 --- a/config/lifecycle_test.go +++ b/config/lifecycle_test.go @@ -14,7 +14,7 @@ func TestLifecycleConfiguration(t *testing.T) { if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: 2s\n drainTimeout: 3s\n cancelTimeout: 4s\n"), 0600); err != nil { t.Fatal(err) } - cfg, err := config.LoadFile(path, "PHASE_TEST_") + cfg, err := config.LoadFile(path) if err != nil { t.Fatal(err) } @@ -24,13 +24,16 @@ func TestLifecycleConfiguration(t *testing.T) { for _, key := range []string{"STOP_TIMEOUT", "DRAIN_TIMEOUT", "CANCEL_TIMEOUT"} { t.Setenv("PHASE_TEST_LIFECYCLE_"+key, "6s") } - cfg, err = config.ApplyEnv(cfg, "PHASE_TEST_") + if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: ${PHASE_TEST_LIFECYCLE_STOP_TIMEOUT}\n drainTimeout: ${PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT}\n cancelTimeout: ${PHASE_TEST_LIFECYCLE_CANCEL_TIMEOUT}\n"), 0600); err != nil { + t.Fatal(err) + } + cfg, err = config.LoadProject(path, cfg) if err != nil || cfg.Lifecycle.StopTimeout != 6*time.Second || cfg.Lifecycle.DrainTimeout != 6*time.Second || cfg.Lifecycle.CancelTimeout != 6*time.Second { t.Fatal(cfg.Lifecycle, err) } for _, v := range []string{"bad", "-1s"} { t.Setenv("PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT", v) - if _, err := config.ApplyEnv(cfg, "PHASE_TEST_"); err == nil { + if _, err := config.LoadProject(path, cfg); err == nil { t.Fatal("invalid duration accepted") } } diff --git a/config/loader.go b/config/loader.go index 0675be0..1823564 100644 --- a/config/loader.go +++ b/config/loader.go @@ -26,6 +26,7 @@ func (f DecoderFunc) Decode(data []byte) (Document, error) { return f(data) } type Loader struct { decoders map[string]Decoder sealed bool + lookup func(string) (string, bool) } type LoaderOption func(*Loader) error @@ -55,7 +56,7 @@ func WithDecoder(extension string, decoder Decoder) LoaderOption { } func NewLoader(options ...LoaderOption) (*Loader, error) { - l := &Loader{decoders: map[string]Decoder{"json": JSONDecoder{}, "yaml": YAMLDecoder{}, "yml": YAMLDecoder{}, "xml": XMLDecoder{}}} + l := &Loader{lookup: os.LookupEnv, decoders: map[string]Decoder{"json": JSONDecoder{}, "yaml": YAMLDecoder{}, "yml": YAMLDecoder{}, "xml": XMLDecoder{}}} for _, o := range options { if o == nil { return nil, errors.New("config: nil loader option") @@ -135,7 +136,7 @@ func (l *Loader) read(path string) (doc Document, err error) { if _, ok := doc.Value.(map[string]any); !ok { return doc, doc.diagnostic("", "root must be an object", nil) } - return doc, nil + return resolveDocument(doc, l.lookup) } func decodeSafely(decoder Decoder, data []byte) (doc Document, err error) { defer func() { @@ -165,11 +166,12 @@ func DecodeFile[T any](path string, defaults T, loaders ...*Loader) (T, error) { if err != nil { return zero, err } + if err := rejectEnvironmentTags(reflect.TypeOf(defaults), map[reflect.Type]bool{}); err != nil { + return zero, err + } result, err := bindDocument(doc, defaults) if err != nil { - d := validationError(err) - d.File = path - return zero, d + return zero, locateError(err, path, doc) } return result, nil } diff --git a/config/loader_test.go b/config/loader_test.go index 32519c2..dbda398 100644 --- a/config/loader_test.go +++ b/config/loader_test.go @@ -16,7 +16,7 @@ import ( type loaderExample struct { Server Config `json:"server"` Upload struct { - MaxBytes int64 `json:"maxBytes" envDefault:"1024"` + MaxBytes int64 `json:"maxBytes"` } `json:"upload"` Values []int64 `json:"values"` Optional *string `json:"optional"` @@ -45,19 +45,19 @@ func configFile(t *testing.T, ext, raw string) string { func TestUnifiedFormatAndPrecedence(t *testing.T) { formats := map[string]string{ - "json": `{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":7},"values":[9007199254740993],"optional":null,"enabled":true,"dynamic":{"id":9007199254740993}}`, - "yaml": "server:\n http:\n readTimeout: 2s\nupload:\n maxBytes: 7\nvalues: [9007199254740993]\noptional: null\nenabled: true\ndynamic:\n id: 9007199254740993\n", - "xml": `2s79007199254740993true9007199254740993`, + "json": `{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":"${APP_UPLOAD_MAX_BYTES}"},"values":[9007199254740993],"optional":null,"enabled":"${APP_ENABLED}","dynamic":{"id":9007199254740993}}`, + "yaml": "server:\n http:\n readTimeout: 2s\nupload:\n maxBytes: \"${APP_UPLOAD_MAX_BYTES}\"\nvalues: [9007199254740993]\noptional: null\nenabled: \"${APP_ENABLED}\"\ndynamic:\n id: 9007199254740993\n", + "xml": `2s${APP_UPLOAD_MAX_BYTES}9007199254740993${APP_ENABLED}9007199254740993`, } var reference loaderExample for _, ext := range []string{"json", "yaml", "xml"} { t.Run(ext, func(t *testing.T) { old := "old" base := loaderExample{Server: Default(), Optional: &old} - cfg, err := LoadProject(configFile(t, ext, formats[ext]), base, EnvOptions{Prefix: "APP_", Lookup: func(key string) (string, bool) { + cfg, err := LoadProject(configFile(t, ext, formats[ext]), base, testLoader(func(key string) (string, bool) { v, ok := map[string]string{"APP_UPLOAD_MAX_BYTES": "0", "APP_ENABLED": "false"}[key] return v, ok - }}) + })) if err != nil { t.Fatal(err) } @@ -77,8 +77,8 @@ func TestUnifiedFormatAndPrecedence(t *testing.T) { } }) } - cfg, err := LoadProject("", loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}) - if err != nil || cfg.Upload.MaxBytes != 1024 { + cfg, err := LoadProject("", loaderExample{Server: Default()}, testLoader(noEnvironment)) + if err != nil || cfg.Upload.MaxBytes != 0 { t.Fatal(cfg, err) } } @@ -97,7 +97,7 @@ func TestConfigDiagnosticsAndCauses(t *testing.T) { } { t.Run(tc.ext+tc.field, func(t *testing.T) { path := configFile(t, tc.ext, tc.body) - _, err := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}) + _, err := LoadProject(path, loaderExample{Server: Default()}, testLoader(noEnvironment)) var d *Error if !errors.As(err, &d) || d.File != path || d.Field != tc.field || d.Reason == "" || tc.line && d.Line < 1 { t.Fatalf("missing location: %#v %v", d, err) @@ -108,7 +108,7 @@ func TestConfigDiagnosticsAndCauses(t *testing.T) { }) } path := filepath.Join(t.TempDir(), "missing.json") - _, err := LoadProject(path, loaderExample{}, EnvOptions{}) + _, err := LoadProject(path, loaderExample{}) if !errors.Is(err, os.ErrNotExist) { t.Fatal("I/O cause discarded", err) } @@ -119,11 +119,12 @@ func TestConfigDiagnosticsAndCauses(t *testing.T) { if !errors.As(err, &typed) { t.Fatal("type cause discarded", err) } - _, err = LoadProject(configFile(t, "yaml", "upload:\n maxBytes: 7"), loaderExample{Server: Default()}, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { return "-1", k == "APP_UPLOAD_MAX_BYTES" }}) + _, err = LoadProject(configFile(t, "yaml", "upload:\n maxBytes: \"${APP_UPLOAD_MAX_BYTES}\""), loaderExample{Server: Default()}, testLoader(func(k string) (string, bool) { return "-1", k == "APP_UPLOAD_MAX_BYTES" })) var d *Error - if !errors.As(err, &d) || d.Env != "APP_UPLOAD_MAX_BYTES" || d.Field != "upload.maxBytes" || d.File != "" || d.Line != 0 { - t.Fatal("environment validation wrongly blamed file", err) + if !errors.As(err, &d) || d.Env != "APP_UPLOAD_MAX_BYTES" || d.Field != "upload.maxBytes" || d.File == "" || d.Line != 2 { + t.Fatal("missing file and variable provenance", err) } + } func TestMalformedFormatsAndLimits(t *testing.T) { @@ -134,7 +135,7 @@ func TestMalformedFormatsAndLimits(t *testing.T) { } { for i, raw := range bodies { t.Run(fmt.Sprintf("%s%d", ext, i), func(t *testing.T) { - _, err := LoadProject(configFile(t, ext, raw), loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}) + _, err := LoadProject(configFile(t, ext, raw), loaderExample{Server: Default()}, testLoader(noEnvironment)) if err == nil || strings.Contains(err.Error(), "private-value") { t.Fatal(err) } @@ -163,7 +164,7 @@ func TestDecoderRegistryAndConcurrentUse(t *testing.T) { wg.Add(1) go func() { defer wg.Done() - cfg, e := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}, l) + cfg, e := LoadProject(path, loaderExample{Server: Default()}, l) if e != nil || cfg.Upload.MaxBytes != 42 { t.Error(e) } @@ -191,25 +192,6 @@ func TestDecoderRegistryAndConcurrentUse(t *testing.T) { } } -func TestUnifiedLoggerNormalization(t *testing.T) { - type project struct { - Server Config `json:"server"` - } - base := project{Server: Default()} - base.Server.Log.File = "old.log" - base.Server.Log.Rotation.Enabled = true - for _, full := range []bool{false, true} { - name := "GO_OMNI_LOG_OUTPUT" - if full { - name = "GO_OMNI_SERVER_LOG_OUTPUT" - } - cfg, err := LoadProject("", base, EnvOptions{Prefix: "GO_OMNI_", Empty: "ignore", Aliases: ServerEnvAliases("GO_OMNI_", "server"), Lookup: func(k string) (string, bool) { return "stdout", k == name }}) - if err != nil || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled || base.Server.Log.File != "old.log" { - t.Fatal(cfg, err) - } - } -} - func FuzzConfigurationDecoders(f *testing.F) { f.Add(uint8(0), []byte(`{"count":1}`)) f.Add(uint8(1), []byte("count: 1")) @@ -332,18 +314,21 @@ func TestConfigDefaultsIsolationAndEmptyValues(t *testing.T) { func TestDefaultStageDoesNotReadEnvironment(t *testing.T) { reads := 0 type C struct { - Value int `json:"value" envDefault:"3"` + Value int `json:"value"` } - got, err := LoadProject("", C{}, EnvOptions{Lookup: func(string) (string, bool) { reads++; return "", false }}) - if err != nil || got.Value != 3 || reads != 1 { + got, err := LoadProject("", C{Value: 3}, testLoader(func(string) (string, bool) { reads++; return "9", true })) + if err != nil || got.Value != 3 || reads != 0 { t.Fatal(got, err, reads) } - d := locateError(Invalid("servers[0].timeout", "must be positive"), "a.json", Document{}, map[string]string{"servers": "APP_SERVERS"}) - if d.Env != "APP_SERVERS" || d.File != "" { - t.Fatal(d) - } } +func testLoader(lookup func(string) (string, bool)) *Loader { + l, err := NewLoader(WithLookupEnv(lookup)) + if err != nil { + panic(err) + } + return l +} func TestDecoderRegistryFrozen(t *testing.T) { l, err := NewLoader() if err != nil { @@ -358,17 +343,12 @@ func BenchmarkConfigurationStartup(b *testing.B) { if err := os.WriteFile(path, []byte(`{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":4096}}`), 0600); err != nil { b.Fatal(err) } - options := EnvOptions{Prefix: "BENCH_", Lookup: func(name string) (string, bool) { - if name == "BENCH_UPLOAD_MAX_BYTES" { - return "8192", true - } - return "", false - }} + defaults := loaderExample{Server: Default()} b.ReportAllocs() b.ResetTimer() for i := 0; i < b.N; i++ { - if _, err := LoadProject(path, defaults, options); err != nil { + if _, err := LoadProject(path, defaults); err != nil { b.Fatal(err) } } diff --git a/config/project.go b/config/project.go index 85ec00e..30548c7 100644 --- a/config/project.go +++ b/config/project.go @@ -1,318 +1,105 @@ package config import ( - "bytes" "encoding" - "encoding/json" "errors" - "fmt" - apierrors "git.yuntuo.hk/k2safe/go-omni/errors" - "git.yuntuo.hk/k2safe/go-omni/logx" "math" - "os" "reflect" "strconv" "strings" "time" - "unicode" ) -// EnvOptions controls typed project configuration overlays. Explicit env tags -// are absolute names; otherwise JSON names become PREFIX_NESTED_FIELD names. -// Empty is allow (default), ignore or reject. No input values appear in errors. -type EnvOptions struct { - Prefix string - Empty string - Lookup func(string) (string, bool) - // Aliases maps canonical absolute environment names to ordered fallback names. - // A present canonical value wins, including an explicit empty value. - Aliases map[string][]string -} - -// ApplyDefaults applies envDefault tags to zero fields before file overlays. -func ApplyDefaults[T any](base T) (T, error) { return projectOverlay(base, EnvOptions{}, true) } - -// LoadProject applies code/tag defaults, a strict optional file, environment, -// then the project's Validate method, before any logger or dependency is opened. -// envDefault is only applied to zero-valued defaults before reading the file. -func LoadProject[T any](path string, defaults T, options EnvOptions, loaders ...*Loader) (T, error) { +// LoadProject clones defaults, resolves only explicit file references, binds the +// provided fields and validates. Empty path uses only defaults. No env tags, +// prefixes, aliases or implicit environment overlays are applied. +func LoadProject[T any](path string, defaults T, loaders ...*Loader) (T, error) { var zero T loader, err := selectLoader(loaders) if err != nil { return zero, err } - cfg, err := projectOverlay(defaults, options, true) - if err != nil { + if err = rejectEnvironmentTags(reflect.TypeOf(defaults), map[reflect.Type]bool{}); err != nil { return zero, err } - var doc Document + doc := Document{Value: map[string]any{}} if path != "" { doc, err = loader.read(path) if err != nil { return zero, err } - cfg, err = bindDocument(doc, cfg) - if err != nil { - return zero, locateError(err, path, doc, nil) - } - } else { - value, e := cloneConfiguration(reflect.ValueOf(&cfg).Elem(), 0) - if e != nil { - return zero, &Error{Reason: "cannot clone defaults", cause: e} - } - cfg = value.Interface().(T) } - origins := map[string]string{} - cfg, err = projectOverlay(cfg, options, false, origins) + cfg, err := bindDocument(doc, defaults) if err != nil { - return zero, err + return zero, locateError(err, path, doc) } - normalizeProject(reflect.ValueOf(&cfg).Elem(), "", origins, 0) if valid, ok := any(&cfg).(interface{ Validate() error }); ok { if err = valid.Validate(); err != nil { - return zero, locateError(err, path, doc, origins) + return zero, locateError(err, path, doc) + } + } else if valid, ok := any(cfg).(interface{ Validate() error }); ok { + if err = valid.Validate(); err != nil { + return zero, locateError(err, path, doc) } } return cfg, nil } - -func locateError(err error, path string, doc Document, origins map[string]string) *Error { +func locateError(err error, path string, doc Document) *Error { d := validationError(err) - field := d.Field - for field != "" && origins[field] == "" { - index := strings.LastIndexAny(field, ".[") - if index < 0 { - field = "" - } else { - field = field[:index] - } - } - if env := origins[field]; env != "" { - d.Env = env - d.File = "" - d.Line = 0 - d.Column = 0 - return d - } d.File = path if d.Line == 0 { p := doc.Positions[d.Field] d.Line = p.Line d.Column = p.Column } + for field := d.Field; field != ""; { + if names := doc.environments[field]; len(names) > 0 { + d.Env = strings.Join(names, ", ") + break + } + i := strings.LastIndexAny(field, ".[") + if i < 0 { + break + } + field = field[:i] + } return d } -// Framework-owned normalization stays inside the common pipeline. Explicit -// environment stream selection clears a file profile unless a file override -// was also provided; project code never needs to repeat this rule. -func normalizeProject(v reflect.Value, path string, origins map[string]string, depth int) { - if depth > 32 { - return +// Reject removed declarations rather than silently ignoring deployment rules. +func rejectEnvironmentTags(t reflect.Type, seen map[reflect.Type]bool) error { + if t == nil || seen[t] { + return nil } - if v.Kind() == reflect.Pointer { - if v.IsNil() { - return + seen[t] = true + switch t.Kind() { + case reflect.Pointer, reflect.Slice, reflect.Array, reflect.Map: + return rejectEnvironmentTags(t.Elem(), seen) + case reflect.Struct: + for i := 0; i < t.NumField(); i++ { + f := t.Field(i) + if f.PkgPath != "" { + continue + } + _, env := f.Tag.Lookup("env") + _, def := f.Tag.Lookup("envDefault") + if env || def { + return &Error{Field: f.Name, Reason: "env/envDefault tags were removed; use Default() and explicit file references; see CONFIG.md"} + } + if f.Tag.Get("json") != "-" { + if err := rejectEnvironmentTags(f.Type, seen); err != nil { + return err + } + } } - v = v.Elem() } - if v.Kind() != reflect.Struct { - return - } - if v.Type() == reflect.TypeOf(Config{}) && v.CanAddr() { - c := v.Addr().Interface().(*Config) - if origins[joinField(path, "log.output")] != "" && c.Log.Output != "" && (origins[joinField(path, "log.file")] == "" || c.Log.File == "") { - c.Log.File = "" - c.Log.Rotation = logx.RotationConfig{} - } - return - } - for i := 0; i < v.NumField(); i++ { - f := v.Type().Field(i) - if f.PkgPath != "" { - continue - } - name := strings.Split(f.Tag.Get("json"), ",")[0] - if name == "-" { - continue - } - if name == "" { - name = f.Name - } - normalizeProject(v.Field(i), joinField(path, name), origins, depth+1) - } -} - -// OverlayEnv overlays the entire project struct without mutating the input. -// Tags: env:"CUSTOM_NAME,required,secret,empty=reject"; env:"-" excludes a -// subtree. required means nonempty string/non-nil pointer or collection; false -// and zero are valid scalars. Use project Validate for semantic constraints. -// Slices/maps use JSON syntax, avoiding ambiguous comma and empty semantics. -func OverlayEnv[T any](base T, options EnvOptions) (T, error) { - return projectOverlay(base, options, false) + return nil } var textType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem() var durationType = reflect.TypeOf(time.Duration(0)) -func projectOverlay[T any](base T, options EnvOptions, defaults bool, sourceMaps ...map[string]string) (T, error) { - var zero T - if options.Lookup == nil { - options.Lookup = os.LookupEnv - } - if options.Empty == "" { - options.Empty = "allow" - } - if options.Empty != "allow" && options.Empty != "ignore" && options.Empty != "reject" { - return zero, errors.New("config: empty policy must be allow, ignore or reject") - } - v := reflect.ValueOf(&base).Elem() - if v.Kind() != reflect.Struct { - return zero, errors.New("config: project configuration must be a struct") - } - result := reflect.New(v.Type()).Elem() - result.Set(v) - seen := map[string]bool{} - var walk func(reflect.Value, string, string, int) error - walk = func(value reflect.Value, prefix, path string, depth int) error { - if depth > 32 { - return errors.New("config: nesting exceeds 32") - } - for i := 0; i < value.NumField(); i++ { - field := value.Type().Field(i) - dst := value.Field(i) - if !dst.CanSet() || field.PkgPath != "" { - continue - } - tag := strings.Split(field.Tag.Get("env"), ",") - if tag[0] == "-" { - continue - } - jsonName := strings.Split(field.Tag.Get("json"), ",")[0] - if jsonName == "-" && tag[0] == "" { - continue - } - if jsonName == "" { - jsonName = field.Name - } - fieldPath := joinField(path, jsonName) - key := prefix + envName(jsonName) - if tag[0] != "" { - key = tag[0] - } - empty, required := options.Empty, false - for _, opt := range tag[1:] { - switch opt { - case "required": - required = true - case "secret", "": - case "empty=allow": - empty = "allow" - case "empty=ignore": - empty = "ignore" - case "empty=reject": - empty = "reject" - default: - return fmt.Errorf("config: invalid env tag for %s", key) - } - } - if dst.Kind() == reflect.Struct && !dst.Addr().Type().Implements(textType) { - if err := walk(dst, key+"_", fieldPath, depth+1); err != nil { - return err - } - continue - } - if dst.Kind() == reflect.Pointer && dst.Type().Elem().Kind() == reflect.Struct && !dst.Type().Implements(textType) { - // Clone before traversing so a failed overlay never edits defaults. - child := reflect.New(dst.Type().Elem()) - if !dst.IsNil() { - child.Elem().Set(dst.Elem()) - } - if err := walk(child.Elem(), key+"_", fieldPath, depth+1); err != nil { - return err - } - if !dst.IsNil() || !child.Elem().IsZero() { - dst.Set(child) - } - if required && !defaults && dst.IsNil() { - return &Error{Env: key, Field: fieldPath, Reason: "is required"} - } - continue - } - // Unexported runtime capabilities are skipped unless explicitly tagged. - if dst.Kind() == reflect.Func || dst.Kind() == reflect.Chan || dst.Kind() == reflect.Interface { - if tag[0] != "" { - return fmt.Errorf("config: %s has unsupported environment type", key) - } - continue - } - if seen[key] { - return fmt.Errorf("config: duplicate environment key %s", key) - } - seen[key] = true - var raw string - var present bool - if !defaults { - raw, present = options.Lookup(key) - } - selectedKey := key - if !present && !defaults { - for _, alias := range options.Aliases[key] { - if raw, present = options.Lookup(alias); present { - selectedKey = alias - break - } - } - } - if defaults { - raw, present = field.Tag.Lookup("envDefault") - present = present && dst.IsZero() - } - if present && raw == "" && !defaults { - if empty == "reject" { - return &Error{Env: selectedKey, Field: fieldPath, Reason: "must not be empty"} - } - if empty == "ignore" { - present = false - } - } - if present { - parsed := reflect.New(dst.Type()).Elem() - if err := parseEnvValue(parsed, raw); err != nil { - origin := selectedKey - if defaults { - origin = "" - } - return &Error{Env: origin, Field: fieldPath, Reason: "expected a valid " + dst.Type().String(), cause: err} - } - dst.Set(parsed) - if !defaults && len(sourceMaps) > 0 { - sourceMaps[0][fieldPath] = selectedKey - } - } - if required && !defaults && emptyRequired(dst) { - return &Error{Env: key, Field: fieldPath, Reason: "is required"} - } - } - return nil - } - if err := walk(result, options.Prefix, "", 0); err != nil { - return zero, err - } - return result.Interface().(T), nil -} - -func emptyRequired(v reflect.Value) bool { - switch v.Kind() { - case reflect.String: - return v.Len() == 0 - case reflect.Pointer, reflect.Map, reflect.Slice: - return v.IsNil() - } - return false -} - -func parseEnvValue(v reflect.Value, raw string) (err error) { +func parseTextValue(v reflect.Value, raw string) (err error) { defer func() { if recover() != nil { err = errors.New("invalid custom value") @@ -320,7 +107,7 @@ func parseEnvValue(v reflect.Value, raw string) (err error) { }() if v.Kind() == reflect.Pointer { v.Set(reflect.New(v.Type().Elem())) - return parseEnvValue(v.Elem(), raw) + return parseTextValue(v.Elem(), raw) } if v.CanAddr() && v.Addr().Type().Implements(textType) { return v.Addr().Interface().(encoding.TextUnmarshaler).UnmarshalText([]byte(raw)) @@ -363,37 +150,6 @@ func parseEnvValue(v reflect.Value, raw string) (err error) { v.SetFloat(x) } return err - case reflect.Slice, reflect.Map: - var value any - if err := apierrors.DecodeCatalogJSON([]byte(raw), &value); err != nil { - return err - } - value, err := durations(value, v.Type()) - if err != nil { - return err - } - encoded, err := json.Marshal(value) - if err != nil { - return err - } - decoder := json.NewDecoder(bytes.NewReader(encoded)) - decoder.DisallowUnknownFields() - return decoder.Decode(v.Addr().Interface()) } - return errors.New("unsupported environment value") -} - -func envName(s string) string { - r := []rune(s) - var out strings.Builder - for i, c := range r { - if i > 0 && unicode.IsUpper(c) && (unicode.IsLower(r[i-1]) || unicode.IsDigit(r[i-1]) || (i+1 < len(r) && unicode.IsLower(r[i+1]))) { - out.WriteByte('_') - } - if c == '-' { - c = '_' - } - out.WriteRune(unicode.ToUpper(c)) - } - return out.String() + return errors.New("expected scalar text") } diff --git a/config/projectAdvanced_test.go b/config/projectAdvanced_test.go index fd5cbf7..e646573 100644 --- a/config/projectAdvanced_test.go +++ b/config/projectAdvanced_test.go @@ -42,20 +42,3 @@ func TestConfigFilePreservesDefaultsWithoutSharingMutableData(t *testing.T) { t.Fatal("decode error leaked contents", err) } } -func TestEnvCollectionPrecisionAndDurationMaps(t *testing.T) { - type cfg struct { - IDs []int64 `json:"ids"` - Timers map[string]time.Duration `json:"timers"` - } - lookup := func(key string) (string, bool) { - v, ok := map[string]string{"IDS": "[9007199254740993]", "TIMERS": `{"backoff":"250ms"}`}[key] - return v, ok - } - c, err := OverlayEnv(cfg{}, EnvOptions{Lookup: lookup}) - if err != nil || c.IDs[0] != 9007199254740993 || c.Timers["backoff"] != 250*time.Millisecond { - t.Fatal(c, err) - } - if _, err = OverlayEnv(cfg{}, EnvOptions{Lookup: func(key string) (string, bool) { return `{"x":"1s","x":"2s"}`, key == "TIMERS" }}); err == nil { - t.Fatal("duplicate map key accepted") - } -} diff --git a/config/project_test.go b/config/project_test.go deleted file mode 100644 index 83119d9..0000000 --- a/config/project_test.go +++ /dev/null @@ -1,124 +0,0 @@ -package config - -import ( - "errors" - "os" - "path/filepath" - "reflect" - "strings" - "testing" - "time" -) - -type projectNested struct { - ReadTimeout time.Duration `json:"readTimeout"` - Names []string `json:"names"` -} -type projectConfig struct { - Nested *projectNested `json:"nested"` - Enabled bool `json:"enabled"` - Count int `json:"count" envDefault:"12"` - Password string `json:"-" env:"ETAX_PASSWORD,secret,required,empty=reject"` - Label string `json:"label" env:",empty=ignore"` -} - -func (c projectConfig) Validate() error { - if c.Count < 0 { - return errors.New("count must be nonnegative") - } - return nil -} -func TestProjectConfigurationOrderAndValues(t *testing.T) { - f := filepath.Join(t.TempDir(), "config.yaml") - if err := os.WriteFile(f, []byte("count: 3\nenabled: true\nnested:\n readTimeout: 2s\n names: [file]\nlabel: keep\n"), 0600); err != nil { - t.Fatal(err) - } - env := map[string]string{"ETAX_PASSWORD": " password ", "APP_ENABLED": "false", "APP_COUNT": "0", "APP_NESTED_READ_TIMEOUT": "7s", "APP_NESTED_NAMES": "[]", "APP_LABEL": ""} - options := EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }} - cfg, err := LoadProject(f, projectConfig{}, options) - if err != nil { - t.Fatal(err) - } - if cfg.Enabled || cfg.Count != 0 || cfg.Password != " password " || cfg.Nested.ReadTimeout != 7*time.Second || cfg.Nested.Names == nil || len(cfg.Nested.Names) != 0 || cfg.Label != "keep" { - t.Fatal("overlay mismatch") - } - delete(env, "APP_COUNT") - cfg, err = LoadProject("", projectConfig{}, options) - if err != nil || cfg.Count != 12 { - t.Fatal(cfg.Count, err) - } - env["APP_COUNT"] = "-1" - if _, err = LoadProject(f, projectConfig{}, options); err == nil { - t.Fatal("project Validate not called") - } -} - -func TestProjectOverlayIsTransactionalAndRedacted(t *testing.T) { - base := projectConfig{Nested: &projectNested{ReadTimeout: time.Second, Names: []string{"old"}}, Password: "original"} - env := map[string]string{"APP_NESTED_READ_TIMEOUT": "2s", "APP_NESTED_NAMES": "[\"new\"]", "APP_COUNT": "secret-value"} - _, err := OverlayEnv(base, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}) - if err == nil || strings.Contains(err.Error(), "secret-value") { - t.Fatal(err) - } - if base.Nested.ReadTimeout != time.Second || !reflect.DeepEqual(base.Nested.Names, []string{"old"}) { - t.Fatal("default pointer modified") - } - delete(env, "APP_COUNT") - env["ETAX_PASSWORD"] = "" - if _, err = OverlayEnv(base, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}); err == nil { - t.Fatal("empty secret accepted") - } - type required struct { - Token string `env:"TOKEN,required"` - } - if _, err = OverlayEnv(required{}, EnvOptions{Lookup: func(string) (string, bool) { return "", false }}); err == nil { - t.Fatal("required absent") - } - type duplicate struct { - A string `env:"SAME"` - B string `env:"SAME"` - } - if _, err = OverlayEnv(duplicate{}, EnvOptions{}); err == nil { - t.Fatal("duplicate env binding accepted") - } -} - -func TestEnvironmentCollectionAndEmptySemantics(t *testing.T) { - type sample struct { - Values []time.Duration - Flag bool - Text string - Optional *int - } - cases := []struct { - raw string - valid bool - }{{`["1s","2s"]`, true}, {`["invalid-secret"]`, false}} - for _, tc := range cases { - cfg, err := OverlayEnv(sample{}, EnvOptions{Lookup: func(k string) (string, bool) { - if k == "VALUES" { - return tc.raw, true - } - return "", false - }}) - if (err == nil) != tc.valid { - t.Fatal(err) - } - if tc.valid && !reflect.DeepEqual(cfg.Values, []time.Duration{time.Second, 2 * time.Second}) { - t.Fatal(cfg) - } - if err != nil && strings.Contains(err.Error(), "secret") { - t.Fatal(err) - } - } - for _, mode := range []string{"allow", "ignore", "reject"} { - cfg, err := OverlayEnv(sample{Text: "old"}, EnvOptions{Empty: mode, Lookup: func(k string) (string, bool) { return "", k == "TEXT" }}) - if mode == "reject" { - if err == nil { - t.Fatal("empty not rejected") - } - } else if err != nil || (mode == "allow" && cfg.Text != "") || (mode == "ignore" && cfg.Text != "old") { - t.Fatal(cfg, err) - } - } -} diff --git a/config/references.go b/config/references.go new file mode 100644 index 0000000..c7e8713 --- /dev/null +++ b/config/references.go @@ -0,0 +1,210 @@ +package config + +import ( + "errors" + "fmt" + "strings" + "unicode/utf8" +) + +// WithLookupEnv supplies an explicit-reference lookup for tests or an embedding +// host. Ordinary projects need no loader options. The function must support +// concurrent loads; each load caches each referenced name independently. +func WithLookupEnv(lookup func(string) (string, bool)) LoaderOption { + return func(l *Loader) error { + if l == nil || l.sealed || lookup == nil { + return errors.New("config: expected a non-nil environment lookup on a new loader") + } + l.lookup = lookup + return nil + } +} + +// resolvedText is textual input bound according to the target scalar type. It +// never becomes an object, collection or null by interpreting environment data. +type resolvedText string + +const maxExpandedBytes = 4 << 20 + +type environmentValue struct { + value string + present bool +} +type referenceResolver struct { + lookup func(string) (string, bool) + cache map[string]environmentValue + doc Document + nodes, bytes, cacheBytes int +} + +func resolveDocument(doc Document, lookup func(string) (string, bool)) (Document, error) { + r := referenceResolver{lookup: lookup, cache: map[string]environmentValue{}, doc: Document{Positions: doc.Positions, environments: map[string][]string{}}} + v, err := r.walk(doc.Value, "", 0) + if err != nil { + return Document{}, err + } + r.doc.Value = v + return r.doc, nil +} +func (r *referenceResolver) walk(v any, path string, depth int) (any, error) { + r.nodes++ + if depth > maxConfigDepth || r.nodes > maxConfigNodes { + return nil, r.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil) + } + switch x := v.(type) { + case string: + return r.text(x, path, false) + case Text: + return r.text(string(x), path, true) + case map[string]any: + out := make(map[string]any, len(x)) + for _, key := range sortedKeys(x) { + value, err := r.walk(x[key], joinField(path, key), depth+1) + if err != nil { + return nil, err + } + out[key] = value + } + return out, nil + case []any: + out := make([]any, len(x)) + for i, value := range x { + var err error + out[i], err = r.walk(value, fmt.Sprintf("%s[%d]", path, i), depth+1) + if err != nil { + return nil, err + } + } + return out, nil + default: + return v, nil + } +} +func (r *referenceResolver) fail(path, name, reason string) error { + d := r.doc.diagnostic(path, reason, nil) + d.Env = name + return d +} +func validEnvironmentName(name string) bool { + if len(name) == 0 { + return false + } + for i := 0; i < len(name); i++ { + c := name[i] + if c == '_' || c >= 'A' && c <= 'Z' || c >= 'a' && c <= 'z' || i > 0 && c >= '0' && c <= '9' { + continue + } + return false + } + return true +} +func (r *referenceResolver) lookupValue(name, path string) (environmentValue, error) { + if v, ok := r.cache[name]; ok { + return v, nil + } + value, present := r.lookup(name) + if len(value) > maxConfigBytes || r.cacheBytes+len(value) > maxExpandedBytes { + return environmentValue{}, r.fail(path, name, "environment value exceeds configuration size limit") + } + if !utf8.ValidString(value) { + return environmentValue{}, r.fail(path, name, "environment value is not valid UTF-8") + } + + r.cacheBytes += len(value) + v := environmentValue{value, present} + r.cache[name] = v + return v, nil +} +func (r *referenceResolver) text(raw, path string, xmlText bool) (any, error) { + var out strings.Builder + var names []string + seen := map[string]bool{} + if !strings.Contains(raw, "$") { + if len(raw) > maxConfigBytes || r.bytes+len(raw) > maxExpandedBytes { + return nil, r.fail(path, "", "expanded configuration exceeds size limit") + } + r.bytes += len(raw) + if xmlText { + return Text(raw), nil + } + return raw, nil + } + appendText := func(s string) error { + if out.Len()+len(s) > maxConfigBytes || r.bytes+len(s) > maxExpandedBytes { + return r.fail(path, strings.Join(names, ", "), "expanded configuration exceeds size limit") + } + out.WriteString(s) + r.bytes += len(s) + return nil + } + // Scan only the file's decoded scalar. Inserted environment values are never + // scanned again, including dollars, quotes, separators or newline characters. + for i := 0; i < len(raw); { + j := strings.IndexByte(raw[i:], '$') + if j < 0 { + if err := appendText(raw[i:]); err != nil { + return nil, err + } + break + } + j += i + if err := appendText(raw[i:j]); err != nil { + return nil, err + } + switch { + case j+1 < len(raw) && raw[j+1] == '$': + if err := appendText("$"); err != nil { + return nil, err + } + i = j + 2 + case j+1 < len(raw) && raw[j+1] == '{': + end := strings.IndexByte(raw[j+2:], '}') + if end < 0 { + return nil, r.fail(path, "", "unclosed environment reference; use ${NAME}, ${NAME:-fallback} or $${...} for a literal") + } + end += j + 2 + expression := raw[j+2 : end] + name, fallback, hasDefault := strings.Cut(expression, ":-") + if !validEnvironmentName(name) || strings.Contains(fallback, "${") { + return nil, r.fail(path, "", "invalid environment reference; use ${NAME} or ${NAME:-fallback}; nested references are not supported") + } + r.nodes++ + if r.nodes > maxConfigNodes { + return nil, r.fail(path, name, "environment reference count exceeds configuration limit") + } + if !seen[name] { + names = append(names, name) + seen[name] = true + } + + value, err := r.lookupValue(name, path) + if err != nil { + return nil, err + } + selected := value.value + if !value.present || selected == "" { + if !hasDefault { + return nil, r.fail(path, name, "required environment variable is unset or empty") + } + selected = fallback + } + if err := appendText(selected); err != nil { + return nil, err + } + i = end + 1 + default: + if err := appendText("$"); err != nil { + return nil, err + } + i = j + 1 + } + } + if len(names) > 0 { + r.doc.environments[path] = names + return resolvedText(out.String()), nil + } + if xmlText { + return Text(out.String()), nil + } + return out.String(), nil +} diff --git a/config/references_test.go b/config/references_test.go new file mode 100644 index 0000000..32a902a --- /dev/null +++ b/config/references_test.go @@ -0,0 +1,230 @@ +package config + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "strings" + "sync" + "testing" + "time" +) + +type referenceConfig struct { + Port int `json:"port"` + BatchSize int `json:"batchSize"` + Enabled bool `json:"enabled"` + Password string `json:"password"` + Address string `json:"address"` + Timeout time.Duration `json:"timeout"` + Optional *int `json:"optional"` + Labels map[string]string `json:"labels"` + Numbers []int64 `json:"numbers"` + Dynamic any `json:"dynamic"` +} + +func (c referenceConfig) Validate() error { + if c.Port < 0 { + return Invalid("port", "must not be negative") + } + return nil +} +func TestExplicitReferencesAcrossFormats(t *testing.T) { + // Delimiters, quotes and newlines belong to the credential, never file syntax. + secret := " DDSFLKJlsf:saf:-123\"\n${DO_NOT_READ}&<>\\ " + sources := map[string]string{ + "json": `{"port":"${PORT:-3306}","enabled":"${ENABLED:-true}","password":"${PASSWORD}","address":"https://${HOST}:${PORT:-3306}/v1","timeout":"${TIMEOUT:-3s}","optional":"${ZERO}","numbers":["${BIG}"],"dynamic":"${ZERO}","labels":{"literal":"$${NOT_READ}","empty":"${OPTIONAL:-}","dollars":"$$cost"}}`, + "yaml": "port: \"${PORT:-3306}\"\nenabled: \"${ENABLED:-true}\"\npassword: \"${PASSWORD}\"\naddress: \"https://${HOST}:${PORT:-3306}/v1\"\ntimeout: \"${TIMEOUT:-3s}\"\noptional: \"${ZERO}\"\nnumbers: [\"${BIG}\"]\ndynamic: \"${ZERO}\"\nlabels:\n literal: \"$${NOT_READ}\"\n empty: \"${OPTIONAL:-}\"\n dollars: \"$$cost\"\n", + "xml": `${PORT:-3306}${ENABLED:-true}${PASSWORD}
https://${HOST}:${PORT:-3306}/v1
${TIMEOUT:-3s}${ZERO}${BIG}${ZERO}$${NOT_READ}${OPTIONAL:-}$$cost`, + } + for ext, source := range sources { + t.Run(ext, func(t *testing.T) { + calls := map[string]int{} + values := map[string]string{"PORT": "0", "ENABLED": "false", "PASSWORD": secret, "HOST": "example.test", "ZERO": "0", "BIG": "9007199254740993"} + l := testLoader(func(name string) (string, bool) { calls[name]++; v, ok := values[name]; return v, ok }) + got, err := LoadProject(configFile(t, ext, source), referenceConfig{Port: 77, BatchSize: 100, Enabled: true, Password: "old"}, l) + if err != nil { + t.Fatal(err) + } + if got.Port != 0 || got.BatchSize != 100 || got.Enabled || got.Password != secret || got.Address != "https://example.test:0/v1" || got.Timeout != 3*time.Second || got.Optional == nil || *got.Optional != 0 || got.Numbers[0] != 9007199254740993 || got.Dynamic != "0" { + t.Fatal("binding changed values") + } + if got.Labels["literal"] != "${NOT_READ}" || got.Labels["empty"] != "" || got.Labels["dollars"] != "$cost" { + t.Fatal(got.Labels) + } + for name, n := range calls { + if n != 1 { + t.Fatal("not cached", name, n) + } + } + if calls["DO_NOT_READ"] != 0 || calls["NOT_READ"] != 0 { + t.Fatal("recursive expansion") + } + }) + } +} +func TestReferenceFallbackAndFixedValues(t *testing.T) { + for _, present := range []bool{false, true} { + l := testLoader(func(string) (string, bool) { return "", present }) + got, err := LoadProject(configFile(t, "yaml", "port: \"${PORT:-3306}\"\npassword: \"${PASSWORD:-DDSFLKJlsf:saf:-123}\""), referenceConfig{}, l) + if err != nil || got.Port != 3306 || got.Password != "DDSFLKJlsf:saf:-123" { + t.Fatal(got, err) + } + for _, ext := range []string{"json", "yaml", "xml"} { + raw := `{"password":"${PASSWORD}"}` + if ext == "yaml" { + raw = "password: '${PASSWORD}'" + } + if ext == "xml" { + raw = "${PASSWORD}" + } + path := configFile(t, ext, raw) + _, err = LoadProject(path, referenceConfig{Password: "initial"}, l) + var d *Error + if !errors.As(err, &d) || d.Env != "PASSWORD" || d.File != path || d.Field != "password" || d.Line < 1 { + t.Fatal("missing required-reference diagnostic", err) + } + } + } + calls := 0 + l := testLoader(func(string) (string, bool) { calls++; return "999", true }) + got, err := LoadProject(configFile(t, "json", `{"port":0,"enabled":false,"password":""}`), referenceConfig{Port: 77, BatchSize: 100, Enabled: true, Password: "old"}, l) + if err != nil || got.Port != 0 || got.BatchSize != 100 || got.Enabled || got.Password != "" || calls != 0 { + t.Fatal(got, err, calls) + } + got, err = LoadProject(configFile(t, "yaml", "password: '${VALUE}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return " ", true })) + if err != nil || got.Password != " " { + t.Fatal("trimmed credential", err) + } +} +func TestReferenceDiagnosticsAndScalarLimits(t *testing.T) { + for _, value := range []string{"private-not-int", "92233720368547758080", "false", "null", "-1"} { + path := configFile(t, "yaml", "port: '${PORT}'") + _, err := LoadProject(path, referenceConfig{}, testLoader(func(string) (string, bool) { return value, true })) + var d *Error + if !errors.As(err, &d) || d.File != path || d.Env != "PORT" || d.Field != "port" || d.Line != 1 || d.Column < 1 || strings.Contains(err.Error(), value) { + t.Fatalf("invalid diagnostic: %v", err) + } + } + for _, raw := range []string{"${BAD-123}", "${}", "${X", "${1BAD}", "${BAD NAME}", "${X:-${Y}}"} { + encoded, _ := json.Marshal(map[string]string{"password": raw}) + if _, err := LoadProject(configFile(t, "json", string(encoded)), referenceConfig{}, testLoader(noEnvironment)); err == nil { + t.Fatal("invalid syntax accepted", raw) + } + } + for _, value := range []string{string([]byte{0xff}), strings.Repeat("x", maxConfigBytes+1)} { + if _, err := LoadProject(configFile(t, "yaml", "password: '${SECRET}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return value, true })); err == nil { + t.Fatal("unbounded/invalid environment") + } + } + path := configFile(t, "yaml", "labels:\n a: '${BIG}'\n b: '${BIG}'\n c: '${BIG}'\n d: '${BIG}'\n e: '${BIG}'") + if _, err := LoadProject(path, referenceConfig{}, testLoader(func(string) (string, bool) { return strings.Repeat("x", maxConfigBytes), true })); err == nil { + t.Fatal("expansion budget bypassed") + } + // An environment value cannot inject an array/object or coerce dynamic null. + for _, raw := range []string{"[1,2]", "{\"x\":1}", "null"} { + _, err := LoadProject(configFile(t, "yaml", "numbers: '${VALUE}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return raw, true })) + if err == nil { + t.Fatal("structure injected") + } + } +} +func TestReferenceCacheIsolationAndRemovedRules(t *testing.T) { + path := configFile(t, "yaml", "password: '${TOKEN}'") + for i := 0; i < 24; i++ { + t.Run(fmt.Sprint(i), func(t *testing.T) { + t.Parallel() + v := fmt.Sprint(i) + l := testLoader(func(string) (string, bool) { return v, true }) + for n := 0; n < 4; n++ { + got, err := LoadProject(path, referenceConfig{}, l) + if err != nil || got.Password != v { + t.Fatal("load leaked", err) + } + } + }) + } +} +func TestReferenceLookupPerLoadAndConcurrentLoader(t *testing.T) { + var mu sync.Mutex + reads := 0 + l := testLoader(func(string) (string, bool) { mu.Lock(); defer mu.Unlock(); reads++; return "secret", true }) + path := configFile(t, "yaml", "password: '${TOKEN}${TOKEN}'") + var wg sync.WaitGroup + for i := 0; i < 24; i++ { + wg.Add(1) + go func() { + defer wg.Done() + got, err := LoadProject(path, referenceConfig{}, l) + if err != nil || got.Password != "secretsecret" { + t.Error(err) + } + }() + } + wg.Wait() + if reads != 24 { + t.Fatal(reads) + } +} +func TestRemovedEnvTagsAndFieldSuggestion(t *testing.T) { + type old struct { + Value int `json:"value" envDefault:"9"` + } + if _, err := LoadProject("", old{}); err == nil || !strings.Contains(err.Error(), "removed") { + t.Fatal(err) + } + type oldSecret struct { + Token string `json:"-" env:"TOKEN"` + } + if _, err := DecodeFile(configFile(t, "json", `{}`), oldSecret{}); err == nil { + t.Fatal("old env tag ignored") + } + _, err := LoadProject(configFile(t, "yaml", "batchSiz: 10"), referenceConfig{}) + if err == nil || !strings.Contains(err.Error(), `did you mean "batchSize"`) { + t.Fatal(err) + } + t.Setenv("GO_OMNI_BATCH_SIZE", "999") + got, err := LoadProject("", referenceConfig{BatchSize: 100}) + if err != nil || got.BatchSize != 100 { + t.Fatal("implicit overlay", err) + } +} +func FuzzConfigurationReferences(f *testing.F) { + for _, s := range []string{"${X}", "$${X}", "${X:-DDS:abc:-123}", "${X:-}", "https://${X}/${X}", "${X:-${Y}}"} { + f.Add(s, " secret${NO}\n\" ") + } + f.Fuzz(func(t *testing.T, raw, value string) { + if len(raw) > 8192 || len(value) > 8192 { + return + } + doc := Document{Value: map[string]any{"password": raw}} + resolved, err := resolveDocument(doc, func(string) (string, bool) { return value, true }) + if err != nil { + return + } + _, _ = bindDocument(resolved, referenceConfig{}) + }) +} +func BenchmarkExplicitConfiguration(b *testing.B) { + for _, refs := range []bool{false, true} { + b.Run(fmt.Sprint(refs), func(b *testing.B) { + source := `{"port":3306,"enabled":false,"password":"fixed","batchSize":0}` + if refs { + source = `{"port":"${PORT:-3306}","enabled":"${ENABLED:-false}","password":"${PASSWORD}","batchSize":0}` + } + path := b.TempDir() + "/config.json" + if err := os.WriteFile(path, []byte(source), 0600); err != nil { + b.Fatal(err) + } + l := testLoader(func(k string) (string, bool) { return "private", k == "PASSWORD" }) + b.ReportAllocs() + b.ResetTimer() + for i := 0; i < b.N; i++ { + if _, err := LoadProject(path, referenceConfig{BatchSize: 100}, l); err != nil { + b.Fatal(err) + } + } + }) + } +} diff --git a/config/suggestions.go b/config/suggestions.go new file mode 100644 index 0000000..c7abb76 --- /dev/null +++ b/config/suggestions.go @@ -0,0 +1,51 @@ +package config + +import ( + "reflect" + "strconv" + "strings" +) + +// Suggest only a unique, nearby declared field. Input values are never used. +func unknownFieldReason(name string, fields map[string]reflect.StructField) string { + if len(name) < 3 || len(name) > 128 { + return "unknown field" + } + best, match, tied := 3, "", false + for candidate := range fields { + if len(candidate) > 128 { + continue + } + d := fieldDistance(strings.ToLower(name), strings.ToLower(candidate)) + if d < best { + best, match, tied = d, candidate, false + } else if d == best { + tied = true + } + } + if match != "" && !tied && best <= 2 { + return "unknown field; did you mean " + strconv.Quote(match) + "?" + } + return "unknown field" +} +func fieldDistance(a, b string) int { + prev := make([]int, len(b)+1) + for i := range prev { + prev[i] = i + } + for i := 0; i < len(a); i++ { + left, diagonal := i+1, prev[0] + prev[0] = left + for j := 0; j < len(b); j++ { + cost := 0 + if a[i] != b[j] { + cost = 1 + } + old := prev[j+1] + left = min(left+1, old+1, diagonal+cost) + prev[j+1] = left + diagonal = old + } + } + return prev[len(b)] +} diff --git a/database/README.md b/database/README.md index db0d5c6..9103bba 100644 --- a/database/README.md +++ b/database/README.md @@ -6,7 +6,7 @@ `database` 包提供基于标准库 `database/sql` 的连接池生命周期，不绑定 ORM 或驱动： -- `Config` / `ConfigFromEnv`：连接池参数和 `GO_OMNI_DB_*` 环境变量。 +- `Config`：驱动、DSN 与连接池参数；嵌入项目配置，通过文件显式引用变量。 - `Open` / `OpenConnector`：通过 DSN 或标准 Connector 创建连接池、设置连接上限并在返回前执行 `PingContext`。 - `Health`：用于就绪检查，等待连接和 Ping 均受 `PingTimeout` 限制。 - `InTx`：统一事务提交和回滚；工作函数返回错误或 panic 时回滚。 @@ -17,11 +17,12 @@ ```go import ( "context" + "git.yuntuo.hk/k2safe/go-omni/config" _ "github.com/go-sql-driver/mysql" "git.yuntuo.hk/k2safe/go-omni/database" ) -cfg, err := database.ConfigFromEnv("GO_OMNI_DB_") +cfg, err := config.LoadProject("etc/database.yaml", database.DefaultConfig()) if err != nil { return err } @@ -32,19 +33,15 @@ if err != nil { defer db.Close() ``` -连接池相关环境变量： +对应文件（框架不规定变量名）： -```text -GO_OMNI_DB_DRIVER=mysql -GO_OMNI_DB_DSN=user:password@tcp(127.0.0.1:3306)/app?parseTime=true -GO_OMNI_DB_MAX_OPEN_CONNS=10 -GO_OMNI_DB_MAX_IDLE_CONNS=5 -GO_OMNI_DB_CONN_MAX_LIFETIME=30m -GO_OMNI_DB_CONN_MAX_IDLE_TIME=5m -GO_OMNI_DB_PING_TIMEOUT=5s +```yaml +driver: mysql +dsn: "${DB_DSN}" +maxOpenConns: "${DB_MAX_OPEN:-10}" ``` -使用 `ConfigFromEnv` / `Open` 时 `DSN` 必填，环境变量入口的驱动默认是 `mysql`。数据库建库、迁移、表结构和具体 Repository 属于业务项目或可选模块，不会被核心包自动执行。 +Open 使用的 Driver、DSN 必填。建库、迁移、表结构和业务 Repository 由项目负责。v0.22 删除 ConfigFromEnv/ApplyEnv；不再隐式扫描数据库变量。 `MaxIdleConns=0` 会禁用空闲连接。初始化 Ping 失败会关闭已创建的连接池。 @@ -62,12 +59,9 @@ python3 scripts/verifyDatabase.py --version v0.2.0 结束时清理容器、匿名数据卷和临时项目。常规 `go test ./...` 不依赖 Docker。 -## 项目配置叠加 +## 项目配置 -`database.ApplyEnv(cfg, prefix)` 在已有文件/默认配置上覆盖非空环境变量，保留未覆盖的池参数。 -`Config` 的 JSON/YAML 字段使用 lowerCamelCase，`DSN` 使用 `json:"-"`，连接密钥从环境变量加载。 -API 脚手架 `--database mysql` 使用此流程，并要求正的 `maxOpenConns`，提供数据库 readiness 与幂等清理。 -具体表结构、迁移和 Repository 留在业务项目，参考 `examples/userService`。 +配置结构作为项目 Config 的字段，Default() 初始化连接池，文件按需覆盖。DSN 可使用 `dsn: "${DB_DSN}"`；database.Config 的 JSON 序列化不输出 DSN 原值。不要打印原始配置对象。当前 API 脚手架使用结构化 MySQL 适配器，由 svc 统一初始化与关闭。 ## Connector 接入（v0.9.1-preview） @@ -103,7 +97,7 @@ func Open(ctx context.Context, address, user, password, name string, } ``` -调用时以 `database.DefaultPoolConfig()` 为起点，按项目需要调整。已有 `database.Config` 可调用 `.PoolConfig()` 提取池设置；`ConfigFromEnv` / `ApplyEnv` 仍是 DSN 入口，不用为 Connector 填假 DSN。`PoolConfig` 支持通过 `config.DecodeFile` 加载项目的 JSON/YAML 嵌套配置，环境覆盖由项目按自己的驱动配置组织。 +调用时以 `database.DefaultPoolConfig()` 为起点，按项目需要调整。已有 `database.Config` 可调用 `.PoolConfig()` 提取池设置；Connector 不需要填写 DSN。`PoolConfig` 支持通过 `config.DecodeFile` 加载项目的 JSON/YAML 嵌套配置，环境引用由项目在文件里显式声明。 | PoolConfig 字段 | 默认值 / 含义 | | --- | --- | diff --git a/database/configOverlay_test.go b/database/configOverlay_test.go index b50fbe7..c461e53 100644 --- a/database/configOverlay_test.go +++ b/database/configOverlay_test.go @@ -2,19 +2,27 @@ package database import ( "encoding/json" + "git.yuntuo.hk/k2safe/go-omni/config" + "os" + "path/filepath" "strings" "testing" "time" ) -func TestApplyEnvPreservesFilePoolAndHidesDSN(t *testing.T) { +func TestExplicitConfigPreservesPoolAndHidesDSN(t *testing.T) { cfg := DefaultConfig() + cfg.Driver = "mysql" + path := filepath.Join(t.TempDir(), "db.yaml") + if err := os.WriteFile(path, []byte("dsn: \"${OMNI_TEST_DSN}\"\nmaxIdleConns: \"${OMNI_TEST_MAX_IDLE_CONNS}\"\nmaxOpenConns: \"${OMNI_TEST_MAX_OPEN_CONNS:-3}\""), 0600); err != nil { + t.Fatal(err) + } cfg.MaxOpenConns = 3 cfg.MaxIdleConns = 1 cfg.PingTimeout = time.Second t.Setenv("OMNI_TEST_DSN", "user:secret@tcp(localhost:3306)/test") t.Setenv("OMNI_TEST_MAX_IDLE_CONNS", "0") - got, err := ApplyEnv(cfg, "OMNI_TEST_") + got, err := config.LoadProject(path, cfg) if err != nil { t.Fatal(err) } @@ -29,7 +37,7 @@ func TestApplyEnvPreservesFilePoolAndHidesDSN(t *testing.T) { t.Fatalf("unsafe config encoding: %s", data) } t.Setenv("OMNI_TEST_MAX_OPEN_CONNS", "bad") - if _, err = ApplyEnv(cfg, "OMNI_TEST_"); err == nil { + if _, err = config.LoadProject(path, cfg); err == nil { t.Fatal("invalid override accepted") } } diff --git a/database/database.go b/database/database.go index 1851d51..cb7d770 100644 --- a/database/database.go +++ b/database/database.go @@ -5,17 +5,16 @@ package database import ( "context" "database/sql" + "encoding/json" "errors" "fmt" - "os" - "strconv" "strings" "time" ) type Config struct { Driver string `json:"driver"` - DSN string `json:"-"` + DSN string `json:"dsn"` MaxOpenConns int `json:"maxOpenConns"` MaxIdleConns int `json:"maxIdleConns"` ConnMaxLifetime time.Duration `json:"connMaxLifetime"` @@ -23,6 +22,14 @@ type Config struct { PingTimeout time.Duration `json:"pingTimeout"` } +// MarshalJSON omits the connection credential from serialized diagnostics. +func (c Config) MarshalJSON() ([]byte, error) { + type plain Config + safe := plain(c) + safe.DSN = "" + return json.Marshal(safe) +} + func DefaultConfig() Config { return DefaultPoolConfig().config() } // PoolConfig returns driver-independent connection pool settings. @@ -34,49 +41,6 @@ func (c Config) PoolConfig() PoolConfig { } } -// ConfigFromEnv reads GO_OMNI_DB_* style settings. DSN is required. -func ConfigFromEnv(prefix string) (Config, error) { - return ApplyEnv(DefaultConfig(), prefix) -} - -// ApplyEnv overlays nonempty environment settings on file/default configuration. -// DSN is environment-only when Config is decoded from JSON or YAML. -func ApplyEnv(cfg Config, prefix string) (Config, error) { - prefix = strings.TrimSpace(prefix) - if prefix == "" { - prefix = "GO_OMNI_DB_" - } - if value := strings.TrimSpace(os.Getenv(prefix + "DRIVER")); value != "" { - cfg.Driver = value - } - if cfg.Driver == "" { - cfg.Driver = "mysql" - } - if value := strings.TrimSpace(os.Getenv(prefix + "DSN")); value != "" { - cfg.DSN = value - } - if cfg.DSN == "" { - return Config{}, fmt.Errorf("%sDSN is required", prefix) - } - var err error - if cfg.MaxOpenConns, err = envInt(prefix+"MAX_OPEN_CONNS", cfg.MaxOpenConns); err != nil { - return Config{}, err - } - if cfg.MaxIdleConns, err = envInt(prefix+"MAX_IDLE_CONNS", cfg.MaxIdleConns); err != nil { - return Config{}, err - } - if cfg.ConnMaxLifetime, err = envDuration(prefix+"CONN_MAX_LIFETIME", cfg.ConnMaxLifetime); err != nil { - return Config{}, err - } - if cfg.ConnMaxIdleTime, err = envDuration(prefix+"CONN_MAX_IDLE_TIME", cfg.ConnMaxIdleTime); err != nil { - return Config{}, err - } - if cfg.PingTimeout, err = envDuration(prefix+"PING_TIMEOUT", cfg.PingTimeout); err != nil { - return Config{}, err - } - return cfg, cfg.Validate() -} - func (c Config) Validate() error { if strings.TrimSpace(c.Driver) == "" { return errors.New("database driver is required") @@ -165,27 +129,3 @@ func (db *DB) InTx(ctx context.Context, options *sql.TxOptions, work func(*sql.T } return nil } - -func envInt(key string, fallback int) (int, error) { - value := strings.TrimSpace(os.Getenv(key)) - if value == "" { - return fallback, nil - } - parsed, err := strconv.Atoi(value) - if err != nil { - return 0, fmt.Errorf("%s must be an integer: %w", key, err) - } - return parsed, nil -} - -func envDuration(key string, fallback time.Duration) (time.Duration, error) { - value := strings.TrimSpace(os.Getenv(key)) - if value == "" { - return fallback, nil - } - parsed, err := time.ParseDuration(value) - if err != nil { - return 0, fmt.Errorf("%s must be a duration: %w", key, err) - } - return parsed, nil -} diff --git a/database/database_test.go b/database/database_test.go index 49dbbe6..963a5fe 100644 --- a/database/database_test.go +++ b/database/database_test.go @@ -2,54 +2,10 @@ package database import ( "context" - "os" "strings" "testing" - "time" ) -func TestConfigFromEnv(t *testing.T) { - keys := []string{ - "GO_OMNI_DB_DRIVER", "GO_OMNI_DB_DSN", "GO_OMNI_DB_MAX_OPEN_CONNS", - "GO_OMNI_DB_MAX_IDLE_CONNS", "GO_OMNI_DB_CONN_MAX_LIFETIME", - "GO_OMNI_DB_CONN_MAX_IDLE_TIME", "GO_OMNI_DB_PING_TIMEOUT", - } - for _, key := range keys { - t.Setenv(key, "") - _ = os.Unsetenv(key) - } - t.Setenv("GO_OMNI_DB_DSN", "root:secret@tcp(127.0.0.1:3306)/demo") - t.Setenv("GO_OMNI_DB_MAX_OPEN_CONNS", "20") - t.Setenv("GO_OMNI_DB_MAX_IDLE_CONNS", "8") - t.Setenv("GO_OMNI_DB_CONN_MAX_LIFETIME", "2m") - t.Setenv("GO_OMNI_DB_PING_TIMEOUT", "750ms") - cfg, err := ConfigFromEnv("") - if err != nil { - t.Fatal(err) - } - if cfg.Driver != "mysql" || cfg.DSN == "" || cfg.MaxOpenConns != 20 || cfg.MaxIdleConns != 8 || cfg.ConnMaxLifetime != 2*time.Minute || cfg.PingTimeout != 750*time.Millisecond { - t.Fatalf("unexpected config: %+v", cfg) - } -} - -func TestConfigFromEnvRejectsInvalidValues(t *testing.T) { - t.Setenv("GO_OMNI_DB_DSN", "demo") - t.Setenv("GO_OMNI_DB_MAX_OPEN_CONNS", "bad") - if _, err := ConfigFromEnv(""); err == nil || !strings.Contains(err.Error(), "MAX_OPEN_CONNS") { - t.Fatalf("expected invalid pool limit error, got %v", err) - } - t.Setenv("GO_OMNI_DB_MAX_OPEN_CONNS", "2") - t.Setenv("GO_OMNI_DB_MAX_IDLE_CONNS", "3") - if _, err := ConfigFromEnv(""); err == nil || !strings.Contains(err.Error(), "max idle") { - t.Fatalf("expected pool relationship error, got %v", err) - } - t.Setenv("GO_OMNI_DB_MAX_IDLE_CONNS", "1") - t.Setenv("GO_OMNI_DB_PING_TIMEOUT", "0s") - if _, err := ConfigFromEnv(""); err == nil || !strings.Contains(err.Error(), "timeouts") { - t.Fatalf("expected timeout error, got %v", err) - } -} - func TestOpenValidatesBeforeLoadingDriver(t *testing.T) { _, err := Open(context.Background(), Config{Driver: "", DSN: "demo"}) if err == nil || !strings.Contains(err.Error(), "driver") { diff --git a/database/mysql/README.md b/database/mysql/README.md index c0fbd1f..4e104ed 100644 --- a/database/mysql/README.md +++ b/database/mysql/README.md @@ -10,13 +10,14 @@ go-omni new --database mysql --module example.com/billing ./billing ``` -`etc/config.yaml` 使用结构化配置，环境覆盖优先： +`etc/config.yaml` 使用结构化配置；需要环境变量时在字段值明确写引用，固定值不被覆盖： ```yaml database: host: db.example.com port: 3306 user: app + password: "${DB_PASSWORD}" database: billing parseTime: true location: UTC @@ -37,7 +38,7 @@ database: pingTimeout: 5s ``` -通过 `GO_OMNI_DB_PASSWORD` 传密码；还有 HOST、PORT、USER、DATABASE、TLS_CA_FILE、TLS_CERT_FILE、TLS_KEY_FILE、POOL_MAX_OPEN_CONNS 等。业务也可自定义 env 标签。新生成服务不再读取 DSN；独立数据库迁移 CLI 和手写 `database.Open/OpenConnector` 的原接口保留。 +示例通过文件里的 `${DB_PASSWORD}` 读取密码；HOST、PORT、TLS 和池参数同样按需在文件写引用，不自动扫描变量、不使用 env 标签。新生成服务使用结构化连接参数；独立迁移 CLI 的显式 DSN 参数及 Open/OpenConnector 接口保留。 默认启用 TLS 并校验系统信任链与主机名；私有 CA 用 caFile，双向 TLS 额外设置 certFile/keyFile。只有明确设置 `tls.mode: disabled` 才使用明文。Unix socket 用 socket，若在 socket 上使用 TLS，必须提供 serverName。不会注册全局 TLS 配置或全局驱动 Logger。 diff --git a/database/mysql/mysql.go b/database/mysql/mysql.go index 17a7cf2..8cbcbeb 100644 --- a/database/mysql/mysql.go +++ b/database/mysql/mysql.go @@ -28,7 +28,7 @@ type Config struct { Port int `json:"port"` Socket string `json:"socket"` User string `json:"user"` - Password string `json:"password" env:",secret"` + Password string `json:"password"` Database string `json:"database"` ParseTime bool `json:"parseTime"` Location string `json:"location"` diff --git a/examples/configuration/README.md b/examples/configuration/README.md index 093d223..7ed9258 100644 --- a/examples/configuration/README.md +++ b/examples/configuration/README.md @@ -10,11 +10,13 @@ DEMO_CLIENT_TIMEOUT=500ms DEMO_MAX_ITEMS=0 go run ./examples/configuration DEMO_MAX_ITEMS=wrong go run ./examples/configuration ``` -前三次都输出 `timeout=2s maxItems=100 allowed=[json xml]`。环境覆盖允许显式零值。 +前三次都输出 `timeout=2s maxItems=100 allowed=[json xml]`。文件显式引用允许零值。 最后一条输出定位 `maxItems` / `DEMO_MAX_ITEMS` 的安全错误，不打印输入原值。 也可 `-config /path/to/app.yaml` 读取自己的文件。示例没有建立网络连接。 -`Config` 定义字段，`envDefault` 定义缺省值，`Validate` 用 `config.Invalid` 返回字段与公开原因。 +`Config` 定义字段，`Default()` 定义初始值，`Validate` 用 `config.Invalid` 返回字段与公开原因。 +文件中的 `${NAME}` 必填，`${NAME:-value}` 支持空/未设置的备用值，`$${NAME}` 保留字面量；字符串不裁剪，也不二次展开。没有出现在文件引用中的变量不会被读取。 + `newLoader` 额外注册 `.custom`（演示复用 JSON decoder）；同一个 `load` 可读取所有格式。 JSON/YAML/XML 直接可用，无需注册。新增格式只修改解析器注册，不改业务结构或校验。 diff --git a/examples/configuration/config.json b/examples/configuration/config.json index 9cf71b9..a29bf0c 100644 --- a/examples/configuration/config.json +++ b/examples/configuration/config.json @@ -1,5 +1 @@ -{ - "client": {"endpoint": "https://example.com", "timeout": "2s"}, - "maxItems": 100, - "allowed": ["json", "xml"] -} +{"client":{"endpoint":"https://${DEMO_HOST:-example.com}","timeout":"${DEMO_CLIENT_TIMEOUT:-2s}","token":"${DEMO_CLIENT_TOKEN:-}"},"maxItems":"${DEMO_MAX_ITEMS:-100}","allowed":["json","xml"]} diff --git a/examples/configuration/config.xml b/examples/configuration/config.xml index ea4492d..1c6fbc6 100644 --- a/examples/configuration/config.xml +++ b/examples/configuration/config.xml @@ -1,8 +1 @@ - - - https://example.com - 2s - - 100 - jsonxml - +https://${DEMO_HOST:-example.com}${DEMO_CLIENT_TIMEOUT:-2s}${DEMO_CLIENT_TOKEN:-}${DEMO_MAX_ITEMS:-100}jsonxml diff --git a/examples/configuration/config.yaml b/examples/configuration/config.yaml index 41a32e3..8b741e6 100644 --- a/examples/configuration/config.yaml +++ b/examples/configuration/config.yaml @@ -1,5 +1,6 @@ client: - endpoint: https://example.com - timeout: 2s -maxItems: 100 + endpoint: "https://${DEMO_HOST:-example.com}" + timeout: "${DEMO_CLIENT_TIMEOUT:-2s}" + token: "${DEMO_CLIENT_TOKEN:-}" +maxItems: "${DEMO_MAX_ITEMS:-100}" allowed: [json, xml] diff --git a/examples/configuration/main.go b/examples/configuration/main.go index 785ca4b..6838d0c 100644 --- a/examples/configuration/main.go +++ b/examples/configuration/main.go @@ -17,16 +17,18 @@ var examples embed.FS type ClientConfig struct { Endpoint string `json:"endpoint"` - Timeout time.Duration `json:"timeout" envDefault:"3s"` - // This field is deliberately unavailable to configuration files. - Token string `json:"-" env:"DEMO_CLIENT_TOKEN,secret"` + Timeout time.Duration `json:"timeout"` + // The file chooses a fixed value or an explicit environment reference. + Token string `json:"token"` } type Config struct { Client ClientConfig `json:"client"` - MaxItems int `json:"maxItems" envDefault:"20"` + MaxItems int `json:"maxItems"` Allowed []string `json:"allowed"` } +func Default() Config { return Config{Client: ClientConfig{Timeout: 3 * time.Second}, MaxItems: 20} } + func (c Config) Validate() error { if c.Client.Timeout <= 0 { return config.Invalid("client.timeout", "must be positive") @@ -48,7 +50,7 @@ func load(path string) (Config, error) { if err != nil { return Config{}, err } - return config.LoadProject(path, Config{}, config.EnvOptions{Prefix: "DEMO_", Empty: "reject"}, loader) + return config.LoadProject(path, Default(), loader) } func run() error { path := flag.String("config", "", "external configuration file") diff --git a/examples/managedService/.goOmni/apiManifest.json b/examples/managedService/.goOmni/apiManifest.json index 319ab01..020ff8b 100644 --- a/examples/managedService/.goOmni/apiManifest.json +++ b/examples/managedService/.goOmni/apiManifest.json @@ -1,6 +1,6 @@ { "generatorConfig": true, - "templateVersion": "v0.20.0-preview", + "templateVersion": "v0.22.0-preview", "profile": { "database": "none", "storage": "none", @@ -12,7 +12,7 @@ "entry": "api/managed.api", "generated": { "docs/openapi.json": "0370fd6bd6a387e2ef21f6b42b009ce66749846b8bea3d481009a34f3af039a5", - "internal/config/loadGen.go": "5d630b894abdbbd66ddfa7d7d30f7573fc4ff995020a32a9f9157cd4af3a5655", + "internal/config/loadGen.go": "d85e3123e4c96c95cb0886e60e4760230426efca4d2bbbfc981822da08b90d13", "internal/handler/appGen.go": "f5251b4dc33e2312b0bdb71ead70d2171977d851265e124f640458d705a2cc9d", "internal/handler/demo/handlersGen.go": "3c76d85dbe89eb5127d9d134dc625203ecf9719dd11d174c1f335691b99e1375", "internal/handler/openapiGen.go": "f24cab3abbb8557a3fc4dd93fba23cc0b093f3e0c944d3ba1bd5b0eeff2d96f3", @@ -22,7 +22,7 @@ "internal/svc/runtimeGen.go": "29f6afb6614881935099c1845ecb3f7902ee773caa077940f566cb8dfdbcdbb1", "internal/types/dtoManagedd28645a2af1cGen.go": "7586e86fa5d67051283178cc578c890f6c02fd2d5c383170a71a39c2e77ce344", "internal/types/typesGen.go": "372d67f23a313b3e77d2a3b7a236315db9721bc63350bf1f71db1293e57570d1", - "main.go": "64e26d97de7e7c6c17f47186785edc73d43c23d6c20b50bfc7fff6ef2e914f17" + "main.go": "10cb6a84b2eb1b68dc10c2dca949c9dd7c68a9f78a8e6d8f56ce9336d4b2d125" }, "contract": { "routes": { diff --git a/examples/managedService/README.md b/examples/managedService/README.md index fd3bbc4..f80ac0d 100644 --- a/examples/managedService/README.md +++ b/examples/managedService/README.md @@ -46,6 +46,6 @@ Configuration, API contracts, svc fields/Init/AppOptions, Logic, Repository and middleware implementations remain developer-owned. Regeneration preserves them. Use `go-omni api format` to format local contracts, then `api generate` and `api generate --check`. See the [framework development guide](../../DEVELOPMENT.md) -for array/null responses, typed multipart, full configuration environment overrides, +for array/null responses, typed multipart, explicit configuration references and -check-config, shared contracts and v0.13 upgrade steps. This example keeps its raw callback, streaming upload and download to exercise those existing extension points. diff --git a/examples/managedService/etc/config.yaml b/examples/managedService/etc/config.yaml index 42f523b..e20cb22 100644 --- a/examples/managedService/etc/config.yaml +++ b/examples/managedService/etc/config.yaml @@ -1,16 +1,3 @@ -# Non-secret settings only. Environment variables override these values. server: - address: '127.0.0.1:8080' - enableLogger: true - slowRequestThreshold: 500ms - shutdownTimeout: 10s - endpoints: - liveness: {enabled: true, path: /health/live, responseMode: app} - readiness: {enabled: true, path: /health/ready, responseMode: app} - openapi: {enabled: true, path: /openapi.json} - log: - service: "example.com/managed" - level: info - format: json - output: stderr - redactKeys: [] + address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}" + transport: "${GO_OMNI_TRANSPORT:-nethttp}" diff --git a/examples/managedService/go.mod b/examples/managedService/go.mod index c16cbb1..269ea61 100644 --- a/examples/managedService/go.mod +++ b/examples/managedService/go.mod @@ -2,7 +2,7 @@ module example.com/managed go 1.22.0 -require git.yuntuo.hk/k2safe/go-omni v0.21.0-preview +require git.yuntuo.hk/k2safe/go-omni v0.22.0-preview require ( github.com/gabriel-vasile/mimetype v1.4.3 // indirect diff --git a/examples/managedService/internal/config/loadGen.go b/examples/managedService/internal/config/loadGen.go index eb7f776..bdcda30 100644 --- a/examples/managedService/internal/config/loadGen.go +++ b/examples/managedService/internal/config/loadGen.go @@ -3,14 +3,12 @@ package config import omniConfig "git.yuntuo.hk/k2safe/go-omni/config" -// Load is maintained by the generator. Extend Config/Default/Validate in -// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline. +// Load resolves explicit file references, binds onto Default and validates. +// Ordinary projects only edit Config, Default, Validate and the selected file. func Load(path string) (Config, error) { loader, err := NewLoader() if err != nil { return Config{}, err } - return omniConfig.LoadProject(path, Default(), omniConfig.EnvOptions{ - Prefix: "GO_OMNI_", Empty: "ignore", Aliases: omniConfig.ServerEnvAliases("GO_OMNI_", "server"), - }, loader) + return omniConfig.LoadProject(path, Default(), loader) } diff --git a/examples/managedService/main.go b/examples/managedService/main.go index c7ecd08..4c281f5 100644 --- a/examples/managedService/main.go +++ b/examples/managedService/main.go @@ -17,8 +17,15 @@ import ( ) func main() { - path := flag.String("config", "etc/config.yaml", "configuration file; empty uses defaults and environment") + path := flag.String("config", "etc/config.yaml", "configuration file; empty uses only Default()") + check := flag.Bool("check-config", false, "validate configuration and exit without starting dependencies") flag.Parse() + if *check { + if _, err := config.Load(*path); err != nil { + log.Fatal(err) + } + return + } ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() if err := run(ctx, *path); err != nil { diff --git a/examples/modules/README.md b/examples/modules/README.md index 7bec617..ff21283 100644 --- a/examples/modules/README.md +++ b/examples/modules/README.md @@ -7,7 +7,7 @@ go run ./examples/modules ``` 默认监听 `127.0.0.1:8080`，使用仅限开发的 `Bearer dev-token`。 -配置 `GO_OMNI_TRANSPORT=native` 可直接切换自研网络层，TLS 配置与主框架一致。 +默认读取 `examples/modules/config.yaml`，也可用 `-config` 选择一个其他文件。文件显式引用了 `GO_OMNI_TRANSPORT`，设置为 native 可切换自研网络层；TLS 在文件的 `tls.certFile` / `tls.keyFile` 中配置。此示例直接加载框架 Config，没有生成项目的 server 外层。 ```sh curl -H 'Authorization: Bearer dev-token' http://127.0.0.1:8080/api/catalog diff --git a/examples/modules/config.yaml b/examples/modules/config.yaml new file mode 100644 index 0000000..e10578e --- /dev/null +++ b/examples/modules/config.yaml @@ -0,0 +1,2 @@ +address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}" +transport: "${GO_OMNI_TRANSPORT:-nethttp}" diff --git a/examples/modules/main.go b/examples/modules/main.go index 6807f32..1796702 100644 --- a/examples/modules/main.go +++ b/examples/modules/main.go @@ -5,6 +5,7 @@ import ( "context" "encoding/json" "errors" + "flag" "log" "os" "os/signal" @@ -106,13 +107,11 @@ func newApp(settings omni.Config, token, upstream string) (application *omni.App } func run(ctx context.Context) error { - settings, err := omni.LoadConfig("") + settings, err := omni.LoadConfig(*configPath) if err != nil { return err } - if os.Getenv("GO_OMNI_HTTP_ADDR") == "" { - settings.Address = "127.0.0.1:8080" - } + token := os.Getenv("GO_OMNI_DEV_TOKEN") if token == "" { token = "dev-token" @@ -124,7 +123,10 @@ func run(ctx context.Context) error { return application.RunContext(ctx) } +var configPath = flag.String("config", "examples/modules/config.yaml", "configuration file") + func main() { + flag.Parse() ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() if err := run(ctx); err != nil { diff --git a/examples/modules/main_test.go b/examples/modules/main_test.go index b85c1f8..299498d 100644 --- a/examples/modules/main_test.go +++ b/examples/modules/main_test.go @@ -28,7 +28,7 @@ func TestModulesConfiguredTransports(t *testing.T) { io.WriteString(w, `{"name":"upstream catalog"}`) })) defer upstream.Close() - settings, err := omni.LoadConfig("MODULES_TEST_") + settings, err := omni.LoadConfig("") if err != nil { t.Fatal(err) } diff --git a/examples/platform/main.go b/examples/platform/main.go index f86fb51..5fe82c1 100644 --- a/examples/platform/main.go +++ b/examples/platform/main.go @@ -88,16 +88,9 @@ func newApp(cfg omni.Config, d dependencies) (*omni.App, error) { return app, nil } func run(ctx context.Context, configFile, root string) error { - var cfg config.Config - var err error - if configFile != "" { - cfg, err = config.LoadFile(configFile, "") - } else { - cfg, err = config.LoadFromEnv("") - if os.Getenv("GO_OMNI_HTTP_ADDR") == "" { - cfg.Address = "127.0.0.1:8080" - } - } + defaults := config.Default() + defaults.Address = "127.0.0.1:8080" + cfg, err := config.LoadProject(configFile, defaults) if err != nil { return err } diff --git a/examples/reference/README.md b/examples/reference/README.md index 870f979..576aae2 100644 --- a/examples/reference/README.md +++ b/examples/reference/README.md @@ -10,10 +10,10 @@ go run ./examples/reference ``` -默认监听 `127.0.0.1:8080`，可通过环境变量调整： +默认读取 `examples/reference/config.yaml`，监听 `127.0.0.1:8080`。文件已显式引用地址变量： ```sh -GO_OMNI_ADDR=127.0.0.1:8081 go run ./examples/reference +GO_OMNI_HTTP_ADDR=127.0.0.1:8081 go run ./examples/reference ``` 如果终端提示找不到 `go`，需要先把 Go 的 `bin` 目录加入 `PATH`。不要与 `examples/basic` 同时占用同一个端口。 @@ -133,14 +133,10 @@ curl -i http://127.0.0.1:8080/api/v1/users/1 - 列表按 ID 排序、返回总数，空列表使用 `[]`。 - 使用传入的 `context.Context` 执行查询，保留创建时间，修改时更新时间。 -MySQL 运行配置通过环境变量传入： +此手写示例显式读取 `GO_OMNI_DB_DSN`；驱动固定 mysql，连接池使用 database.DefaultConfig()。项目可将数据库配置放入自己的 Config 并通过 LoadProject 统一加载（见 [配置指南](../../CONFIG.md)），框架不再自动读取数据库环境变量： ```sh -export GO_OMNI_DB_DRIVER=mysql export GO_OMNI_DB_DSN='root:password@tcp(127.0.0.1:3306)/go_omni?parseTime=true&loc=UTC' -export GO_OMNI_DB_MAX_OPEN_CONNS=10 -export GO_OMNI_DB_MAX_IDLE_CONNS=5 -export GO_OMNI_DB_PING_TIMEOUT=5s go run ./examples/reference ``` diff --git a/examples/reference/config.yaml b/examples/reference/config.yaml new file mode 100644 index 0000000..3344e4a --- /dev/null +++ b/examples/reference/config.yaml @@ -0,0 +1,9 @@ +address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}" +transport: "${GO_OMNI_TRANSPORT:-nethttp}" +tls: + certFile: "${GO_OMNI_TLS_CERT_FILE:-}" + keyFile: "${GO_OMNI_TLS_KEY_FILE:-}" +log: + service: reference-api + environment: local + output: stdout diff --git a/examples/reference/main.go b/examples/reference/main.go index c386a1d..300525e 100644 --- a/examples/reference/main.go +++ b/examples/reference/main.go @@ -2,6 +2,7 @@ package main import ( "context" + "flag" "log" "os" "os/signal" @@ -42,6 +43,7 @@ func registerRoutes(application *omni.App, users service.UserRepository, logger } func main() { + flag.Parse() ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) defer stop() if err := runReference(ctx); err != nil { @@ -49,29 +51,9 @@ func main() { } } -func referenceConfig() (omni.Config, error) { - settings, err := omni.LoadConfig("") - if err != nil { - return settings, err - } - // Preserve the example's loopback default and legacy address variable. - if strings.TrimSpace(os.Getenv("GO_OMNI_HTTP_ADDR")) == "" { - settings.Address = strings.TrimSpace(os.Getenv("GO_OMNI_ADDR")) - if settings.Address == "" { - settings.Address = "127.0.0.1:8080" - } - } - if settings.Log.Service == "" { - settings.Log.Service = "reference-api" - } - if settings.Log.Environment == "" { - settings.Log.Environment = "local" - } - if strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT")) == "" { - settings.Log.Output = "stdout" - } - return settings, nil -} +var configPath = flag.String("config", "examples/reference/config.yaml", "configuration file") + +func referenceConfig() (omni.Config, error) { return omni.LoadConfig(*configPath) } func runReference(ctx context.Context) error { settings, err := referenceConfig() @@ -106,10 +88,9 @@ func openUsers(ctx context.Context, logger *logx.Logger) (service.UserRepository if strings.TrimSpace(os.Getenv("GO_OMNI_DB_DSN")) == "" { return repository.NewMemoryUsers(), func() error { return nil }, "memory", nil } - cfg, err := database.ConfigFromEnv("GO_OMNI_DB_") - if err != nil { - return nil, nil, "", err - } + cfg := database.DefaultConfig() + cfg.Driver = "mysql" + cfg.DSN = os.Getenv("GO_OMNI_DB_DSN") db, err := database.Open(ctx, cfg) if err != nil { return nil, nil, "", err diff --git a/examples/reference/transport_test.go b/examples/reference/transport_test.go index 5b17889..94225cb 100644 --- a/examples/reference/transport_test.go +++ b/examples/reference/transport_test.go @@ -7,6 +7,8 @@ import ( "io" "net" "net/http" + "os" + "path/filepath" "strings" "sync" "testing" @@ -99,7 +101,11 @@ func TestBusinessConfiguredTransports(t *testing.T) { t.Setenv("REFERENCE_TEST_TLS_CERT_FILE", cert) t.Setenv("REFERENCE_TEST_TLS_KEY_FILE", key) } - settings, err := omni.LoadConfig("REFERENCE_TEST_") + path := filepath.Join(t.TempDir(), "config.yaml") + if err := os.WriteFile(path, []byte("transport: ${REFERENCE_TEST_TRANSPORT}\nshutdownTimeout: 3s\ntls:\n certFile: \"${REFERENCE_TEST_TLS_CERT_FILE:-}\"\n keyFile: \"${REFERENCE_TEST_TLS_KEY_FILE:-}\""), 0600); err != nil { + t.Fatal(err) + } + settings, err := omni.LoadConfig(path) if err != nil { t.Fatal(err) } @@ -159,24 +165,26 @@ func TestBusinessConfiguredTransports(t *testing.T) { } } -func TestReferenceAddressCompatibility(t *testing.T) { +func TestReferenceExplicitConfig(t *testing.T) { + previous := *configPath + *configPath = "config.yaml" + defer func() { *configPath = previous }() t.Setenv("GO_OMNI_TRANSPORT", "native") + t.Setenv("GO_OMNI_HTTP_ADDR", "") t.Setenv("GO_OMNI_TLS_CERT_FILE", "") t.Setenv("GO_OMNI_TLS_KEY_FILE", "") - t.Setenv("GO_OMNI_HTTP_ADDR", "") - t.Setenv("GO_OMNI_ADDR", "") - settings, err := referenceConfig() - if err != nil || settings.Address != "127.0.0.1:8080" || settings.Transport != "native" { - t.Fatalf("default: %+v %v", settings, err) + c, err := referenceConfig() + if err != nil || c.Address != "127.0.0.1:8080" || c.Transport != "native" { + t.Fatal(c, err) } - t.Setenv("GO_OMNI_ADDR", "127.0.0.1:8081") - settings, err = referenceConfig() - if err != nil || settings.Address != "127.0.0.1:8081" { - t.Fatalf("legacy: %+v %v", settings, err) + t.Setenv("GO_OMNI_ADDR", "127.0.0.1:1234") + c, err = referenceConfig() + if err != nil || c.Address != "127.0.0.1:8080" { + t.Fatal("implicit alias", err) } t.Setenv("GO_OMNI_HTTP_ADDR", "127.0.0.1:8082") - settings, err = referenceConfig() - if err != nil || settings.Address != "127.0.0.1:8082" { - t.Fatalf("standard: %+v %v", settings, err) + c, err = referenceConfig() + if err != nil || c.Address != "127.0.0.1:8082" { + t.Fatal(c, err) } } diff --git a/examples/userService/README.md b/examples/userService/README.md index 0b98a30..d1c07d9 100644 --- a/examples/userService/README.md +++ b/examples/userService/README.md @@ -18,13 +18,12 @@ go-omni migrate --dir migrations --apply go run ./cmd/server ``` -源码示例的 go.mod 使用 `replace ... => ../..`。生成代码使用 `v0.8.0-preview` 响应策略入口。 +源码示例的 go.mod 使用 `replace ... => ../..`。生成代码使用 `v0.22.0-preview` 显式配置加载入口。 复制到独立仓库时删除 replace，保留同版本固定依赖，再执行 go mod tidy；CLI 也固定到同一版本。 远程版本验证脚本会检查没有本地替换。 `GO_OMNI_TRANSPORT=native` 可切换为 native，业务代码无需调整。 -配置是默认值 → etc/config.yaml → 环境变量。日志默认写入 var/log/server.log 并轮转， -`GO_OMNI_LOG_OUTPUT=stderr` 可以切换为标准错误输出。MySQL 默认最多 10 条连接。 +配置是 Default() → etc/config.yaml 的显式引用 → 类型绑定与校验。文件里的固定值不会被环境自动覆盖。日志默认写入 var/log/server.log 并轮转；要改用 stderr，在文件明确设置 log.file 为空、log.rotation.enabled 为 false、log.output 为 stderr。MySQL 默认最多 10 条连接。 应用检查 users 和 user_audit 表，缺失时拒绝启动，不自动改表。 ## 接口 diff --git a/examples/userService/etc/config.yaml b/examples/userService/etc/config.yaml index cb45f01..79eca66 100644 --- a/examples/userService/etc/config.yaml +++ b/examples/userService/etc/config.yaml @@ -1,6 +1,7 @@ -# Non-secret settings only. Environment variables override these values. +# Defaults are overridden only by fields in this file. server: - address: '127.0.0.1:8080' + transport: "${GO_OMNI_TRANSPORT:-nethttp}" + address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}" enableLogger: true slowRequestThreshold: 500ms shutdownTimeout: 10s @@ -17,16 +18,19 @@ server: compress: true redactKeys: [] storage: + filesToken: "${GO_OMNI_FILES_TOKEN:-}" + signingKey: "${GO_OMNI_STORAGE_SIGNING_KEY}" provider: local maxBytes: 33554432 contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf'] linkTtl: 5m requestTimeout: 30s root: './var/files' - publicUrl: 'http://127.0.0.1:8080/storage/object' + publicUrl: "${GO_OMNI_STORAGE_PUBLIC_URL:-http://127.0.0.1:8080/storage/object}" database: + dsn: "${GO_OMNI_DB_DSN}" driver: mysql maxOpenConns: 10 maxIdleConns: 5 diff --git a/examples/userService/go.mod b/examples/userService/go.mod index b9f37c6..40e9a71 100644 --- a/examples/userService/go.mod +++ b/examples/userService/go.mod @@ -3,7 +3,7 @@ module example.com/userService go 1.22.0 require ( - git.yuntuo.hk/k2safe/go-omni v0.8.0-preview + git.yuntuo.hk/k2safe/go-omni v0.22.0-preview github.com/go-sql-driver/mysql v1.9.3 ) diff --git a/examples/userService/internal/repository/user/transaction_test.go b/examples/userService/internal/repository/user/transaction_test.go index a8846ec..f324220 100644 --- a/examples/userService/internal/repository/user/transaction_test.go +++ b/examples/userService/internal/repository/user/transaction_test.go @@ -15,10 +15,9 @@ func TestTransactions(t *testing.T) { if os.Getenv("USER_SERVICE_TEST_DATABASE") != "1" { t.Skip("isolated MySQL required") } - cfg, err := database.ConfigFromEnv("") - if err != nil { - t.Fatal(err) - } + cfg := database.DefaultConfig() + cfg.Driver = "mysql" + cfg.DSN = os.Getenv("GO_OMNI_DB_DSN") db, err := database.Open(context.Background(), cfg) if err != nil { t.Fatal(err) diff --git a/examples/userService/internal/settings/config.go b/examples/userService/internal/settings/config.go index 12d4107..6e38133 100644 --- a/examples/userService/internal/settings/config.go +++ b/examples/userService/internal/settings/config.go @@ -3,14 +3,11 @@ package settings import ( "errors" - "fmt" "git.yuntuo.hk/k2safe/go-omni/config" "git.yuntuo.hk/k2safe/go-omni/database" "git.yuntuo.hk/k2safe/go-omni/logx" "git.yuntuo.hk/k2safe/go-omni/storage" "net/url" - "os" - "strconv" "strings" "time" ) @@ -38,40 +35,7 @@ func Default() Config { }} } -// Load uses defaults -> strict YAML/JSON -> nonempty environment variables. -// Pass an empty path to use only defaults and environment. -func Load(path string) (Config, error) { - cfg := Default() - var err error - if path != "" { - cfg, err = config.DecodeFile(path, cfg) - if err != nil { - return Config{}, err - } - } - cfg.Server, err = config.ApplyEnv(cfg.Server, "") - if err != nil { - return Config{}, err - } - // A stream explicitly selected by environment overrides a file profile. - if output := strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT")); output != "" && strings.TrimSpace(os.Getenv("GO_OMNI_LOG_FILE")) == "" { - cfg.Server.Log.File = "" - cfg.Server.Log.Rotation = logx.RotationConfig{} - } - - if err = cfg.Storage.applyEnv(); err != nil { - return Config{}, err - } - - cfg.Database, err = database.ApplyEnv(cfg.Database, "") - if err != nil { - return Config{}, err - } - if err = cfg.Validate(); err != nil { - return Config{}, err - } - return cfg, nil -} +func Load(path string) (Config, error) { return config.LoadProject(path, Default()) } func (c Config) Validate() error { if err := c.Database.Validate(); err != nil { @@ -108,62 +72,17 @@ type StorageConfig struct { ContentTypes []string `json:"contentTypes"` LinkTTL time.Duration `json:"linkTtl"` RequestTimeout time.Duration `json:"requestTimeout"` - // Secrets are environment-only and cannot be populated by config files. - FilesToken string `json:"-"` - SigningKey string `json:"-"` - AccessKey string `json:"-"` - SecretKey string `json:"-"` - SessionToken string `json:"-"` + // The file explicitly selects secret environment references. + FilesToken string `json:"filesToken"` + SigningKey string `json:"signingKey"` + AccessKey string `json:"accessKey"` + SecretKey string `json:"secretKey"` + SessionToken string `json:"sessionToken"` } func (s StorageConfig) Policy() storage.UploadPolicy { return storage.UploadPolicy{MaxBytes: s.MaxBytes, ContentTypes: s.ContentTypes} } -func (s *StorageConfig) applyEnv() error { - values := map[string]*string{ - "GO_OMNI_STORAGE_ROOT": &s.Root, "GO_OMNI_STORAGE_PUBLIC_URL": &s.PublicURL, - "GO_OMNI_STORAGE_ENDPOINT": &s.Endpoint, "GO_OMNI_STORAGE_REGION": &s.Region, - "GO_OMNI_STORAGE_BUCKET": &s.Bucket, "GO_OMNI_STORAGE_ACCOUNT_ID": &s.AccountID, - "GO_OMNI_STORAGE_SIGNING_KEY": &s.SigningKey, "GO_OMNI_FILES_TOKEN": &s.FilesToken, - "GO_OMNI_STORAGE_ACCESS_KEY": &s.AccessKey, "GO_OMNI_STORAGE_SECRET_KEY": &s.SecretKey, - "GO_OMNI_STORAGE_SESSION_TOKEN": &s.SessionToken, - } - for name, target := range values { - if value := strings.TrimSpace(os.Getenv(name)); value != "" { - *target = value - } - } - for name, target := range map[string]*time.Duration{"GO_OMNI_STORAGE_LINK_TTL": &s.LinkTTL, "GO_OMNI_STORAGE_REQUEST_TIMEOUT": &s.RequestTimeout} { - if value := strings.TrimSpace(os.Getenv(name)); value != "" { - v, err := time.ParseDuration(value) - if err != nil { - return fmt.Errorf("%s must be a duration", name) - } - *target = v - } - } - if value := os.Getenv("GO_OMNI_STORAGE_MAX_BYTES"); value != "" { - v, err := strconv.ParseInt(value, 10, 64) - if err != nil { - return errors.New("GO_OMNI_STORAGE_MAX_BYTES must be an integer") - } - s.MaxBytes = v - } - if value := os.Getenv("GO_OMNI_STORAGE_SECURE"); value != "" { - v, err := strconv.ParseBool(value) - if err != nil { - return errors.New("GO_OMNI_STORAGE_SECURE must be a boolean") - } - s.Secure = v - } - if value := strings.TrimSpace(os.Getenv("GO_OMNI_STORAGE_CONTENT_TYPES")); value != "" { - s.ContentTypes = strings.Split(value, ",") - for i := range s.ContentTypes { - s.ContentTypes[i] = strings.TrimSpace(s.ContentTypes[i]) - } - } - return nil -} func (s StorageConfig) Validate() error { if s.Provider != "local" { return errors.New("storage.provider must match the adapter selected at generation") @@ -186,7 +105,7 @@ func (s StorageConfig) Validate() error { return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object") } if len(s.SigningKey) < 32 { - return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes") + return errors.New("storage.signingKey requires a stable secret of at least 32 bytes") } return nil diff --git a/httpc/README.md b/httpc/README.md index 796da96..a6df538 100644 --- a/httpc/README.md +++ b/httpc/README.md @@ -116,7 +116,7 @@ outbound: maxResponseBytes: 1048576 ``` -现有统一加载入口可读取 JSON/YAML/XML，并自动覆盖 `GO_OMNI_OUTBOUND_PROXY_MODE`、`GO_OMNI_OUTBOUND_DIAL_TIMEOUT` 等环境变量。无需为每个参数手写环境解析。详见 [统一配置](../CONFIG.md)。 +统一加载入口读取 JSON/YAML/XML。需要环境变量时在文件明确写 `mode: "${HTTP_PROXY_MODE:-direct}"`、`dialTimeout: "${HTTP_DIAL_TIMEOUT:-500ms}"`；固定值不会被环境自动覆盖。无需手写解析。详见 [统一配置](../CONFIG.md)。 ## 标准请求、特殊 TLS 与生命周期 diff --git a/omni.go b/omni.go index 19555f5..263a914 100644 --- a/omni.go +++ b/omni.go @@ -76,7 +76,7 @@ func WithLogConfig(settings LogConfig) Option { return config.WithLogConfig(sett func WithShutdownTimeout(timeout time.Duration) Option { return config.WithShutdownTimeout(timeout) } -func LoadConfig(prefix string) (Config, error) { return config.LoadFromEnv(prefix) } +func LoadConfig(path string) (Config, error) { return config.LoadFile(path) } func SetupLogger(cfg LogConfig) (func() error, error) { return logx.Setup(cfg) } diff --git a/reports/explicitConfigurationAcceptance.md b/reports/explicitConfigurationAcceptance.md new file mode 100644 index 0000000..96f5154 --- /dev/null +++ b/reports/explicitConfigurationAcceptance.md @@ -0,0 +1,53 @@ +# v0.22 显式配置引用验收（2026-09-26） + +配置文件明确决定值的来源。框架只处理通用加载、类型绑定和安全诊断，不包含接入项目的字段、环境前缀或业务规则。 + +## 回归结果 + +- `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race ./...`：48 个有测试的包通过，包含实际生成项目、错误目录、生命周期失败清理、上传下载、后台收尾和 copyx 回归。 +- 生成模板最后调整后，CLI、apigen、scaffold 的实际消费者 race 复测通过。 +- `go vet ./...`、`go build ./...` 通过。 +- `GOTOOLCHAIN=go1.22.12 go test ./config ./database ./examples/configuration` 通过。 +- managedService 的生成检查、race 与根目录构建通过；userService 的 race 通过。外部 MySQL 测试未配置时按原规则跳过。 +- 三种格式示例分别运行默认值与显式 0/含特殊字符凭据；结果一致。类型错误非零退出，指出字段和环境名，未输出输入值。 + +| 场景 | 验证 | +| --- | --- | +| 固定值与 Default() | 不读取同名环境变量；文件缺失保留默认，显式 0/false/空字符串覆盖 | +| `${NAME}` / `${NAME:-fallback}` | 未设置与空值区分规则明确；必填失败不退回 Default，0/false/空格属于有效值 | +| 特殊字符与转义 | 冒号、`:-`、引号、换行、XML 字符、`${OTHER}` 原样保留；`$$` 转义，无二次展开 | +| YAML/JSON/XML/自定义 Decoder | 解码标量后解析引用，不替换文件源文本；同一绑定/验证流程 | +| 类型、null 与集合 | 按目标类型绑定，保留 int64 精度；不把环境文本隐式解析为对象、集合或 null | +| 失败诊断 | 文件、字段、环境名、可获取行列；类型/溢出/Validate 错误不输出值；相近未知字段给出建议 | +| 并发和默认值隔离 | 每次加载独立缓存，每个引用变量最多读取一次；24 个并发加载 race 通过 | +| 资源限制 | 文件和单值上限、总展开预算、深度/节点/引用上限、非法 UTF-8、恶意格式拒绝 | +| 生成维护边界 | 重复生成、dry-run/check 结果一致；业务配置和实现保留；main/loadGen 漂移拒绝 | +| `-check-config` | 使用同一 Load，成功退出 0；失败非零；不初始化依赖、不创建日志文件、不监听 | +| 单文件选择 | `-config` 选择一份，邻接 local/test 文件不自动叠加；空路径仅 Default | +| 删除旧模式 | env/envDefault 检出即提示迁移；不保留前缀、别名或隐式二次覆盖 | + +`FuzzConfigurationReferences` 使用 4 worker、30 秒：284783 次执行，通过，总运行 31.445 秒（包含收尾）。另有确定性超限与格式安全测试；执行次数不代表形式化证明。 + +## 性能 + +Go 1.27.1、darwin/arm64、Apple M2 Max；同一实现、无 race、每场景三轮，benchtime=500ms。原始结果：[explicitConfigurationBenchmark.txt](explicitConfigurationBenchmark.txt)。 + +| 场景 | 中位数 | 典型分配 | +| --- | ---: | ---: | +| 小配置固定值：读文件＋解析＋绑定＋校验 | 24.932 µs | 10.24 KB / 123 allocs | +| 相同小配置带显式环境引用 | 25.420 µs | 11.12 KB / 133 allocs | +| 约 28 KB、1000 条目 JSON 解码＋绑定 | 2.390 ms | 2.40 MB / 31582 allocs | + +小配置引用增加约 0.49 µs / 0.88 KB / 10 次分配；测量有波动，不承诺固定比例。配置只在启动/显式重新加载时执行，不进入业务请求路径。这是本机暖文件缓存测试，不是线上并发、冷磁盘或请求 QPS 测量。 + +## 迁移与范围 + +- 本版有意移除旧自动环境覆盖、标签、前缀/别名及数据库环境加载 API；完整清单见 [CONFIG.md](../CONFIG.md#v022-迁移旧模式已移除)。 +- 配置文件里的其他 `${...}` 模板改写 `$${...}`；`.api` 响应模板不变。固定文件值不再被进程环境偷偷覆盖。 +- XML 占位符使用普通叶元素或 type=string；显式 number/bool 表示文件常量。原有解析器的配置子集和安全限制保留。 +- Decoder、Validate、自定义反序列化是可信 Go 扩展，项目负责其副作用与公开原因脱敏。 +- 本轮没有修改接入项目业务代码，没有使用线上服务器压测，也没有连接业务数据库。框架回归不能代替项目升级后的实际部署配置和业务验收。 + +## 远程验收 + +发布后使用空 module 缓存安装固定版本，验证无本地 replace 的生成、编译、退出、配置及消费者回归。具体版本、校验和、检查清单与源码哈希写入远程验收记录。 diff --git a/reports/explicitConfigurationBenchmark.txt b/reports/explicitConfigurationBenchmark.txt new file mode 100644 index 0000000..fe02a19 --- /dev/null +++ b/reports/explicitConfigurationBenchmark.txt @@ -0,0 +1,15 @@ +goos: darwin +goarch: arm64 +pkg: git.yuntuo.hk/k2safe/go-omni/config +cpu: Apple M2 Max +BenchmarkConfigurationLarge-12 246 2408597 ns/op 11.63 MB/s 2403405 B/op 31582 allocs/op +BenchmarkConfigurationLarge-12 249 2368666 ns/op 11.83 MB/s 2403394 B/op 31582 allocs/op +BenchmarkConfigurationLarge-12 249 2389703 ns/op 11.72 MB/s 2403361 B/op 31582 allocs/op +BenchmarkExplicitConfiguration/false-12 23610 24932 ns/op 10234 B/op 123 allocs/op +BenchmarkExplicitConfiguration/false-12 23541 25349 ns/op 10237 B/op 123 allocs/op +BenchmarkExplicitConfiguration/false-12 24046 24805 ns/op 10237 B/op 123 allocs/op +BenchmarkExplicitConfiguration/true-12 23520 25420 ns/op 11119 B/op 133 allocs/op +BenchmarkExplicitConfiguration/true-12 23612 24984 ns/op 11118 B/op 133 allocs/op +BenchmarkExplicitConfiguration/true-12 23516 25432 ns/op 11116 B/op 133 allocs/op +PASS +ok git.yuntuo.hk/k2safe/go-omni/config 8.469s diff --git a/response/codeTypes_test.go b/response/codeTypes_test.go index e28243f..7875d76 100644 --- a/response/codeTypes_test.go +++ b/response/codeTypes_test.go @@ -101,7 +101,7 @@ func TestInt64CodesFromConfigurationKeepPrecision(t *testing.T) { if err := os.WriteFile(path, []byte(body), 0600); err != nil { t.Fatal(err) } - cfg, err := config.LoadFile(path, "RESPONSE_TEST_") + cfg, err := config.LoadFile(path) if err != nil { t.Fatalf("%s: %v", ext, err) } diff --git a/response/config_test.go b/response/config_test.go index 1fcb933..638e5e0 100644 --- a/response/config_test.go +++ b/response/config_test.go @@ -52,14 +52,14 @@ func TestResponseConfigLoadsFromJSONAndYAML(t *testing.T) { for filename, body := range map[string]string{ "flat.json": `{"response":{"codeType":"integer","successCode":0,"errorCode":9999,"messageField":"msg","requestIdField":"-","dataMode":"always","codes":{"DENIED":403001}}}`, "flat.yaml": "response:\n codeType: integer\n successCode: 0\n errorCode: 9999\n messageField: msg\n requestIdField: '-'\n dataMode: always\n codes:\n DENIED: 403001\n", - "body.json": `{"response":{"body":{"code":"${code}","msg":"${message}","data":"${data}"},"codeType":"integer","successCode":0,"errorCode":9999,"dataMode":"always","codes":{"DENIED":403001}}}`, - "body.yaml": "response:\n body:\n code: '${code}'\n msg: '${message}'\n data: '${data}'\n codeType: integer\n successCode: 0\n errorCode: 9999\n dataMode: always\n codes:\n DENIED: 403001\n", + "body.json": `{"response":{"body":{"code":"$${code}","msg":"$${message}","data":"$${data}"},"codeType":"integer","successCode":0,"errorCode":9999,"dataMode":"always","codes":{"DENIED":403001}}}`, + "body.yaml": "response:\n body:\n code: '$${code}'\n msg: '$${message}'\n data: '$${data}'\n codeType: integer\n successCode: 0\n errorCode: 9999\n dataMode: always\n codes:\n DENIED: 403001\n", } { path := filepath.Join(t.TempDir(), filename) if err := os.WriteFile(path, []byte(body), 0600); err != nil { t.Fatal(err) } - cfg, err := config.LoadFile(path, "RESPONSE_TEST_") + cfg, err := config.LoadFile(path) if err != nil { t.Fatal(err) } diff --git a/scripts/verifyManagedRelease.py b/scripts/verifyManagedRelease.py index f8de10f..d597605 100644 --- a/scripts/verifyManagedRelease.py +++ b/scripts/verifyManagedRelease.py @@ -146,6 +146,16 @@ def main(): checks += ["remoteHTTPClientRaceTests", "remoteHTTPProxyAndTimeouts", "remoteHTTPHeaderIsolation", "remoteHTTPPoolAndDrainBudgets", "remoteHTTPExample"] + config_output = run(["go", "test", "-race", "-count=1", "-json", MODULE + "/config"], starter, env) + config_events = [json.loads(line) for line in config_output.splitlines() if line.startswith("{")] + config_passed = {e.get("Test") for e in config_events if e.get("Action") == "pass"} + assert {"TestExplicitReferencesAcrossFormats", "TestReferenceFallbackAndFixedValues", + "TestReferenceDiagnosticsAndScalarLimits", "TestRemovedEnvTagsAndFieldSuggestion"} <= config_passed + run([cli, "api", "generate", "--check"], starter, env) + run(["go", "run", ".", "-config", "etc/config.yaml", "-check-config"], starter, env) + checks += ["remoteExplicitConfigurationRaceTests", "literalAndRequiredDefaultReferences", + "remoteCheckConfigBeforeDependencies", "removedImplicitEnvironmentRules"] + checks += ["flatFieldComposition", "copyxThreeEndpointConsumer", "explicitNullAndMoneyPolicies", "fieldAdditionWithoutCopyRuleChanges", "middlewareFactoryInjectionAndCleanup", "optionalRepositoryPreservesBusiness"] checks += ["generatedUnifiedConfig", "jsonYamlXmlAndCustomDecoder", "configValidationBeforeDependencies", "configEnvProvenance", "configExtensionPreserved", "configLoaderDriftCheck"] @@ -153,7 +163,7 @@ def main(): sources = ["lifecycle/runtime.go", "lifecycle/phases.go", "config/lifecycle_test.go", "task/task.go", "cmd/go-omni/scaffold/templates/api-runtime.go.tmpl", "cmd/go-omni/apigen/templates.go", "config/project.go", "contract/bind.go", "contract/policy.go", "contract/multipart.go", "contract/file.go", "cmd/go-omni/apigen/business_test.go", "cmd/go-omni/apigen/generate.go", "lifecycle/controller.go", "config/endpoints.go", "health/health.go", "httpx/response.go", "app/app.go", "app/transport.go", "scripts/verifyManagedRelease.py", "scripts/verifyManagedProject.py", "cmd/go-omni/apigen/managed_test.go"] sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/managed").glob("*.go")] sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/business").glob("*.go")] - sources += ["cmd/go-omni/apigen/responseShape.go", "cmd/go-omni/apigen/documentRefs.go", "cmd/go-omni/apigen/catalogDocument.go", "cmd/go-omni/apigen/workspaceCatalog.go", "config/env.go", "cmd/go-omni/apigen/arrayResponse_test.go"] + sources += ["cmd/go-omni/apigen/responseShape.go", "cmd/go-omni/apigen/documentRefs.go", "cmd/go-omni/apigen/catalogDocument.go", "cmd/go-omni/apigen/workspaceCatalog.go", "config/references.go", "cmd/go-omni/apigen/arrayResponse_test.go"] sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/arrays").glob("*.go")] sources += ["cmd/go-omni/apigen/" + name for name in ("composition.go", "generatorConfig.go", "copyConsumer_test.go", "generatorConfig_test.go", "middlewareFactory.go")] sources += [str(p.relative_to(ROOT)) for p in (ROOT / "cmd/go-omni/apigen/testdata/composition").iterdir() if p.is_file()] diff --git a/transport/README.md b/transport/README.md index 9dbc27f..a2bfc33 100644 --- a/transport/README.md +++ b/transport/README.md @@ -150,6 +150,6 @@ return application.RunContext(ctx, ":8080") I/O 超时不会强制终止 CPU 计算或数据库调用；业务和下游调用继续通过 `context` 控制时限。native 需要流式输出时显式 `Flush()`，否则保持原有缓冲响应方式。不要依赖不同网络层在慢请求时返回完全相同的 HTTP 错误体；有些超时会直接断开连接。 -YAML/JSON 字段与上述字段对应的 lowerCamelCase 一致，位于 `config.Config` 的 `http` 对象。脚手架将框架配置包在 `server` 下，因此生成项目填写 `server.http`；直接用 `config.LoadFile` 加载框架配置时填写根级 `http`。环境变量为 `GO_OMNI_HTTP_READ_HEADER_TIMEOUT`、`GO_OMNI_HTTP_READ_TIMEOUT`、`GO_OMNI_HTTP_WRITE_TIMEOUT`、`GO_OMNI_HTTP_IDLE_TIMEOUT`（如 `30s`），以及 `GO_OMNI_HTTP_MAX_HEADER_BYTES`（整数）。`omni.New` 不自动加载环境；先 `config.LoadFile` / `config.ApplyEnv`，再 `WithConfig`。Go-only 的 `ErrorLog` 在文件加载之后设置。 +YAML/JSON 字段与上述字段对应的 lowerCamelCase 一致，位于 `config.Config` 的 `http` 对象。脚手架将框架配置包在 `server` 下，因此生成项目填写 `server.http`；直接用 `config.LoadFile` 加载框架配置时填写根级 `http`。环境变量由文件显式声明，例如 `readTimeout: "${HTTP_READ_TIMEOUT:-30s}"`，不再按字段自动覆盖。`omni.New` 不自动读取文件；先 `config.LoadFile` / `config.LoadProject`，再 `WithConfig`。Go-only 的 `ErrorLog` 在文件加载之后设置。 `NewServer` 始终返回普通 `*http.Server`，应用 HTTP 字段但不负责加载 TLS、监听、退出或资源清理；非法 HTTP 值会 panic。`RunContext` / `ServeContext` 的非法配置则作为 error 返回并关闭 App 资源。手动修改返回服务器的字段仍然有效。项目自己的 Runner 通过 `RunWith` 接入时，配置由该 Runner 持有，这组 HTTP 配置不会覆盖它。