Compare commits

...

4 Commits

112 changed files with 3281 additions and 1890 deletions

View File

@@ -11,7 +11,7 @@
| `openapi` | 路由与 OpenAPI 3.0.3 同时注册、DTO schema、响应信封 | 显式声明请求/响应;不自动推断任意 Handler 行为,不附带 Swagger UI | | `openapi` | 路由与 OpenAPI 3.0.3 同时注册、DTO schema、响应信封 | 显式声明请求/响应;不自动推断任意 Handler 行为,不附带 Swagger UI |
| `auth` | RS256/EdDSA JWT 验证与公钥轮换、可撤销 Session、Policy 接口 | 登录验证和业务授权规则由接入方提供;不自动获取 token 指定的密钥 | | `auth` | RS256/EdDSA JWT 验证与公钥轮换、可撤销 Session、Policy 接口 | 登录验证和业务授权规则由接入方提供;不自动获取 token 指定的密钥 |
| `telemetry` | HTTP 入站/出站、Redis、SQL spans,HTTP 指标和 traceId 日志关联 | 注入 OTel exporter/reader 后才外发;SQL Query span 不覆盖后续遍历;不接管全局 provider | | `telemetry` | HTTP 入站/出站、Redis、SQL spans,HTTP 指标和 traceId 日志关联 | 注入 OTel exporter/reader 后才外发;SQL Query span 不覆盖后续遍历;不接管全局 provider |
| `config` / `logx` | JSON/YAML 文件、环境变量覆盖、动态日志级别、轮转/脱敏/堆栈/慢请求 | 热更新只开放日志级别;连接池、监听、认证等静态变更要求重启 | | `config` / `logx` | JSON/YAML/XML 文件、显式环境引用、动态日志级别、轮转/脱敏/堆栈/慢请求 | 热更新只开放日志级别;连接池、监听、认证等静态变更要求重启 |
| `resilience` | 失败阈值、熔断冷却、单探针恢复、上游并发隔离 | 一个实例对应一个依赖;默认内存状态,不做跨进程全局熔断 | | `resilience` | 失败阈值、熔断冷却、单探针恢复、上游并发隔离 | 一个实例对应一个依赖;默认内存状态,不做跨进程全局熔断 |
| `middleware/redis` | 跨副本限流、幂等 claim/回放、容量限制 | 专用 Redis namespace;TTL 窗口内保护不等于永久业务去重 | | `middleware/redis` | 跨副本限流、幂等 claim/回放、容量限制 | 专用 Redis namespace;TTL 窗口内保护不等于永久业务去重 |
| `queue` / `queue/redis` | 持久消息接口、Streams 接管、有限重试、失败消息管理 | 至少一次交付,处理器必须幂等;持久性取决于 Redis 部署 | | `queue` / `queue/redis` | 持久消息接口、Streams 接管、有限重试、失败消息管理 | 至少一次交付,处理器必须幂等;持久性取决于 Redis 部署 |
@@ -102,7 +102,7 @@ err = consumer.Run(serviceContext, func(ctx context.Context, item queue.Delivery
## 配置与日志热更新 ## 配置与日志热更新
`config.LoadFile(path, prefix)` 按默认值→文件→非空环境变量加载并验证。两种文件格式都使用 JSON tag 的 lowerCamelCase,duration 可以写 `"5s"`;文件上限 1 MiB,拒绝未知字段、多文档和非法值。自定义 prefix 要包含期望的尾部下划线,例如 `BILLING_`。空环境变量沿用文件值,不清空已配置字段。 `config.LoadFile(path)` 使用默认值→文件字段显式引用→类型绑定→校验;没有自动环境覆盖。用 `${NAME}` 或 `${NAME:-fallback}` 指定来源,完整规则与迁移见 [CONFIG.md](CONFIG.md)。
```yaml ```yaml
address: '127.0.0.1:8080' address: '127.0.0.1:8080'
@@ -113,7 +113,7 @@ log:
format: json format: json
``` ```
先按配置创建 `logx.New(cfg.Log)`,再 `config.NewReloader(path, prefix, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`;后者阻塞,生命周期由应用负责。只有 `log.level` 允许变化,校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。 先按配置创建 `logx.New(cfg.Log)`,再 `config.NewReloader(path, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`;后者阻塞,生命周期由应用负责。只有 `log.level` 允许变化,校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。
## 熔断与观测 ## 熔断与观测

View File

@@ -6,3 +6,5 @@
- examples 使用通用场景,避免依赖业务项目、私有数据库或线上服务。必要的外部基础设施配置应明确说明。 - examples 使用通用场景,避免依赖业务项目、私有数据库或线上服务。必要的外部基础设施配置应明确说明。
- 2026-09-25 用户决定:移除 mappings/mappingFiles 及结构映射代码生成机制,改用可选 copyx 工具包。不得恢复另一套 JSON 或 Go 映射声明语言。 - 2026-09-25 用户决定:移除 mappings/mappingFiles 及结构映射代码生成机制,改用可选 copyx 工具包。不得恢复另一套 JSON 或 Go 映射声明语言。
- 深拷贝的自定义转换结果不得与输入共享可变引用;发现共享应报错,不能降级成浅拷贝或静默接受。 - 深拷贝的自定义转换结果不得与输入共享可变引用;发现共享应报错,不能降级成浅拷贝或静默接受。
- 2026-09-26 用户决定:配置文件显式声明固定值或 `${NAME}` / `${NAME:-fallback}`,旧自动环境覆盖、前缀/别名及 env/envDefault 标签模式移除,不得重新引入隐式二次覆盖。项目维护 Config/Default/Validate/文件,生成器维护 Load/main。

4
API.md
View File

@@ -48,7 +48,7 @@ go run ./cmd/go-omni new --api examples/contracts/user.api \
--log-output file --log-format json ../userService --log-output file --log-format json ../userService
``` ```
`api generate` 在新目录中也会生成完整项目。默认只监听 localhost;`GO_OMNI_HTTP_ADDR` 修改地址,`GO_OMNI_TRANSPORT=native` 切换网络层。配置顺序为默认值 → `etc/config.yaml` → 环境变量。文件日志支持现有轮转配置。远程试用使用与 CLI 一致的固定版本,见 [发布说明](RELEASE.md)。 `api generate` 在新目录中也会生成完整项目。默认只监听 localhost;`GO_OMNI_HTTP_ADDR` 修改地址,`GO_OMNI_TRANSPORT=native` 切换网络层。配置顺序为 Default() → `etc/config.yaml` 中的显式环境引用 → 类型绑定 → Validate();固定值不再被隐式覆盖。文件日志支持现有轮转配置。远程试用使用与 CLI 一致的固定版本,见 [发布说明](RELEASE.md)。
API 项目现在支持组合数据库、存储和日志: API 项目现在支持组合数据库、存储和日志:
@@ -63,7 +63,7 @@ go-omni new --api user.api --module example.com/userService \
初次选项记录在 `.goOmni/apiManifest.json`,以后直接 `api generate` 会继承。已有项目通过参数切换选项会报错,因为初始化代码属于业务方;可在新目录生成对照后手动迁移。`--force` 不允许覆盖 API 项目。new 与 api new 统一生成契约项目,不再使用旧式 CRUD 目录。 初次选项记录在 `.goOmni/apiManifest.json`,以后直接 `api generate` 会继承。已有项目通过参数切换选项会报错,因为初始化代码属于业务方;可在新目录生成对照后手动迁移。`--force` 不允许覆盖 API 项目。new 与 api new 统一生成契约项目,不再使用旧式 CRUD 目录。
MySQL 使用官方 `database/mysql` 结构化适配器,密码从 `GO_OMNI_DB_PASSWORD` 覆盖,非秘密池配置位于 YAML,环境变量优先。存储使用原有 `GO_OMNI_STORAGE_*`;API 项目不会额外生成公开的签名发放接口,鉴权和对象权限由业务实现。初始化失败会清理已打开资源,退出清理可重复调用。 MySQL 使用官方 `database/mysql` 结构化适配器,生成文件以 `password: "${GO_OMNI_DB_PASSWORD}"` 显式读取密码;其他字段可写固定值或引用,缺失项保留 Default()。存储配置也只读取文件明确引用的变量;API 项目不会额外生成公开的签名发放接口,鉴权和对象权限由业务实现。初始化失败会清理已打开资源,退出清理可重复调用。
HTTP 请求类型不能推导数据库表结构;迁移和 Repository 由业务自己维护。当前目录的完整示例见 [managedService](examples/managedService/README.md)。MySQL、事务审计和鉴权实现可参考 [旧目录用户管理示例](examples/userService/README.md),接入时按 [项目指南](PROJECT.md) 装配到 config/svc/logic。 HTTP 请求类型不能推导数据库表结构;迁移和 Repository 由业务自己维护。当前目录的完整示例见 [managedService](examples/managedService/README.md)。MySQL、事务审计和鉴权实现可参考 [旧目录用户管理示例](examples/userService/README.md),接入时按 [项目指南](PROJECT.md) 装配到 config/svc/logic。

View File

@@ -5,7 +5,7 @@
## 实施与验收清单 ## 实施与验收清单
- [x] 运行期:资源登记、逆序释放、幂等关闭、失败回滚、故障原因保留归框架;生成入口与项目依赖初始化边界明确。多 App 日志隔离、标准 log.Logger 适配、可信请求元数据与依赖中间件示例。 - [x] 运行期:资源登记、逆序释放、幂等关闭、失败回滚、故障原因保留归框架;生成入口与项目依赖初始化边界明确。多 App 日志隔离、标准 log.Logger 适配、可信请求元数据与依赖中间件示例。
- [x] 配置:项目完整结构体环境覆盖;默认值→文件→环境;嵌套、duration、slice、显式零值、空值策略、必填与脱敏;日志无副作用完整校验。 - [x] 配置:Default()→文件显式环境引用→类型绑定→Validate;嵌套、duration、slice、显式零值、必填/备用值、转义与安全诊断;日志无副作用完整校验。
- [x] 数据库:官方 MySQL 结构化配置/TLS/驱动日志适配,继续复用 OpenConnector 和唯一连接池;独立启动检查、失败清理、生成器接入;不内置版本/表结构/业务错误策略。 - [x] 数据库:官方 MySQL 结构化配置/TLS/驱动日志适配,继续复用 OpenConnector 和唯一连接池;独立启动检查、失败清理、生成器接入;不内置版本/表结构/业务错误策略。
- [x] 响应契约:数组、可空对象、成功 null、独立 HTTP 204;Logic/Handler/OpenAPI/漂移检查一致。 - [x] 响应契约:数组、可空对象、成功 null、独立 HTTP 204;Logic/Handler/OpenAPI/漂移检查一致。
- [x] 请求契约:未知 query、JSON UTF-8/charset、无 Body/必填空对象/可选 Body,App 默认与路由覆盖;scalar/any/数组/大小限制一致,普通 JSON 只绑定一次。 - [x] 请求契约:未知 query、JSON UTF-8/charset、无 Body/必填空对象/可选 Body,App 默认与路由覆盖;scalar/any/数组/大小限制一致,普通 JSON 只绑定一次。

View File

@@ -1,5 +1,21 @@
# Changelog # Changelog
## v0.22.0-preview(2026-09-26)
- 文件显式环境引用与备用值,字符串组合/转义,安全有界展开和类型绑定;变量值原样插入、不递归处理。
- 移除隐式环境覆盖、EnvOptions/前缀/别名及 env/envDefault;数据库统一使用文件加载,不再单独扫描环境。
- 生成 main/Load 新增 -check-config,精简配置文件,保留开发者扩展;错误字段建议和迁移检查。
- README、CONFIG、数据库/HTTP 客户端指南及示例同步新规则;配置中的响应模板需要 $$ 转义,.api 不变。
## v0.21.0-preview(2026-09-26)
- httpc 可配置环境/直连/固定代理、常用连接阶段超时和独立空闲池参数。
- JSON 新增逐请求 Header 快照与响应大小选项,保持有界读取、关闭和认证隔离。
- 成功请求少复制和退避计算,丢弃响应流式处理,错误体有界回收以改善连接复用。
- 增加真实本地代理、超时、连接限制/回收、并发、日志脱敏、gzip、fuzz 和修改前后基准。
- JSON 自定义方法接口签名需同步 variadic 参数;借用 Transport 的参数混配现在报错。详见 httpc/README.md。
## v0.20.0-preview(2026-09-26) ## v0.20.0-preview(2026-09-26)
- 生成项目配置加载入口持续由生成器维护;保留业务字段、默认值、校验与 NewLoader 扩展文件。 - 生成项目配置加载入口持续由生成器维护;保留业务字段、默认值、校验与 NewLoader 扩展文件。

263
CONFIG.md
View File

@@ -1,178 +1,177 @@
# 统一配置加载 # 配置:文件明确声明值的来源
v0.20 起,新生成服务的 `main.go` 调用生成器维护的 `config.Load`。项目只需要定义业务字段、默认值和校验;日志、HTTP、数据库等配置使用同一流程,配置加载时不打开数据库、监听端口或创建日志文件。 v0.22 起只使用这一条流程:
```text ```text
代码默认值 → envDefault(仅填零值)→ 可选配置文件 项目 Default() → 解析选中的一个文件 → 解析字段值中的环境引用
→ 完整结构体环境覆盖(含别名)→ 框架规范化 → 项目 Validate → 按 Go 类型绑定已提供的字段 → 项目 Validate() → 初始化日志和依赖
→ 初始化日志及项目依赖 → 启动服务
``` ```
## 生成项目要维护什么 没有环境前缀、别名表或隐式二次覆盖。业务只维护 `Config`、`Default()`、`Validate()` 和配置文件。项目 Validate 中继续调用嵌套框架配置的 Validate;生成模板已包含这些调用。验证函数只检查值,不应打开资源。
| 文件 | 维护方 | 用途 | ## 直接使用
| --- | --- | --- |
| `main.go` | 生成器 | 启动、信号、请求排空、依赖关闭 |
| `internal/config/loadGen.go` | 生成器 | 调用统一加载入口;纳入 `generate --check` |
| `internal/config/config.go` | 项目 | `Config`、`Default()`、`Validate()` |
| `internal/config/loader.go` | 项目 | `NewLoader()`,只有增加文件格式时才需修改 |
| `etc/config.yaml` | 项目 | 部署配置;可改用 JSON/XML 并传 `-config` |
| `internal/svc` | 项目与生成器分文件 | 初始化依赖、登记清理、AppOptions |
比如增加业务的批次数量:在 Config 增加一个字段,在 Validate 加范围校验;需要部署值时改配置文件或环境变量。不修改 main、Load 或 Handler。
```go ```go
// 添加到项目 Config。
BatchSize int `json:"batchSize" envDefault:"100"`
// 添加到项目 Validate。
if c.BatchSize < 1 || c.BatchSize > 1000 {
return omniConfig.Invalid("batchSize", "must be between 1 and 1000")
}
```
自动支持 `GO_OMNI_BATCH_SIZE`。生成服务的空环境值策略为 `ignore`,0 和 false 是有效覆盖;范围由 Validate 决定。
重新生成保留 config.go、loader.go 和 svc 的手写扩展。
## 手写项目用法
```go
package main
import (
"time"
"git.yuntuo.hk/k2safe/go-omni/config"
)
type Config struct { type Config struct {
Timeout time.Duration `json:"timeout" envDefault:"3s"` Server omniConfig.Config `json:"server"`
Limit int `json:"limit" envDefault:"100"` BatchSize int `json:"batchSize"`
Token string `json:"-" env:"APP_TOKEN,secret"` Enabled bool `json:"enabled"`
Password string `json:"password"`
Timeout time.Duration `json:"timeout"`
}
func Default() Config {
return Config{Server: omniConfig.Default(), BatchSize: 100, Enabled: true,
Timeout: 3*time.Second}
} }
func (c Config) Validate() error { func (c Config) Validate() error {
if c.Timeout <= 0 { if err := c.Server.Validate(); err != nil {
return config.Invalid("timeout", "must be positive") return omniConfig.AtField("server", err)
} }
if c.BatchSize < 0 { return omniConfig.Invalid("batchSize", "must not be negative") }
return nil return nil
} }
// 手写项目;生成项目的 Load 已自动调用此入口。
// cfg, err := config.LoadProject("etc/config.yaml", Config{}, cfg, err := omniConfig.LoadProject("etc/config.yaml", Default())
// config.EnvOptions{Prefix: "APP_", Empty: "reject"})
```
`LoadProject` 接收结构体默认值,可实现值接收器或指针接收器的 `Validate() error`。空路径表示不读文件。
`DecodeFile` 只做文件覆盖,不做 envDefault、环境覆盖或业务 Validate;旧的两参数调用仍有效,可用结构体或结构体指针。低层 `ApplyDefaults`、`OverlayEnv`、`LoadFile` 继续存在。
嵌套校验用 `config.AtField("client", err)` 加路径前缀。公共原因通过 `config.Invalid` 提供,原因中不要包含密码、DSN、Token 或输入原值。
## JSON / YAML / XML
按文件扩展名选择解析器:`.json`、`.yaml`、`.yml`、`.xml`,大小写不敏感。所有格式使用 Go 结构体的 **json 字段名**,无需为 XML 复制一份结构或标签。
以下配置等价:
```json
{"client":{"timeout":"2s"},"enabled":false,"ports":[8080,8081],"optional":null}
``` ```
```yaml ```yaml
client: server:
timeout: 2s address: "${HTTP_ADDR:-127.0.0.1:8080}"
password: "${DB_PASSWORD}"
timeout: "${REQUEST_TIMEOUT:-3s}"
enabled: false enabled: false
ports: [8080, 8081] ```
optional: null
文件未写 batchSize,保留 100;enabled 明确写 false,就使用 false。变量名字完全由文件决定,不需要对应 Go 字段名。配置文件可只列出部署需要变动的字段;不会自动导出/打印包含凭据的完整配置。
## 引用语法
| 文件里的值 | 含义 |
| --- | --- |
| `host: 127.0.0.1` | 固定值,不读取任何环境变量 |
| `password: "${DB_PASSWORD}"` | 未设置或空字符串报错,不回退到 Default() |
| `port: "${DB_PORT:-3306}"` | 未设置或为空时使用 3306 |
| `sessionToken: "${SESSION_TOKEN:-}"` | 未设置或为空时使用空字符串,是否合法由目标类型和 Validate 决定 |
| `url: "https://${HOST}:${PORT:-443}/v1"` | 普通字符串中可出现多个引用 |
| `literal: "$${DB_PASSWORD}"` | 得到字面量 `${DB_PASSWORD}`,不读取变量 |
| `literal: "$$cost"` | `$$` 转义一个 `$`,得到 `$cost` |
变量名遵循 `[A-Za-z_][A-Za-z0-9_]*`。只支持 `${NAME}`、`${NAME:-fallback}` 和 `$$` 转义,不支持 `${NAME-default}`、嵌套引用或 shell 表达式。普通单独的 `$` 保留。fallback 到第一个 `}` 结束,可包含冒号、`:-`、引号等文件格式允许的字符;需要包含 `}` 的值可使用固定值或环境变量。
解析只扫描文件的**已解码字段值**,不替换 YAML/JSON/XML 源文本,不扩展字段名,也不扫描 Default() 的字符串。插入的环境值和 fallback 不再展开。密码 `DDSFLKJlsf:saf:-123`、引号、换行、`${OTHER}` 均原样保留,不破坏文档结构。字符串不统一 TrimSpace;`"0"`、`"false"` 和全空格字符串均属于已提供。
同一变量每次加载最多读取一次;下一次加载会重新读取。没有全局变量值缓存。并发加载各自维护结果,不能把调用者正在修改的默认配置当作共享可变输入。
## 类型与覆盖
目标 Go 类型决定引用结果的转换:整数、浮点数、布尔值、`time.Duration`、字符串、指针及已有自定义 JSON/Text 反序列化。数值范围按目标位宽检查,整数不经过 float64。自定义反序列化仍由自定义类型解释文本,不猜测其业务含义。
- 文件缺失字段保留初始值;显式 0、false、空字符串正常覆盖,不用零值判断字段是否存在。
- `${MISSING}` 失败就报错;需要备用值必须写 `${MISSING:-value}`。
- 字符串编号 `"001"` 保持 `001`;绑定到 int 才转整数。环境里的 `null` 不自动变成空指针,`[1,2]` 不自动变成数组。
- 集合直接在文件里声明,可在每个标量元素中使用引用,例如 `ports: ["${PORT_A:-8080}", "${PORT_B:-8081}"]`。
- 数组整体替换;空 slice 保持非 nil;固定长度 Go 数组要求长度一致。map 合并键,已提供键的值整体替换。
- 固定文件 null 沿用 JSON 语义:清空指针/map/slice/interface,普通值字段保持初始值。需要区分 null 时使用指针。
- `any` 的引用结果保持字符串,不猜测动态类型。固定 JSON/YAML 数字使用 json.Number 保留精度。
- 引用成功后的值不会再覆盖一次。旧 `GO_OMNI_*` 名称只有出现在文件引用中才有作用;不隐式重置日志 file/rotation。
普通固定字符串仍按原有严格绑定规则处理;不会为了支持变量,把所有 `"123"` 都自动转换成数字。
## YAML、JSON、XML 一致
使用 Go 的 `json` 字段名,无需维护三份 tag。以下三份文件等价:
```yaml
port: "${PORT:-3306}"
password: "${PASSWORD}"
```
```json
{"port":"${PORT:-3306}","password":"${PASSWORD}"}
``` ```
```xml ```xml
<config> <config><port>${PORT:-3306}</port><password>${PASSWORD}</password></config>
<client><timeout>2s</timeout></client>
<enabled>false</enabled>
<ports type="array"><item>8080</item><item>8081</item></ports>
<optional type="null"/>
</config>
``` ```
XML 必须使用一个 `<config>` 根;普通叶节点按目标 Go 类型转换。数组显式 `type="array"` 加 `<item>`,不根据重复元素猜测类型。`type="array"` 空元素表示空数组,`type="object"` 表示空对象,`type="null"` 表示 null。`any` 中的 XML 叶节点默认是字符串,需要动态数字/布尔时标注 `type="number"` / `type="bool"`;也可用 `type="string"`。 XML 使用 `<config>` 根,标量占位符写在普通叶元素或 `type="string"` 元素中,由 Go 字段决定类型;显式 `type="number"` / `type="bool"` 表示文件里已经有合法数字/布尔常量。XML 数组用 `type="array"` 加 `<item>`;空对象 `type="object"`、null 用 `type="null"`。不从重复元素猜测数组。
类型和覆盖规则: `.api` 不经过配置加载器,响应模板 `${code}` 不变。**如果把响应模板写在 YAML/JSON/XML 配置文件里,须转义:**
- 缺失字段保留默认值;显式 0、false、空字符串正常覆盖。 ```yaml
- null 清空指针、map、slice、interface;普通值字段保持默认值,沿用 Go JSON 解码语义。需要区分 null 时用指针。 server:
- 数组整体替换;空 slice 保持非 nil;固定长度 Go 数组要求长度一致,避免丢弃多余配置。 response:
- map 合并键,已提供的键整体覆盖对应值;map 中未提供的键保留。 body:
- duration 支持 `250ms`、`3s`;整数保持精度,动态 JSON 数字使用 `json.Number`,不先转 float64。 code: "$${code}"
- 自定义 `json.Unmarshaler` / `encoding.TextUnmarshaler` 仍可接入。这些回调是可信项目代码,必须自己保证并发和副作用安全;不要修改共享外部状态。 msg: "$${message}"
- JSON `,string` 字段、嵌入字段遵循 JSON 字段选择;未知字段、重复绑定、冲突均拒绝。 data: "$${data}"
内置解析器限制:单文件最多 1 MiB,解析最多 64 层、100000 个节点;绑定和默认值复制也有深度限制。只接受普通文件,不读取设备或命名管道。不支持多文档、重复字段、无效 UTF-8;YAML 拒绝 alias/merge/custom tag,XML 拒绝 DTD/外部实体、命名空间、业务属性和混合内容。XML 仅支持配置所需的子集,不是通用 XML 文档映射器。
这些限制针对启动配置,避免意外超大文件、别名扩展和模糊覆盖。自定义值解码器的内部字段校验由该解码器负责;例如现有错误目录使用其专有解码规则。
## 环境覆盖
统一入口的 EnvOptions 控制前缀、空值策略、别名和 Lookup。完整 JSON 路径自动转换为 UPPER_SNAKE_CASE:`client.timeout` → `APP_CLIENT_TIMEOUT`。
```go
config.EnvOptions{
Prefix: "GO_OMNI_", Empty: "ignore",
Aliases: config.ServerEnvAliases("GO_OMNI_", "server"),
}
``` ```
生成项目同时接受完整的 `GO_OMNI_SERVER_HTTP_READ_TIMEOUT` 与短别名 `GO_OMNI_HTTP_READ_TIMEOUT`。完整名存在时优先,即便其空值被 ignore,也不回退别名。不会再二次调用 ApplyEnv 覆盖业务字段。 加载后恢复为 `${code}` 等响应占位符,仍由响应编码器处理。
显式 `env:"CUSTOM_NAME"` 是绝对名称;`env:"-"` 跳过子树。`json:"-"` 只在声明显式 env 名时参与,可让密钥只能从环境加载。`secret` 标签表达用途,解析错误对所有字段都不打印值。slice/map 的环境值采用 JSON。`required` 检查字符串非空、指针/集合非 nil;复杂范围由 Validate 负责。详见 [完整业务环境覆盖](DEVELOPMENT.md#完整业务配置的环境覆盖)。 ## 生成项目与检查
显式环境选择日志 stdout/stderr 会清除文件 profile 的 File/Rotation,除非同时提供非空 File 覆盖。该兼容规则由框架处理。 | 文件 | 维护方 |
| --- | --- |
| `main.go`、`internal/config/loadGen.go` | 生成器维护,纳入 `generate --check` |
| `internal/config/config.go` | 项目维护 Config、Default、Validate |
| `etc/config.yaml` 等选定文件 | 项目维护,重新生成不覆盖 |
| `internal/config/loader.go` | 初次生成即可使用,仅添加自定义格式时才修改 |
## 扩展解析器 ```sh
go run . -config etc/config.test.yaml -check-config
go run . -config etc/config.local.yaml
go build .
go-omni api generate --check
```
内置格式直接可用。新增格式只在 `internal/config/loader.go` 的 `NewLoader` 注册: `-check-config` 使用与启动完全相同的 Load,通过时退出 0、失败时非零;不打印配置,不创建日志文件、数据库、后台组件或 HTTP 监听。它只校验配置,不检测真实数据库连接或证书握手。
`-config` 只选一个文件,不自动叠加 local/test/default。指定文件不存在时报错;`-config ''` 只使用 Default() 后校验,不读取任何环境变量,也不寻找其他文件。示例:[examples/configuration](examples/configuration/README.md)。
## 安全诊断与限制
```text
config "etc/config.yaml":7:9 env "DB_PORT" field "database.port": expected a valid int
```
`*config.Error` 包含 File/Field/Env/Line/Column/Reason。引用错误同时保留文件与环境变量名;组合引用失败可列出多个名字。行列定位原文件字段,不把环境值伪装成源文件文本。无法取得位置时仍提供可用的文件/字段信息。
用 `config.Invalid(field, publicReason)` 或 `config.AtField(prefix, err)` 报业务校验错误。未知字段给出唯一且接近的已声明字段建议,不自动修改。错误默认不打印原值;自定义 Reason 不应拼入秘密值。原始 cause 仍支持 errors.Is/As,不要把 Unwrap 后的错误或整个配置写到日志。
文件最多 1 MiB;单个环境值/展开字段最多 1 MiB,展开字符串合计/变量缓存分别最多 4 MiB;解析/展开限制 64 层、100000 个节点与引用计数。拒绝无效 UTF-8、多文档、重复字段、未知字段、非普通文件;YAML 不接受 alias/merge/custom tag,XML 不接受 DTD/外部实体。不执行 shell,不自动加载 `.env`。
## 可选解析器与测试来源
内置格式无需项目手写加载器。需要额外格式时在已有 NewLoader 注册 Decoder,后续仍自动经过同一引用/绑定/校验流程:
```go ```go
func NewLoader() (*omniConfig.Loader, error) { func NewLoader() (*omniConfig.Loader, error) {
return omniConfig.NewLoader( return omniConfig.NewLoader(
omniConfig.WithDecoder("custom", omniConfig.DecoderFunc( omniConfig.WithDecoder("custom", omniConfig.JSONDecoder{}),
func(data []byte) (omniConfig.Document, error) {
// 演示注册扩展名;实际可在这里调用项目选定的 TOML 等解析库。
return (omniConfig.JSONDecoder{}).Decode(data)
},
)),
) )
} }
``` ```
Decoder 返回 Document:Value 是 map[string]any 根,子值为 string/bool/json.Number/nil/[]any/map[string]any;无自身类型的叶节点可返回 config.Text。Positions 可选,用 `client.timeout` 或 `servers[0].timeout` 标记 1-based 行列。不提供位置时仍有文件、字段、原因。 Decoder 返回 Document(map/string/bool/json.Number/nil/[]any,XML 类标量可用 Text),可提供 Positions。实例注册无全局 init;插件必须可并发调用、不修改共享输入,并且是可信 Go 代码。
自己的语法错误可返回 `&config.Error{Field: "client.timeout", Line: 3, Column: 5, Reason: "expected a duration"}`。Reason 必须安全;普通第三方错误仅公开 `invalid configuration syntax`,内部 cause 通过 errors.Is/As 保留。 测试或嵌入宿主可以用 `WithLookupEnv(func(string)(string,bool))` 替换引用查找函数。它仍然只读取文件明确引用的名称,不启用任何自动覆盖。普通业务不需要这个选项。
注册表属于 Loader 实例,无全局 init 注册;不能覆盖已有扩展名。Loader 可共享,但 Decoder 必须支持并发且不保留/修改输入。解析插件是可信 Go 代码,框架不是插件沙箱。新格式依赖由选择它的项目显式引入,核心不强制引入 TOML 等库。 ## v0.22 迁移:旧模式已移除
## 定位配置错误 这是不兼容调整,不保留两套加载逻辑:
```text - 删除 `EnvOptions`、`LoadFromEnv`、`ApplyEnv`、`OverlayEnv`、`ApplyDefaults`、`ServerEnvAliases` 和 `env`/`envDefault` 标签处理;检出遗留标签会提示迁移。
config "etc/config.yaml":4:12 field "client.timeout": expected a duration such as 5s - 删除 `database.ConfigFromEnv` / `database.ApplyEnv`;数据库配置作为项目 Config 的字段绑定。DSN 字段可在文件里使用 `dsn: "${DB_DSN}"`,序列化 database.Config 不输出 DSN 原值。
config env "APP_CLIENT_TIMEOUT" field "client.timeout": expected a valid time.Duration - `LoadProject(path, defaults[, loader])` 不再接收 EnvOptions;`LoadFile(path)`、`NewReloader(path, logger)` 不再接收前缀。
``` - `omni.LoadConfig(path)` 的参数现在是文件路径;空路径只用框架默认值。
`errors.As(err, &diagnostic)` 可获取 `*config.Error` 的 File/Field/Env/Line/Column/Reason;原 `DecodeError` 名称保留为别名。`errors.Is` 可判断文件不存在等 I/O 原因。 迁移顺序:
解析失败显示文件和可获取的行列;Validate 使用 Invalid/AtField 才能定位业务字段。环境覆盖导致的错误优先标环境变量,不错误地归因到文件;数组元素验证也会追溯到覆盖它的集合环境变量。默认值未出现在文件里时没有行列。YAML 语法错误通常只有行号,列显示 1。 1. 将 envDefault 默认值搬入 Default(),移除 env/envDefault 标签。需要文件绑定的旧 `json:"-"` 密钥字段改成明确 json 名称;不打印配置对象。
2. 按实际部署需要,在选定文件写环境引用。固定值保持固定,不按原前缀猜测。原先依赖环境切换日志输出的项目,应在文件明确配置 file/output/rotation。
3. 配置文件中的其他 `${...}` 模板改成 `$${...}` 字面量;`.api` 无需修改。
4. 手写项目改为 LoadProject。生成项目保留 Config/Default/Validate/NewLoader,删除旧手写 Load 后重新生成;main/loadGen 由生成器更新,业务配置文件不会被静默改写。
5. 升级匹配的 CLI 与框架,执行 generate、generate --check、go test、go build 和 `-check-config`,再使用真实部署配置启动。
不打印底层错误原文或配置值;任意普通 `errors.New(...)` 校验失败的公开原因是 `validation failed`,内部原因仍在链中。项目应改用 Invalid 给出安全的原因。第三方子配置校验若没有字段元数据,只定位到对应子树,框架不会从任意错误文案猜字段。 低层 `DecodeFile(path, defaults[, loader])` 也解析显式引用,但不调用 Validate;默认值和原始 Document 都不会因失败而被部分修改。日志 Reloader 仍只允许 log.level 变化,其他静态字段变更要求重启。
不要把 Unwrap 后的原始错误、整个配置对象写入公开日志/HTTP;自定义解析器与校验器传入的 Reason 由项目负责脱敏。
## v0.19 及更早项目迁移
先提交现有代码,再升级匹配的 CLI 与框架。旧项目的 `internal/config/config.go` 含手写 Load,新生成器会明确阻止 generate/check/dry-run,不会直接覆盖它。
1. 保留 Config、Default、Validate、业务配置结构及辅助函数。
2. 删除旧 Load。普通的 defaults/file/env/Validate 流程已由 LoadProject 接管;移除由此不再使用的 import。将自定义默认值和校验分别迁入 Default / Validate,自定义文件格式迁入 NewLoader。
3. 框架只在 `loader.go` 不存在时创建 NewLoader;已有项目文件同名时需要保证它提供同签名函数。无需手写 loadGen.go。
4. 重新生成,执行 `generate --check`、`go test ./...`、`go build .`,再用真实配置验证启动。
保留业务需要的显式 env 标签/规则;如果原 Load 有解密或其他项目特殊步骤,不要直接删除,应先明确其归属,必要时在自定义 Decoder 中处理。生成器不自动搬动或猜测手写行为。`api upgrade` 仍可预览较早版本的基础设施差异,但已修改的配置必须按上述步骤迁移。
不会移除低层配置 API;本次移除的是**新项目里重复手写的加载流程**,不删除旧项目的业务文件。
完整可运行例子:[examples/configuration](examples/configuration/README.md)。

View File

@@ -1,4 +1,4 @@
# 通用业务开发指南(v0.20.0-preview) # 通用业务开发指南(v0.22.0-preview)
从 [README](README.md) 创建服务后,业务主要维护 `.api`、配置、`svc.Init`、Logic、Repository 和项目中间件。本页说明通用接入能力;字段、错误码、权限、表结构和事务边界都由项目定义。 从 [README](README.md) 创建服务后,业务主要维护 `.api`、配置、`svc.Init`、Logic、Repository 和项目中间件。本页说明通用接入能力;字段、错误码、权限、表结构和事务边界都由项目定义。
@@ -100,44 +100,13 @@ defer reader.Close()
不声明 multipart 字段时,保留原来的 `body *httpx.Body` 流式入口,不创建临时文件。该方式适合特殊协议;项目应在不可撤销副作用之前完成校验并调用 `body.Complete()`。下载和原始回调继续使用 `@produces` / `@body(mode: raw)`。 不声明 multipart 字段时,保留原来的 `body *httpx.Body` 流式入口,不创建临时文件。该方式适合特殊协议;项目应在不可撤销副作用之前完成校验并调用 `body.Complete()`。下载和原始回调继续使用 `@produces` / `@body(mode: raw)`。
## 完整业务配置的环境覆盖 ## 文件声明配置来源
生成器维护 `internal/config/loadGen.go`;项目维护 Config/Default/Validate 和可选 NewLoader。三种格式、可扩展解析器、字段错误定位及旧 Load 迁移见 [统一配置指南](CONFIG.md)。 项目维护 Config、Default()、Validate() 和配置文件;生成器维护 Load/main。流程为默认值→选定文件→显式 `${NAME}` / `${NAME:-fallback}` 引用→类型绑定→校验。没有前缀、别名或 env/envDefault 隐式覆盖。
生成项目默认执行:代码与 `envDefault` 默认值 → JSON/YAML/XML 文件 → 全结构体环境覆盖(含别名)→ 规范化 → `Validate`。增加业务字段不需要另外写 `os.Getenv`。 固定值保持固定,缺失字段保留 Default(),0/false/空字符串正常覆盖;引用失败不回退初始化默认值。`$$` 转义 `$`,字符串凭据保留空格与特殊字符。
```go `go run . -config etc/config.test.yaml -check-config` 只校验,不打开日志、数据库或端口。生成/check/dry-run 保留业务文件;新的模板要求 v0.22。完整格式、错误定位、解析器与移除清单见 [CONFIG.md](CONFIG.md)。
type ClientConfig struct {
Endpoint string `json:"endpoint" env:"CLIENT_ENDPOINT,required"`
Token string `json:"-" env:"CLIENT_TOKEN,required,secret,empty=reject"`
Timeout time.Duration `json:"timeout" envDefault:"3s"`
Retries []int `json:"retries"`
Enabled bool `json:"enabled"`
}
```
把字段加入项目 `Config` 后,默认环境名取 `GO_OMNI_` 加 JSON 路径的 UPPER_SNAKE_CASE,如 `GO_OMNI_CLIENT_TIMEOUT`。显式 env 名是完整名称,可以使用自己的业务前缀。`env:"-"` 排除整个子树;`json:"-"` 字段只有显式 env 标签才参与。
duration 使用 `250ms`、`3s`;slice/map 使用 JSON,如 `[1,3,5]`、`{"retry":"2s"}`。支持嵌套、零/false、空数组、自定义 TextUnmarshaler。空环境值默认允许;可选择 `empty=ignore` 或 `empty=reject`。required 要求字符串非空、指针/集合非 nil;0 和 false 可以是合法业务值,范围由项目 Validate 决定。
手写项目可使用 `config.LoadProject(path, defaults, config.EnvOptions{Prefix:"MY_APP_"})`,或分别调用 `ApplyDefaults/DecodeFile/OverlayEnv`。读取失败不修改输入默认值;诊断不会打印输入值。不要主动打印含密钥的整个配置,或把底层解析错误展开到公开响应。Validate 不打开日志文件、数据库或存储连接;证书的可读性和握手信任在依赖初始化阶段验证。
### 一次覆盖完整配置与 HTTP/日志短名称
```go
cfg, err = config.OverlayEnv(cfg, config.EnvOptions{
Prefix: "MY_APP_",
Empty: "reject", // allow / ignore / reject,项目自行选择
Aliases: config.ServerEnvAliases("MY_APP_", "server"),
// Lookup 可注入测试或项目配置源;默认 os.LookupEnv。
})
```
不要再紧接着调用 `ApplyEnv(cfg.Server, ...)`。上面的调用同时处理完整配置和 HTTP_ADDR、SERVICE、LOG_LEVEL 等框架短名称。例如 `MY_APP_SERVER_ADDRESS` 与 `MY_APP_HTTP_ADDR` 指向同一字段,**完整名称优先**;未设置时才读取短名称。完整名称显式设空仍优先:allow 交给该字段解析(string 可清空;数字等仍要求合法值),ignore 保留文件/默认值,reject 报错。不会因 ignore 再退回别名。字段 `env` 标签的 empty 规则优先于全局规则。
自定义别名用 `EnvOptions.Aliases`,键是完整环境名、值是按优先级排列的别名数组。两种名称都使用同一 Lookup、空值策略和类型转换;布尔 false、数值 0 与未设置不同。集合始终使用 JSON,包括 `MY_APP_TRUSTED_PROXIES='["127.0.0.1"]'`。
新项目模板一次调用 OverlayEnv,默认 `Empty: "ignore"`。旧 `ApplyEnv` 保留原有忽略空值、修剪空白及 TRUSTED_PROXIES 逗号列表行为。已有项目的 `internal/config` 属于开发者文件,升级不覆盖它;按上例合并原来的两次覆盖调用后,保留项目自己的 Validate。日志输出的显式完整名称也优先于短名称。
## 依赖、日志与中间件 ## 依赖、日志与中间件
@@ -280,7 +249,7 @@ go-omni workspace check --file omni.workspace.json
## v0.13 项目升级 ## v0.13 项目升级
先使用与目标框架匹配的 CLI;CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.20.0-preview(或显式引用当前源码)。 先使用与目标框架匹配的 CLI;CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.22.0-preview(或显式引用当前源码)。
```sh ```sh
go-omni api upgrade --dir . --dry-run go-omni api upgrade --dir . --dry-run

View File

@@ -70,30 +70,26 @@ application.OnShutdown(client.Close)
```go ```go
client, err := httpc.New(httpc.Config{ client, err := httpc.New(httpc.Config{
Timeout: 3*time.Second, MaxRetries: 2, MaxConnsPerHost: 32, Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
MaxResponseBytes: 1 << 20, Timeout: 3*time.Second, DialTimeout: time.Second,
TLSHandshakeTimeout: time.Second, ResponseHeaderTimeout: time.Second,
MaxConnsPerHost: 32, MaxIdleConnsPerHost: 16, MaxIdleConns: 64,
IdleConnTimeout: 60*time.Second, MaxResponseBytes: 1<<20,
}) })
if err != nil { return err } if err != nil { return err }
application.OnShutdown(client.Close) application.OnShutdown(client.Close)
err = client.JSON(ctx, "GET", serviceURL, nil, &result) err = client.JSON(ctx, "GET", serviceURL, nil, &result,
httpc.WithHeader("Authorization", "Bearer "+token),
)
``` ```
客户端应在服务启动时创建并复用。默认使用独立的 `net/http.Transport` 连接池,每个目标主机 启动时创建并复用 Client。代理可选环境、直连、固定地址;不配置时保留标准库原代理策略。JSON 支持单次请求头和响应限额,不保存 Client 全局 Token。新阶段超时与空闲连接参数可选,0 沿用默认。
有连接上限;该上限不是整个进程的全局连接数上限。也可以注入自有 `http.RoundTripper`,
其连接限制和生命周期由调用方负责。这样出站实现同样可以替换。
总超时覆盖建连、重试、等待和读取响应体。`Do` 返回后调用方必须读取/关闭 `response.Body`; 成功响应读取有界;output=nil 流式丢弃,不整体缓存。小错误体只在大小/时间预算内尝试读完以复用连接;未知或大错误体及时关闭。Do 保留标准流式 Request/Response,由调用者限量读取并关闭 body。Close 拒绝新调用并关闭自有池空闲连接,项目先排空在途工作。
`JSON` 自动关闭,并限制解码前读取的字节数。`Do` 允许流式读取,本身不限制响应体大小。
`Close` 拒绝新请求、关闭自有池的空闲连接;在途请求应先排空。
`MaxRetries` 默认 0,最多 5。封装只对可重放的 GET/HEAD/OPTIONS 重试网络错误及 自定义 Transport 借用、不修改、不关闭;不能同时指定池/代理/阶段选项。总超时持续覆盖 HTTP 等待、重试和读取 body,但不抢占 JSON 编解码或不遵守 context 的自定义代码。MaxRetries 默认 0,只对可重放 GET/HEAD/OPTIONS 重试;3xx 不跟随。标准库和借用 Transport 自身仍可能重放,不能据此承诺业务只执行一次。
429/502/503/504;POST/PUT/PATCH/DELETE 不做应用层自动重试。标准库 Transport 和借用的
RoundTripper 自身的重放规则仍生效,不能据此承诺“只执行一次”。退避带抖动,尊重
`Retry-After`;服务器要求等待超过 `MaxBackoff` 时直接返回该响应,不提前重试。
重定向交给业务处理,3xx 不会自动跟随。
默认从 context 传递 `X-Request-ID`,日志只包含方法、主机、状态、尝试次数和收到响应头的耗时, 完整参数、NO_PROXY、TLS、安全日志、重试、签名兼容说明见 [httpc 指南](httpc/README.md),可运行 [HTTP Client 示例](examples/httpClient/README.md)。
不记录完整 URL、查询参数、凭证和响应体。非 2xx 的 JSON 调用返回只带状态码的 `StatusError`。
## 请求保护:middleware ## 请求保护:middleware

View File

@@ -1,6 +1,6 @@
# 生成项目与文件维护边界 # 生成项目与文件维护边界
v0.20.0-preview 提供受管理的生成项目。安装同版远程 CLI,生成的项目直接引用该版本,无需本地 replace。根目录入口支持 `go run .` 和 `go build .`;框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。 v0.22.0-preview 提供受管理的生成项目。安装同版远程 CLI,生成的项目直接引用该版本,无需本地 replace。根目录入口支持 `go run .` 和 `go build .`;框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。
## 创建和日常开发 ## 创建和日常开发
@@ -143,7 +143,7 @@ server:
`responseMode: raw` 返回 `{"status":"ready"}` 等健康数据,有依赖时包含检查名称、结果与耗时;不包装公共体。存活检查不执行依赖探针;服务开始退出后,就绪检查返回 503,存活仍可返回 200,直到监听关闭。OpenAPI 始终输出规范文档本身,不使用业务响应包装;关闭 HTTP 文档端点不影响生成 `docs/openapi.json`。 `responseMode: raw` 返回 `{"status":"ready"}` 等健康数据,有依赖时包含检查名称、结果与耗时;不包装公共体。存活检查不执行依赖探针;服务开始退出后,就绪检查返回 503,存活仍可返回 200,直到监听关闭。OpenAPI 始终输出规范文档本身,不使用业务响应包装;关闭 HTTP 文档端点不影响生成 `docs/openapi.json`。
环境变量覆盖配置文件,例如 `GO_OMNI_ENDPOINTS_READINESS_PATH=/readyz`、`GO_OMNI_ENDPOINTS_READINESS_RESPONSE_MODE=raw`、`GO_OMNI_ENDPOINTS_OPENAPI_ENABLED=false`;liveness/readiness/openapi 都支持对应的 ENABLED 和 PATH。Config/AppOptions 的最终有效配置决定是否注册。 内置路由配置由项目文件明确设置,也可以使用 `${NAME:-fallback}` 引用;不再按字段名自动读取环境变量。Config/AppOptions 的最终有效配置决定是否注册。
关闭或改名后,旧路径不再由框架保留,项目可以在 `.api` 自己声明。启用的内置 GET 路由与业务路由相同或被业务参数/通配路径覆盖时,在创建 App、开始监听之前报错;不会覆盖业务路由。内置接口仍经过全局中间件;项目在鉴权中间件中明确决定这些路径的访问规则,框架不自动绕过鉴权。 关闭或改名后,旧路径不再由框架保留,项目可以在 `.api` 自己声明。启用的内置 GET 路由与业务路由相同或被业务参数/通配路径覆盖时,在创建 App、开始监听之前报错;不会覆盖业务路由。内置接口仍经过全局中间件;项目在鉴权中间件中明确决定这些路径的访问规则,框架不自动绕过鉴权。
@@ -215,7 +215,7 @@ get /download (DownloadReq)
## 已有服务接入步骤 ## 已有服务接入步骤
每个独立服务生成一个项目,module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.20.0-preview: 每个独立服务生成一个项目,module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.22.0-preview:
1. 先保留旧服务,整理实际路由及现有响应、鉴权和超时要求。在新目录执行 `go-omni new --api /path/to/service.api --module example.com/service ./service`;不对旧入口强制覆盖。 1. 先保留旧服务,整理实际路由及现有响应、鉴权和超时要求。在新目录执行 `go-omni new --api /path/to/service.api --module example.com/service ./service`;不对旧入口强制覆盖。
2. 在 `.api` 中选择 `@app(defaultResponse: ...)`,显式保留原项目的字段、成功码、错误码和 `dataMode`。需要独立回执的路由单独声明 raw/produces。普通路由、404/405 和提前返回的中间件一起验收。 2. 在 `.api` 中选择 `@app(defaultResponse: ...)`,显式保留原项目的字段、成功码、错误码和 `dataMode`。需要独立回执的路由单独声明 raw/produces。普通路由、404/405 和提前返回的中间件一起验收。

View File

@@ -2,7 +2,7 @@
GoOmni 是面向多个业务项目复用的 Go 后端框架:用 `.api` 定义接口、请求和返回类型,生成项目目录、HTTP 接入代码及 OpenAPI,业务代码负责 Logic 和 Repository。也可以在已有项目中直接使用 `omni.New()`,或单独引入日志、存储等模块。 GoOmni 是面向多个业务项目复用的 Go 后端框架:用 `.api` 定义接口、请求和返回类型,生成项目目录、HTTP 接入代码及 OpenAPI,业务代码负责 Logic 和 Repository。也可以在已有项目中直接使用 `omni.New()`,或单独引入日志、存储等模块。
**从这份 README 开始接入。** 当前版本 **`v0.20.0-preview`**,新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入;文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md),版本变化与验收见 [发布说明](RELEASE.md)。 **从这份 README 开始接入。** 当前版本 **`v0.22.0-preview`**,新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入;文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md),版本变化与验收见 [发布说明](RELEASE.md)。
**v0.19 改动**:新增可选 `copyx` 工具,用普通 Go 调用完成浅拷贝、深拷贝和显式类型转换;深拷贝会拒绝自定义转换中残留的源对象可变引用。**移除了 `mappings` / `mappingFiles` 和映射代码生成**,已有使用者须按 [迁移步骤](COPY.md#从旧-mapping-迁移) 调整,不能直接删除仍在调用的旧文件。 **v0.19 改动**:新增可选 `copyx` 工具,用普通 Go 调用完成浅拷贝、深拷贝和显式类型转换;深拷贝会拒绝自定义转换中残留的源对象可变引用。**移除了 `mappings` / `mappingFiles` 和映射代码生成**,已有使用者须按 [迁移步骤](COPY.md#从旧-mapping-迁移) 调整,不能直接删除仍在调用的旧文件。
@@ -12,9 +12,9 @@ Go **1.22+**。框架不依赖 Gin,默认使用标准库网络层,提供可
响应字段、业务码、登录方式、权限规则和数据库表结构由接入项目决定。文中的 `code/msg/data`、`200/9999` 都是示例配置,可以替换。 响应字段、业务码、登录方式、权限规则和数据库表结构由接入项目决定。文中的 `code/msg/data`、`200/9999` 都是示例配置,可以替换。
契约支持 `returns([]string/[]int64/[]bool/[]any)`、OpenAPI 公共引用、workspace 一次导出共享 Go 错误目录,以及统一环境变量别名覆盖。详细配置与升级步骤见 [业务开发指南](DEVELOPMENT.md)。 契约支持 `returns([]string/[]int64/[]bool/[]any)`、OpenAPI 公共引用、workspace 一次导出共享 Go 错误目录,以及文件显式环境引用与配置检查。详细配置与升级步骤见 [业务开发指南](DEVELOPMENT.md)。
已有通用开发能力见 [业务开发指南](DEVELOPMENT.md):数组/null、严格请求策略、声明式上传、完整配置环境覆盖、资源生命周期、共享契约和升级工具。MySQL 直接使用 [官方适配器](database/mysql/README.md)。 已有通用开发能力见 [业务开发指南](DEVELOPMENT.md):数组/null、严格请求策略、声明式上传、文件显式配置引用、资源生命周期、共享契约和升级工具。MySQL 直接使用 [官方适配器](database/mysql/README.md)。
后台组件可在 svc 登记 [分阶段关闭](lifecycle/README.md#后台组件分阶段关闭),先停止接收工作并等待收尾,再关闭共享资源;无需绑定队列或 ORM。 后台组件可在 svc 登记 [分阶段关闭](lifecycle/README.md#后台组件分阶段关闭),先停止接收工作并等待收尾,再关闭共享资源;无需绑定队列或 ORM。
@@ -38,13 +38,13 @@ Go **1.22+**。框架不依赖 Gin,默认使用标准库网络层,提供可
| 错误目录与动态提示 | `WithErrors(catalog)` 注册一次;`ByCode(...).WithParams(...).WithCause(err)` | [错误目录](errors/README.md)、[模板参数示例](examples/contracts/errorParams.api) | | 错误目录与动态提示 | `WithErrors(catalog)` 注册一次;`ByCode(...).WithParams(...).WithCause(err)` | [错误目录](errors/README.md)、[模板参数示例](examples/contracts/errorParams.api) |
| OpenAPI 与错误公共引用 | `.api` 自动生成;手写接口可使用 `openapi` 注册器;Go 错误目录显式导出 | [契约工具](DEVELOPMENT.md)、[注册示例](examples/platform/main.go) | | OpenAPI 与错误公共引用 | `.api` 自动生成;手写接口可使用 `openapi` 注册器;Go 错误目录显式导出 | [契约工具](DEVELOPMENT.md)、[注册示例](examples/platform/main.go) |
| 结构化业务日志 | `logx.Info/Error`、`logx.FromContext(ctx)`、`logx.New`;支持轮转、脱敏、慢请求和级别更新 | [下方日志示例](#日志)、[完整配置](logx/README.md) | | 结构化业务日志 | `logx.Info/Error`、`logx.FromContext(ctx)`、`logx.New`;支持轮转、脱敏、慢请求和级别更新 | [下方日志示例](#日志)、[完整配置](logx/README.md) |
| 文件配置与环境覆盖 | 默认值→JSON/YAML/XML→环境→校验;完整业务结构用 `config.LoadProject[T]` / 环境别名 | [统一配置指南](CONFIG.md)、[完整可运行示例](examples/configuration/README.md)、[动态级别](ADVANCED.md#配置与日志热更新) | | 文件配置与环境引用 | Default()→JSON/YAML/XML 字段引用→类型绑定→校验;无隐式覆盖,支持 `-check-config` | [统一配置指南](CONFIG.md)、[完整可运行示例](examples/configuration/README.md)、[动态级别](ADVANCED.md#配置与日志热更新) |
| 生命周期与健康检查 | svc 登记 OnClose/OnStop/OnDrain,RequestShutdown 通知故障;健康路径/开关可配 | [生命周期用法](lifecycle/README.md)、[健康检查](health/README.md)、[生成项目扩展](PROJECT.md) | | 生命周期与健康检查 | svc 登记 OnClose/OnStop/OnDrain,RequestShutdown 通知故障;健康路径/开关可配 | [生命周期用法](lifecycle/README.md)、[健康检查](health/README.md)、[生成项目扩展](PROJECT.md) |
| 数据库、连接池与事务 | `database.Open` / `OpenConnector`;事务与检查由项目显式调用 | [通用数据库](database/README.md)、[MySQL TLS/Logger](database/mysql/README.md) | | 数据库、连接池与事务 | `database.Open` / `OpenConnector`;事务与检查由项目显式调用 | [通用数据库](database/README.md)、[MySQL TLS/Logger](database/mysql/README.md) |
| 可选 GORM 与迁移 | gormx 复用已有连接池;`go-omni migrate` 预览,显式 apply | [GORM 示例](database/gormx/README.md)、[迁移用法](ADVANCED.md#迁移审计与分页) | | 可选 GORM 与迁移 | gormx 复用已有连接池;`go-omni migrate` 预览,显式 apply | [GORM 示例](database/gormx/README.md)、[迁移用法](ADVANCED.md#迁移审计与分页) |
| 认证与权限接口 | 注入 Authenticator / Policy;JWT 或 Session 只是可选实现 | [认证说明](auth/README.md)、[鉴权完整例子](examples/platform/README.md) | | 认证与权限接口 | 注入 Authenticator / Policy;JWT 或 Session 只是可选实现 | [认证说明](auth/README.md)、[鉴权完整例子](examples/platform/README.md) |
| 缓存 | `cache.NewMemory` 或 Redis 实现;业务面向 Store 的 Get/Set/Delete | [缓存配置/代码](MODULES.md)、[可运行模块服务](examples/modules/README.md) | | 缓存 | `cache.NewMemory` 或 Redis 实现;业务面向 Store 的 Get/Set/Delete | [缓存配置/代码](MODULES.md)、[可运行模块服务](examples/modules/README.md) |
| 出站 HTTP、熔断 | `httpc.New` 复用连接池与截止时间;`resilience.New` 隔离故障依赖 | [HTTP 用法](MODULES.md)、[熔断/观测](ADVANCED.md#熔断与观测) | | 出站 HTTP、熔断 | `httpc.New` 配置代理、连接池和阶段超时;JSON 支持单次请求头/限额;`resilience.New` 隔离故障 | [HTTP 完整用法](httpc/README.md)、[可运行示例](examples/httpClient/README.md)、[熔断/观测](ADVANCED.md#熔断与观测) |
| 请求追踪、访问日志、恢复、限流、幂等、超时 | `app.Use(...)` 按需组合 middleware;跨副本选择 Redis 实现 | [模块说明与限制](MODULES.md#请求追踪与基础指标)、[分布式实现](ADVANCED.md#分布式限流与幂等) | | 请求追踪、访问日志、恢复、限流、幂等、超时 | `app.Use(...)` 按需组合 middleware;跨副本选择 Redis 实现 | [模块说明与限制](MODULES.md#请求追踪与基础指标)、[分布式实现](ADVANCED.md#分布式限流与幂等) |
| 本地后台任务与调度 | `task.New`、Submit、Every;有界队列、排空关闭 | [task 示例与生命周期](MODULES.md#后台执行task) | | 本地后台任务与调度 | `task.New`、Submit、Every;有界队列、排空关闭 | [task 示例与生命周期](MODULES.md#后台执行task) |
| 持久队列 | 业务使用 Publisher / Consumer;Redis Streams 提供至少一次交付 | [队列初始化/消费示例](ADVANCED.md#可靠队列) | | 持久队列 | 业务使用 Publisher / Consumer;Redis Streams 提供至少一次交付 | [队列初始化/消费示例](ADVANCED.md#可靠队列) |
@@ -54,28 +54,57 @@ Go **1.22+**。框架不依赖 Gin,默认使用标准库网络层,提供可
| 可选结构转换与深拷贝 | `copyx.Copy` / `Clone` / `New`;Go converter 处理差异 | [下方示例](#可选结构转换与深拷贝)、[完整 copyx 指南](COPY.md)、[可运行代码](examples/copying) | | 可选结构转换与深拷贝 | `copyx.Copy` / `Clone` / `New`;Go converter 处理差异 | [下方示例](#可选结构转换与深拷贝)、[完整 copyx 指南](COPY.md)、[可运行代码](examples/copying) |
| 自定义适配器验收 | `transporttest.Run`、`storagetest.Run` / `RunUploads` 验证扩展契约 | [网络契约测试入口](transport/transporttest/router.go)、[存储契约测试入口](storage/storagetest/contract.go) | | 自定义适配器验收 | `transporttest.Run`、`storagetest.Run` / `RunUploads` 验证扩展契约 | [网络契约测试入口](transport/transporttest/router.go)、[存储契约测试入口](storage/storagetest/contract.go) |
## 统一配置加载 ## 带认证的出站 HTTP
新生成项目的加载流程由框架管理。业务在 `internal/config/config.go` 增加字段、默认值和 `Validate`,无需改 main 或复制环境变量解析。JSON、YAML、XML 共用字段和校验;扩展格式在 `loader.go` 注册。 HTTP Client 在 svc 初始化一次,业务重复使用。普通内部请求可直接选择直连和连接参数,无需手写 Transport:
```go ```go
// 项目 Config 中增加字段,自动支持 GO_OMNI_BATCH_SIZE: client, err := httpc.New(httpc.Config{
BatchSize int `json:"batchSize" envDefault:"100"` Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
Timeout: 3*time.Second, DialTimeout: time.Second,
// 项目 Validate 中增加业务范围检查: TLSHandshakeTimeout: time.Second, ResponseHeaderTimeout: time.Second,
if c.BatchSize < 1 { MaxConnsPerHost: 32, MaxIdleConnsPerHost: 16, MaxIdleConns: 64,
return omniConfig.Invalid("batchSize", "must be positive") })
} if err != nil { return err }
// 在 svc 生命周期登记 client.Close;关闭前先排空使用它的工作。
err = client.JSON(ctx, "GET", endpoint, nil, &result,
httpc.WithHeader("Authorization", "Bearer "+token),
httpc.WithMaxResponseBytes(64<<10),
)
``` ```
错误包含文件、字段、可用的行列或环境变量名;输入原值不自动写入错误。 请求头只作用于本次调用,不把用户 Token 留在共享 Client。默认代理/参数保持原行为;也可选择环境代理或固定代理。JSON 自动关闭并限制响应,Do 保留下载/流式读取扩展。完整配置、重试边界和已有自定义 Transport 的迁移见 [httpc](httpc/README.md)。
```sh
go run ./examples/httpClient
```
## 统一配置加载
业务只维护 Config、Default()、Validate() 和配置文件。生成器维护 Load/main;文件缺失字段保留默认值,显式 0/false/空字符串正常覆盖,固定值不会被同名环境变量再次覆盖。
```go
BatchSize int `json:"batchSize"` // 加入 Config;在 Default() 中设置 BatchSize: 100。
```
```yaml
server:
address: "${HTTP_ADDR:-127.0.0.1:8080}"
batchSize: "${BATCH_SIZE:-100}"
password: "${DB_PASSWORD}"
sessionToken: "${SESSION_TOKEN:-}"
```
`${NAME}` 未设置或为空就报错;`${NAME:-value}` 明确指定备用值。`$$` 转义 `$`;密码中的冒号、换行、引号和 `${...}` 原样插入,不二次解释。JSON/YAML/XML 共用解析后的字段处理和 Go 类型绑定,错误同时定位文件、字段、变量及可用行列,不打印原值。
```sh ```sh
go run ./examples/configuration -format xml go run ./examples/configuration -format xml
DEMO_CLIENT_TIMEOUT=500ms go run ./examples/configuration DEMO_CLIENT_TIMEOUT=500ms go run ./examples/configuration
# 在生成项目中,仅检查选中的文件,不初始化依赖:
go run . -config etc/config.test.yaml -check-config
``` ```
详见 [统一加载、解析器、迁移](CONFIG.md) 和 [完整示例](examples/configuration/README.md)。**旧项目须先迁移手写 Load,再重新生成**;生成器会保留业务配置并拒绝冲突,不静默覆盖。 完整 [语义与迁移](CONFIG.md)、[可运行示例](examples/configuration/README.md)。**v0.22 移除旧自动环境覆盖、前缀/别名及 env/envDefault 标签**,生成项目先迁移开发者配置再重新生成;文件中的响应模板 `${code}` 写成 `$${code}`,`.api` 保持原样。
## 选择接入方式 ## 选择接入方式
@@ -97,7 +126,7 @@ module 地址是 `git.yuntuo.hk/k2safe/go-omni`。先确保本机 Go 和 Git 可
```sh ```sh
export GOPRIVATE="$(go env GOPRIVATE),git.yuntuo.hk" export GOPRIVATE="$(go env GOPRIVATE),git.yuntuo.hk"
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.20.0-preview go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview
export PATH="$(go env GOPATH)/bin:$PATH" export PATH="$(go env GOPATH)/bin:$PATH"
go-omni --help go-omni --help
``` ```
@@ -254,7 +283,7 @@ go-omni new --api catalog.api \
在业务项目的 **go.mod 所在目录**执行: 在业务项目的 **go.mod 所在目录**执行:
```sh ```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.20.0-preview go get git.yuntuo.hk/k2safe/go-omni@v0.22.0-preview
``` ```
已有项目可直接复用现有目录、Service、Repository 和数据库连接,先选择一个接口接入。也可以只用 `logx`、`storage` 等包,暂时保留原来的 HTTP 框架。希望入口和 HTTP 接入都交给生成器的项目,按 [已有服务接入步骤](PROJECT.md#已有服务接入步骤) 在新目录生成后迁入业务代码。 已有项目可直接复用现有目录、Service、Repository 和数据库连接,先选择一个接口接入。也可以只用 `logx`、`storage` 等包,暂时保留原来的 HTTP 框架。希望入口和 HTTP 接入都交给生成器的项目,按 [已有服务接入步骤](PROJECT.md#已有服务接入步骤) 在新目录生成后迁入业务代码。
@@ -498,20 +527,22 @@ API 项目的数据库迁移需要单独维护并显式执行;启动不会自
## 配置与运行 ## 配置与运行
生成项目从 `etc/config.yaml` 读取配置,优先级为 **默认值 → 配置文件 → 环境变量**。密钥通过环境变量提供。手写入口需显式调用配置加载函数并传给 `omni.New(omni.WithConfig(settings))`;单独调用 `omni.New()` 不会自动读取环境变量。 生成项目使用 **Default() → 选定文件的显式引用 → 类型绑定 → Validate()**。固定值不会被环境变量覆盖;配置规则、转义及迁移见 [配置指南](CONFIG.md)。手写入口显式加载配置,再传入 `omni.New(omni.WithConfig(settings))`。
| 环境变量 | 用途 | | 文件字段 | 用途 |
| --- | --- | | --- | --- |
| `GO_OMNI_HTTP_ADDR` | 监听地址;生成项目默认 `127.0.0.1:8080` | | `server.address` | 监听地址,生成项目默认 127.0.0.1:8080 |
| `GO_OMNI_TRANSPORT` | `nethttp`(默认)、`native`、`http2` | | `server.transport` | nethttp(默认)、native、http2 |
| `GO_OMNI_TLS_CERT_FILE` / `GO_OMNI_TLS_KEY_FILE` | 成对设置 TLS 证书和私钥路径 | | `server.tls.certFile/keyFile` | 成对配置 TLS 证书和私钥路径 |
| `GO_OMNI_HTTP_READ_HEADER_TIMEOUT` / `GO_OMNI_HTTP_READ_TIMEOUT` | 请求头 / 请求读取超时,如 `5s` / `30s` | | `server.http.readHeaderTimeout/readTimeout` | 请求头 / 请求读取超时 |
| `GO_OMNI_HTTP_WRITE_TIMEOUT` / `GO_OMNI_HTTP_IDLE_TIMEOUT` | 响应写入 / Keep-Alive 空闲超时 | | `server.http.writeTimeout/idleTimeout` | 响应写入 / Keep-Alive 空闲超时 |
| `GO_OMNI_HTTP_MAX_HEADER_BYTES` | 请求头大小限制,字节数 | | `server.http.maxHeaderBytes` | 请求头大小限制,字节数 |
| `GO_OMNI_SHUTDOWN_TIMEOUT` | 优雅退出等待时间,默认 `10s` | | `server.shutdownTimeout` | 优雅退出等待时间,默认 10s |
| `GO_OMNI_ENABLE_LOGGER` / `GO_OMNI_LOG_LEVEL` | 访问日志开关、日志级别 | | `server.enableLogger` / `server.log.level` | 访问日志开关、日志级别 |
| `GO_OMNI_TRUSTED_PROXIES` | 可解析转发地址头的可信代理 IP/CIDR,逗号分隔 | | `server.trustedProxies` | 可信代理 IP/CIDR,文件数组 |
| `GO_OMNI_DB_HOST/USER/PASSWORD/DATABASE` | 官方 MySQL 适配器的结构化连接配置 | | `database.host/user/password/database` | 官方 MySQL 结构化连接配置 |
初始生成文件已为地址、网络层和选中模块的必要配置声明环境引用;其他字段按需要添加固定值或引用。
例如在生成项目中切换 native: 例如在生成项目中切换 native:
@@ -556,7 +587,7 @@ go test ./...
`main.go`、`Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理,手动修改会报告冲突。Logic、Repository、中间件实现、配置和依赖装配属于项目,重复生成保留手写内容。契约变化会提示受影响的 Logic 文件和 DTO 字段,明确的签名错误会给出原签名与期望签名;生成后 `--check` 仍能检出未修复的问题。将 `.api`、`omni.gen.json`、生成文件和 `.goOmni/apiManifest.json` 一起提交,CI 同时运行 `generate --check` 与 `go test ./...`。详见 [契约变更提示与检查边界](PROJECT.md#契约变更提示)。 `main.go`、`Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理,手动修改会报告冲突。Logic、Repository、中间件实现、配置和依赖装配属于项目,重复生成保留手写内容。契约变化会提示受影响的 Logic 文件和 DTO 字段,明确的签名错误会给出原签名与期望签名;生成后 `--check` 仍能检出未修复的问题。将 `.api`、`omni.gen.json`、生成文件和 `.goOmni/apiManifest.json` 一起提交,CI 同时运行 `generate --check` 与 `go test ./...`。详见 [契约变更提示与检查边界](PROJECT.md#契约变更提示)。
CLI 与业务 module 固定使用 `v0.20.0-preview`。v0.14–v0.19 项目先按 [配置入口迁移](CONFIG.md#v019-及更早项目迁移) 整理旧 Load,再升级依赖与 CLI 并重新生成,框架维护的 main/runtimeGen 自动更新;v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理,不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移),旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 CLI 与业务 module 固定使用 `v0.22.0-preview`。v0.21 及更早项目先按 [配置迁移](CONFIG.md#v022-迁移旧模式已移除) 整理旧标签、文件与 Load,再升级依赖与 CLI 并重新生成,框架维护的 main/runtimeGen 自动更新;v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理,不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移),旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。
## 按需深入 ## 按需深入

View File

@@ -1,4 +1,50 @@
# v0.20.0-preview # v0.22.0-preview
配置文件明确决定每个值的来源,项目只维护 Config、Default()、Validate() 和文件。
- `${NAME}` 必填,`${NAME:-fallback}` 在未设置/空字符串时使用备用值,`$$` 转义 `$`。支持字符串组合;环境值不二次展开,凭据空格和特殊字符原样保留。
- 在 JSON/YAML/XML 已解析的字段值上展开,再按目标 Go 类型绑定;缺失字段保留默认值,0/false/空字符串可显式覆盖。字段错误定位文件、环境名和可用行列,不回显值。
- 生成 Load/main 自动接入;新增 `-check-config`,选定一个文件检查,成功退出,不启动日志/依赖/监听。精简生成配置,重新生成保留业务 Config、Default、Validate 和文件。
- **不兼容变更:旧自动环境覆盖、前缀/别名、env/envDefault、database.ConfigFromEnv/ApplyEnv 已移除。** LoadFile/NewReloader 不再接收 prefix,omni.LoadConfig 参数改为文件路径。发现旧标签提示迁移。
- 配置文件中的响应模板用 `$${code}` 等转义;`.api` 中的 `${code}` 不变。日志 file/output/rotation 均由显式文件配置决定,不再隐式清除。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.22.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview
# 按 CONFIG.md 迁移开发者配置后:
go-omni api generate
go-omni api generate --check
go run . -config etc/config.yaml -check-config
```
[完整规则与迁移](CONFIG.md)、[可运行示例](examples/configuration/README.md)、[场景与性能验收](reports/explicitConfigurationAcceptance.md) 已同步。48 个测试包全量 race、实际生成消费者、Go 1.22、vet/build 和约 28 万次引用 fuzz 通过;小配置固定值/带引用加载约 25 µs,属于本机启动阶段基准。当前仍为 preview。
---
# v0.21.0-preview(历史版本)
让带认证的 HTTP 调用和内部直连可以直接使用 httpc,减少手写 Transport 与 JSON 响应处理。
- Proxy 支持环境、直连、固定代理;默认保留既有策略。
- 建连、TCP keepalive、TLS 握手、响应头、100-continue、空闲超时和独立空闲连接数量可选配置;不修改全局 Transport,不关闭借用实例。
- JSON 支持 WithHeader / WithHeaders / WithMaxResponseBytes;认证信息逐请求隔离,自动关闭和有界响应处理保留。
- 成功路径减少请求复制和无用退避;output=nil 按限额流式丢弃;小错误体在大小和时间预算内尝试复用连接。
- 不新增依赖、不增加业务规则;Do 保留标准流式扩展。CLI 最新版本与模板最低版本分开,本轮模板兼容门槛仍是 v0.20。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.21.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.21.0-preview
```
**兼容注意**:JSON 旧的直接调用仍有效,自定义 interface/函数值需要同步新增的 variadic 参数。自定义 Transport 与显式连接/代理/阶段参数混用现在报错,避免参数被忽略;迁移时在自己的 Transport 配置,或改用内置字段。没有删除原调用入口。
[参数与使用说明](httpc/README.md)、[可运行示例](examples/httpClient/README.md)、[场景与性能验收](reports/httpcAcceptance.md) 已同步到 README。48 个测试包全量 race、Go 1.22 兼容、vet/build、约 110 万次请求头 fuzz 通过。性能来自本机同条件基准,不能代替项目实际流量验证;仍为 preview。
空缓存远程安装、下载模块的 httpc race 与示例、8 组生成消费者验收通过,无本地 replace。标签固定在 `96e7d9629b5c8b78e9cfefbcb05da0ab702c242b`,见 [远程记录](reports/httpcRemote.json)。
---
# v0.20.0-preview(历史版本)
统一配置加载,减少生成项目重复维护的启动代码。 统一配置加载,减少生成项目重复维护的启动代码。
@@ -18,7 +64,7 @@ go test ./...
go build . go build .
``` ```
完整用法、格式、脱敏边界和迁移见 [统一配置指南](CONFIG.md),可运行例子见 [examples/configuration](examples/configuration/README.md)。回归、模糊测试与启动加载性能见 [验收记录](reports/configurationAcceptance.md)。仍为 preview;项目的实际配置和业务依赖应自行验收。 完整用法、格式、脱敏边界和迁移见 [统一配置指南](CONFIG.md),可运行例子见 [examples/configuration](examples/configuration/README.md)。回归、模糊测试与启动加载性能见 [验收记录](reports/configurationAcceptance.md)。47 个测试包的全量 race、Go 1.22 兼容、vet/build、30 秒 fuzz 与 5 个基准场景三轮测量通过。空缓存远程安装和 8 组生成消费者回归通过,无本地 replace;标签固定在 `c60841f901a3bcc3e9582eece6cfb2cad008fb9e`,见 [远程记录](reports/configurationRemote.json)。仍为 preview;项目的实际配置和业务依赖应自行验收。
--- ---
@@ -116,7 +162,7 @@ go-omni api generate --check
go test ./... go test ./...
``` ```
旧项目 config/Logic/svc 继续由开发者维护,不会被重新生成覆盖。已有环境配置需按 [一次覆盖完整配置](DEVELOPMENT.md#一次覆盖完整配置与-http日志短名称) 合并 OverlayEnv 与 ApplyEnv;共享错误目录接入见 [workspace 配置](DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。已有手写权限 Handler 切换自动生成前,先迁出实现并显式移除 Manual 文件,生成器不会静默删除。 旧项目 config/Logic/svc 继续由开发者维护,不会被重新生成覆盖。已有环境配置需按 本版当时的统一覆盖方式合并 OverlayEnv 与 ApplyEnv(该模式已在 v0.22 移除,当前升级见 [配置迁移](CONFIG.md#v022-迁移旧模式已移除));共享错误目录接入见 [workspace 配置](DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。已有手写权限 Handler 切换自动生成前,先迁出实现并显式移除 Manual 文件,生成器不会静默删除。
全量 race(含实际生成消费者)、vet、build、根入口/退出/再生成验证通过。eTax 只读契约验收:API 文档 20,751,149 → 847,171 字节,ID 文档 1,008,210 → 392,481 字节;70 个操作展开引用后与原文档完全一致。空缓存远程 CLI / 固定版本消费者验收也已通过,没有本地 replace;标签固定在 `64e3bd6cfced225476d58e95037249cd0a4fe695`。详见 [验收记录](reports/contractFollowupAcceptance.md)。这不代表已完成业务项目迁移。 全量 race(含实际生成消费者)、vet、build、根入口/退出/再生成验证通过。eTax 只读契约验收:API 文档 20,751,149 → 847,171 字节,ID 文档 1,008,210 → 392,481 字节;70 个操作展开引用后与原文档完全一致。空缓存远程 CLI / 固定版本消费者验收也已通过,没有本地 replace;标签固定在 `64e3bd6cfced225476d58e95037249cd0a4fe695`。详见 [验收记录](reports/contractFollowupAcceptance.md)。这不代表已完成业务项目迁移。

View File

@@ -14,7 +14,7 @@ v0.14 新增 `api format [--check]`、`api upgrade [--dry-run|--preview <空目
```sh ```sh
# 仓库读取权限与 PATH 配置见根目录 README # 仓库读取权限与 PATH 配置见根目录 README
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.20.0-preview go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview
go-omni new --module example.com/hello ./hello go-omni new --module example.com/hello ./hello
# 从自己的契约生成;service.api 及其相对 import 文件需要在本地存在 # 从自己的契约生成;service.api 及其相对 import 文件需要在本地存在
@@ -49,14 +49,14 @@ main、Gen.go、OpenAPI 和 manifest 由生成器维护;业务文件按字节
## 版本与迁移 ## 版本与迁移
CLI 与框架依赖固定使用 v0.20.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod,不添加本地 replace;`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换: CLI 与框架依赖固定使用 v0.22.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod,不添加本地 replace;`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换:
```sh ```sh
# 在 go-omni 源码仓库中执行 # 在 go-omni 源码仓库中执行
go run ./cmd/go-omni new --module example.com/hello --framework-dir . ../hello go run ./cmd/go-omni new --module example.com/hello --framework-dir . ../hello
``` ```
当前模板与依赖以 v0.20.0-preview 为基线,继续使用 manifest v2;GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级):原始模板可自动替换,含业务修改的基础文件按预览合并;通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务,不使用旧 `add` 命令的手动路由装配。 当前模板与依赖以 v0.22.0-preview 为基线,继续使用 manifest v2;GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级):原始模板可自动替换,含业务修改的基础文件按预览合并;通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务,不使用旧 `add` 命令的手动路由装配。
## 数据库迁移命令 ## 数据库迁移命令
@@ -70,6 +70,8 @@ go-omni migrate --dir migrations --apply
SQL 命名示例 `0001_createUsers.up.sql`;预览只显示版本和 SHA-256,不连接数据库。失败保留 dirty 状态,不自动 down/force。细节见 [扩展模块](../../ADVANCED.md)。 SQL 命名示例 `0001_createUsers.up.sql`;预览只显示版本和 SHA-256,不连接数据库。失败保留 dirty 状态,不自动 down/force。细节见 [扩展模块](../../ADVANCED.md)。
### v0.20 配置入口 ### v0.22 文件显式配置
新生成 `internal/config/loadGen.go` 调用统一加载,纳入生成检查;项目 config.go 只维护字段、默认值和校验,loader.go 用于注册额外格式。JSON/YAML/XML 内置。旧手写 Load 会阻止生成,请先按 [迁移步骤](../../CONFIG.md#v019-及更早项目迁移) 整理,不会自动覆盖业务代码。 新生成 `internal/config/loadGen.go` 调用统一加载,纳入生成检查;项目 config.go 只维护字段、默认值和校验,loader.go 用于注册额外格式。JSON/YAML/XML 内置。旧手写 Load 会阻止生成,请先按 [迁移步骤](../../CONFIG.md#v022-迁移旧模式已移除) 整理,不会自动覆盖业务代码。
`go run . -config etc/config.test.yaml -check-config` 只做配置解析和校验,不初始化日志、数据库或监听。文件中的固定值不受环境自动覆盖;`${NAME}` 为必填引用,`${NAME:-fallback}` 提供备用值。

View File

@@ -26,8 +26,8 @@ func TestGeneratedConfigurationConsumer(t *testing.T) {
// The business only changes fields/defaults/validation, not the loading flow. // The business only changes fields/defaults/validation, not the loading flow.
writeFile(t, filepath.Join(dir, "internal/config/config.go"), `package config writeFile(t, filepath.Join(dir, "internal/config/config.go"), `package config
import("git.yuntuo.hk/k2safe/go-omni/config";"git.yuntuo.hk/k2safe/go-omni/logx") import("git.yuntuo.hk/k2safe/go-omni/config";"git.yuntuo.hk/k2safe/go-omni/logx")
type Config struct { Server config.Config `+"`json:\"server\"`"+`;Limit int `+"`json:\"limit\" envDefault:\"10\"`"+` } type Config struct { Server config.Config `+"`json:\"server\"`"+`;Limit int `+"`json:\"limit\"`"+` }
func Default()Config{ c:=config.Default();c.Log.File="./var/log/app.log";c.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:10};return Config{Server:c} } func Default()Config{ c:=config.Default();c.Log.File="./var/log/app.log";c.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:10};return Config{Server:c,Limit:10} }
func(c Config)Validate()error{if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)};if c.Limit<0{return config.Invalid("limit","must not be negative")};return nil} func(c Config)Validate()error{if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)};if c.Limit<0{return config.Invalid("limit","must not be negative")};return nil}
`) `)
writeFile(t, filepath.Join(dir, "internal/config/loader.go"), `package config writeFile(t, filepath.Join(dir, "internal/config/loader.go"), `package config
@@ -111,3 +111,26 @@ func TestConfigLoadMigrationGuard(t *testing.T) {
} }
} }
} }
func TestRemovedEnvironmentTagsRejectBeforeGeneration(t *testing.T) {
dir := filepath.Join(t.TempDir(), "app")
o := Options{Dir: dir, Module: "example.com/migrate", FrameworkDir: frameworkRoot(t)}
if _, err := New(o); err != nil {
t.Fatal(err)
}
o.API = filepath.Join(dir, "api/service.api")
path := filepath.Join(dir, "internal/config/config.go")
source := readFile(t, path)
writeFile(t, path, strings.Replace(source, "json:\"server\"", "json:\"server\" env:\"OLD_SERVER\"", 1))
before := snapshot(t, dir)
for _, mode := range []string{"dry", "generate", "check"} {
o.DryRun = mode == "dry"
o.Check = mode == "check"
if _, err := Generate(o); err == nil || !strings.Contains(err.Error(), "env/envDefault") {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("migration rejection wrote files")
}
}
}

View File

@@ -19,7 +19,10 @@ import (
) )
const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" const frameworkModule = "git.yuntuo.hk/k2safe/go-omni"
const minimumVersion = "v0.20.0-preview"
// Runtime-only releases need not raise the managed template compatibility floor.
const minimumVersion = "v0.22.0-preview"
const currentVersion = "v0.22.0-preview"
const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n" const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n"
type Options struct { type Options struct {
@@ -155,7 +158,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
if !filepath.IsAbs(path) { if !filepath.IsAbs(path) {
path = filepath.Join(directory, path) path = filepath.Join(directory, path)
} }
if source, err := os.ReadFile(filepath.Join(path, "config", "loader.go")); err == nil && strings.Contains(string(source), "func NewLoader(") { if source, err := os.ReadFile(filepath.Join(path, "config", "references.go")); err == nil && strings.Contains(string(source), "func WithLookupEnv(") {
localCurrent = true localCurrent = true
} }
} }
@@ -265,7 +268,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
return Output{}, errors.New("generation configuration changed during generation; retry") return Output{}, errors.New("generation configuration changed during generation; retry")
} }
out, err := apply(directory, o.Module, entry, generated, seeds, o.Check, o.DryRun, o.profile(), snapshotContract(c.Spec), o.lockHeld) out, err := apply(directory, o.Module, entry, generated, seeds, o.Check, o.DryRun, o.profile(), snapshotContract(c.Spec), o.lockHeld)
expectedVersion := minimumVersion expectedVersion := currentVersion
if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" && semver.IsValid(info.Main.Version) { if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" && semver.IsValid(info.Main.Version) {
expectedVersion = info.Main.Version expectedVersion = info.Main.Version
} }
@@ -304,7 +307,7 @@ func moduleFile(o Options) ([]byte, error) {
if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" { if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" {
version = info.Main.Version version = info.Main.Version
} else { } else {
version = minimumVersion version = currentVersion
} }
} }
if !semver.IsValid(version) || semver.Major(version) != "v0" && semver.Major(version) != "v1" { if !semver.IsValid(version) || semver.Major(version) != "v0" && semver.Major(version) != "v1" {

View File

@@ -63,18 +63,23 @@ func TestAPIInfrastructureConsumers(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
test := `package config test := `package config
import("testing";"os") import("testing";"os";"strings")
func TestConfiguration(t *testing.T){ func TestConfiguration(t *testing.T){
t.Setenv("GO_OMNI_DB_DSN","test:test@tcp(127.0.0.1:1)/test") t.Setenv("GO_OMNI_DB_DATABASE","test");t.Setenv("GO_OMNI_DB_USER","test");t.Setenv("GO_OMNI_DB_PASSWORD","test")
t.Setenv("GO_OMNI_STORAGE_SIGNING_KEY","01234567890123456789012345678901") t.Setenv("GO_OMNI_STORAGE_SIGNING_KEY","01234567890123456789012345678901")
t.Setenv("GO_OMNI_STORAGE_ACCESS_KEY","test");t.Setenv("GO_OMNI_STORAGE_SECRET_KEY","test") t.Setenv("GO_OMNI_STORAGE_ACCESS_KEY","test");t.Setenv("GO_OMNI_STORAGE_SECRET_KEY","test")
t.Setenv("GO_OMNI_STORAGE_BUCKET","test");t.Setenv("GO_OMNI_STORAGE_ENDPOINT","ENDPOINT") t.Setenv("GO_OMNI_STORAGE_BUCKET","test");t.Setenv("GO_OMNI_STORAGE_ENDPOINT","ENDPOINT")
t.Setenv("GO_OMNI_STORAGE_ACCOUNT_ID","01234567890123456789012345678901") t.Setenv("GO_OMNI_STORAGE_ACCOUNT_ID","01234567890123456789012345678901")
cfg,err:=Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)} cfg,err:=Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
if cfg.Server.Log.File==""{t.Fatal("file logger absent")} if cfg.Server.Log.File==""{t.Fatal("file logger absent")}
t.Setenv("GO_OMNI_LOG_OUTPUT","stdout");cfg,err=Load("");if err!=nil||cfg.Server.Log.File!=""{t.Fatalf("override: %v",err)} t.Setenv("GO_OMNI_LOG_OUTPUT","stdout");t.Setenv("GO_OMNI_SERVER_LOG_FILE","./implicit.log")
t.Setenv("GO_OMNI_SERVER_LOG_FILE","./explicit.log");cfg,err=Load("");if err!=nil||cfg.Server.Log.File!="./explicit.log"{t.Fatalf("canonical log file: %v",err)} cfg,err=Load("../../etc/config.yaml");if err!=nil||cfg.Server.Log.File!=Default().Server.Log.File{t.Fatalf("implicit environment override: %v",err)}
t.Setenv("GO_OMNI_SERVER_LOG_OUTPUT","stderr");cfg,err=Load("");if err!=nil||cfg.Server.Log.Output!="stderr"{t.Fatalf("canonical output: %v",err)} raw,err:=os.ReadFile("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
// Add explicit fields while keeping storage and DB requirements.
raw=[]byte(strings.Replace(string(raw),"server:\n","server:\n log:\n output: '${OUTPUT:-stdout}'\n file: ''\n rotation:\n enabled: false\n",1))
path:="explicit.yaml";if err=os.WriteFile(path,raw,0600);err!=nil{t.Fatal(err)};defer os.Remove(path)
cfg,err=Load(path);if err!=nil||cfg.Server.Log.Output!="stdout"||cfg.Server.Log.File!=""||cfg.Server.Log.Rotation.Enabled{t.Fatalf("explicit file override: %v",err)}
_=os.Getenv("unused") _=os.Getenv("unused")
}` }`
endpoint := "" endpoint := ""

View File

@@ -19,8 +19,10 @@ import (
"MODULE/internal/svc" "MODULE/internal/svc"
) )
func main(){ func main(){
path:=flag.String("config","etc/config.yaml","configuration file; empty uses defaults and environment") path:=flag.String("config","etc/config.yaml","configuration file; empty uses only Default()")
check:=flag.Bool("check-config",false,"validate configuration and exit without starting dependencies")
flag.Parse() flag.Parse()
if *check {if _,err:=config.Load(*path);err!=nil{log.Fatal(err)};return}
ctx,stop:=signal.NotifyContext(context.Background(),os.Interrupt,syscall.SIGTERM) ctx,stop:=signal.NotifyContext(context.Background(),os.Interrupt,syscall.SIGTERM)
defer stop() defer stop()
if err:=run(ctx,*path);err!=nil{log.Fatal(err)} if err:=run(ctx,*path);err!=nil{log.Fatal(err)}
@@ -58,7 +60,7 @@ Unimplemented/nil factories fail startup and clean registered dependencies.
Existing projects retain middleware: handlers until explicitly migrated. Existing projects retain middleware: handlers until explicitly migrated.
Ordinary Logic uses the generated Request/Response types directly. Where an Ordinary Logic uses the generated Request/Response types directly. Where an
independent business type is needed, use ordinary Go or the optional copyx package. independent business type is needed, use ordinary Go or the optional copyx package.
Configuration loading is generator-managed in internal/config/loadGen.go. Extend fields/defaults/Validate in config.go; additional decoders belong in loader.go/NewLoader. JSON/YAML/XML use one pipeline. See the framework CONFIG.md. Configuration loading is generator-managed in internal/config/loadGen.go. Extend fields/defaults/Validate in config.go; additional decoders belong in loader.go/NewLoader. JSON/YAML/XML use explicit ${NAME} / ${NAME:-fallback} references; fixed values are never overridden by inferred environment names. Use -check-config to validate without opening dependencies. See the framework CONFIG.md.
There are no mappings/mappingFiles settings or generated mapping files in v0.19. There are no mappings/mappingFiles settings or generated mapping files in v0.19.
See COPY.md in the framework repository for copy policies and migration steps, See COPY.md in the framework repository for copy policies and migration steps,
@@ -77,7 +79,7 @@ Configure server.endpoints to disable or move each built-in route. Health
responseMode is app by default (project format), or raw for plain health data. responseMode is app by default (project format), or raw for plain health data.
Probe HTTP statuses remain 200/503 even when business errors use HTTP 200. Probe HTTP statuses remain 200/503 even when business errors use HTTP 200.
OpenAPI remains an unwrapped document; disabled endpoints reserve no paths. OpenAPI remains an unwrapped document; disabled endpoints reserve no paths.
Set GO_OMNI_TRANSPORT=native to select the native transport without changing services. The default file explicitly references GO_OMNI_TRANSPORT; set it to native or edit server.transport.
The greeting starter works immediately; other new operation stubs return 501 until The greeting starter works immediately; other new operation stubs return 501 until
their business logic is implemented. Authentication hooks also fail closed with 501. their business logic is implemented. Authentication hooks also fail closed with 501.
@@ -139,7 +141,7 @@ Full reference: https://git.yuntuo.hk/k2safe/go-omni/src/branch/main/DEVELOPMENT
api/ # editable contract and imported types api/ # editable contract and imported types
main.go # GENERATED startup, lifecycle and route registration main.go # GENERATED startup, lifecycle and route registration
etc/config.yaml # lowerCamelCase configuration etc/config.yaml # lowerCamelCase configuration
internal/config/ # defaults -> file -> environment internal/config/ # defaults -> file references -> typed binding -> validation
internal/svc/ # dependency initialization and cleanup internal/svc/ # dependency initialization and cleanup
internal/types/*Gen.go # DTOs split by source API, one types package internal/types/*Gen.go # DTOs split by source API, one types package
internal/svc/runtimeGen.go # GENERATED lifecycle and infrastructure startup internal/svc/runtimeGen.go # GENERATED lifecycle and infrastructure startup
@@ -240,13 +242,16 @@ func infrastructureREADME(o Options) string {
Selected database: %s; storage: %s; logging: %s/%s. Selected database: %s; storage: %s; logging: %s/%s.
MySQL uses database/sql through deps.DB, including InTx and a readiness probe. MySQL uses database/sql through deps.DB, including InTx and a readiness probe.
Use structured database/mysql configuration and GO_OMNI_DB_PASSWORD. Other connection/pool/TLS settings live in etc/config.yaml and GO_OMNI_DB_*. Use structured database/mysql configuration. The generated etc/config.yaml explicitly
references GO_OMNI_DB_PASSWORD and other required connection values; add fixed
values or explicit references for pool/TLS fields. No implicit environment overlay.
No tables are invented or migrated automatically from request/response DTOs. No tables are invented or migrated automatically from request/response DTOs.
The none/memory selection opens no database; implement repositories per business. The none/memory selection opens no database; implement repositories per business.
With storage enabled, services receive deps.Files, Uploads, Downloads and With storage enabled, services receive deps.Files, Uploads, Downloads and
UploadLinks capability interfaces. Local signed GET/HEAD/PUT uses /storage/object; UploadLinks capability interfaces. Local signed GET/HEAD/PUT uses /storage/object;
set GO_OMNI_STORAGE_SIGNING_KEY (32+ bytes) and GO_OMNI_STORAGE_PUBLIC_URL. the generated file references GO_OMNI_STORAGE_SIGNING_KEY (32+ bytes) and
GO_OMNI_STORAGE_PUBLIC_URL. File values may instead be fixed explicitly.
Cloud credentials use GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY; set bucket and endpoint Cloud credentials use GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY; set bucket and endpoint
or R2 accountId as appropriate. Signed URLs grant temporary access: authenticate or R2 accountId as appropriate. Signed URLs grant temporary access: authenticate
and authorize before issuing them. No public upload-link route is added. and authorize before issuing them. No public upload-link route is added.

View File

@@ -6,28 +6,26 @@ import (
"example.com/configuration/internal/config" "example.com/configuration/internal/config"
omniConfig "git.yuntuo.hk/k2safe/go-omni/config" omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
"os" "os"
"os/exec"
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
) )
func TestGeneratedConfiguration(t *testing.T) { func TestGeneratedConfiguration(t *testing.T) {
for ext, source := range map[string]string{ for ext, source := range map[string]string{"json": `{"limit":"${LIMIT:-7}"}`, "yaml": "limit: '${LIMIT:-7}'\n", "xml": "<config><limit>${LIMIT:-7}</limit></config>", "custom": `{"limit":"${LIMIT:-7}"}`} {
"json": `{"limit":7}`, "yaml": "limit: 7\n", "xml": "<config><limit>7</limit></config>", "custom": `{"limit":7}`,
} {
t.Run(ext, func(t *testing.T) { t.Run(ext, func(t *testing.T) {
t.Setenv("LIMIT", "")
t.Setenv("GO_OMNI_LIMIT", "999")
path := filepath.Join(t.TempDir(), "app."+ext) path := filepath.Join(t.TempDir(), "app."+ext)
if err := os.WriteFile(path, []byte(source), 0600); err != nil { if err := os.WriteFile(path, []byte(source), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
cfg, err := config.Load(path) cfg, err := config.Load(path)
if err != nil { if err != nil || cfg.Limit != 7 {
t.Fatal(err) t.Fatal(cfg.Limit, err)
} }
if cfg.Limit != 7 { t.Setenv("LIMIT", "0")
t.Fatal(cfg.Limit)
}
t.Setenv("GO_OMNI_LIMIT", "0")
cfg, err = config.Load(path) cfg, err = config.Load(path)
if err != nil || cfg.Limit != 0 { if err != nil || cfg.Limit != 0 {
t.Fatal(cfg.Limit, err) t.Fatal(cfg.Limit, err)
@@ -36,24 +34,60 @@ func TestGeneratedConfiguration(t *testing.T) {
} }
t.Setenv("GO_OMNI_LOG_OUTPUT", "stderr") t.Setenv("GO_OMNI_LOG_OUTPUT", "stderr")
cfg, err := config.Load("") cfg, err := config.Load("")
if err != nil { if err != nil || cfg.Limit != 10 || cfg.Server.Log.File != "./var/log/app.log" || !cfg.Server.Log.Rotation.Enabled {
t.Fatal(err) t.Fatal("implicit overlay", err)
}
if cfg.Limit != 10 || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled {
t.Fatal("defaults/log normalization", cfg, err)
} }
path := filepath.Join(t.TempDir(), "bad.yaml") path := filepath.Join(t.TempDir(), "bad.yaml")
os.WriteFile(path, []byte("limit: -1\n"), 0600) os.WriteFile(path, []byte("limit: '${LIMIT}'\n"), 0600)
// Validation must run before svc.Initialize; the project initializer panics t.Setenv("LIMIT", "-1")
// below if reached, so this also detects accidentally starting dependencies. // svc.Init panics if reached. Both invalid startup and successful check must
// stop before opening the configured file logger or project dependencies.
err = run(context.Background(), path) err = run(context.Background(), path)
var diagnostic *omniConfig.Error var d *omniConfig.Error
if !errors.As(err, &diagnostic) || diagnostic.Field != "limit" || diagnostic.File != path || diagnostic.Line != 1 { if !errors.As(err, &d) || d.Field != "limit" || d.File != path || d.Line != 1 || d.Env != "LIMIT" {
t.Fatal(err) t.Fatal(err)
} }
t.Setenv("GO_OMNI_LIMIT", "private-not-a-number") t.Setenv("LIMIT", "private-invalid")
_, err = config.Load(path) _, err = config.Load(path)
if !errors.As(err, &diagnostic) || diagnostic.Env != "GO_OMNI_LIMIT" || strings.Contains(err.Error(), "private-not-a-number") { if !errors.As(err, &d) || d.Env != "LIMIT" || strings.Contains(err.Error(), "private-invalid") {
t.Fatal(err) t.Fatal(err)
} }
} }
func TestConfigurationCheckProcess(t *testing.T) {
exe := filepath.Join(t.TempDir(), "server")
cmd := exec.Command("go", "build", "-o", exe, ".")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(err, string(out))
}
dir := t.TempDir()
path := filepath.Join(dir, "selected.yaml")
if err := os.WriteFile(path, []byte("limit: '${LIMIT:-0}'\nserver:\n address: 127.0.0.1:1\n"), 0600); err != nil {
t.Fatal(err)
}
// A broken neighboring default/local/test file must never be layered in.
for _, name := range []string{"config.yaml", "config.local.yaml", "config.test.yaml"} {
os.WriteFile(filepath.Join(dir, name), []byte("broken: ["), 0600)
}
cmd = exec.Command(exe, "-config", path, "-check-config")
cmd.Dir = dir
if out, err := cmd.CombinedOutput(); err != nil || len(out) != 0 {
t.Fatal(err, string(out))
}
if _, err := os.Stat(filepath.Join(dir, "var")); !os.IsNotExist(err) {
t.Fatal("check opened logger/dependencies")
}
os.WriteFile(path, []byte("limit: '${MISSING_REQUIRED}'"), 0600)
cmd = exec.Command(exe, "-config", path, "-check-config")
cmd.Dir = dir
cmd.Env = append(os.Environ(), "MISSING_REQUIRED=")
out, err := cmd.CombinedOutput()
if err == nil || !strings.Contains(string(out), "MISSING_REQUIRED") || strings.Contains(string(out), "panic") {
t.Fatal(err, string(out))
}
cmd = exec.Command(exe, "-config", path)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "MISSING_REQUIRED=")
if out, err := cmd.CombinedOutput(); err == nil || strings.Contains(string(out), "panic") {
t.Fatal(err, string(out))
}
}

View File

@@ -123,7 +123,7 @@ func TestGeneratedTransfers(t *testing.T) {
func configFile(t *testing.T, address string, fail bool) string { func configFile(t *testing.T, address string, fail bool) string {
t.Helper() t.Helper()
path := filepath.Join(t.TempDir(), "config.json") path := filepath.Join(t.TempDir(), "config.json")
data, _ := json.Marshal(map[string]any{"server": map[string]any{"address": address}, "failInit": fail}) data, _ := json.Marshal(map[string]any{"server": map[string]any{"address": address, "transport": "${GO_OMNI_TRANSPORT:-nethttp}"}, "failInit": fail})
if err := os.WriteFile(path, data, 0600); err != nil { if err := os.WriteFile(path, data, 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -24,7 +24,7 @@ func phasedConfig(t *testing.T, address, mode string, fail bool) string {
if mode == "stuck" { if mode == "stuck" {
drainTimeout = "50ms" drainTimeout = "50ms"
} }
data, _ := json.Marshal(map[string]any{"backgroundMode": mode, "failInit": fail, "server": map[string]any{"address": address, "lifecycle": map[string]string{"stopTimeout": "1s", "drainTimeout": drainTimeout, "cancelTimeout": "50ms"}}}) data, _ := json.Marshal(map[string]any{"backgroundMode": mode, "failInit": fail, "server": map[string]any{"address": address, "transport": "${GO_OMNI_TRANSPORT:-nethttp}", "lifecycle": map[string]string{"stopTimeout": "1s", "drainTimeout": drainTimeout, "cancelTimeout": "50ms"}}})
if err := os.WriteFile(path, data, 0600); err != nil { if err := os.WriteFile(path, data, 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -14,7 +14,9 @@ import (
"golang.org/x/mod/semver" "golang.org/x/mod/semver"
"os" "os"
"path/filepath" "path/filepath"
"reflect"
"sort" "sort"
"strconv"
"strings" "strings"
) )
@@ -54,6 +56,22 @@ func checkConfigLoadExtension(root string) error {
if err != nil { if err != nil {
return fmt.Errorf("parse %s: %w", rel, err) return fmt.Errorf("parse %s: %w", rel, err)
} }
var removed bool
ast.Inspect(f, func(n ast.Node) bool {
field, ok := n.(*ast.Field)
if ok && field.Tag != nil {
if raw, e := strconv.Unquote(field.Tag.Value); e == nil {
tag := reflect.StructTag(raw)
_, a := tag.Lookup("env")
_, b := tag.Lookup("envDefault")
removed = removed || a || b
}
}
return true
})
if removed {
return fmt.Errorf("%s: env/envDefault tags were removed; migrate defaults and explicit file references before generating; see CONFIG.md", rel)
}
for _, decl := range f.Decls { for _, decl := range f.Decls {
if fn, ok := decl.(*ast.FuncDecl); ok && fn.Recv == nil && fn.Name.Name == "Load" { if fn, ok := decl.(*ast.FuncDecl); ok && fn.Recv == nil && fn.Name.Name == "Load" {
return fmt.Errorf("%s: migrate the project-owned Load function before generating internal/config/loadGen.go; retain Config/Default/Validate and move decoder options to NewLoader; see CONFIG.md", rel) return fmt.Errorf("%s: migrate the project-owned Load function before generating internal/config/loadGen.go; retain Config/Default/Validate and move decoder options to NewLoader; see CONFIG.md", rel)

View File

@@ -39,7 +39,7 @@ func oldInfrastructureProject(t *testing.T) string {
raw, _ := json.Marshal(m) raw, _ := json.Marshal(m)
writeFile(t, filepath.Join(dir, manifestPath), string(raw)) writeFile(t, filepath.Join(dir, manifestPath), string(raw))
path := filepath.Join(dir, "go.mod") path := filepath.Join(dir, "go.mod")
writeFile(t, path, strings.ReplaceAll(readFile(t, path), minimumVersion, "v0.13.0-preview")) writeFile(t, path, strings.ReplaceAll(readFile(t, path), currentVersion, "v0.13.0-preview"))
return dir return dir
} }
@@ -74,7 +74,7 @@ func TestUpgradeV014PreservesExtensions(t *testing.T) {
raw, _ := json.Marshal(m) raw, _ := json.Marshal(m)
writeFile(t, filepath.Join(dir, manifestPath), string(raw)) writeFile(t, filepath.Join(dir, manifestPath), string(raw))
mod := filepath.Join(dir, "go.mod") mod := filepath.Join(dir, "go.mod")
writeFile(t, mod, strings.ReplaceAll(readFile(t, mod), minimumVersion, "v0.14.1-preview")) writeFile(t, mod, strings.ReplaceAll(readFile(t, mod), currentVersion, "v0.14.1-preview"))
owned := map[string]string{} owned := map[string]string{}
for _, name := range []string{"internal/svc/serviceContext.go", "internal/config/config.go", "etc/config.yaml"} { for _, name := range []string{"internal/svc/serviceContext.go", "internal/config/config.go", "etc/config.yaml"} {
path := filepath.Join(dir, name) path := filepath.Join(dir, name)

View File

@@ -8,7 +8,7 @@ import (
) )
func TestNewWithRemoteFrameworkVersion(t *testing.T) { func TestNewWithRemoteFrameworkVersion(t *testing.T) {
for _, flag := range [][]string{{"--framework-version", "v0.20.0-preview"}, {"--framework-version=v0.20.0-preview"}} { for _, flag := range [][]string{{"--framework-version", "v0.22.0-preview"}, {"--framework-version=v0.22.0-preview"}} {
project := filepath.Join(t.TempDir(), "billing") project := filepath.Join(t.TempDir(), "billing")
args := append([]string{"--module", "example.com/billing"}, flag...) args := append([]string{"--module", "example.com/billing"}, flag...)
if err := runNew(append(args, project)); err != nil { if err := runNew(append(args, project)); err != nil {
@@ -18,7 +18,7 @@ func TestNewWithRemoteFrameworkVersion(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if !strings.Contains(string(content), "go-omni v0.20.0-preview") || strings.Contains(string(content), "replace ") { if !strings.Contains(string(content), "go-omni v0.22.0-preview") || strings.Contains(string(content), "replace ") {
t.Fatalf("CLI did not pin remote dependency: %s", content) t.Fatalf("CLI did not pin remote dependency: %s", content)
} }
} }

View File

@@ -66,9 +66,13 @@ func TestGenerateModuleProfiles(t *testing.T) {
t.Fatal("replacement does not resolve", err) t.Fatal("replacement does not resolve", err)
} }
cfg, _ := os.ReadFile(filepath.Join(project, "etc/config.yaml")) cfg, _ := os.ReadFile(filepath.Join(project, "etc/config.yaml"))
if strings.Contains(string(cfg), "secretKey:") || strings.Contains(string(cfg), "filesToken:") { for _, line := range strings.Split(string(cfg), "\n") {
t.Fatal("secret defaults emitted") key, value, ok := strings.Cut(strings.TrimSpace(line), ":")
if ok && (key == "secretKey" || key == "filesToken" || key == "signingKey") && !strings.Contains(value, "${GO_OMNI_") {
t.Fatal("secret must be an explicit reference")
}
} }
}) })
} }
} }
@@ -76,6 +80,7 @@ func TestModuleOptionsRejectBeforeWriting(t *testing.T) {
for _, options := range []Options{ for _, options := range []Options{
{LogOutput: "invalid"}, {LogFormat: "xml"}, {Storage: "ftp"}, {LogOutput: "invalid"}, {LogFormat: "xml"}, {Storage: "ftp"},
{Storage: "local", FrameworkVersion: "v0.6.0-preview"}, {Storage: "local", FrameworkVersion: "v0.6.0-preview"},
{FrameworkVersion: "v0.21.0-preview"},
{Module: "example.com/x\nrequire malicious v1.0.0"}, {Module: "example.com/x\nrequire malicious v1.0.0"},
} { } {
options.Dir = filepath.Join(t.TempDir(), "project") options.Dir = filepath.Join(t.TempDir(), "project")

View File

@@ -34,9 +34,13 @@ type Options struct {
} }
var files = map[string]string{ var files = map[string]string{
"internal/settings/config.go": "templates/modules-settings.go.tmpl",
"internal/settings/config_test.go": "templates/modules-settings_test.go.tmpl",
"etc/config.yaml": "templates/modules-config.yaml.tmpl",
"go.mod": "templates/go.mod.tmpl", "go.mod": "templates/go.mod.tmpl",
"README.md": "templates/README.md.tmpl", "README.md": "templates/README.md.tmpl",
"cmd/server/main.go": "templates/cmd-server-main.go.tmpl", "cmd/server/main.go": "templates/modules-main.go.tmpl",
"internal/model/user.go": "templates/internal-model-user.go.tmpl", "internal/model/user.go": "templates/internal-model-user.go.tmpl",
"internal/model/errors.go": "templates/internal-model-errors.go.tmpl", "internal/model/errors.go": "templates/internal-model-errors.go.tmpl",
"internal/repository/users.go": "templates/internal-repository-users.go.tmpl", "internal/repository/users.go": "templates/internal-repository-users.go.tmpl",
@@ -109,7 +113,7 @@ func Generate(options Options) error {
if options.FrameworkDir != "" || options.FrameworkVersion == "" { if options.FrameworkDir != "" || options.FrameworkVersion == "" {
replace = frameworkReplace(directory, options.FrameworkDir) replace = frameworkReplace(directory, options.FrameworkDir)
} }
if modules && replace == "" && semver.Compare(version, modulesFrameworkVersion) < 0 && version != installedFrameworkVersion() { if replace == "" && semver.Compare(version, modulesFrameworkVersion) < 0 && version != installedFrameworkVersion() {
return fmt.Errorf("module templates require %s or newer; use --framework-dir for unpublished source", modulesFrameworkVersion) return fmt.Errorf("module templates require %s or newer; use --framework-dir for unpublished source", modulesFrameworkVersion)
} }
data := templateData{ data := templateData{

View File

@@ -39,7 +39,7 @@ func TestGenerateCreatesServiceLayout(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("read generated server: %v", err) t.Fatalf("read generated server: %v", err)
} }
for _, expected := range []string{"config.LoadFromEnv(\"\")", "logx.SetDefault(logger)", "omni.WithConfig(settings)", "servicecontext.New(ctx, logger)", "services.Health.ReadinessHandler()", "application.RunContext(ctx, settings.Address)"} { for _, expected := range []string{"settings.Load(path)", "logx.SetDefault(logger)", "omni.WithConfig(cfg.Server)", "servicecontext.New(ctx, logger, cfg)", "services.Health.ReadinessHandler()", "application.RunContext(ctx, cfg.Server.Address)"} {
if !strings.Contains(string(mainContent), expected) { if !strings.Contains(string(mainContent), expected) {
t.Fatalf("generated server missing %q: %s", expected, mainContent) t.Fatalf("generated server missing %q: %s", expected, mainContent)
} }
@@ -110,7 +110,7 @@ func TestFrameworkReplaceUsesCanonicalPaths(t *testing.T) {
} }
func TestGenerateRemoteProjectPinsVersion(t *testing.T) { func TestGenerateRemoteProjectPinsVersion(t *testing.T) {
for _, version := range []string{"v0.1.0", "v1.2.3-rc.1", "v0.0.0-20260920140624-8b19aa79eea0"} { for _, version := range []string{"v0.22.0-preview", "v1.2.3-rc.1", "v0.22.1-0.20260926000000-8b19aa79eea0"} {
t.Run(version, func(t *testing.T) { t.Run(version, func(t *testing.T) {
project := filepath.Join(t.TempDir(), "billing") project := filepath.Join(t.TempDir(), "billing")
if err := Generate(Options{Dir: project, Module: "example.com/billing", FrameworkVersion: version}); err != nil { if err := Generate(Options{Dir: project, Module: "example.com/billing", FrameworkVersion: version}); err != nil {

View File

@@ -32,46 +32,44 @@ go mod tidy
go run ./cmd/server go run ./cmd/server
``` ```
运行时配置从 `GO_OMNI_*` 环境变量加载,例如: 配置入口是 `internal/settings.Config`、`Default()`、`Validate()` 和 `etc/config.yaml`。
只有文件里的 `${NAME}` / `${NAME:-fallback}` 会读取环境变量,固定值不受进程环境覆盖。生成的文件已声明地址、网络层,以及所选数据库/存储的必要引用;其他参数直接写进文件。
```sh ```yaml
GO_OMNI_ENABLE_LOGGER=true GO_OMNI_LOG_LEVEL=debug go run ./cmd/server server:
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
transport: "${GO_OMNI_TRANSPORT:-nethttp}"
log:
level: debug
shutdownTimeout: 10s
``` ```
`GO_OMNI_HTTP_ADDR` 可以修改监听地址。生成项目默认提供 `/health/live` 和 用 `-config` 选择一个 YAML/JSON/XML 文件;`-config ''` 只使用 Default();`-check-config` 仅加载和校验,不创建依赖或监听:
`/health/ready`,日志、请求 ID 和优雅退出等基础能力由 GoOmni 统一初始化;业务依赖集中放在项目自己的 `internal/servicecontext`。
服务使用 `signal.NotifyContext` 处理 `SIGINT` / `SIGTERM`。部署时构建并运行服务二进制,
使信号直接到达服务进程:
```sh ```sh
go run ./cmd/server -config etc/config.yaml -check-config
go build -o server ./cmd/server go build -o server ./cmd/server
./server ./server -config etc/config.yaml
``` ```
`GO_OMNI_SHUTDOWN_TIMEOUT` 控制等待请求完成的时间,默认 `10s`。 服务通过 `signal.NotifyContext` 处理 `SIGINT` / `SIGTERM`,先排空 HTTP 请求,再关闭依赖和日志。生成项目默认提供 `/health/live` 和 `/health/ready`,业务依赖集中在 `internal/servicecontext`。
使用支持统一网络配置的框架版本时,设置 `GO_OMNI_TRANSPORT=native` 即可切换为自研 HTTP/1.1; 通过文件中的引用设置 `GO_OMNI_TRANSPORT=native` 可切换自研 HTTP/1.1,默认 nethttp。HTTPS 在文件的 `server.tls.certFile` / `server.tls.keyFile` 成对配置;实验 HTTP/2 选择 `http2` 且必须配置 TLS。不会自动读取未在文件声明的 TLS 环境变量。
默认 `nethttp`。`GO_OMNI_TLS_CERT_FILE` 和 `GO_OMNI_TLS_KEY_FILE` 配对启用 HTTPS。
实验版 HTTP/2 使用 `GO_OMNI_TRANSPORT=http2`,必须同时配置 TLS 证书和私钥。
业务 Handler / Service / Repository 不需要修改;三种适配器共用上述退出等待时间。
{{if eq .Database "mysql"}} {{if eq .Database "mysql"}}
`/health/ready` 已注册数据库探针:连接失败返回 `503`,数据库恢复后重新返回 `200`。 `/health/ready` 已注册数据库探针,连接失败返回 503,恢复后返回 200;`/health/live` 只检查进程存活。连接池参数直接写入 `database`,缺失字段保留 Default():
`/health/live` 始终只检查服务存活。`GO_OMNI_DB_PING_TIMEOUT` 限制初始化检查和数据库探针的等待时间。
退出时先排空 HTTP 请求,再关闭数据库连接池,最后关闭日志。
数据库连接池参数: ```yaml
database:
```text driver: mysql
GO_OMNI_DB_MAX_OPEN_CONNS=10 dsn: "${GO_OMNI_DB_DSN}"
GO_OMNI_DB_MAX_IDLE_CONNS=5 maxOpenConns: 10
GO_OMNI_DB_CONN_MAX_LIFETIME=30m maxIdleConns: 5
GO_OMNI_DB_CONN_MAX_IDLE_TIME=5m connMaxLifetime: 30m
GO_OMNI_DB_PING_TIMEOUT=5s connMaxIdleTime: 5m
pingTimeout: 5s
``` ```
`MAX_IDLE_CONNS=0` 禁用空闲连接,`MAX_OPEN_CONNS=0` 表示不限制连接数。 `maxIdleConns: 0` 禁用空闲连接;此项目要求 maxOpenConns 为正数。项目启用 MySQL parseTime,并按 UTC 读取 DATETIME;JSON 字段保持 lowerCamelCase。
项目会启用 MySQL `parseTime` 并按 UTC 读取 DATETIME 字段,JSON 字段保持 lowerCamelCase。
事务内复用 Repository(在业务代码中导入 `database/sql` 和项目的 `internal/repository`): 事务内复用 Repository(在业务代码中导入 `database/sql` 和项目的 `internal/repository`):

View File

@@ -11,8 +11,8 @@ import (
) )
type Config struct { type Config struct {
Server config.Config `json:"server"` Server config.Config `json:"server"`
{{if eq .Database "mysql"}}Database mysqladapter.Config `json:"database" env:"GO_OMNI_DB"`{{end}} {{if eq .Database "mysql"}}Database mysqladapter.Config `json:"database"`{{end}}
{{if ne .Storage "none"}}Storage StorageConfig `json:"storage" env:"GO_OMNI_STORAGE"`{{end}} {{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}}
} }
func Default() Config { func Default() Config {
server:=config.Default() server:=config.Default()
@@ -54,12 +54,12 @@ type StorageConfig struct {
ContentTypes []string `json:"contentTypes"` ContentTypes []string `json:"contentTypes"`
LinkTTL time.Duration `json:"linkTtl"` LinkTTL time.Duration `json:"linkTtl"`
RequestTimeout time.Duration `json:"requestTimeout"` RequestTimeout time.Duration `json:"requestTimeout"`
// Secrets are environment-only and cannot be populated by config files. // File values may explicitly reference environment variables. Never log this struct.
FilesToken string `json:"-" env:"GO_OMNI_FILES_TOKEN,secret"` FilesToken string `json:"filesToken"`
SigningKey string `json:"-" env:"GO_OMNI_STORAGE_SIGNING_KEY,secret"` SigningKey string `json:"signingKey"`
AccessKey string `json:"-" env:"GO_OMNI_STORAGE_ACCESS_KEY,secret"` AccessKey string `json:"accessKey"`
SecretKey string `json:"-" env:"GO_OMNI_STORAGE_SECRET_KEY,secret"` SecretKey string `json:"secretKey"`
SessionToken string `json:"-" env:"GO_OMNI_STORAGE_SESSION_TOKEN,secret"` SessionToken string `json:"sessionToken"`
} }
func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}} func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}}
func (s StorageConfig) Validate()error { func (s StorageConfig) Validate()error {
@@ -67,14 +67,14 @@ func (s StorageConfig) Validate()error {
if err:=s.Policy().Validate();err!=nil{return err} if err:=s.Policy().Validate();err!=nil{return err}
if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")} if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")}
if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")} if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")}
{{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("set GO_OMNI_FILES_TOKEN to a token of at least 32 bytes without whitespace")}{{end}} {{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("storage.filesToken requires at least 32 bytes without whitespace")}{{end}}
{{if eq .Storage "local"}} {{if eq .Storage "local"}}
if s.Root=="" {return errors.New("storage.root is required")} if s.Root=="" {return errors.New("storage.root is required")}
u,err:=url.Parse(s.PublicURL) u,err:=url.Parse(s.PublicURL)
if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")} if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")}
if len(s.SigningKey)<32 {return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")} if len(s.SigningKey)<32 {return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")}
{{else}} {{else}}
if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region and GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY are required")} if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region, accessKey and secretKey are required")}
{{if eq .Storage "r2"}} {{if eq .Storage "r2"}}
if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")} if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")}
if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")} if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")}

View File

@@ -1,13 +1,8 @@
package config package config
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config" import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
// Load resolves explicit file references, binds onto Default and validates.
// Load is maintained by the generator. Extend Config/Default/Validate in // Ordinary projects only edit Config, Default, Validate and the selected file.
// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline. func Load(path string)(Config,error){
func Load(path string) (Config, error) { loader,err:=NewLoader();if err!=nil{return Config{},err}
loader,err:=NewLoader() return omniConfig.LoadProject(path,Default(),loader)
if err!=nil{return Config{},err}
return omniConfig.LoadProject(path,Default(),omniConfig.EnvOptions{
Prefix:"GO_OMNI_",Empty:"ignore",Aliases:omniConfig.ServerEnvAliases("GO_OMNI_","server"),
},loader)
} }

View File

@@ -1,57 +0,0 @@
package main
import (
"context"
"log"
"os"
"os/signal"
"strings"
"syscall"
omni "git.yuntuo.hk/k2safe/go-omni"
"git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/logx"
"{{.Module}}/internal/handler"
"{{.Module}}/internal/service"
"{{.Module}}/internal/servicecontext"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
settings, err := config.LoadFromEnv("")
if err != nil {
return err
}
if strings.TrimSpace(settings.Log.Service) == "" {
settings.Log.Service = "{{.Module}}"
}
logger, err := logx.New(settings.Log)
if err != nil {
return err
}
defer logger.Close()
// Access logs and business logs share the same configured sink.
logx.SetDefault(logger)
services, err := servicecontext.New(ctx, logger)
if err != nil { return err }
defer services.Close()
userService := service.NewUsers(services)
userHandler := handler.NewUsers(userService)
application := omni.New(omni.WithConfig(settings))
application.OnShutdown(logger.Close)
application.OnShutdown(services.Close)
application.GET("/health/live", services.Health.LivenessHandler())
application.GET("/health/ready", services.Health.ReadinessHandler())
userHandler.RegisterRoutes(application)
return application.RunContext(ctx, settings.Address)
}

View File

@@ -13,6 +13,7 @@ import (
"git.yuntuo.hk/k2safe/go-omni/database" "git.yuntuo.hk/k2safe/go-omni/database"
"git.yuntuo.hk/k2safe/go-omni/logx" "git.yuntuo.hk/k2safe/go-omni/logx"
"{{.Module}}/internal/model" "{{.Module}}/internal/model"
"{{.Module}}/internal/settings"
"{{.Module}}/internal/repository" "{{.Module}}/internal/repository"
"{{.Module}}/internal/servicecontext" "{{.Module}}/internal/servicecontext"
) )
@@ -23,15 +24,12 @@ func databaseContext(t *testing.T) *servicecontext.Context {
t.Helper() t.Helper()
dsn := os.Getenv("GO_OMNI_TEST_MYSQL_DSN") dsn := os.Getenv("GO_OMNI_TEST_MYSQL_DSN")
if dsn == "" { t.Skip("set GO_OMNI_TEST_MYSQL_DSN to a dedicated test database") } if dsn == "" { t.Skip("set GO_OMNI_TEST_MYSQL_DSN to a dedicated test database") }
for _, key := range []string{"MAX_OPEN_CONNS", "MAX_IDLE_CONNS", "CONN_MAX_LIFETIME", "CONN_MAX_IDLE_TIME", "PING_TIMEOUT"} {
t.Setenv("GO_OMNI_DB_"+key, "")
}
t.Setenv("GO_OMNI_DB_DSN", dsn) t.Setenv("GO_OMNI_DB_DSN", dsn)
t.Setenv("GO_OMNI_DB_DRIVER", "mysql")
logger, err := logx.New(logx.Config{Writer: io.Discard}) logger, err := logx.New(logx.Config{Writer: io.Discard})
if err != nil { t.Fatal(err) } if err != nil { t.Fatal(err) }
t.Cleanup(func() { _ = logger.Close() }) t.Cleanup(func() { _ = logger.Close() })
services, err := servicecontext.New(context.Background(), logger) cfg,err:=settings.Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
services, err := servicecontext.New(context.Background(), logger,cfg)
if err != nil { t.Fatal(err) } if err != nil { t.Fatal(err) }
t.Cleanup(func() { _ = services.Close() }) t.Cleanup(func() { _ = services.Close() })
return services return services

View File

@@ -2,6 +2,7 @@ package servicecontext
import ( import (
"context" "context"
"{{.Module}}/internal/settings"
{{if and .Modules (ne .Storage "none")}}"errors" {{if and .Modules (ne .Storage "none")}}"errors"
"net/http" "net/http"
"sync" "sync"
@@ -38,16 +39,16 @@ type Context struct {
// New initializes the project's dependencies before the HTTP listener opens. // New initializes the project's dependencies before the HTTP listener opens.
// The caller owns Logger; Close releases only resources created here. // The caller owns Logger; Close releases only resources created here.
func New(ctx context.Context, logger *logx.Logger) (*Context, error) { func New(ctx context.Context, logger *logx.Logger, project settings.Config) (*Context, error) {
if ctx == nil { ctx = context.Background() } if ctx == nil { ctx = context.Background() }
if err := ctx.Err(); err != nil { return nil, err } if err := ctx.Err(); err != nil { return nil, err }
checker, err := health.New() checker, err := health.New()
if err != nil { return nil, err } if err != nil { return nil, err }
services := &Context{Logger: logger, Health: checker} services := &Context{Logger: logger, Health: checker}
{{if eq .Database "mysql"}} {{if eq .Database "mysql"}}
cfg, err := database.ConfigFromEnv("") cfg:=project.Database
if err != nil { return nil, err } if err:=cfg.Validate();err!=nil{return nil,err}
if cfg.Driver != "mysql" { return nil, fmt.Errorf("this project requires GO_OMNI_DB_DRIVER=mysql") } if cfg.Driver != "mysql" { return nil, fmt.Errorf("this project requires database.driver=mysql") }
dsn, err := mysql.ParseDSN(cfg.DSN) dsn, err := mysql.ParseDSN(cfg.DSN)
if err != nil { return nil, fmt.Errorf("parse mysql connection settings: %w", err) } if err != nil { return nil, fmt.Errorf("parse mysql connection settings: %w", err) }
if dsn.DBName == "" { return nil, fmt.Errorf("mysql connection settings require a database name") } if dsn.DBName == "" { return nil, fmt.Errorf("mysql connection settings require a database name") }

View File

@@ -2,7 +2,7 @@
本项目选用日志输出 `{{.LogOutput}}`、格式 `{{.LogFormat}}`、对象存储 `{{.Storage}}`。数据库选择 `{{.Database}}` 独立于对象存储。 本项目选用日志输出 `{{.LogOutput}}`、格式 `{{.LogFormat}}`、对象存储 `{{.Storage}}`。数据库选择 `{{.Database}}` 独立于对象存储。
配置入口为 `etc/config.yaml`,通过 `go run ./cmd/server -config /path/to/config.yaml` 指定其他 JSON/YAML 文件;`-config ''` 只用代码默认值和环境变量。优先级:默认值 → 文件 → 非空环境变量。未知字段、无效参数会在监听前拒绝。 配置入口为 `etc/config.yaml`,通过 `go run ./cmd/server -config /path/to/config.yaml` 指定其他 JSON/YAML/XML 文件;`-config ''` 只用代码默认值。文件用 `${NAME}` 或 `${NAME:-fallback}` 显式引用变量,固定值不受隐式环境覆盖;`-check-config` 只检查、不初始化依赖。未知字段、无效参数会在监听前拒绝。
```sh ```sh
go mod tidy go mod tidy
@@ -24,7 +24,7 @@ go run ./cmd/server
{{if eq .Database "mysql"}}还需按 README 配置独立 MySQL 数据库并应用 schema.sql;应用不自动迁移。{{end}} {{if eq .Database "mysql"}}还需按 README 配置独立 MySQL 数据库并应用 schema.sql;应用不自动迁移。{{end}}
默认仅监听 `127.0.0.1:8080`。`GO_OMNI_HTTP_ADDR` 覆盖地址;本地签名 URL 的外部地址通过 `GO_OMNI_STORAGE_PUBLIC_URL` 单独配置,始终以 `/storage/object` 结尾,不使用请求 Host 自动拼接可信链接。 默认仅监听 `127.0.0.1:8080`。文件中的 `${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}` 设置地址;本地签名 URL 的外部地址通过 `GO_OMNI_STORAGE_PUBLIC_URL` 单独配置,始终以 `/storage/object` 结尾,不使用请求 Host 自动拼接可信链接。
## 日志 ## 日志
@@ -35,7 +35,7 @@ curl -H 'Content-Type: application/json' -H 'X-Request-ID: example-request' \
-d '{"name":"Example","email":"example@example.com"}' http://127.0.0.1:8080/users -d '{"name":"Example","email":"example@example.com"}' http://127.0.0.1:8080/users
``` ```
`GO_OMNI_LOG_LEVEL`、`GO_OMNI_LOG_FORMAT`、`GO_OMNI_LOG_FILE` 覆盖对应项;显式设置 `GO_OMNI_LOG_OUTPUT=stdout` 且未设置 LOG_FILE 时,切换到控制台并关闭文件轮转。慢请求阈值为 500ms,可通过 `GO_OMNI_SLOW_REQUEST_THRESHOLD` 调整。字段脱敏不扫描 msg/error 中的自由文本;不要把凭据或签名 URL 拼到日志中。 日志参数直接写在 `server.log`,例如 `level: debug`;也可显式写 `level: "${LOG_LEVEL:-info}"`。切换文件日志为控制台时,明确设置 `file: ""`、`output: stdout` 和 `rotation.enabled: false`,不会隐式重置其他字段。慢请求阈值使用 `server.slowRequestThreshold`,默认 500ms。字段脱敏不扫描 msg/error 自由文本;不要把凭据或签名 URL 拼到日志中。
{{if ne .Storage "none"}}## 文件示例 {{if ne .Storage "none"}}## 文件示例
@@ -66,18 +66,18 @@ curl -H "Authorization: Bearer $GO_OMNI_FILES_TOKEN" -H 'Content-Type: applicati
响应包括随机 `data.key` 和 `data.request`(method/url/headers/expiresAt)。按返回的 method 和 headers 用原始文件字节上传到 URL;不能额外包 FormData。浏览器自动填写 Content-Length,其余签名 Header 原样发送;云桶需配置允许业务域名的 CORS。云端直传只绑定声明类型,实际内容需上传后验证;本地和 API 中转上传会检查允许类型及前缀嗅探结果。未完成内容验证前不要把暂存文件当作可信文件公开。 响应包括随机 `data.key` 和 `data.request`(method/url/headers/expiresAt)。按返回的 method 和 headers 用原始文件字节上传到 URL;不能额外包 FormData。浏览器自动填写 Content-Length,其余签名 Header 原样发送;云桶需配置允许业务域名的 CORS。云端直传只绑定声明类型,实际内容需上传后验证;本地和 API 中转上传会检查允许类型及前缀嗅探结果。未完成内容验证前不要把暂存文件当作可信文件公开。
| 参数 | 默认值 / 环境变量 | | 文件字段 | 默认值 / 含义 |
| --- | --- | | --- | --- |
| 大小 | 32 MiB / GO_OMNI_STORAGE_MAX_BYTES | | `storage.maxBytes` | 32 MiB |
| 类型 | text/plain、image/png、image/jpeg、application/pdf / GO_OMNI_STORAGE_CONTENT_TYPES(逗号分隔) | | `storage.contentTypes` | text/plain、image/png、image/jpeg、application/pdf;文件数组 |
| 链接有效期 | 5m / GO_OMNI_STORAGE_LINK_TTL(1s..1h) | | `storage.linkTtl` | 5m,允许 1s..1h |
| 请求超时 | 30s / GO_OMNI_STORAGE_REQUEST_TIMEOUT(>0 且 <=5m) | | `storage.requestTimeout` | 30s,大于 0 且不超过 5m |
| 本地目录 | var/files / GO_OMNI_STORAGE_ROOT | | `storage.root` | var/files |
| 本地外部 URL | http://127.0.0.1:8080/storage/object / GO_OMNI_STORAGE_PUBLIC_URL | | `storage.publicUrl` | http://127.0.0.1:8080/storage/object |
| 云配置 | GO_OMNI_STORAGE_ENDPOINT、REGION、BUCKET、ACCOUNT_ID、SECURE | | `storage.endpoint/region/bucket/accountId/secure` | 对应平台连接配置 |
| 环境密钥 | GO_OMNI_FILES_TOKEN、GO_OMNI_STORAGE_SIGNING_KEY、GO_OMNI_STORAGE_ACCESS_KEY、SECRET_KEY、SESSION_TOKEN | | `storage.filesToken/signingKey/accessKey/secretKey/sessionToken` | 文件显式引用环境变量提供凭据 |
全部 JSON/YAML 字段使用 lowerCamelCase。密钥字段仅允许环境变量,配置文件中写入会报未知字段;不要把运行环境文件提交到仓库。S3 的 endpoint 为 host[:port],OSS 的可选 endpoint 为 HTTPS origin;R2 从 accountId 派生官方 endpoint,region 为 auto。仅 S3/MinIO 的本地测试可设置 SECURE=false。 全部 JSON/YAML/XML 字段使用 lowerCamelCase。文件可使用固定值或显式环境引用,不按字段名自动推导环境变量;凭据建议使用 `${NAME}`,不把含秘密的配置提交到仓库。S3 endpoint 为 host[:port],OSS 可选 endpoint 为 HTTPS origin;R2 从 accountId 派生官方 endpoint,region 为 auto。仅 S3/MinIO 本地测试可设置 `storage.secure: false`。
错误响应:未授权 401、无 Content-Length 411、超限 413、类型不允许 415、长度不符/非法 key 400、对象不存在 404、存储超时 504,其余内部错误统一隐藏细节。底层签名入口遵循存储协议返回 201/403 等状态,流式传输且访问日志保留实际状态。 错误响应:未授权 401、无 Content-Length 411、超限 413、类型不允许 415、长度不符/非法 key 400、对象不存在 404、存储超时 504,其余内部错误统一隐藏细节。底层签名入口遵循存储协议返回 201/403 等状态,流式传输且访问日志保留实际状态。

View File

@@ -1,68 +1,29 @@
# Non-secret settings only. Environment variables override these values. # Only fields listed here override Default(). Environment names are explicit.
server: server:
address: '127.0.0.1:8080' address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
enableLogger: true transport: "${GO_OMNI_TRANSPORT:-nethttp}"
slowRequestThreshold: 500ms
shutdownTimeout: 10s
{{if .API}} lifecycle:
stopTimeout: 5s
drainTimeout: 10s
cancelTimeout: 5s
endpoints:
liveness:
enabled: true
path: /health/live
responseMode: app
readiness:
enabled: true
path: /health/ready
responseMode: app
openapi:
enabled: true
path: /openapi.json
{{end}}
log:
service: {{printf "%q" .Module}}
level: info
format: {{.LogFormat}}
{{if eq .LogOutput "file"}} file: './var/log/server.log'
rotation:
enabled: true
maxSizeMb: 100
maxBackups: 10
maxAgeDays: 7
compress: true
{{else}} output: {{.LogOutput}}
{{end}} redactKeys: []
{{if ne .Storage "none"}}storage: {{if ne .Storage "none"}}storage:
provider: {{.Storage}} {{if not .API}} filesToken: "${GO_OMNI_FILES_TOKEN}"
maxBytes: 33554432 {{end}}{{if eq .Storage "local"}} root: "${GO_OMNI_STORAGE_ROOT:-var/files}"
contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf'] publicUrl: "${GO_OMNI_STORAGE_PUBLIC_URL:-http://127.0.0.1:8080/storage/object}"
linkTtl: 5m signingKey: "${GO_OMNI_STORAGE_SIGNING_KEY}"
requestTimeout: 30s {{else}} bucket: "${GO_OMNI_STORAGE_BUCKET}"
{{if eq .Storage "local"}} root: './var/files' accessKey: "${GO_OMNI_STORAGE_ACCESS_KEY}"
publicUrl: 'http://127.0.0.1:8080/storage/object' secretKey: "${GO_OMNI_STORAGE_SECRET_KEY}"
{{else}} bucket: '' # set GO_OMNI_STORAGE_BUCKET sessionToken: "${GO_OMNI_STORAGE_SESSION_TOKEN:-}"
secure: true {{if eq .Storage "r2"}} accountId: "${GO_OMNI_STORAGE_ACCOUNT_ID}"
{{if eq .Storage "r2"}} accountId: '' # set GO_OMNI_STORAGE_ACCOUNT_ID {{else if eq .Storage "s3"}} endpoint: "${GO_OMNI_STORAGE_ENDPOINT}"
region: auto region: "${GO_OMNI_STORAGE_REGION:-us-east-1}"
{{else if eq .Storage "s3"}} endpoint: '' # set GO_OMNI_STORAGE_ENDPOINT (host[:port]) {{else}} region: "${GO_OMNI_STORAGE_REGION:-cn-hangzhou}"
region: us-east-1
{{else}} region: cn-hangzhou
endpoint: '' # empty uses the official regional endpoint
{{end}}{{end}}{{end}} {{end}}{{end}}{{end}}
{{if and .API (eq .Database "mysql")}}database: {{if and .API (eq .Database "mysql")}}database:
host: 127.0.0.1 host: "${GO_OMNI_DB_HOST:-127.0.0.1}"
port: 3306 port: "${GO_OMNI_DB_PORT:-3306}"
database: '' # GO_OMNI_DB_DATABASE database: "${GO_OMNI_DB_DATABASE}"
user: '' # GO_OMNI_DB_USER; password via GO_OMNI_DB_PASSWORD user: "${GO_OMNI_DB_USER}"
tls: password: "${GO_OMNI_DB_PASSWORD}"
mode: required {{end}}
pool: {{if and (not .API) (eq .Database "mysql")}}database:
maxOpenConns: 10 driver: mysql
maxIdleConns: 5 dsn: "${GO_OMNI_DB_DSN}"
connMaxLifetime: 30m
connMaxIdleTime: 5m
pingTimeout: 5s
{{end}} {{end}}

View File

@@ -18,8 +18,10 @@ import (
) )
func main(){ func main(){
path:=flag.String("config","etc/config.yaml","JSON/YAML settings; empty uses defaults and environment") path:=flag.String("config","etc/config.yaml","configuration file; empty uses only Default()")
check:=flag.Bool("check-config",false,"validate without starting dependencies")
flag.Parse() flag.Parse()
if *check {if _,err:=settings.Load(*path);err!=nil{log.Fatal(err)};return}
if err:=run(*path);err!=nil{log.Fatal(err)} if err:=run(*path);err!=nil{log.Fatal(err)}
} }
func run(path string)(runErr error){ func run(path string)(runErr error){
@@ -31,7 +33,7 @@ func run(path string)(runErr error){
if err!=nil{return err} if err!=nil{return err}
defer func(){runErr=errors.Join(runErr,logger.Close())}() defer func(){runErr=errors.Join(runErr,logger.Close())}()
logx.SetDefault(logger) logx.SetDefault(logger)
services,err:=servicecontext.New(ctx,logger) services,err:=servicecontext.New(ctx,logger,cfg)
if err!=nil{return err} if err!=nil{return err}
defer func(){runErr=errors.Join(runErr,services.Close())}() defer func(){runErr=errors.Join(runErr,services.Close())}()
{{if ne .Storage "none"}}if err=services.InitStorage(ctx,cfg.Storage);err!=nil{return err}{{end}} {{if ne .Storage "none"}}if err=services.InitStorage(ctx,cfg.Storage);err!=nil{return err}{{end}}

View File

@@ -3,21 +3,19 @@ package settings
import ( import (
"errors" "errors"
"os"
"strings" "strings"
"time" "time"
{{if ne .Storage "none"}}"fmt" {{if ne .Storage "none"}}
{{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}} {{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}}
"strconv"
"git.yuntuo.hk/k2safe/go-omni/storage"{{end}} "git.yuntuo.hk/k2safe/go-omni/storage"{{end}}
{{if and .API (eq .Database "mysql")}}"git.yuntuo.hk/k2safe/go-omni/database"{{end}} {{if eq .Database "mysql"}}"git.yuntuo.hk/k2safe/go-omni/database"{{end}}
"git.yuntuo.hk/k2safe/go-omni/config" "git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/logx" {{if eq .LogOutput "file"}}"git.yuntuo.hk/k2safe/go-omni/logx"{{end}}
) )
type Config struct { type Config struct {
Server config.Config `json:"server"` Server config.Config `json:"server"`
{{if and .API (eq .Database "mysql")}}Database database.Config `json:"database"`{{end}} {{if eq .Database "mysql"}}Database database.Config `json:"database"`{{end}}
{{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}} {{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}}
} }
@@ -31,7 +29,8 @@ func Default() Config {
{{if eq .LogOutput "file"}}server.Log.File="./var/log/server.log" {{if eq .LogOutput "file"}}server.Log.File="./var/log/server.log"
server.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:100,MaxBackups:10,MaxAgeDays:7,Compress:true} server.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:100,MaxBackups:10,MaxAgeDays:7,Compress:true}
{{else}}server.Log.Output={{printf "%q" .LogOutput}}{{end}} {{else}}server.Log.Output={{printf "%q" .LogOutput}}{{end}}
return Config{Server:server{{if and .API (eq .Database "mysql")}},Database:database.DefaultConfig(){{end}}{{if ne .Storage "none"}},Storage:StorageConfig{ {{if eq .Database "mysql"}}db:=database.DefaultConfig();db.Driver="mysql"{{end}}
return Config{Server:server{{if eq .Database "mysql"}},Database:db{{end}}{{if ne .Storage "none"}},Storage:StorageConfig{
Provider:{{printf "%q" .Storage}}, Root:"./var/files", PublicURL:"http://127.0.0.1:8080/storage/object", Provider:{{printf "%q" .Storage}}, Root:"./var/files", PublicURL:"http://127.0.0.1:8080/storage/object",
Secure:true, MaxBytes:32<<20, ContentTypes:[]string{"text/plain","image/png","image/jpeg","application/pdf"}, Secure:true, MaxBytes:32<<20, ContentTypes:[]string{"text/plain","image/png","image/jpeg","application/pdf"},
LinkTTL:5*time.Minute, RequestTimeout:30*time.Second, LinkTTL:5*time.Minute, RequestTimeout:30*time.Second,
@@ -41,27 +40,10 @@ func Default() Config {
}{{end}}} }{{end}}}
} }
// Load uses defaults -> strict YAML/JSON -> nonempty environment variables. // Load uses Default(), the selected file's explicit references, and Validate.
// Pass an empty path to use only defaults and environment. func Load(path string)(Config,error){return config.LoadProject(path,Default())}
func Load(path string)(Config,error) {
cfg:=Default()
var err error
if path!="" { cfg,err=config.DecodeFile(path,cfg);if err!=nil{return Config{},err} }
cfg.Server,err=config.ApplyEnv(cfg.Server,"")
if err!=nil{return Config{},err}
// A stream explicitly selected by environment overrides a file profile.
if output:=strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT"));output!="" && strings.TrimSpace(os.Getenv("GO_OMNI_LOG_FILE"))=="" {
cfg.Server.Log.File="";cfg.Server.Log.Rotation=logx.RotationConfig{}
}
{{if ne .Storage "none"}}
if err=cfg.Storage.applyEnv();err!=nil{return Config{},err}
{{end}}
{{if and .API (eq .Database "mysql")}}cfg.Database,err=database.ApplyEnv(cfg.Database,"");if err!=nil{return Config{},err}{{end}}
if err=cfg.Validate();err!=nil{return Config{},err}
return cfg,nil
}
func (c Config) Validate()error { func (c Config) Validate()error {
{{if and .API (eq .Database "mysql")}} {{if eq .Database "mysql"}}
if err:=c.Database.Validate();err!=nil{return err} if err:=c.Database.Validate();err!=nil{return err}
if c.Database.Driver!="mysql" || c.Database.MaxOpenConns<1 {return errors.New("database requires driver mysql and a positive maxOpenConns")} if c.Database.Driver!="mysql" || c.Database.MaxOpenConns<1 {return errors.New("database requires driver mysql and a positive maxOpenConns")}
{{end}} {{end}}
@@ -85,45 +67,27 @@ type StorageConfig struct {
ContentTypes []string `json:"contentTypes"` ContentTypes []string `json:"contentTypes"`
LinkTTL time.Duration `json:"linkTtl"` LinkTTL time.Duration `json:"linkTtl"`
RequestTimeout time.Duration `json:"requestTimeout"` RequestTimeout time.Duration `json:"requestTimeout"`
// Secrets are environment-only and cannot be populated by config files. // Use explicit environment references in the configuration file for secrets.
FilesToken string `json:"-"` FilesToken string `json:"filesToken"`
SigningKey string `json:"-"` SigningKey string `json:"signingKey"`
AccessKey string `json:"-"` AccessKey string `json:"accessKey"`
SecretKey string `json:"-"` SecretKey string `json:"secretKey"`
SessionToken string `json:"-"` SessionToken string `json:"sessionToken"`
} }
func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}} func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}}
func (s *StorageConfig) applyEnv()error {
values:=map[string]*string{
"GO_OMNI_STORAGE_ROOT":&s.Root,"GO_OMNI_STORAGE_PUBLIC_URL":&s.PublicURL,
"GO_OMNI_STORAGE_ENDPOINT":&s.Endpoint,"GO_OMNI_STORAGE_REGION":&s.Region,
"GO_OMNI_STORAGE_BUCKET":&s.Bucket,"GO_OMNI_STORAGE_ACCOUNT_ID":&s.AccountID,
"GO_OMNI_STORAGE_SIGNING_KEY":&s.SigningKey,"GO_OMNI_FILES_TOKEN":&s.FilesToken,
"GO_OMNI_STORAGE_ACCESS_KEY":&s.AccessKey,"GO_OMNI_STORAGE_SECRET_KEY":&s.SecretKey,
"GO_OMNI_STORAGE_SESSION_TOKEN":&s.SessionToken,
}
for name,target:=range values {if value:=strings.TrimSpace(os.Getenv(name));value!=""{*target=value}}
for name,target:=range map[string]*time.Duration{"GO_OMNI_STORAGE_LINK_TTL":&s.LinkTTL,"GO_OMNI_STORAGE_REQUEST_TIMEOUT":&s.RequestTimeout} {
if value:=strings.TrimSpace(os.Getenv(name));value!="" {v,err:=time.ParseDuration(value);if err!=nil{return fmt.Errorf("%s must be a duration",name)};*target=v}
}
if value:=os.Getenv("GO_OMNI_STORAGE_MAX_BYTES");value!=""{v,err:=strconv.ParseInt(value,10,64);if err!=nil{return errors.New("GO_OMNI_STORAGE_MAX_BYTES must be an integer")};s.MaxBytes=v}
if value:=os.Getenv("GO_OMNI_STORAGE_SECURE");value!=""{v,err:=strconv.ParseBool(value);if err!=nil{return errors.New("GO_OMNI_STORAGE_SECURE must be a boolean")};s.Secure=v}
if value:=strings.TrimSpace(os.Getenv("GO_OMNI_STORAGE_CONTENT_TYPES"));value!="" {s.ContentTypes=strings.Split(value,",");for i:=range s.ContentTypes{s.ContentTypes[i]=strings.TrimSpace(s.ContentTypes[i])}}
return nil
}
func (s StorageConfig) Validate()error { func (s StorageConfig) Validate()error {
if s.Provider!={{printf "%q" .Storage}} {return errors.New("storage.provider must match the adapter selected at generation")} if s.Provider!={{printf "%q" .Storage}} {return errors.New("storage.provider must match the adapter selected at generation")}
if err:=s.Policy().Validate();err!=nil{return err} if err:=s.Policy().Validate();err!=nil{return err}
if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")} if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")}
if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")} if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")}
{{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("set GO_OMNI_FILES_TOKEN to a token of at least 32 bytes without whitespace")}{{end}} {{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("storage.filesToken requires at least 32 bytes without whitespace")}{{end}}
{{if eq .Storage "local"}} {{if eq .Storage "local"}}
if s.Root=="" {return errors.New("storage.root is required")} if s.Root=="" {return errors.New("storage.root is required")}
u,err:=url.Parse(s.PublicURL) u,err:=url.Parse(s.PublicURL)
if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")} if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")}
if len(s.SigningKey)<32 {return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")} if len(s.SigningKey)<32 {return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")}
{{else}} {{else}}
if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region and GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY are required")} if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region, accessKey and secretKey are required")}
{{if eq .Storage "r2"}} {{if eq .Storage "r2"}}
if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")} if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")}
if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")} if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")}

View File

@@ -21,18 +21,18 @@ func environment(t *testing.T){
} }
func TestSettingsPrecedenceAndStrictness(t *testing.T){ func TestSettingsPrecedenceAndStrictness(t *testing.T){
environment(t) environment(t)
{{if eq .Database "mysql"}}t.Setenv("GO_OMNI_DB_DSN","test:secret@tcp(localhost)/demo"){{end}}
source,err:=os.ReadFile("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
path:=filepath.Join(t.TempDir(),"config.yaml") path:=filepath.Join(t.TempDir(),"config.yaml")
if err:=os.WriteFile(path,[]byte("server:\n address: '127.0.0.1:8123'\n log:\n level: debug\n"),0600);err!=nil{t.Fatal(err)} source=[]byte(strings.Replace(string(source),"${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}","127.0.0.1:8123",1))
if err=os.WriteFile(path,source,0600);err!=nil{t.Fatal(err)}
t.Setenv("GO_OMNI_HTTP_ADDR","127.0.0.1:9123") t.Setenv("GO_OMNI_HTTP_ADDR","127.0.0.1:9123")
t.Setenv("GO_OMNI_LOG_OUTPUT","stdout") t.Setenv("GO_OMNI_LOG_OUTPUT","stdout")
cfg,err:=Load(path);if err!=nil{t.Fatal(err)} cfg,err:=Load(path);if err!=nil{t.Fatal(err)}
if cfg.Server.Address!="127.0.0.1:9123" || cfg.Server.Log.Level!="debug" || cfg.Server.Log.Output!="stdout" || cfg.Server.Log.File!="" || cfg.Server.Log.Rotation.Enabled {t.Fatal("configuration precedence failed")} initial:=Default()
for _,body:=range []string{"unexpected: true","server:\n slowRequestThreshold: -1s","server:\n log:\n format: invalid"{{if ne .Storage "none"}},"storage:\n filesToken: forbidden","storage:\n maxBytes: -1","storage:\n provider: unsupported"{{end}}}{ if cfg.Server.Address!="127.0.0.1:8123"||cfg.Server.Log.Output!=initial.Server.Log.Output||cfg.Server.Log.File!=initial.Server.Log.File {t.Fatal("implicit overwrite")}
if err:=os.WriteFile(path,[]byte(body),0600);err!=nil{t.Fatal(err)} for _,body:=range []string{"unexpected: true","server:\n slowRequestThreshold: -1s","server:\n log:\n format: invalid"}{
if _,err:=Load(path);err==nil{t.Fatal("invalid configuration accepted")} if err=os.WriteFile(path,[]byte(body),0600);err!=nil{t.Fatal(err)}
if _,err=Load(path);err==nil{t.Fatal("invalid configuration accepted")}
} }
{{if ne .Storage "none"}}
t.Setenv("GO_OMNI_FILES_TOKEN","")
if _,err:=Load("");err==nil{t.Fatal("missing file token accepted")}
{{end}}
} }

View File

@@ -10,11 +10,11 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni"
// Source builds can carry an unpublished VCS version (including +dirty). // Source builds can carry an unpublished VCS version (including +dirty).
// Keep this fallback in sync with the next release; installed CLIs use their // Keep this fallback in sync with the next release; installed CLIs use their
// downloaded module version, which also has a module checksum. // downloaded module version, which also has a module checksum.
const defaultFrameworkVersion = "v0.20.0-preview" const defaultFrameworkVersion = "v0.22.0-preview"
// This target is not a claim that the release already exists. Source users // This target is not a claim that the release already exists. Source users
// select a local framework checkout until the module templates are published. // select a local framework checkout until the module templates are published.
const modulesFrameworkVersion = "v0.7.0-preview" const modulesFrameworkVersion = "v0.22.0-preview"
func installedFrameworkVersion() string { func installedFrameworkVersion() string {
info, _ := debug.ReadBuildInfo() info, _ := debug.ReadBuildInfo()

View File

@@ -1,66 +0,0 @@
package config
import (
"strings"
"testing"
"time"
)
func TestUnifiedEnvAliases(t *testing.T) {
type project struct {
Server Config `json:"server"`
Label string `json:"label"`
}
base := project{Server: Default(), Label: "base"}
for _, policy := range []string{"allow", "ignore", "reject"} {
t.Run(policy, func(t *testing.T) {
env := map[string]string{"TEST_HTTP_ADDR": "127.0.0.1:8099", "TEST_LOG_SERVICE": "", "TEST_SERVICE": "", "TEST_LABEL": ""}
opts := EnvOptions{Prefix: "TEST_", Empty: policy, Aliases: ServerEnvAliases("TEST_", "server"), Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}
got, err := OverlayEnv(base, opts)
if policy == "reject" {
if err == nil || !strings.Contains(err.Error(), "empty") {
t.Fatal(err)
}
return
}
if err != nil {
t.Fatal(err)
}
if got.Server.Address != "127.0.0.1:8099" {
t.Fatal("alias ignored")
}
if policy == "allow" && got.Label != "" || policy == "ignore" && got.Label != "base" {
t.Fatal("empty policy differs")
}
env["TEST_SERVER_ADDRESS"] = "127.0.0.1:9001"
env["TEST_HTTP_READ_TIMEOUT"] = "3s"
env["TEST_ENABLE_LOGGER"] = "false"
env["TEST_TRUSTED_PROXIES"] = `["127.0.0.1"]`
got, err = OverlayEnv(base, opts)
if err != nil || got.Server.Address != "127.0.0.1:9001" || got.Server.HTTP.ReadTimeout != 3*time.Second || got.Server.EnableLogger || len(got.Server.TrustedProxies) != 1 {
t.Fatal(got, err)
}
env["TEST_SERVER_ADDRESS"] = ""
got, err = OverlayEnv(base, opts)
if err != nil {
t.Fatal(err)
}
expected := ""
if policy == "ignore" {
expected = base.Server.Address
}
if got.Server.Address != expected {
t.Fatal("empty canonical unexpectedly fell through to alias")
}
})
}
if base.Server.Address != ":8080" || base.Label != "base" {
t.Fatal("defaults mutated")
}
}
func TestAliasTypeErrorsDoNotExposeValue(t *testing.T) {
opts := EnvOptions{Prefix: "X_", Empty: "reject", Aliases: ServerEnvAliases("X_", ""), Lookup: func(k string) (string, bool) { return "private-invalid", k == "X_HTTP_READ_TIMEOUT" }}
if _, err := OverlayEnv(Default(), opts); err == nil || strings.Contains(err.Error(), "private-invalid") {
t.Fatal(err)
}
}

View File

@@ -59,6 +59,20 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int,
if err := b.step(path, depth); err != nil { if err := b.step(path, depth); err != nil {
return err return err
} }
if text, ok := v.(resolvedText); ok {
if dst.Kind() != reflect.Pointer && dst.Type() != durationType && dst.CanAddr() && !dst.Addr().Type().Implements(jsonUnmarshalerType) && !dst.Addr().Type().Implements(textType) {
switch dst.Kind() {
case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64:
if err := parseTextValue(dst, string(text)); err != nil {
return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err)
}
return nil
}
}
if dst.Kind() != reflect.Pointer {
v = string(text)
}
}
if v == nil { if v == nil {
return b.decode(nil, dst, path) return b.decode(nil, dst, path)
} // Preserve encoding/json null semantics. } // Preserve encoding/json null semantics.
@@ -78,7 +92,7 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int,
default: default:
return b.decode(v, dst, path) return b.decode(v, dst, path)
} }
if err := parseEnvValue(dst, strings.TrimSpace(raw)); err != nil { if err := parseTextValue(dst, strings.TrimSpace(raw)); err != nil {
return b.doc.diagnostic(path, "expected a duration such as 5s", err) return b.doc.diagnostic(path, "expected a duration such as 5s", err)
} }
return nil return nil
@@ -102,7 +116,7 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int,
} }
switch dst.Kind() { switch dst.Kind() {
case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64: case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64:
if err := parseEnvValue(dst, strings.TrimSpace(string(text))); err != nil { if err := parseTextValue(dst, strings.TrimSpace(string(text))); err != nil {
return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err) return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err)
} }
return nil return nil
@@ -143,7 +157,7 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int,
} }
} }
if !ok { if !ok {
return b.doc.diagnostic(child, "unknown field", nil) return b.doc.diagnostic(child, unknownFieldReason(name, fields), nil)
} }
if used[canonical] { if used[canonical] {
return b.doc.diagnostic(child, "duplicate field binding", nil) return b.doc.diagnostic(child, "duplicate field binding", nil)
@@ -237,6 +251,8 @@ func (b *documentBinder) dynamic(v any, path string, depth int) (any, error) {
switch x := v.(type) { switch x := v.(type) {
case Text: case Text:
return string(x), nil return string(x), nil
case resolvedText:
return string(x), nil
case map[string]any: case map[string]any:
out := make(map[string]any, len(x)) out := make(map[string]any, len(x))
for _, k := range sortedKeys(x) { for _, k := range sortedKeys(x) {

View File

@@ -1,132 +1,6 @@
package config package config
import ( import "testing"
"strings"
"testing"
"time"
)
func TestLoadFromEnvUsesDefaults(t *testing.T) {
for _, key := range []string{
"GO_OMNI_HTTP_ADDR", "GO_OMNI_REQUEST_ID_HEADER", "GO_OMNI_ENABLE_LOGGER",
"GO_OMNI_TRUSTED_PROXIES",
"GO_OMNI_TRANSPORT", "GO_OMNI_TLS_CERT_FILE", "GO_OMNI_TLS_KEY_FILE",
"GO_OMNI_LOG_LEVEL", "GO_OMNI_LOG_FORMAT", "GO_OMNI_LOG_OUTPUT",
"GO_OMNI_LOG_FILE", "GO_OMNI_SERVICE", "GO_OMNI_ENVIRONMENT",
"GO_OMNI_LOG_ADD_SOURCE", "GO_OMNI_SHUTDOWN_TIMEOUT",
"GO_OMNI_SLOW_REQUEST_THRESHOLD",
} {
t.Setenv(key, "")
}
cfg, err := LoadFromEnv("")
if err != nil {
t.Fatal(err)
}
if cfg.Address != ":8080" || cfg.Transport != TransportNetHTTP || cfg.TLS != (TLSConfig{}) || cfg.RequestIDHeader != "X-Request-ID" || cfg.ShutdownTimeout != 10*time.Second {
t.Fatalf("unexpected defaults: %+v", cfg)
}
if cfg.EnableLogger || cfg.Log.Level != "info" || cfg.Log.Format != "json" || cfg.Log.Output != "stderr" {
t.Fatalf("unexpected logger defaults: %+v", cfg.Log)
}
}
func TestTransportEnvironment(t *testing.T) {
for _, test := range []struct {
name, transport, cert, key, errorField string
}{
{"default", "", "", "", ""},
{"native", "native", "", "", ""},
{"nativeTLS", "native", "server.crt", "server.key", ""},
{"http2", "http2", "server.crt", "server.key", ""},
{"unknown", "typo", "", "", "transport"},
{"certOnly", "nethttp", "server.crt", "", "tls"},
{"keyOnly", "native", "", "server.key", "tls"},
{"http2WithoutTLS", "http2", "", "", "tls"},
} {
t.Run(test.name, func(t *testing.T) {
t.Setenv("TRANSPORT_TEST_TRANSPORT", test.transport)
t.Setenv("TRANSPORT_TEST_TLS_CERT_FILE", test.cert)
t.Setenv("TRANSPORT_TEST_TLS_KEY_FILE", test.key)
cfg, err := LoadFromEnv("TRANSPORT_TEST_")
if test.errorField != "" {
if err == nil || !strings.Contains(err.Error(), test.errorField) {
t.Fatalf("unexpected error: %v", err)
}
return
}
if err != nil {
t.Fatal(err)
}
want := test.transport
if want == "" {
want = TransportNetHTTP
}
if cfg.Transport != want || cfg.TLS.CertFile != test.cert || cfg.TLS.KeyFile != test.key {
t.Fatalf("settings: %+v", cfg)
}
})
}
}
func TestLoadFromEnvOverridesSettings(t *testing.T) {
t.Setenv("GO_OMNI_HTTP_ADDR", "127.0.0.1:9090")
t.Setenv("GO_OMNI_REQUEST_ID_HEADER", "X-Correlation-ID")
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "127.0.0.1, 10.0.0.0/8")
t.Setenv("GO_OMNI_ENABLE_LOGGER", "true")
t.Setenv("GO_OMNI_LOG_LEVEL", "debug")
t.Setenv("GO_OMNI_LOG_FORMAT", "text")
t.Setenv("GO_OMNI_LOG_OUTPUT", "stdout")
t.Setenv("GO_OMNI_SERVICE", "billing")
t.Setenv("GO_OMNI_ENVIRONMENT", "test")
t.Setenv("GO_OMNI_LOG_ADD_SOURCE", "true")
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "3s")
t.Setenv("GO_OMNI_SLOW_REQUEST_THRESHOLD", "500ms")
cfg, err := LoadFromEnv("")
if err != nil {
t.Fatal(err)
}
if cfg.Address != "127.0.0.1:9090" || cfg.RequestIDHeader != "X-Correlation-ID" || !cfg.EnableLogger || len(cfg.TrustedProxies) != 2 || cfg.TrustedProxies[1] != "10.0.0.0/8" {
t.Fatalf("unexpected application settings: %+v", cfg)
}
if cfg.ShutdownTimeout != 3*time.Second || cfg.Log.Level != "debug" || cfg.Log.Format != "text" || cfg.Log.Output != "stdout" || cfg.Log.Service != "billing" || cfg.Log.Environment != "test" || !cfg.Log.AddSource {
t.Fatalf("unexpected logger settings: %+v", cfg.Log)
}
if cfg.SlowRequestThreshold != 500*time.Millisecond {
t.Fatal(cfg.SlowRequestThreshold)
}
}
func TestLoadFromEnvRejectsInvalidValues(t *testing.T) {
t.Setenv("GO_OMNI_ENABLE_LOGGER", "sometimes")
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "ENABLE_LOGGER") {
t.Fatalf("expected invalid logger flag error, got %v", err)
}
t.Setenv("GO_OMNI_ENABLE_LOGGER", "")
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "later")
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "SHUTDOWN_TIMEOUT") {
t.Fatalf("expected invalid timeout error, got %v", err)
}
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "1s")
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "proxy.example.com")
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "trustedProxies") {
t.Fatalf("expected invalid trusted proxy error, got %v", err)
}
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "")
cfg := Default()
cfg.Address = ""
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "address") {
t.Fatalf("expected invalid address error, got %v", err)
}
cfg = Default()
cfg.RequestIDHeader = " "
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "requestIdHeader") {
t.Fatalf("expected invalid request header error, got %v", err)
}
}
func TestWithConfigAndWithLogConfig(t *testing.T) { func TestWithConfigAndWithLogConfig(t *testing.T) {
settings := Default() settings := Default()
@@ -147,23 +21,3 @@ func TestWithConfigAndWithLogConfig(t *testing.T) {
t.Fatalf("log config was not replaced: %+v", cfg.Log) t.Fatalf("log config was not replaced: %+v", cfg.Log)
} }
} }
func TestApplyEnvPreservesProjectDefaults(t *testing.T) {
cfg := Default()
cfg.Address = "127.0.0.1:8123"
cfg.Log.RedactKeys = []string{"bankAccount"}
cfg.TrustedProxies = []string{"127.0.0.1"}
t.Setenv("OVERLAY_TEST_LOG_LEVEL", "debug")
got, err := ApplyEnv(cfg, "OVERLAY_TEST_")
if err != nil {
t.Fatal(err)
}
if got.Address != cfg.Address || got.Log.Level != "debug" {
t.Fatal(got)
}
got.Log.RedactKeys[0] = "changed"
got.TrustedProxies[0] = "192.0.2.1"
if cfg.Log.RedactKeys[0] != "bankAccount" || cfg.TrustedProxies[0] != "127.0.0.1" {
t.Fatal("caller-owned defaults mutated")
}
}

View File

@@ -82,8 +82,9 @@ type Position struct{ Line, Column int }
// string, bool, json.Number, nil, []any, map[string]any. Text is available for // string, bool, json.Number, nil, []any, map[string]any. Text is available for
// formats such as XML whose scalar tokens have no inherent JSON type. // formats such as XML whose scalar tokens have no inherent JSON type.
type Document struct { type Document struct {
Value any Value any
Positions map[string]Position Positions map[string]Position
environments map[string][]string
} }
type Text string type Text string

View File

@@ -18,14 +18,15 @@ func TestEndpointConfiguration(t *testing.T) {
if cfg.Endpoints.Liveness.Enabled || cfg.Endpoints.OpenAPI.Enabled || !cfg.Endpoints.Readiness.Enabled || cfg.Endpoints.Readiness.ResponseMode != "raw" { if cfg.Endpoints.Liveness.Enabled || cfg.Endpoints.OpenAPI.Enabled || !cfg.Endpoints.Readiness.Enabled || cfg.Endpoints.Readiness.ResponseMode != "raw" {
t.Fatal(cfg.Endpoints) t.Fatal(cfg.Endpoints)
} }
refs := configFile(t, "yaml", "endpoints:\n readiness:\n path: ${TEST_ENDPOINTS_READINESS_PATH}\n openapi:\n enabled: ${TEST_ENDPOINTS_OPENAPI_ENABLED}\n")
t.Setenv("TEST_ENDPOINTS_READINESS_PATH", "/internal/ready") t.Setenv("TEST_ENDPOINTS_READINESS_PATH", "/internal/ready")
t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "true") t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "true")
cfg, err = ApplyEnv(cfg, "TEST_") cfg, err = LoadProject(refs, cfg)
if err != nil || cfg.Endpoints.Readiness.Path != "/internal/ready" || !cfg.Endpoints.OpenAPI.Enabled { if err != nil || cfg.Endpoints.Readiness.Path != "/internal/ready" || !cfg.Endpoints.OpenAPI.Enabled {
t.Fatal(cfg.Endpoints, err) t.Fatal(cfg.Endpoints, err)
} }
t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "invalid") t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "invalid")
if _, err = ApplyEnv(cfg, "TEST_"); err == nil { if _, err = LoadProject(refs, cfg); err == nil {
t.Fatal("invalid bool accepted") t.Fatal("invalid bool accepted")
} }
} }

View File

@@ -1,83 +0,0 @@
package config
import (
"encoding/json"
"os"
"strings"
)
const defaultEnvPrefix = "GO_OMNI_"
// ServerEnvAliases returns the documented short names for Config fields.
// field is the JSON path of Config in a project (usually "server"), or ""
// for Config itself. Canonical names win over aliases. Both use the same
// EnvOptions.Empty/Lookup and typed parser; collections use JSON syntax.
func ServerEnvAliases(prefix, field string) map[string][]string {
root := prefix
if field != "" {
for _, part := range strings.Split(field, ".") {
root += envName(part) + "_"
}
}
names := map[string]string{
"ADDRESS": "HTTP_ADDR", "TRANSPORT": "TRANSPORT", "TLS_CERT_FILE": "TLS_CERT_FILE", "TLS_KEY_FILE": "TLS_KEY_FILE",
"REQUEST_ID_HEADER": "REQUEST_ID_HEADER", "ENABLE_LOGGER": "ENABLE_LOGGER", "SLOW_REQUEST_THRESHOLD": "SLOW_REQUEST_THRESHOLD",
"TRUSTED_PROXIES": "TRUSTED_PROXIES", "SHUTDOWN_TIMEOUT": "SHUTDOWN_TIMEOUT",
"LOG_LEVEL": "LOG_LEVEL", "LOG_FORMAT": "LOG_FORMAT", "LOG_OUTPUT": "LOG_OUTPUT", "LOG_FILE": "LOG_FILE", "LOG_SERVICE": "SERVICE", "LOG_ENVIRONMENT": "ENVIRONMENT", "LOG_ADD_SOURCE": "LOG_ADD_SOURCE",
"HTTP_READ_HEADER_TIMEOUT": "HTTP_READ_HEADER_TIMEOUT", "HTTP_READ_TIMEOUT": "HTTP_READ_TIMEOUT", "HTTP_WRITE_TIMEOUT": "HTTP_WRITE_TIMEOUT", "HTTP_IDLE_TIMEOUT": "HTTP_IDLE_TIMEOUT", "HTTP_MAX_HEADER_BYTES": "HTTP_MAX_HEADER_BYTES",
"LIFECYCLE_STOP_TIMEOUT": "LIFECYCLE_STOP_TIMEOUT", "LIFECYCLE_DRAIN_TIMEOUT": "LIFECYCLE_DRAIN_TIMEOUT", "LIFECYCLE_CANCEL_TIMEOUT": "LIFECYCLE_CANCEL_TIMEOUT",
}
for _, endpoint := range []string{"LIVENESS", "READINESS", "OPENAPI"} {
for _, part := range []string{"ENABLED", "PATH", "RESPONSE_MODE"} {
if endpoint == "OPENAPI" && part == "RESPONSE_MODE" {
continue
}
name := "ENDPOINTS_" + endpoint + "_" + part
names[name] = name
}
}
aliases := map[string][]string{}
for key, alias := range names {
if root+key != prefix+alias {
aliases[root+key] = []string{prefix + alias}
}
}
return aliases
}
// LoadFromEnv loads defaults using the legacy short-name conventions.
func LoadFromEnv(prefix string) (Config, error) { return ApplyEnv(Default(), prefix) }
// ApplyEnv retains legacy whitespace trimming, ignored empty variables and
// comma-separated TRUSTED_PROXIES. New projects should use OverlayEnv with
// ServerEnvAliases for one explicit empty-value policy across the whole config.
func ApplyEnv(defaults Config, prefix string) (Config, error) {
prefix = strings.TrimSpace(prefix)
if prefix == "" {
prefix = defaultEnvPrefix
}
lookup := func(key string) (string, bool) {
raw, ok := os.LookupEnv(key)
raw = strings.TrimSpace(raw)
if ok && raw != "" && key == prefix+"TRUSTED_PROXIES" && !strings.HasPrefix(raw, "[") {
values := strings.Split(raw, ",")
for i := range values {
values[i] = strings.TrimSpace(values[i])
}
data, _ := json.Marshal(values)
raw = string(data)
}
return raw, ok
}
defaults.Errors = defaults.Errors.Clone()
defaults.TrustedProxies = append([]string(nil), defaults.TrustedProxies...)
defaults.Log.RedactKeys = append([]string(nil), defaults.Log.RedactKeys...)
cfg, err := OverlayEnv(defaults, EnvOptions{Prefix: prefix, Empty: "ignore", Lookup: lookup, Aliases: ServerEnvAliases(prefix, "")})
if err != nil {
return Config{}, err
}
if err = cfg.Validate(); err != nil {
return Config{}, err
}
return cfg, nil
}

View File

@@ -21,7 +21,7 @@ func TestCatalogConfigFiles(t *testing.T) {
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
cfg, err := config.LoadFile(path, "CATALOG_FIXTURE_") cfg, err := config.LoadFile(path)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -40,7 +40,7 @@ func TestCatalogConfigFiles(t *testing.T) {
if err := os.WriteFile(path, []byte(body), 0600); err != nil { if err := os.WriteFile(path, []byte(body), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := config.LoadFile(path, "CATALOG_FIXTURE_"); err == nil { if _, err := config.LoadFile(path); err == nil {
t.Fatal("accepted invalid catalog", body) t.Fatal("accepted invalid catalog", body)
} }
} }
@@ -57,7 +57,7 @@ func TestCatalogReloadSnapshotIsolation(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
defer logger.Close() defer logger.Close()
reloader, err := config.NewReloader(path, "CATALOG_RELOAD_", logger) reloader, err := config.NewReloader(path, logger)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -20,7 +20,7 @@ func TestTemplateConfigFiles(t *testing.T) {
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
cfg, err := config.LoadFile(path, "TEMPLATE_TEST_") cfg, err := config.LoadFile(path)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -4,103 +4,38 @@ import (
"context" "context"
"errors" "errors"
"reflect" "reflect"
"strings"
"sync" "sync"
"time" "time"
"git.yuntuo.hk/k2safe/go-omni/logx" "git.yuntuo.hk/k2safe/go-omni/logx"
) )
func durations(v any, t reflect.Type) (any, error) { // LoadFile loads and validates framework configuration using explicit file references.
if t == nil { func LoadFile(path string) (Config, error) { return LoadProject(path, Default()) }
return v, nil
}
for t.Kind() == reflect.Pointer {
t = t.Elem()
}
if t == reflect.TypeOf(time.Duration(0)) {
if s, ok := v.(string); ok {
d, err := time.ParseDuration(s)
return int64(d), err
}
return v, nil
}
if t.Kind() == reflect.Struct {
if m, ok := v.(map[string]any); ok {
for n := 0; n < t.NumField(); n++ {
field := t.Field(n)
name := strings.Split(field.Tag.Get("json"), ",")[0]
if name == "-" {
continue
}
if name == "" {
name = field.Name
}
if item, exists := m[name]; exists {
converted, err := durations(item, field.Type)
if err != nil {
return nil, err
}
m[name] = converted
}
}
}
}
if t.Kind() == reflect.Map && t.Key().Kind() == reflect.String {
if values, ok := v.(map[string]any); ok {
for key, value := range values {
converted, err := durations(value, t.Elem())
if err != nil {
return nil, err
}
values[key] = converted
}
}
}
if t.Kind() == reflect.Slice {
if items, ok := v.([]any); ok {
for n, item := range items {
converted, err := durations(item, t.Elem())
if err != nil {
return nil, err
}
items[n] = converted
}
}
}
return v, nil
}
func LoadFile(path, prefix string) (Config, error) {
cfg, err := DecodeFile(path, Default())
if err != nil {
return Config{}, err
}
return ApplyEnv(cfg, prefix)
}
// Reloader only accepts log.level changes. Static changes and invalid files // Reloader only accepts log.level changes. Static changes and invalid files
// leave the current snapshot and logger untouched. Watch does not own a goroutine. // leave the current snapshot and logger untouched. Watch does not own a goroutine.
type Reloader struct { type Reloader struct {
mu sync.Mutex mu sync.Mutex
path, prefix string path string
current Config current Config
logger *logx.Logger logger *logx.Logger
} }
func NewReloader(path, prefix string, logger *logx.Logger) (*Reloader, error) { func NewReloader(path string, logger *logx.Logger) (*Reloader, error) {
cfg, err := LoadFile(path, prefix) cfg, err := LoadFile(path)
if err != nil { if err != nil {
return nil, err return nil, err
} }
if err = logger.SetLevel(cfg.Log.Level); err != nil { if err = logger.SetLevel(cfg.Log.Level); err != nil {
return nil, err return nil, err
} }
return &Reloader{path: path, prefix: prefix, current: cfg, logger: logger}, nil return &Reloader{path: path, current: cfg, logger: logger}, nil
} }
func (r *Reloader) Reload() error { func (r *Reloader) Reload() error {
r.mu.Lock() r.mu.Lock()
defer r.mu.Unlock() defer r.mu.Unlock()
next, err := LoadFile(r.path, r.prefix) next, err := LoadFile(r.path)
if err != nil { if err != nil {
return err return err
} }

View File

@@ -20,8 +20,8 @@ func TestFileMergeAndSafeReload(t *testing.T) {
} }
write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: info\n") write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: info\n")
t.Setenv("TEST_MODULE_HTTP_ADDR", ":9123") t.Setenv("TEST_MODULE_HTTP_ADDR", ":9123")
cfg, err := LoadFile(path, "TEST_MODULE_") cfg, err := LoadFile(path)
if err != nil || cfg.Address != ":9123" || cfg.ShutdownTimeout != 7*time.Second { if err != nil || cfg.Address != ":8123" || cfg.ShutdownTimeout != 7*time.Second {
t.Fatal(cfg, err) t.Fatal(cfg, err)
} }
var output bytes.Buffer var output bytes.Buffer
@@ -33,7 +33,7 @@ func TestFileMergeAndSafeReload(t *testing.T) {
} }
defer l.Close() defer l.Close()
child := l.With("scope", "test") child := l.With("scope", "test")
r, err := NewReloader(path, "TEST_MODULE_", l) r, err := NewReloader(path, l)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -22,7 +22,7 @@ func TestHTTPFileAndEnvironment(t *testing.T) {
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
cfg, err := config.LoadFile(path, "HTTP_TEST_") cfg, err := config.LoadFile(path)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -35,7 +35,10 @@ func TestHTTPFileAndEnvironment(t *testing.T) {
t.Setenv("HTTP_TEST_HTTP_WRITE_TIMEOUT", "8s") t.Setenv("HTTP_TEST_HTTP_WRITE_TIMEOUT", "8s")
t.Setenv("HTTP_TEST_HTTP_IDLE_TIMEOUT", "9s") t.Setenv("HTTP_TEST_HTTP_IDLE_TIMEOUT", "9s")
t.Setenv("HTTP_TEST_HTTP_MAX_HEADER_BYTES", "16384") t.Setenv("HTTP_TEST_HTTP_MAX_HEADER_BYTES", "16384")
got, err := config.ApplyEnv(cfg, "HTTP_TEST_") if err := os.WriteFile(path, []byte(`{"http":{"readHeaderTimeout":"${HTTP_TEST_HTTP_READ_HEADER_TIMEOUT}","readTimeout":"${HTTP_TEST_HTTP_READ_TIMEOUT}","writeTimeout":"${HTTP_TEST_HTTP_WRITE_TIMEOUT}","idleTimeout":"${HTTP_TEST_HTTP_IDLE_TIMEOUT}","maxHeaderBytes":"${HTTP_TEST_HTTP_MAX_HEADER_BYTES}"}}`), 0600); err != nil {
t.Fatal(err)
}
got, err := config.LoadProject(path, cfg)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -55,7 +58,12 @@ func TestHTTPInvalidAndUnsupportedSettings(t *testing.T) {
value = "-1s" value = "-1s"
} }
t.Setenv("HTTP_INVALID_HTTP_"+key, value) t.Setenv("HTTP_INVALID_HTTP_"+key, value)
if _, err := config.LoadFromEnv("HTTP_INVALID_"); err == nil { names := map[string]string{"READ_HEADER_TIMEOUT": "readHeaderTimeout", "READ_TIMEOUT": "readTimeout", "WRITE_TIMEOUT": "writeTimeout", "IDLE_TIMEOUT": "idleTimeout", "MAX_HEADER_BYTES": "maxHeaderBytes"}
path := filepath.Join(t.TempDir(), "config.json")
if err := os.WriteFile(path, []byte(`{"http":{"`+names[key]+`":"${HTTP_INVALID_HTTP_`+key+`}"}}`), 0600); err != nil {
t.Fatal(err)
}
if _, err := config.LoadFile(path); err == nil {
t.Fatal("invalid HTTP value accepted") t.Fatal("invalid HTTP value accepted")
} }
}) })

View File

@@ -14,7 +14,7 @@ func TestLifecycleConfiguration(t *testing.T) {
if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: 2s\n drainTimeout: 3s\n cancelTimeout: 4s\n"), 0600); err != nil { if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: 2s\n drainTimeout: 3s\n cancelTimeout: 4s\n"), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
cfg, err := config.LoadFile(path, "PHASE_TEST_") cfg, err := config.LoadFile(path)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -24,13 +24,16 @@ func TestLifecycleConfiguration(t *testing.T) {
for _, key := range []string{"STOP_TIMEOUT", "DRAIN_TIMEOUT", "CANCEL_TIMEOUT"} { for _, key := range []string{"STOP_TIMEOUT", "DRAIN_TIMEOUT", "CANCEL_TIMEOUT"} {
t.Setenv("PHASE_TEST_LIFECYCLE_"+key, "6s") t.Setenv("PHASE_TEST_LIFECYCLE_"+key, "6s")
} }
cfg, err = config.ApplyEnv(cfg, "PHASE_TEST_") if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: ${PHASE_TEST_LIFECYCLE_STOP_TIMEOUT}\n drainTimeout: ${PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT}\n cancelTimeout: ${PHASE_TEST_LIFECYCLE_CANCEL_TIMEOUT}\n"), 0600); err != nil {
t.Fatal(err)
}
cfg, err = config.LoadProject(path, cfg)
if err != nil || cfg.Lifecycle.StopTimeout != 6*time.Second || cfg.Lifecycle.DrainTimeout != 6*time.Second || cfg.Lifecycle.CancelTimeout != 6*time.Second { if err != nil || cfg.Lifecycle.StopTimeout != 6*time.Second || cfg.Lifecycle.DrainTimeout != 6*time.Second || cfg.Lifecycle.CancelTimeout != 6*time.Second {
t.Fatal(cfg.Lifecycle, err) t.Fatal(cfg.Lifecycle, err)
} }
for _, v := range []string{"bad", "-1s"} { for _, v := range []string{"bad", "-1s"} {
t.Setenv("PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT", v) t.Setenv("PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT", v)
if _, err := config.ApplyEnv(cfg, "PHASE_TEST_"); err == nil { if _, err := config.LoadProject(path, cfg); err == nil {
t.Fatal("invalid duration accepted") t.Fatal("invalid duration accepted")
} }
} }

View File

@@ -26,6 +26,7 @@ func (f DecoderFunc) Decode(data []byte) (Document, error) { return f(data) }
type Loader struct { type Loader struct {
decoders map[string]Decoder decoders map[string]Decoder
sealed bool sealed bool
lookup func(string) (string, bool)
} }
type LoaderOption func(*Loader) error type LoaderOption func(*Loader) error
@@ -55,7 +56,7 @@ func WithDecoder(extension string, decoder Decoder) LoaderOption {
} }
func NewLoader(options ...LoaderOption) (*Loader, error) { func NewLoader(options ...LoaderOption) (*Loader, error) {
l := &Loader{decoders: map[string]Decoder{"json": JSONDecoder{}, "yaml": YAMLDecoder{}, "yml": YAMLDecoder{}, "xml": XMLDecoder{}}} l := &Loader{lookup: os.LookupEnv, decoders: map[string]Decoder{"json": JSONDecoder{}, "yaml": YAMLDecoder{}, "yml": YAMLDecoder{}, "xml": XMLDecoder{}}}
for _, o := range options { for _, o := range options {
if o == nil { if o == nil {
return nil, errors.New("config: nil loader option") return nil, errors.New("config: nil loader option")
@@ -135,7 +136,7 @@ func (l *Loader) read(path string) (doc Document, err error) {
if _, ok := doc.Value.(map[string]any); !ok { if _, ok := doc.Value.(map[string]any); !ok {
return doc, doc.diagnostic("", "root must be an object", nil) return doc, doc.diagnostic("", "root must be an object", nil)
} }
return doc, nil return resolveDocument(doc, l.lookup)
} }
func decodeSafely(decoder Decoder, data []byte) (doc Document, err error) { func decodeSafely(decoder Decoder, data []byte) (doc Document, err error) {
defer func() { defer func() {
@@ -165,11 +166,12 @@ func DecodeFile[T any](path string, defaults T, loaders ...*Loader) (T, error) {
if err != nil { if err != nil {
return zero, err return zero, err
} }
if err := rejectEnvironmentTags(reflect.TypeOf(defaults), map[reflect.Type]bool{}); err != nil {
return zero, err
}
result, err := bindDocument(doc, defaults) result, err := bindDocument(doc, defaults)
if err != nil { if err != nil {
d := validationError(err) return zero, locateError(err, path, doc)
d.File = path
return zero, d
} }
return result, nil return result, nil
} }

View File

@@ -16,7 +16,7 @@ import (
type loaderExample struct { type loaderExample struct {
Server Config `json:"server"` Server Config `json:"server"`
Upload struct { Upload struct {
MaxBytes int64 `json:"maxBytes" envDefault:"1024"` MaxBytes int64 `json:"maxBytes"`
} `json:"upload"` } `json:"upload"`
Values []int64 `json:"values"` Values []int64 `json:"values"`
Optional *string `json:"optional"` Optional *string `json:"optional"`
@@ -45,19 +45,19 @@ func configFile(t *testing.T, ext, raw string) string {
func TestUnifiedFormatAndPrecedence(t *testing.T) { func TestUnifiedFormatAndPrecedence(t *testing.T) {
formats := map[string]string{ formats := map[string]string{
"json": `{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":7},"values":[9007199254740993],"optional":null,"enabled":true,"dynamic":{"id":9007199254740993}}`, "json": `{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":"${APP_UPLOAD_MAX_BYTES}"},"values":[9007199254740993],"optional":null,"enabled":"${APP_ENABLED}","dynamic":{"id":9007199254740993}}`,
"yaml": "server:\n http:\n readTimeout: 2s\nupload:\n maxBytes: 7\nvalues: [9007199254740993]\noptional: null\nenabled: true\ndynamic:\n id: 9007199254740993\n", "yaml": "server:\n http:\n readTimeout: 2s\nupload:\n maxBytes: \"${APP_UPLOAD_MAX_BYTES}\"\nvalues: [9007199254740993]\noptional: null\nenabled: \"${APP_ENABLED}\"\ndynamic:\n id: 9007199254740993\n",
"xml": `<config><server><http><readTimeout>2s</readTimeout></http></server><upload><maxBytes>7</maxBytes></upload><values type="array"><item>9007199254740993</item></values><optional type="null"/><enabled>true</enabled><dynamic><id type="number">9007199254740993</id></dynamic></config>`, "xml": `<config><server><http><readTimeout>2s</readTimeout></http></server><upload><maxBytes>${APP_UPLOAD_MAX_BYTES}</maxBytes></upload><values type="array"><item>9007199254740993</item></values><optional type="null"/><enabled>${APP_ENABLED}</enabled><dynamic><id type="number">9007199254740993</id></dynamic></config>`,
} }
var reference loaderExample var reference loaderExample
for _, ext := range []string{"json", "yaml", "xml"} { for _, ext := range []string{"json", "yaml", "xml"} {
t.Run(ext, func(t *testing.T) { t.Run(ext, func(t *testing.T) {
old := "old" old := "old"
base := loaderExample{Server: Default(), Optional: &old} base := loaderExample{Server: Default(), Optional: &old}
cfg, err := LoadProject(configFile(t, ext, formats[ext]), base, EnvOptions{Prefix: "APP_", Lookup: func(key string) (string, bool) { cfg, err := LoadProject(configFile(t, ext, formats[ext]), base, testLoader(func(key string) (string, bool) {
v, ok := map[string]string{"APP_UPLOAD_MAX_BYTES": "0", "APP_ENABLED": "false"}[key] v, ok := map[string]string{"APP_UPLOAD_MAX_BYTES": "0", "APP_ENABLED": "false"}[key]
return v, ok return v, ok
}}) }))
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -77,8 +77,8 @@ func TestUnifiedFormatAndPrecedence(t *testing.T) {
} }
}) })
} }
cfg, err := LoadProject("", loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}) cfg, err := LoadProject("", loaderExample{Server: Default()}, testLoader(noEnvironment))
if err != nil || cfg.Upload.MaxBytes != 1024 { if err != nil || cfg.Upload.MaxBytes != 0 {
t.Fatal(cfg, err) t.Fatal(cfg, err)
} }
} }
@@ -97,7 +97,7 @@ func TestConfigDiagnosticsAndCauses(t *testing.T) {
} { } {
t.Run(tc.ext+tc.field, func(t *testing.T) { t.Run(tc.ext+tc.field, func(t *testing.T) {
path := configFile(t, tc.ext, tc.body) path := configFile(t, tc.ext, tc.body)
_, err := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}) _, err := LoadProject(path, loaderExample{Server: Default()}, testLoader(noEnvironment))
var d *Error var d *Error
if !errors.As(err, &d) || d.File != path || d.Field != tc.field || d.Reason == "" || tc.line && d.Line < 1 { if !errors.As(err, &d) || d.File != path || d.Field != tc.field || d.Reason == "" || tc.line && d.Line < 1 {
t.Fatalf("missing location: %#v %v", d, err) t.Fatalf("missing location: %#v %v", d, err)
@@ -108,7 +108,7 @@ func TestConfigDiagnosticsAndCauses(t *testing.T) {
}) })
} }
path := filepath.Join(t.TempDir(), "missing.json") path := filepath.Join(t.TempDir(), "missing.json")
_, err := LoadProject(path, loaderExample{}, EnvOptions{}) _, err := LoadProject(path, loaderExample{})
if !errors.Is(err, os.ErrNotExist) { if !errors.Is(err, os.ErrNotExist) {
t.Fatal("I/O cause discarded", err) t.Fatal("I/O cause discarded", err)
} }
@@ -119,11 +119,12 @@ func TestConfigDiagnosticsAndCauses(t *testing.T) {
if !errors.As(err, &typed) { if !errors.As(err, &typed) {
t.Fatal("type cause discarded", err) t.Fatal("type cause discarded", err)
} }
_, err = LoadProject(configFile(t, "yaml", "upload:\n maxBytes: 7"), loaderExample{Server: Default()}, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { return "-1", k == "APP_UPLOAD_MAX_BYTES" }}) _, err = LoadProject(configFile(t, "yaml", "upload:\n maxBytes: \"${APP_UPLOAD_MAX_BYTES}\""), loaderExample{Server: Default()}, testLoader(func(k string) (string, bool) { return "-1", k == "APP_UPLOAD_MAX_BYTES" }))
var d *Error var d *Error
if !errors.As(err, &d) || d.Env != "APP_UPLOAD_MAX_BYTES" || d.Field != "upload.maxBytes" || d.File != "" || d.Line != 0 { if !errors.As(err, &d) || d.Env != "APP_UPLOAD_MAX_BYTES" || d.Field != "upload.maxBytes" || d.File == "" || d.Line != 2 {
t.Fatal("environment validation wrongly blamed file", err) t.Fatal("missing file and variable provenance", err)
} }
} }
func TestMalformedFormatsAndLimits(t *testing.T) { func TestMalformedFormatsAndLimits(t *testing.T) {
@@ -134,7 +135,7 @@ func TestMalformedFormatsAndLimits(t *testing.T) {
} { } {
for i, raw := range bodies { for i, raw := range bodies {
t.Run(fmt.Sprintf("%s%d", ext, i), func(t *testing.T) { t.Run(fmt.Sprintf("%s%d", ext, i), func(t *testing.T) {
_, err := LoadProject(configFile(t, ext, raw), loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}) _, err := LoadProject(configFile(t, ext, raw), loaderExample{Server: Default()}, testLoader(noEnvironment))
if err == nil || strings.Contains(err.Error(), "private-value") { if err == nil || strings.Contains(err.Error(), "private-value") {
t.Fatal(err) t.Fatal(err)
} }
@@ -163,7 +164,7 @@ func TestDecoderRegistryAndConcurrentUse(t *testing.T) {
wg.Add(1) wg.Add(1)
go func() { go func() {
defer wg.Done() defer wg.Done()
cfg, e := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}, l) cfg, e := LoadProject(path, loaderExample{Server: Default()}, l)
if e != nil || cfg.Upload.MaxBytes != 42 { if e != nil || cfg.Upload.MaxBytes != 42 {
t.Error(e) t.Error(e)
} }
@@ -191,25 +192,6 @@ func TestDecoderRegistryAndConcurrentUse(t *testing.T) {
} }
} }
func TestUnifiedLoggerNormalization(t *testing.T) {
type project struct {
Server Config `json:"server"`
}
base := project{Server: Default()}
base.Server.Log.File = "old.log"
base.Server.Log.Rotation.Enabled = true
for _, full := range []bool{false, true} {
name := "GO_OMNI_LOG_OUTPUT"
if full {
name = "GO_OMNI_SERVER_LOG_OUTPUT"
}
cfg, err := LoadProject("", base, EnvOptions{Prefix: "GO_OMNI_", Empty: "ignore", Aliases: ServerEnvAliases("GO_OMNI_", "server"), Lookup: func(k string) (string, bool) { return "stdout", k == name }})
if err != nil || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled || base.Server.Log.File != "old.log" {
t.Fatal(cfg, err)
}
}
}
func FuzzConfigurationDecoders(f *testing.F) { func FuzzConfigurationDecoders(f *testing.F) {
f.Add(uint8(0), []byte(`{"count":1}`)) f.Add(uint8(0), []byte(`{"count":1}`))
f.Add(uint8(1), []byte("count: 1")) f.Add(uint8(1), []byte("count: 1"))
@@ -332,18 +314,21 @@ func TestConfigDefaultsIsolationAndEmptyValues(t *testing.T) {
func TestDefaultStageDoesNotReadEnvironment(t *testing.T) { func TestDefaultStageDoesNotReadEnvironment(t *testing.T) {
reads := 0 reads := 0
type C struct { type C struct {
Value int `json:"value" envDefault:"3"` Value int `json:"value"`
} }
got, err := LoadProject("", C{}, EnvOptions{Lookup: func(string) (string, bool) { reads++; return "", false }}) got, err := LoadProject("", C{Value: 3}, testLoader(func(string) (string, bool) { reads++; return "9", true }))
if err != nil || got.Value != 3 || reads != 1 { if err != nil || got.Value != 3 || reads != 0 {
t.Fatal(got, err, reads) t.Fatal(got, err, reads)
} }
d := locateError(Invalid("servers[0].timeout", "must be positive"), "a.json", Document{}, map[string]string{"servers": "APP_SERVERS"})
if d.Env != "APP_SERVERS" || d.File != "" {
t.Fatal(d)
}
} }
func testLoader(lookup func(string) (string, bool)) *Loader {
l, err := NewLoader(WithLookupEnv(lookup))
if err != nil {
panic(err)
}
return l
}
func TestDecoderRegistryFrozen(t *testing.T) { func TestDecoderRegistryFrozen(t *testing.T) {
l, err := NewLoader() l, err := NewLoader()
if err != nil { if err != nil {
@@ -358,17 +343,12 @@ func BenchmarkConfigurationStartup(b *testing.B) {
if err := os.WriteFile(path, []byte(`{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":4096}}`), 0600); err != nil { if err := os.WriteFile(path, []byte(`{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":4096}}`), 0600); err != nil {
b.Fatal(err) b.Fatal(err)
} }
options := EnvOptions{Prefix: "BENCH_", Lookup: func(name string) (string, bool) {
if name == "BENCH_UPLOAD_MAX_BYTES" {
return "8192", true
}
return "", false
}}
defaults := loaderExample{Server: Default()} defaults := loaderExample{Server: Default()}
b.ReportAllocs() b.ReportAllocs()
b.ResetTimer() b.ResetTimer()
for i := 0; i < b.N; i++ { for i := 0; i < b.N; i++ {
if _, err := LoadProject(path, defaults, options); err != nil { if _, err := LoadProject(path, defaults); err != nil {
b.Fatal(err) b.Fatal(err)
} }
} }

View File

@@ -1,318 +1,105 @@
package config package config
import ( import (
"bytes"
"encoding" "encoding"
"encoding/json"
"errors" "errors"
"fmt"
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
"git.yuntuo.hk/k2safe/go-omni/logx"
"math" "math"
"os"
"reflect" "reflect"
"strconv" "strconv"
"strings" "strings"
"time" "time"
"unicode"
) )
// EnvOptions controls typed project configuration overlays. Explicit env tags // LoadProject clones defaults, resolves only explicit file references, binds the
// are absolute names; otherwise JSON names become PREFIX_NESTED_FIELD names. // provided fields and validates. Empty path uses only defaults. No env tags,
// Empty is allow (default), ignore or reject. No input values appear in errors. // prefixes, aliases or implicit environment overlays are applied.
type EnvOptions struct { func LoadProject[T any](path string, defaults T, loaders ...*Loader) (T, error) {
Prefix string
Empty string
Lookup func(string) (string, bool)
// Aliases maps canonical absolute environment names to ordered fallback names.
// A present canonical value wins, including an explicit empty value.
Aliases map[string][]string
}
// ApplyDefaults applies envDefault tags to zero fields before file overlays.
func ApplyDefaults[T any](base T) (T, error) { return projectOverlay(base, EnvOptions{}, true) }
// LoadProject applies code/tag defaults, a strict optional file, environment,
// then the project's Validate method, before any logger or dependency is opened.
// envDefault is only applied to zero-valued defaults before reading the file.
func LoadProject[T any](path string, defaults T, options EnvOptions, loaders ...*Loader) (T, error) {
var zero T var zero T
loader, err := selectLoader(loaders) loader, err := selectLoader(loaders)
if err != nil { if err != nil {
return zero, err return zero, err
} }
cfg, err := projectOverlay(defaults, options, true) if err = rejectEnvironmentTags(reflect.TypeOf(defaults), map[reflect.Type]bool{}); err != nil {
if err != nil {
return zero, err return zero, err
} }
var doc Document doc := Document{Value: map[string]any{}}
if path != "" { if path != "" {
doc, err = loader.read(path) doc, err = loader.read(path)
if err != nil { if err != nil {
return zero, err return zero, err
} }
cfg, err = bindDocument(doc, cfg)
if err != nil {
return zero, locateError(err, path, doc, nil)
}
} else {
value, e := cloneConfiguration(reflect.ValueOf(&cfg).Elem(), 0)
if e != nil {
return zero, &Error{Reason: "cannot clone defaults", cause: e}
}
cfg = value.Interface().(T)
} }
origins := map[string]string{} cfg, err := bindDocument(doc, defaults)
cfg, err = projectOverlay(cfg, options, false, origins)
if err != nil { if err != nil {
return zero, err return zero, locateError(err, path, doc)
} }
normalizeProject(reflect.ValueOf(&cfg).Elem(), "", origins, 0)
if valid, ok := any(&cfg).(interface{ Validate() error }); ok { if valid, ok := any(&cfg).(interface{ Validate() error }); ok {
if err = valid.Validate(); err != nil { if err = valid.Validate(); err != nil {
return zero, locateError(err, path, doc, origins) return zero, locateError(err, path, doc)
}
} else if valid, ok := any(cfg).(interface{ Validate() error }); ok {
if err = valid.Validate(); err != nil {
return zero, locateError(err, path, doc)
} }
} }
return cfg, nil return cfg, nil
} }
func locateError(err error, path string, doc Document) *Error {
func locateError(err error, path string, doc Document, origins map[string]string) *Error {
d := validationError(err) d := validationError(err)
field := d.Field
for field != "" && origins[field] == "" {
index := strings.LastIndexAny(field, ".[")
if index < 0 {
field = ""
} else {
field = field[:index]
}
}
if env := origins[field]; env != "" {
d.Env = env
d.File = ""
d.Line = 0
d.Column = 0
return d
}
d.File = path d.File = path
if d.Line == 0 { if d.Line == 0 {
p := doc.Positions[d.Field] p := doc.Positions[d.Field]
d.Line = p.Line d.Line = p.Line
d.Column = p.Column d.Column = p.Column
} }
for field := d.Field; field != ""; {
if names := doc.environments[field]; len(names) > 0 {
d.Env = strings.Join(names, ", ")
break
}
i := strings.LastIndexAny(field, ".[")
if i < 0 {
break
}
field = field[:i]
}
return d return d
} }
// Framework-owned normalization stays inside the common pipeline. Explicit // Reject removed declarations rather than silently ignoring deployment rules.
// environment stream selection clears a file profile unless a file override func rejectEnvironmentTags(t reflect.Type, seen map[reflect.Type]bool) error {
// was also provided; project code never needs to repeat this rule. if t == nil || seen[t] {
func normalizeProject(v reflect.Value, path string, origins map[string]string, depth int) { return nil
if depth > 32 {
return
} }
if v.Kind() == reflect.Pointer { seen[t] = true
if v.IsNil() { switch t.Kind() {
return case reflect.Pointer, reflect.Slice, reflect.Array, reflect.Map:
return rejectEnvironmentTags(t.Elem(), seen)
case reflect.Struct:
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
if f.PkgPath != "" {
continue
}
_, env := f.Tag.Lookup("env")
_, def := f.Tag.Lookup("envDefault")
if env || def {
return &Error{Field: f.Name, Reason: "env/envDefault tags were removed; use Default() and explicit file references; see CONFIG.md"}
}
if f.Tag.Get("json") != "-" {
if err := rejectEnvironmentTags(f.Type, seen); err != nil {
return err
}
}
} }
v = v.Elem()
} }
if v.Kind() != reflect.Struct { return nil
return
}
if v.Type() == reflect.TypeOf(Config{}) && v.CanAddr() {
c := v.Addr().Interface().(*Config)
if origins[joinField(path, "log.output")] != "" && c.Log.Output != "" && (origins[joinField(path, "log.file")] == "" || c.Log.File == "") {
c.Log.File = ""
c.Log.Rotation = logx.RotationConfig{}
}
return
}
for i := 0; i < v.NumField(); i++ {
f := v.Type().Field(i)
if f.PkgPath != "" {
continue
}
name := strings.Split(f.Tag.Get("json"), ",")[0]
if name == "-" {
continue
}
if name == "" {
name = f.Name
}
normalizeProject(v.Field(i), joinField(path, name), origins, depth+1)
}
}
// OverlayEnv overlays the entire project struct without mutating the input.
// Tags: env:"CUSTOM_NAME,required,secret,empty=reject"; env:"-" excludes a
// subtree. required means nonempty string/non-nil pointer or collection; false
// and zero are valid scalars. Use project Validate for semantic constraints.
// Slices/maps use JSON syntax, avoiding ambiguous comma and empty semantics.
func OverlayEnv[T any](base T, options EnvOptions) (T, error) {
return projectOverlay(base, options, false)
} }
var textType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem() var textType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()
var durationType = reflect.TypeOf(time.Duration(0)) var durationType = reflect.TypeOf(time.Duration(0))
func projectOverlay[T any](base T, options EnvOptions, defaults bool, sourceMaps ...map[string]string) (T, error) { func parseTextValue(v reflect.Value, raw string) (err error) {
var zero T
if options.Lookup == nil {
options.Lookup = os.LookupEnv
}
if options.Empty == "" {
options.Empty = "allow"
}
if options.Empty != "allow" && options.Empty != "ignore" && options.Empty != "reject" {
return zero, errors.New("config: empty policy must be allow, ignore or reject")
}
v := reflect.ValueOf(&base).Elem()
if v.Kind() != reflect.Struct {
return zero, errors.New("config: project configuration must be a struct")
}
result := reflect.New(v.Type()).Elem()
result.Set(v)
seen := map[string]bool{}
var walk func(reflect.Value, string, string, int) error
walk = func(value reflect.Value, prefix, path string, depth int) error {
if depth > 32 {
return errors.New("config: nesting exceeds 32")
}
for i := 0; i < value.NumField(); i++ {
field := value.Type().Field(i)
dst := value.Field(i)
if !dst.CanSet() || field.PkgPath != "" {
continue
}
tag := strings.Split(field.Tag.Get("env"), ",")
if tag[0] == "-" {
continue
}
jsonName := strings.Split(field.Tag.Get("json"), ",")[0]
if jsonName == "-" && tag[0] == "" {
continue
}
if jsonName == "" {
jsonName = field.Name
}
fieldPath := joinField(path, jsonName)
key := prefix + envName(jsonName)
if tag[0] != "" {
key = tag[0]
}
empty, required := options.Empty, false
for _, opt := range tag[1:] {
switch opt {
case "required":
required = true
case "secret", "":
case "empty=allow":
empty = "allow"
case "empty=ignore":
empty = "ignore"
case "empty=reject":
empty = "reject"
default:
return fmt.Errorf("config: invalid env tag for %s", key)
}
}
if dst.Kind() == reflect.Struct && !dst.Addr().Type().Implements(textType) {
if err := walk(dst, key+"_", fieldPath, depth+1); err != nil {
return err
}
continue
}
if dst.Kind() == reflect.Pointer && dst.Type().Elem().Kind() == reflect.Struct && !dst.Type().Implements(textType) {
// Clone before traversing so a failed overlay never edits defaults.
child := reflect.New(dst.Type().Elem())
if !dst.IsNil() {
child.Elem().Set(dst.Elem())
}
if err := walk(child.Elem(), key+"_", fieldPath, depth+1); err != nil {
return err
}
if !dst.IsNil() || !child.Elem().IsZero() {
dst.Set(child)
}
if required && !defaults && dst.IsNil() {
return &Error{Env: key, Field: fieldPath, Reason: "is required"}
}
continue
}
// Unexported runtime capabilities are skipped unless explicitly tagged.
if dst.Kind() == reflect.Func || dst.Kind() == reflect.Chan || dst.Kind() == reflect.Interface {
if tag[0] != "" {
return fmt.Errorf("config: %s has unsupported environment type", key)
}
continue
}
if seen[key] {
return fmt.Errorf("config: duplicate environment key %s", key)
}
seen[key] = true
var raw string
var present bool
if !defaults {
raw, present = options.Lookup(key)
}
selectedKey := key
if !present && !defaults {
for _, alias := range options.Aliases[key] {
if raw, present = options.Lookup(alias); present {
selectedKey = alias
break
}
}
}
if defaults {
raw, present = field.Tag.Lookup("envDefault")
present = present && dst.IsZero()
}
if present && raw == "" && !defaults {
if empty == "reject" {
return &Error{Env: selectedKey, Field: fieldPath, Reason: "must not be empty"}
}
if empty == "ignore" {
present = false
}
}
if present {
parsed := reflect.New(dst.Type()).Elem()
if err := parseEnvValue(parsed, raw); err != nil {
origin := selectedKey
if defaults {
origin = ""
}
return &Error{Env: origin, Field: fieldPath, Reason: "expected a valid " + dst.Type().String(), cause: err}
}
dst.Set(parsed)
if !defaults && len(sourceMaps) > 0 {
sourceMaps[0][fieldPath] = selectedKey
}
}
if required && !defaults && emptyRequired(dst) {
return &Error{Env: key, Field: fieldPath, Reason: "is required"}
}
}
return nil
}
if err := walk(result, options.Prefix, "", 0); err != nil {
return zero, err
}
return result.Interface().(T), nil
}
func emptyRequired(v reflect.Value) bool {
switch v.Kind() {
case reflect.String:
return v.Len() == 0
case reflect.Pointer, reflect.Map, reflect.Slice:
return v.IsNil()
}
return false
}
func parseEnvValue(v reflect.Value, raw string) (err error) {
defer func() { defer func() {
if recover() != nil { if recover() != nil {
err = errors.New("invalid custom value") err = errors.New("invalid custom value")
@@ -320,7 +107,7 @@ func parseEnvValue(v reflect.Value, raw string) (err error) {
}() }()
if v.Kind() == reflect.Pointer { if v.Kind() == reflect.Pointer {
v.Set(reflect.New(v.Type().Elem())) v.Set(reflect.New(v.Type().Elem()))
return parseEnvValue(v.Elem(), raw) return parseTextValue(v.Elem(), raw)
} }
if v.CanAddr() && v.Addr().Type().Implements(textType) { if v.CanAddr() && v.Addr().Type().Implements(textType) {
return v.Addr().Interface().(encoding.TextUnmarshaler).UnmarshalText([]byte(raw)) return v.Addr().Interface().(encoding.TextUnmarshaler).UnmarshalText([]byte(raw))
@@ -363,37 +150,6 @@ func parseEnvValue(v reflect.Value, raw string) (err error) {
v.SetFloat(x) v.SetFloat(x)
} }
return err return err
case reflect.Slice, reflect.Map:
var value any
if err := apierrors.DecodeCatalogJSON([]byte(raw), &value); err != nil {
return err
}
value, err := durations(value, v.Type())
if err != nil {
return err
}
encoded, err := json.Marshal(value)
if err != nil {
return err
}
decoder := json.NewDecoder(bytes.NewReader(encoded))
decoder.DisallowUnknownFields()
return decoder.Decode(v.Addr().Interface())
} }
return errors.New("unsupported environment value") return errors.New("expected scalar text")
}
func envName(s string) string {
r := []rune(s)
var out strings.Builder
for i, c := range r {
if i > 0 && unicode.IsUpper(c) && (unicode.IsLower(r[i-1]) || unicode.IsDigit(r[i-1]) || (i+1 < len(r) && unicode.IsLower(r[i+1]))) {
out.WriteByte('_')
}
if c == '-' {
c = '_'
}
out.WriteRune(unicode.ToUpper(c))
}
return out.String()
} }

View File

@@ -42,20 +42,3 @@ func TestConfigFilePreservesDefaultsWithoutSharingMutableData(t *testing.T) {
t.Fatal("decode error leaked contents", err) t.Fatal("decode error leaked contents", err)
} }
} }
func TestEnvCollectionPrecisionAndDurationMaps(t *testing.T) {
type cfg struct {
IDs []int64 `json:"ids"`
Timers map[string]time.Duration `json:"timers"`
}
lookup := func(key string) (string, bool) {
v, ok := map[string]string{"IDS": "[9007199254740993]", "TIMERS": `{"backoff":"250ms"}`}[key]
return v, ok
}
c, err := OverlayEnv(cfg{}, EnvOptions{Lookup: lookup})
if err != nil || c.IDs[0] != 9007199254740993 || c.Timers["backoff"] != 250*time.Millisecond {
t.Fatal(c, err)
}
if _, err = OverlayEnv(cfg{}, EnvOptions{Lookup: func(key string) (string, bool) { return `{"x":"1s","x":"2s"}`, key == "TIMERS" }}); err == nil {
t.Fatal("duplicate map key accepted")
}
}

View File

@@ -1,124 +0,0 @@
package config
import (
"errors"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
type projectNested struct {
ReadTimeout time.Duration `json:"readTimeout"`
Names []string `json:"names"`
}
type projectConfig struct {
Nested *projectNested `json:"nested"`
Enabled bool `json:"enabled"`
Count int `json:"count" envDefault:"12"`
Password string `json:"-" env:"ETAX_PASSWORD,secret,required,empty=reject"`
Label string `json:"label" env:",empty=ignore"`
}
func (c projectConfig) Validate() error {
if c.Count < 0 {
return errors.New("count must be nonnegative")
}
return nil
}
func TestProjectConfigurationOrderAndValues(t *testing.T) {
f := filepath.Join(t.TempDir(), "config.yaml")
if err := os.WriteFile(f, []byte("count: 3\nenabled: true\nnested:\n readTimeout: 2s\n names: [file]\nlabel: keep\n"), 0600); err != nil {
t.Fatal(err)
}
env := map[string]string{"ETAX_PASSWORD": " password ", "APP_ENABLED": "false", "APP_COUNT": "0", "APP_NESTED_READ_TIMEOUT": "7s", "APP_NESTED_NAMES": "[]", "APP_LABEL": ""}
options := EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}
cfg, err := LoadProject(f, projectConfig{}, options)
if err != nil {
t.Fatal(err)
}
if cfg.Enabled || cfg.Count != 0 || cfg.Password != " password " || cfg.Nested.ReadTimeout != 7*time.Second || cfg.Nested.Names == nil || len(cfg.Nested.Names) != 0 || cfg.Label != "keep" {
t.Fatal("overlay mismatch")
}
delete(env, "APP_COUNT")
cfg, err = LoadProject("", projectConfig{}, options)
if err != nil || cfg.Count != 12 {
t.Fatal(cfg.Count, err)
}
env["APP_COUNT"] = "-1"
if _, err = LoadProject(f, projectConfig{}, options); err == nil {
t.Fatal("project Validate not called")
}
}
func TestProjectOverlayIsTransactionalAndRedacted(t *testing.T) {
base := projectConfig{Nested: &projectNested{ReadTimeout: time.Second, Names: []string{"old"}}, Password: "original"}
env := map[string]string{"APP_NESTED_READ_TIMEOUT": "2s", "APP_NESTED_NAMES": "[\"new\"]", "APP_COUNT": "secret-value"}
_, err := OverlayEnv(base, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }})
if err == nil || strings.Contains(err.Error(), "secret-value") {
t.Fatal(err)
}
if base.Nested.ReadTimeout != time.Second || !reflect.DeepEqual(base.Nested.Names, []string{"old"}) {
t.Fatal("default pointer modified")
}
delete(env, "APP_COUNT")
env["ETAX_PASSWORD"] = ""
if _, err = OverlayEnv(base, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}); err == nil {
t.Fatal("empty secret accepted")
}
type required struct {
Token string `env:"TOKEN,required"`
}
if _, err = OverlayEnv(required{}, EnvOptions{Lookup: func(string) (string, bool) { return "", false }}); err == nil {
t.Fatal("required absent")
}
type duplicate struct {
A string `env:"SAME"`
B string `env:"SAME"`
}
if _, err = OverlayEnv(duplicate{}, EnvOptions{}); err == nil {
t.Fatal("duplicate env binding accepted")
}
}
func TestEnvironmentCollectionAndEmptySemantics(t *testing.T) {
type sample struct {
Values []time.Duration
Flag bool
Text string
Optional *int
}
cases := []struct {
raw string
valid bool
}{{`["1s","2s"]`, true}, {`["invalid-secret"]`, false}}
for _, tc := range cases {
cfg, err := OverlayEnv(sample{}, EnvOptions{Lookup: func(k string) (string, bool) {
if k == "VALUES" {
return tc.raw, true
}
return "", false
}})
if (err == nil) != tc.valid {
t.Fatal(err)
}
if tc.valid && !reflect.DeepEqual(cfg.Values, []time.Duration{time.Second, 2 * time.Second}) {
t.Fatal(cfg)
}
if err != nil && strings.Contains(err.Error(), "secret") {
t.Fatal(err)
}
}
for _, mode := range []string{"allow", "ignore", "reject"} {
cfg, err := OverlayEnv(sample{Text: "old"}, EnvOptions{Empty: mode, Lookup: func(k string) (string, bool) { return "", k == "TEXT" }})
if mode == "reject" {
if err == nil {
t.Fatal("empty not rejected")
}
} else if err != nil || (mode == "allow" && cfg.Text != "") || (mode == "ignore" && cfg.Text != "old") {
t.Fatal(cfg, err)
}
}
}

210
config/references.go Normal file
View File

@@ -0,0 +1,210 @@
package config
import (
"errors"
"fmt"
"strings"
"unicode/utf8"
)
// WithLookupEnv supplies an explicit-reference lookup for tests or an embedding
// host. Ordinary projects need no loader options. The function must support
// concurrent loads; each load caches each referenced name independently.
func WithLookupEnv(lookup func(string) (string, bool)) LoaderOption {
return func(l *Loader) error {
if l == nil || l.sealed || lookup == nil {
return errors.New("config: expected a non-nil environment lookup on a new loader")
}
l.lookup = lookup
return nil
}
}
// resolvedText is textual input bound according to the target scalar type. It
// never becomes an object, collection or null by interpreting environment data.
type resolvedText string
const maxExpandedBytes = 4 << 20
type environmentValue struct {
value string
present bool
}
type referenceResolver struct {
lookup func(string) (string, bool)
cache map[string]environmentValue
doc Document
nodes, bytes, cacheBytes int
}
func resolveDocument(doc Document, lookup func(string) (string, bool)) (Document, error) {
r := referenceResolver{lookup: lookup, cache: map[string]environmentValue{}, doc: Document{Positions: doc.Positions, environments: map[string][]string{}}}
v, err := r.walk(doc.Value, "", 0)
if err != nil {
return Document{}, err
}
r.doc.Value = v
return r.doc, nil
}
func (r *referenceResolver) walk(v any, path string, depth int) (any, error) {
r.nodes++
if depth > maxConfigDepth || r.nodes > maxConfigNodes {
return nil, r.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil)
}
switch x := v.(type) {
case string:
return r.text(x, path, false)
case Text:
return r.text(string(x), path, true)
case map[string]any:
out := make(map[string]any, len(x))
for _, key := range sortedKeys(x) {
value, err := r.walk(x[key], joinField(path, key), depth+1)
if err != nil {
return nil, err
}
out[key] = value
}
return out, nil
case []any:
out := make([]any, len(x))
for i, value := range x {
var err error
out[i], err = r.walk(value, fmt.Sprintf("%s[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
}
return out, nil
default:
return v, nil
}
}
func (r *referenceResolver) fail(path, name, reason string) error {
d := r.doc.diagnostic(path, reason, nil)
d.Env = name
return d
}
func validEnvironmentName(name string) bool {
if len(name) == 0 {
return false
}
for i := 0; i < len(name); i++ {
c := name[i]
if c == '_' || c >= 'A' && c <= 'Z' || c >= 'a' && c <= 'z' || i > 0 && c >= '0' && c <= '9' {
continue
}
return false
}
return true
}
func (r *referenceResolver) lookupValue(name, path string) (environmentValue, error) {
if v, ok := r.cache[name]; ok {
return v, nil
}
value, present := r.lookup(name)
if len(value) > maxConfigBytes || r.cacheBytes+len(value) > maxExpandedBytes {
return environmentValue{}, r.fail(path, name, "environment value exceeds configuration size limit")
}
if !utf8.ValidString(value) {
return environmentValue{}, r.fail(path, name, "environment value is not valid UTF-8")
}
r.cacheBytes += len(value)
v := environmentValue{value, present}
r.cache[name] = v
return v, nil
}
func (r *referenceResolver) text(raw, path string, xmlText bool) (any, error) {
var out strings.Builder
var names []string
seen := map[string]bool{}
if !strings.Contains(raw, "$") {
if len(raw) > maxConfigBytes || r.bytes+len(raw) > maxExpandedBytes {
return nil, r.fail(path, "", "expanded configuration exceeds size limit")
}
r.bytes += len(raw)
if xmlText {
return Text(raw), nil
}
return raw, nil
}
appendText := func(s string) error {
if out.Len()+len(s) > maxConfigBytes || r.bytes+len(s) > maxExpandedBytes {
return r.fail(path, strings.Join(names, ", "), "expanded configuration exceeds size limit")
}
out.WriteString(s)
r.bytes += len(s)
return nil
}
// Scan only the file's decoded scalar. Inserted environment values are never
// scanned again, including dollars, quotes, separators or newline characters.
for i := 0; i < len(raw); {
j := strings.IndexByte(raw[i:], '$')
if j < 0 {
if err := appendText(raw[i:]); err != nil {
return nil, err
}
break
}
j += i
if err := appendText(raw[i:j]); err != nil {
return nil, err
}
switch {
case j+1 < len(raw) && raw[j+1] == '$':
if err := appendText("$"); err != nil {
return nil, err
}
i = j + 2
case j+1 < len(raw) && raw[j+1] == '{':
end := strings.IndexByte(raw[j+2:], '}')
if end < 0 {
return nil, r.fail(path, "", "unclosed environment reference; use ${NAME}, ${NAME:-fallback} or $${...} for a literal")
}
end += j + 2
expression := raw[j+2 : end]
name, fallback, hasDefault := strings.Cut(expression, ":-")
if !validEnvironmentName(name) || strings.Contains(fallback, "${") {
return nil, r.fail(path, "", "invalid environment reference; use ${NAME} or ${NAME:-fallback}; nested references are not supported")
}
r.nodes++
if r.nodes > maxConfigNodes {
return nil, r.fail(path, name, "environment reference count exceeds configuration limit")
}
if !seen[name] {
names = append(names, name)
seen[name] = true
}
value, err := r.lookupValue(name, path)
if err != nil {
return nil, err
}
selected := value.value
if !value.present || selected == "" {
if !hasDefault {
return nil, r.fail(path, name, "required environment variable is unset or empty")
}
selected = fallback
}
if err := appendText(selected); err != nil {
return nil, err
}
i = end + 1
default:
if err := appendText("$"); err != nil {
return nil, err
}
i = j + 1
}
}
if len(names) > 0 {
r.doc.environments[path] = names
return resolvedText(out.String()), nil
}
if xmlText {
return Text(out.String()), nil
}
return out.String(), nil
}

230
config/references_test.go Normal file
View File

@@ -0,0 +1,230 @@
package config
import (
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"sync"
"testing"
"time"
)
type referenceConfig struct {
Port int `json:"port"`
BatchSize int `json:"batchSize"`
Enabled bool `json:"enabled"`
Password string `json:"password"`
Address string `json:"address"`
Timeout time.Duration `json:"timeout"`
Optional *int `json:"optional"`
Labels map[string]string `json:"labels"`
Numbers []int64 `json:"numbers"`
Dynamic any `json:"dynamic"`
}
func (c referenceConfig) Validate() error {
if c.Port < 0 {
return Invalid("port", "must not be negative")
}
return nil
}
func TestExplicitReferencesAcrossFormats(t *testing.T) {
// Delimiters, quotes and newlines belong to the credential, never file syntax.
secret := " DDSFLKJlsf:saf:-123\"\n${DO_NOT_READ}&<>\\ "
sources := map[string]string{
"json": `{"port":"${PORT:-3306}","enabled":"${ENABLED:-true}","password":"${PASSWORD}","address":"https://${HOST}:${PORT:-3306}/v1","timeout":"${TIMEOUT:-3s}","optional":"${ZERO}","numbers":["${BIG}"],"dynamic":"${ZERO}","labels":{"literal":"$${NOT_READ}","empty":"${OPTIONAL:-}","dollars":"$$cost"}}`,
"yaml": "port: \"${PORT:-3306}\"\nenabled: \"${ENABLED:-true}\"\npassword: \"${PASSWORD}\"\naddress: \"https://${HOST}:${PORT:-3306}/v1\"\ntimeout: \"${TIMEOUT:-3s}\"\noptional: \"${ZERO}\"\nnumbers: [\"${BIG}\"]\ndynamic: \"${ZERO}\"\nlabels:\n literal: \"$${NOT_READ}\"\n empty: \"${OPTIONAL:-}\"\n dollars: \"$$cost\"\n",
"xml": `<config><port>${PORT:-3306}</port><enabled>${ENABLED:-true}</enabled><password>${PASSWORD}</password><address>https://${HOST}:${PORT:-3306}/v1</address><timeout>${TIMEOUT:-3s}</timeout><optional>${ZERO}</optional><numbers type="array"><item>${BIG}</item></numbers><dynamic>${ZERO}</dynamic><labels><literal>$${NOT_READ}</literal><empty>${OPTIONAL:-}</empty><dollars>$$cost</dollars></labels></config>`,
}
for ext, source := range sources {
t.Run(ext, func(t *testing.T) {
calls := map[string]int{}
values := map[string]string{"PORT": "0", "ENABLED": "false", "PASSWORD": secret, "HOST": "example.test", "ZERO": "0", "BIG": "9007199254740993"}
l := testLoader(func(name string) (string, bool) { calls[name]++; v, ok := values[name]; return v, ok })
got, err := LoadProject(configFile(t, ext, source), referenceConfig{Port: 77, BatchSize: 100, Enabled: true, Password: "old"}, l)
if err != nil {
t.Fatal(err)
}
if got.Port != 0 || got.BatchSize != 100 || got.Enabled || got.Password != secret || got.Address != "https://example.test:0/v1" || got.Timeout != 3*time.Second || got.Optional == nil || *got.Optional != 0 || got.Numbers[0] != 9007199254740993 || got.Dynamic != "0" {
t.Fatal("binding changed values")
}
if got.Labels["literal"] != "${NOT_READ}" || got.Labels["empty"] != "" || got.Labels["dollars"] != "$cost" {
t.Fatal(got.Labels)
}
for name, n := range calls {
if n != 1 {
t.Fatal("not cached", name, n)
}
}
if calls["DO_NOT_READ"] != 0 || calls["NOT_READ"] != 0 {
t.Fatal("recursive expansion")
}
})
}
}
func TestReferenceFallbackAndFixedValues(t *testing.T) {
for _, present := range []bool{false, true} {
l := testLoader(func(string) (string, bool) { return "", present })
got, err := LoadProject(configFile(t, "yaml", "port: \"${PORT:-3306}\"\npassword: \"${PASSWORD:-DDSFLKJlsf:saf:-123}\""), referenceConfig{}, l)
if err != nil || got.Port != 3306 || got.Password != "DDSFLKJlsf:saf:-123" {
t.Fatal(got, err)
}
for _, ext := range []string{"json", "yaml", "xml"} {
raw := `{"password":"${PASSWORD}"}`
if ext == "yaml" {
raw = "password: '${PASSWORD}'"
}
if ext == "xml" {
raw = "<config><password>${PASSWORD}</password></config>"
}
path := configFile(t, ext, raw)
_, err = LoadProject(path, referenceConfig{Password: "initial"}, l)
var d *Error
if !errors.As(err, &d) || d.Env != "PASSWORD" || d.File != path || d.Field != "password" || d.Line < 1 {
t.Fatal("missing required-reference diagnostic", err)
}
}
}
calls := 0
l := testLoader(func(string) (string, bool) { calls++; return "999", true })
got, err := LoadProject(configFile(t, "json", `{"port":0,"enabled":false,"password":""}`), referenceConfig{Port: 77, BatchSize: 100, Enabled: true, Password: "old"}, l)
if err != nil || got.Port != 0 || got.BatchSize != 100 || got.Enabled || got.Password != "" || calls != 0 {
t.Fatal(got, err, calls)
}
got, err = LoadProject(configFile(t, "yaml", "password: '${VALUE}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return " ", true }))
if err != nil || got.Password != " " {
t.Fatal("trimmed credential", err)
}
}
func TestReferenceDiagnosticsAndScalarLimits(t *testing.T) {
for _, value := range []string{"private-not-int", "92233720368547758080", "false", "null", "-1"} {
path := configFile(t, "yaml", "port: '${PORT}'")
_, err := LoadProject(path, referenceConfig{}, testLoader(func(string) (string, bool) { return value, true }))
var d *Error
if !errors.As(err, &d) || d.File != path || d.Env != "PORT" || d.Field != "port" || d.Line != 1 || d.Column < 1 || strings.Contains(err.Error(), value) {
t.Fatalf("invalid diagnostic: %v", err)
}
}
for _, raw := range []string{"${BAD-123}", "${}", "${X", "${1BAD}", "${BAD NAME}", "${X:-${Y}}"} {
encoded, _ := json.Marshal(map[string]string{"password": raw})
if _, err := LoadProject(configFile(t, "json", string(encoded)), referenceConfig{}, testLoader(noEnvironment)); err == nil {
t.Fatal("invalid syntax accepted", raw)
}
}
for _, value := range []string{string([]byte{0xff}), strings.Repeat("x", maxConfigBytes+1)} {
if _, err := LoadProject(configFile(t, "yaml", "password: '${SECRET}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return value, true })); err == nil {
t.Fatal("unbounded/invalid environment")
}
}
path := configFile(t, "yaml", "labels:\n a: '${BIG}'\n b: '${BIG}'\n c: '${BIG}'\n d: '${BIG}'\n e: '${BIG}'")
if _, err := LoadProject(path, referenceConfig{}, testLoader(func(string) (string, bool) { return strings.Repeat("x", maxConfigBytes), true })); err == nil {
t.Fatal("expansion budget bypassed")
}
// An environment value cannot inject an array/object or coerce dynamic null.
for _, raw := range []string{"[1,2]", "{\"x\":1}", "null"} {
_, err := LoadProject(configFile(t, "yaml", "numbers: '${VALUE}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return raw, true }))
if err == nil {
t.Fatal("structure injected")
}
}
}
func TestReferenceCacheIsolationAndRemovedRules(t *testing.T) {
path := configFile(t, "yaml", "password: '${TOKEN}'")
for i := 0; i < 24; i++ {
t.Run(fmt.Sprint(i), func(t *testing.T) {
t.Parallel()
v := fmt.Sprint(i)
l := testLoader(func(string) (string, bool) { return v, true })
for n := 0; n < 4; n++ {
got, err := LoadProject(path, referenceConfig{}, l)
if err != nil || got.Password != v {
t.Fatal("load leaked", err)
}
}
})
}
}
func TestReferenceLookupPerLoadAndConcurrentLoader(t *testing.T) {
var mu sync.Mutex
reads := 0
l := testLoader(func(string) (string, bool) { mu.Lock(); defer mu.Unlock(); reads++; return "secret", true })
path := configFile(t, "yaml", "password: '${TOKEN}${TOKEN}'")
var wg sync.WaitGroup
for i := 0; i < 24; i++ {
wg.Add(1)
go func() {
defer wg.Done()
got, err := LoadProject(path, referenceConfig{}, l)
if err != nil || got.Password != "secretsecret" {
t.Error(err)
}
}()
}
wg.Wait()
if reads != 24 {
t.Fatal(reads)
}
}
func TestRemovedEnvTagsAndFieldSuggestion(t *testing.T) {
type old struct {
Value int `json:"value" envDefault:"9"`
}
if _, err := LoadProject("", old{}); err == nil || !strings.Contains(err.Error(), "removed") {
t.Fatal(err)
}
type oldSecret struct {
Token string `json:"-" env:"TOKEN"`
}
if _, err := DecodeFile(configFile(t, "json", `{}`), oldSecret{}); err == nil {
t.Fatal("old env tag ignored")
}
_, err := LoadProject(configFile(t, "yaml", "batchSiz: 10"), referenceConfig{})
if err == nil || !strings.Contains(err.Error(), `did you mean "batchSize"`) {
t.Fatal(err)
}
t.Setenv("GO_OMNI_BATCH_SIZE", "999")
got, err := LoadProject("", referenceConfig{BatchSize: 100})
if err != nil || got.BatchSize != 100 {
t.Fatal("implicit overlay", err)
}
}
func FuzzConfigurationReferences(f *testing.F) {
for _, s := range []string{"${X}", "$${X}", "${X:-DDS:abc:-123}", "${X:-}", "https://${X}/${X}", "${X:-${Y}}"} {
f.Add(s, " secret${NO}\n\" ")
}
f.Fuzz(func(t *testing.T, raw, value string) {
if len(raw) > 8192 || len(value) > 8192 {
return
}
doc := Document{Value: map[string]any{"password": raw}}
resolved, err := resolveDocument(doc, func(string) (string, bool) { return value, true })
if err != nil {
return
}
_, _ = bindDocument(resolved, referenceConfig{})
})
}
func BenchmarkExplicitConfiguration(b *testing.B) {
for _, refs := range []bool{false, true} {
b.Run(fmt.Sprint(refs), func(b *testing.B) {
source := `{"port":3306,"enabled":false,"password":"fixed","batchSize":0}`
if refs {
source = `{"port":"${PORT:-3306}","enabled":"${ENABLED:-false}","password":"${PASSWORD}","batchSize":0}`
}
path := b.TempDir() + "/config.json"
if err := os.WriteFile(path, []byte(source), 0600); err != nil {
b.Fatal(err)
}
l := testLoader(func(k string) (string, bool) { return "private", k == "PASSWORD" })
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := LoadProject(path, referenceConfig{BatchSize: 100}, l); err != nil {
b.Fatal(err)
}
}
})
}
}

51
config/suggestions.go Normal file
View File

@@ -0,0 +1,51 @@
package config
import (
"reflect"
"strconv"
"strings"
)
// Suggest only a unique, nearby declared field. Input values are never used.
func unknownFieldReason(name string, fields map[string]reflect.StructField) string {
if len(name) < 3 || len(name) > 128 {
return "unknown field"
}
best, match, tied := 3, "", false
for candidate := range fields {
if len(candidate) > 128 {
continue
}
d := fieldDistance(strings.ToLower(name), strings.ToLower(candidate))
if d < best {
best, match, tied = d, candidate, false
} else if d == best {
tied = true
}
}
if match != "" && !tied && best <= 2 {
return "unknown field; did you mean " + strconv.Quote(match) + "?"
}
return "unknown field"
}
func fieldDistance(a, b string) int {
prev := make([]int, len(b)+1)
for i := range prev {
prev[i] = i
}
for i := 0; i < len(a); i++ {
left, diagonal := i+1, prev[0]
prev[0] = left
for j := 0; j < len(b); j++ {
cost := 0
if a[i] != b[j] {
cost = 1
}
old := prev[j+1]
left = min(left+1, old+1, diagonal+cost)
prev[j+1] = left
diagonal = old
}
}
return prev[len(b)]
}

View File

@@ -6,7 +6,7 @@
`database` 包提供基于标准库 `database/sql` 的连接池生命周期,不绑定 ORM 或驱动: `database` 包提供基于标准库 `database/sql` 的连接池生命周期,不绑定 ORM 或驱动:
- `Config` / `ConfigFromEnv`:连接池参数和 `GO_OMNI_DB_*` 环境变量。 - `Config`:驱动、DSN 与连接池参数;嵌入项目配置,通过文件显式引用变量。
- `Open` / `OpenConnector`:通过 DSN 或标准 Connector 创建连接池、设置连接上限并在返回前执行 `PingContext`。 - `Open` / `OpenConnector`:通过 DSN 或标准 Connector 创建连接池、设置连接上限并在返回前执行 `PingContext`。
- `Health`:用于就绪检查,等待连接和 Ping 均受 `PingTimeout` 限制。 - `Health`:用于就绪检查,等待连接和 Ping 均受 `PingTimeout` 限制。
- `InTx`:统一事务提交和回滚;工作函数返回错误或 panic 时回滚。 - `InTx`:统一事务提交和回滚;工作函数返回错误或 panic 时回滚。
@@ -17,11 +17,12 @@
```go ```go
import ( import (
"context" "context"
"git.yuntuo.hk/k2safe/go-omni/config"
_ "github.com/go-sql-driver/mysql" _ "github.com/go-sql-driver/mysql"
"git.yuntuo.hk/k2safe/go-omni/database" "git.yuntuo.hk/k2safe/go-omni/database"
) )
cfg, err := database.ConfigFromEnv("GO_OMNI_DB_") cfg, err := config.LoadProject("etc/database.yaml", database.DefaultConfig())
if err != nil { if err != nil {
return err return err
} }
@@ -32,19 +33,15 @@ if err != nil {
defer db.Close() defer db.Close()
``` ```
连接池相关环境变量: 对应文件(框架不规定变量名):
```text ```yaml
GO_OMNI_DB_DRIVER=mysql driver: mysql
GO_OMNI_DB_DSN=user:password@tcp(127.0.0.1:3306)/app?parseTime=true dsn: "${DB_DSN}"
GO_OMNI_DB_MAX_OPEN_CONNS=10 maxOpenConns: "${DB_MAX_OPEN:-10}"
GO_OMNI_DB_MAX_IDLE_CONNS=5
GO_OMNI_DB_CONN_MAX_LIFETIME=30m
GO_OMNI_DB_CONN_MAX_IDLE_TIME=5m
GO_OMNI_DB_PING_TIMEOUT=5s
``` ```
使用 `ConfigFromEnv` / `Open` 时 `DSN` 必填,环境变量入口的驱动默认是 `mysql`。数据库建库、迁移、表结构和具体 Repository 属于业务项目或可选模块,不会被核心包自动执行。 Open 使用的 Driver、DSN 必填。建库、迁移、表结构和业务 Repository 由项目负责。v0.22 删除 ConfigFromEnv/ApplyEnv;不再隐式扫描数据库变量。
`MaxIdleConns=0` 会禁用空闲连接。初始化 Ping 失败会关闭已创建的连接池。 `MaxIdleConns=0` 会禁用空闲连接。初始化 Ping 失败会关闭已创建的连接池。
@@ -62,12 +59,9 @@ python3 scripts/verifyDatabase.py --version v0.2.0
结束时清理容器、匿名数据卷和临时项目。常规 `go test ./...` 不依赖 Docker。 结束时清理容器、匿名数据卷和临时项目。常规 `go test ./...` 不依赖 Docker。
## 项目配置叠加 ## 项目配置
`database.ApplyEnv(cfg, prefix)` 在已有文件/默认配置上覆盖非空环境变量,保留未覆盖的池参数。 配置结构作为项目 Config 的字段,Default() 初始化连接池,文件按需覆盖。DSN 可使用 `dsn: "${DB_DSN}"`;database.Config 的 JSON 序列化不输出 DSN 原值。不要打印原始配置对象。当前 API 脚手架使用结构化 MySQL 适配器,由 svc 统一初始化与关闭。
`Config` 的 JSON/YAML 字段使用 lowerCamelCase,`DSN` 使用 `json:"-"`,连接密钥从环境变量加载。
API 脚手架 `--database mysql` 使用此流程,并要求正的 `maxOpenConns`,提供数据库 readiness 与幂等清理。
具体表结构、迁移和 Repository 留在业务项目,参考 `examples/userService`。
## Connector 接入(v0.9.1-preview) ## Connector 接入(v0.9.1-preview)
@@ -103,7 +97,7 @@ func Open(ctx context.Context, address, user, password, name string,
} }
``` ```
调用时以 `database.DefaultPoolConfig()` 为起点,按项目需要调整。已有 `database.Config` 可调用 `.PoolConfig()` 提取池设置;`ConfigFromEnv` / `ApplyEnv` 仍是 DSN 入口,不用为 Connector 填假 DSN。`PoolConfig` 支持通过 `config.DecodeFile` 加载项目的 JSON/YAML 嵌套配置,环境覆盖由项目按自己的驱动配置组织。 调用时以 `database.DefaultPoolConfig()` 为起点,按项目需要调整。已有 `database.Config` 可调用 `.PoolConfig()` 提取池设置;Connector 不需要填写 DSN。`PoolConfig` 支持通过 `config.DecodeFile` 加载项目的 JSON/YAML 嵌套配置,环境引用由项目在文件里显式声明。
| PoolConfig 字段 | 默认值 / 含义 | | PoolConfig 字段 | 默认值 / 含义 |
| --- | --- | | --- | --- |

View File

@@ -2,19 +2,27 @@ package database
import ( import (
"encoding/json" "encoding/json"
"git.yuntuo.hk/k2safe/go-omni/config"
"os"
"path/filepath"
"strings" "strings"
"testing" "testing"
"time" "time"
) )
func TestApplyEnvPreservesFilePoolAndHidesDSN(t *testing.T) { func TestExplicitConfigPreservesPoolAndHidesDSN(t *testing.T) {
cfg := DefaultConfig() cfg := DefaultConfig()
cfg.Driver = "mysql"
path := filepath.Join(t.TempDir(), "db.yaml")
if err := os.WriteFile(path, []byte("dsn: \"${OMNI_TEST_DSN}\"\nmaxIdleConns: \"${OMNI_TEST_MAX_IDLE_CONNS}\"\nmaxOpenConns: \"${OMNI_TEST_MAX_OPEN_CONNS:-3}\""), 0600); err != nil {
t.Fatal(err)
}
cfg.MaxOpenConns = 3 cfg.MaxOpenConns = 3
cfg.MaxIdleConns = 1 cfg.MaxIdleConns = 1
cfg.PingTimeout = time.Second cfg.PingTimeout = time.Second
t.Setenv("OMNI_TEST_DSN", "user:secret@tcp(localhost:3306)/test") t.Setenv("OMNI_TEST_DSN", "user:secret@tcp(localhost:3306)/test")
t.Setenv("OMNI_TEST_MAX_IDLE_CONNS", "0") t.Setenv("OMNI_TEST_MAX_IDLE_CONNS", "0")
got, err := ApplyEnv(cfg, "OMNI_TEST_") got, err := config.LoadProject(path, cfg)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -29,7 +37,7 @@ func TestApplyEnvPreservesFilePoolAndHidesDSN(t *testing.T) {
t.Fatalf("unsafe config encoding: %s", data) t.Fatalf("unsafe config encoding: %s", data)
} }
t.Setenv("OMNI_TEST_MAX_OPEN_CONNS", "bad") t.Setenv("OMNI_TEST_MAX_OPEN_CONNS", "bad")
if _, err = ApplyEnv(cfg, "OMNI_TEST_"); err == nil { if _, err = config.LoadProject(path, cfg); err == nil {
t.Fatal("invalid override accepted") t.Fatal("invalid override accepted")
} }
} }

View File

@@ -5,17 +5,16 @@ package database
import ( import (
"context" "context"
"database/sql" "database/sql"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"os"
"strconv"
"strings" "strings"
"time" "time"
) )
type Config struct { type Config struct {
Driver string `json:"driver"` Driver string `json:"driver"`
DSN string `json:"-"` DSN string `json:"dsn"`
MaxOpenConns int `json:"maxOpenConns"` MaxOpenConns int `json:"maxOpenConns"`
MaxIdleConns int `json:"maxIdleConns"` MaxIdleConns int `json:"maxIdleConns"`
ConnMaxLifetime time.Duration `json:"connMaxLifetime"` ConnMaxLifetime time.Duration `json:"connMaxLifetime"`
@@ -23,6 +22,14 @@ type Config struct {
PingTimeout time.Duration `json:"pingTimeout"` PingTimeout time.Duration `json:"pingTimeout"`
} }
// MarshalJSON omits the connection credential from serialized diagnostics.
func (c Config) MarshalJSON() ([]byte, error) {
type plain Config
safe := plain(c)
safe.DSN = ""
return json.Marshal(safe)
}
func DefaultConfig() Config { return DefaultPoolConfig().config() } func DefaultConfig() Config { return DefaultPoolConfig().config() }
// PoolConfig returns driver-independent connection pool settings. // PoolConfig returns driver-independent connection pool settings.
@@ -34,49 +41,6 @@ func (c Config) PoolConfig() PoolConfig {
} }
} }
// ConfigFromEnv reads GO_OMNI_DB_* style settings. DSN is required.
func ConfigFromEnv(prefix string) (Config, error) {
return ApplyEnv(DefaultConfig(), prefix)
}
// ApplyEnv overlays nonempty environment settings on file/default configuration.
// DSN is environment-only when Config is decoded from JSON or YAML.
func ApplyEnv(cfg Config, prefix string) (Config, error) {
prefix = strings.TrimSpace(prefix)
if prefix == "" {
prefix = "GO_OMNI_DB_"
}
if value := strings.TrimSpace(os.Getenv(prefix + "DRIVER")); value != "" {
cfg.Driver = value
}
if cfg.Driver == "" {
cfg.Driver = "mysql"
}
if value := strings.TrimSpace(os.Getenv(prefix + "DSN")); value != "" {
cfg.DSN = value
}
if cfg.DSN == "" {
return Config{}, fmt.Errorf("%sDSN is required", prefix)
}
var err error
if cfg.MaxOpenConns, err = envInt(prefix+"MAX_OPEN_CONNS", cfg.MaxOpenConns); err != nil {
return Config{}, err
}
if cfg.MaxIdleConns, err = envInt(prefix+"MAX_IDLE_CONNS", cfg.MaxIdleConns); err != nil {
return Config{}, err
}
if cfg.ConnMaxLifetime, err = envDuration(prefix+"CONN_MAX_LIFETIME", cfg.ConnMaxLifetime); err != nil {
return Config{}, err
}
if cfg.ConnMaxIdleTime, err = envDuration(prefix+"CONN_MAX_IDLE_TIME", cfg.ConnMaxIdleTime); err != nil {
return Config{}, err
}
if cfg.PingTimeout, err = envDuration(prefix+"PING_TIMEOUT", cfg.PingTimeout); err != nil {
return Config{}, err
}
return cfg, cfg.Validate()
}
func (c Config) Validate() error { func (c Config) Validate() error {
if strings.TrimSpace(c.Driver) == "" { if strings.TrimSpace(c.Driver) == "" {
return errors.New("database driver is required") return errors.New("database driver is required")
@@ -165,27 +129,3 @@ func (db *DB) InTx(ctx context.Context, options *sql.TxOptions, work func(*sql.T
} }
return nil return nil
} }
func envInt(key string, fallback int) (int, error) {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback, nil
}
parsed, err := strconv.Atoi(value)
if err != nil {
return 0, fmt.Errorf("%s must be an integer: %w", key, err)
}
return parsed, nil
}
func envDuration(key string, fallback time.Duration) (time.Duration, error) {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback, nil
}
parsed, err := time.ParseDuration(value)
if err != nil {
return 0, fmt.Errorf("%s must be a duration: %w", key, err)
}
return parsed, nil
}

View File

@@ -2,54 +2,10 @@ package database
import ( import (
"context" "context"
"os"
"strings" "strings"
"testing" "testing"
"time"
) )
func TestConfigFromEnv(t *testing.T) {
keys := []string{
"GO_OMNI_DB_DRIVER", "GO_OMNI_DB_DSN", "GO_OMNI_DB_MAX_OPEN_CONNS",
"GO_OMNI_DB_MAX_IDLE_CONNS", "GO_OMNI_DB_CONN_MAX_LIFETIME",
"GO_OMNI_DB_CONN_MAX_IDLE_TIME", "GO_OMNI_DB_PING_TIMEOUT",
}
for _, key := range keys {
t.Setenv(key, "")
_ = os.Unsetenv(key)
}
t.Setenv("GO_OMNI_DB_DSN", "root:secret@tcp(127.0.0.1:3306)/demo")
t.Setenv("GO_OMNI_DB_MAX_OPEN_CONNS", "20")
t.Setenv("GO_OMNI_DB_MAX_IDLE_CONNS", "8")
t.Setenv("GO_OMNI_DB_CONN_MAX_LIFETIME", "2m")
t.Setenv("GO_OMNI_DB_PING_TIMEOUT", "750ms")
cfg, err := ConfigFromEnv("")
if err != nil {
t.Fatal(err)
}
if cfg.Driver != "mysql" || cfg.DSN == "" || cfg.MaxOpenConns != 20 || cfg.MaxIdleConns != 8 || cfg.ConnMaxLifetime != 2*time.Minute || cfg.PingTimeout != 750*time.Millisecond {
t.Fatalf("unexpected config: %+v", cfg)
}
}
func TestConfigFromEnvRejectsInvalidValues(t *testing.T) {
t.Setenv("GO_OMNI_DB_DSN", "demo")
t.Setenv("GO_OMNI_DB_MAX_OPEN_CONNS", "bad")
if _, err := ConfigFromEnv(""); err == nil || !strings.Contains(err.Error(), "MAX_OPEN_CONNS") {
t.Fatalf("expected invalid pool limit error, got %v", err)
}
t.Setenv("GO_OMNI_DB_MAX_OPEN_CONNS", "2")
t.Setenv("GO_OMNI_DB_MAX_IDLE_CONNS", "3")
if _, err := ConfigFromEnv(""); err == nil || !strings.Contains(err.Error(), "max idle") {
t.Fatalf("expected pool relationship error, got %v", err)
}
t.Setenv("GO_OMNI_DB_MAX_IDLE_CONNS", "1")
t.Setenv("GO_OMNI_DB_PING_TIMEOUT", "0s")
if _, err := ConfigFromEnv(""); err == nil || !strings.Contains(err.Error(), "timeouts") {
t.Fatalf("expected timeout error, got %v", err)
}
}
func TestOpenValidatesBeforeLoadingDriver(t *testing.T) { func TestOpenValidatesBeforeLoadingDriver(t *testing.T) {
_, err := Open(context.Background(), Config{Driver: "", DSN: "demo"}) _, err := Open(context.Background(), Config{Driver: "", DSN: "demo"})
if err == nil || !strings.Contains(err.Error(), "driver") { if err == nil || !strings.Contains(err.Error(), "driver") {

View File

@@ -10,13 +10,14 @@
go-omni new --database mysql --module example.com/billing ./billing go-omni new --database mysql --module example.com/billing ./billing
``` ```
`etc/config.yaml` 使用结构化配置,环境覆盖优先: `etc/config.yaml` 使用结构化配置;需要环境变量时在字段值明确写引用,固定值不被覆盖:
```yaml ```yaml
database: database:
host: db.example.com host: db.example.com
port: 3306 port: 3306
user: app user: app
password: "${DB_PASSWORD}"
database: billing database: billing
parseTime: true parseTime: true
location: UTC location: UTC
@@ -37,7 +38,7 @@ database:
pingTimeout: 5s pingTimeout: 5s
``` ```
通过 `GO_OMNI_DB_PASSWORD` 传密码;还有 HOST、PORT、USER、DATABASE、TLS_CA_FILE、TLS_CERT_FILE、TLS_KEY_FILE、POOL_MAX_OPEN_CONNS 等。业务也可自定义 env 标签。新生成服务不再读取 DSN;独立数据库迁移 CLI 和手写 `database.Open/OpenConnector` 的原接口保留。 示例通过文件里的 `${DB_PASSWORD}` 读取密码;HOST、PORT、TLS 和池参数同样按需在文件写引用,不自动扫描变量、不使用 env 标签。新生成服务使用结构化连接参数;独立迁移 CLI 的显式 DSN 参数及 Open/OpenConnector 接口保留。
默认启用 TLS 并校验系统信任链与主机名;私有 CA 用 caFile,双向 TLS 额外设置 certFile/keyFile。只有明确设置 `tls.mode: disabled` 才使用明文。Unix socket 用 socket,若在 socket 上使用 TLS,必须提供 serverName。不会注册全局 TLS 配置或全局驱动 Logger。 默认启用 TLS 并校验系统信任链与主机名;私有 CA 用 caFile,双向 TLS 额外设置 certFile/keyFile。只有明确设置 `tls.mode: disabled` 才使用明文。Unix socket 用 socket,若在 socket 上使用 TLS,必须提供 serverName。不会注册全局 TLS 配置或全局驱动 Logger。

View File

@@ -28,7 +28,7 @@ type Config struct {
Port int `json:"port"` Port int `json:"port"`
Socket string `json:"socket"` Socket string `json:"socket"`
User string `json:"user"` User string `json:"user"`
Password string `json:"password" env:",secret"` Password string `json:"password"`
Database string `json:"database"` Database string `json:"database"`
ParseTime bool `json:"parseTime"` ParseTime bool `json:"parseTime"`
Location string `json:"location"` Location string `json:"location"`

View File

@@ -10,11 +10,13 @@ DEMO_CLIENT_TIMEOUT=500ms DEMO_MAX_ITEMS=0 go run ./examples/configuration
DEMO_MAX_ITEMS=wrong go run ./examples/configuration DEMO_MAX_ITEMS=wrong go run ./examples/configuration
``` ```
前三次都输出 `timeout=2s maxItems=100 allowed=[json xml]`。环境覆盖允许显式零值。 前三次都输出 `timeout=2s maxItems=100 allowed=[json xml]`。文件显式引用允许零值。
最后一条输出定位 `maxItems` / `DEMO_MAX_ITEMS` 的安全错误,不打印输入原值。 最后一条输出定位 `maxItems` / `DEMO_MAX_ITEMS` 的安全错误,不打印输入原值。
也可 `-config /path/to/app.yaml` 读取自己的文件。示例没有建立网络连接。 也可 `-config /path/to/app.yaml` 读取自己的文件。示例没有建立网络连接。
`Config` 定义字段,`envDefault` 定义缺省值,`Validate` 用 `config.Invalid` 返回字段与公开原因。 `Config` 定义字段,`Default()` 定义初始值,`Validate` 用 `config.Invalid` 返回字段与公开原因。
文件中的 `${NAME}` 必填,`${NAME:-value}` 支持空/未设置的备用值,`$${NAME}` 保留字面量;字符串不裁剪,也不二次展开。没有出现在文件引用中的变量不会被读取。
`newLoader` 额外注册 `.custom`(演示复用 JSON decoder);同一个 `load` 可读取所有格式。 `newLoader` 额外注册 `.custom`(演示复用 JSON decoder);同一个 `load` 可读取所有格式。
JSON/YAML/XML 直接可用,无需注册。新增格式只修改解析器注册,不改业务结构或校验。 JSON/YAML/XML 直接可用,无需注册。新增格式只修改解析器注册,不改业务结构或校验。

View File

@@ -1,5 +1 @@
{ {"client":{"endpoint":"https://${DEMO_HOST:-example.com}","timeout":"${DEMO_CLIENT_TIMEOUT:-2s}","token":"${DEMO_CLIENT_TOKEN:-}"},"maxItems":"${DEMO_MAX_ITEMS:-100}","allowed":["json","xml"]}
"client": {"endpoint": "https://example.com", "timeout": "2s"},
"maxItems": 100,
"allowed": ["json", "xml"]
}

View File

@@ -1,8 +1 @@
<config> <config><client><endpoint>https://${DEMO_HOST:-example.com}</endpoint><timeout>${DEMO_CLIENT_TIMEOUT:-2s}</timeout><token>${DEMO_CLIENT_TOKEN:-}</token></client><maxItems>${DEMO_MAX_ITEMS:-100}</maxItems><allowed type="array"><item>json</item><item>xml</item></allowed></config>
<client>
<endpoint>https://example.com</endpoint>
<timeout>2s</timeout>
</client>
<maxItems>100</maxItems>
<allowed type="array"><item>json</item><item>xml</item></allowed>
</config>

View File

@@ -1,5 +1,6 @@
client: client:
endpoint: https://example.com endpoint: "https://${DEMO_HOST:-example.com}"
timeout: 2s timeout: "${DEMO_CLIENT_TIMEOUT:-2s}"
maxItems: 100 token: "${DEMO_CLIENT_TOKEN:-}"
maxItems: "${DEMO_MAX_ITEMS:-100}"
allowed: [json, xml] allowed: [json, xml]

View File

@@ -17,16 +17,18 @@ var examples embed.FS
type ClientConfig struct { type ClientConfig struct {
Endpoint string `json:"endpoint"` Endpoint string `json:"endpoint"`
Timeout time.Duration `json:"timeout" envDefault:"3s"` Timeout time.Duration `json:"timeout"`
// This field is deliberately unavailable to configuration files. // The file chooses a fixed value or an explicit environment reference.
Token string `json:"-" env:"DEMO_CLIENT_TOKEN,secret"` Token string `json:"token"`
} }
type Config struct { type Config struct {
Client ClientConfig `json:"client"` Client ClientConfig `json:"client"`
MaxItems int `json:"maxItems" envDefault:"20"` MaxItems int `json:"maxItems"`
Allowed []string `json:"allowed"` Allowed []string `json:"allowed"`
} }
func Default() Config { return Config{Client: ClientConfig{Timeout: 3 * time.Second}, MaxItems: 20} }
func (c Config) Validate() error { func (c Config) Validate() error {
if c.Client.Timeout <= 0 { if c.Client.Timeout <= 0 {
return config.Invalid("client.timeout", "must be positive") return config.Invalid("client.timeout", "must be positive")
@@ -48,7 +50,7 @@ func load(path string) (Config, error) {
if err != nil { if err != nil {
return Config{}, err return Config{}, err
} }
return config.LoadProject(path, Config{}, config.EnvOptions{Prefix: "DEMO_", Empty: "reject"}, loader) return config.LoadProject(path, Default(), loader)
} }
func run() error { func run() error {
path := flag.String("config", "", "external configuration file") path := flag.String("config", "", "external configuration file")

View File

@@ -0,0 +1,12 @@
# HTTP Client 示例
```sh
go run ./examples/httpClient
```
只创建本机临时 HTTP 服务,不访问业务系统、数据库或公网;结束后关闭连接和监听器。
示例创建并复用一个 Client,显式直连,配置连接池/阶段超时,通过 JSON 选项传入 Authorization 和 Accept,并按单次请求收紧响应上限。
输出 `authenticated request succeeded`。示例 token 是固定演示字符串,实际项目从配置或请求上下文取得认证信息。
完整参数、代理规则、流式调用、生命周期、重试边界和兼容说明见 [httpc](../../httpc/README.md)。

View File

@@ -0,0 +1,57 @@
// Run with go run ./examples/httpClient. All HTTP traffic stays on loopback.
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"time"
"git.yuntuo.hk/k2safe/go-omni/httpc"
)
type result struct {
Message string `json:"message"`
}
func run() error {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer example-token" {
w.WriteHeader(http.StatusUnauthorized)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(result{Message: "authenticated request succeeded"})
}))
defer upstream.Close()
client, err := httpc.New(httpc.Config{
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
Timeout: 3 * time.Second, DialTimeout: time.Second, TLSHandshakeTimeout: time.Second,
ResponseHeaderTimeout: time.Second, IdleConnTimeout: 30 * time.Second,
MaxConnsPerHost: 16, MaxIdleConnsPerHost: 8, MaxIdleConns: 32,
MaxResponseBytes: 1 << 20, DisableLogging: true,
})
if err != nil {
return err
}
defer client.Close()
var output result
if err = client.JSON(context.Background(), http.MethodGet, upstream.URL, nil, &output,
httpc.WithHeader("Authorization", "Bearer example-token"),
httpc.WithHeaders(http.Header{"Accept": []string{"application/json"}}),
httpc.WithMaxResponseBytes(64<<10),
); err != nil {
return err
}
fmt.Println(output.Message)
return nil
}
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}

View File

@@ -0,0 +1,9 @@
package main
import "testing"
func TestExample(t *testing.T) {
if err := run(); err != nil {
t.Fatal(err)
}
}

View File

@@ -1,6 +1,6 @@
{ {
"generatorConfig": true, "generatorConfig": true,
"templateVersion": "v0.20.0-preview", "templateVersion": "v0.22.0-preview",
"profile": { "profile": {
"database": "none", "database": "none",
"storage": "none", "storage": "none",
@@ -12,7 +12,7 @@
"entry": "api/managed.api", "entry": "api/managed.api",
"generated": { "generated": {
"docs/openapi.json": "0370fd6bd6a387e2ef21f6b42b009ce66749846b8bea3d481009a34f3af039a5", "docs/openapi.json": "0370fd6bd6a387e2ef21f6b42b009ce66749846b8bea3d481009a34f3af039a5",
"internal/config/loadGen.go": "5d630b894abdbbd66ddfa7d7d30f7573fc4ff995020a32a9f9157cd4af3a5655", "internal/config/loadGen.go": "d85e3123e4c96c95cb0886e60e4760230426efca4d2bbbfc981822da08b90d13",
"internal/handler/appGen.go": "f5251b4dc33e2312b0bdb71ead70d2171977d851265e124f640458d705a2cc9d", "internal/handler/appGen.go": "f5251b4dc33e2312b0bdb71ead70d2171977d851265e124f640458d705a2cc9d",
"internal/handler/demo/handlersGen.go": "3c76d85dbe89eb5127d9d134dc625203ecf9719dd11d174c1f335691b99e1375", "internal/handler/demo/handlersGen.go": "3c76d85dbe89eb5127d9d134dc625203ecf9719dd11d174c1f335691b99e1375",
"internal/handler/openapiGen.go": "f24cab3abbb8557a3fc4dd93fba23cc0b093f3e0c944d3ba1bd5b0eeff2d96f3", "internal/handler/openapiGen.go": "f24cab3abbb8557a3fc4dd93fba23cc0b093f3e0c944d3ba1bd5b0eeff2d96f3",
@@ -22,7 +22,7 @@
"internal/svc/runtimeGen.go": "29f6afb6614881935099c1845ecb3f7902ee773caa077940f566cb8dfdbcdbb1", "internal/svc/runtimeGen.go": "29f6afb6614881935099c1845ecb3f7902ee773caa077940f566cb8dfdbcdbb1",
"internal/types/dtoManagedd28645a2af1cGen.go": "7586e86fa5d67051283178cc578c890f6c02fd2d5c383170a71a39c2e77ce344", "internal/types/dtoManagedd28645a2af1cGen.go": "7586e86fa5d67051283178cc578c890f6c02fd2d5c383170a71a39c2e77ce344",
"internal/types/typesGen.go": "372d67f23a313b3e77d2a3b7a236315db9721bc63350bf1f71db1293e57570d1", "internal/types/typesGen.go": "372d67f23a313b3e77d2a3b7a236315db9721bc63350bf1f71db1293e57570d1",
"main.go": "64e26d97de7e7c6c17f47186785edc73d43c23d6c20b50bfc7fff6ef2e914f17" "main.go": "10cb6a84b2eb1b68dc10c2dca949c9dd7c68a9f78a8e6d8f56ce9336d4b2d125"
}, },
"contract": { "contract": {
"routes": { "routes": {

View File

@@ -46,6 +46,6 @@ Configuration, API contracts, svc fields/Init/AppOptions, Logic, Repository and
middleware implementations remain developer-owned. Regeneration preserves them. middleware implementations remain developer-owned. Regeneration preserves them.
Use `go-omni api format` to format local contracts, then `api generate` and Use `go-omni api format` to format local contracts, then `api generate` and
`api generate --check`. See the [framework development guide](../../DEVELOPMENT.md) `api generate --check`. See the [framework development guide](../../DEVELOPMENT.md)
for array/null responses, typed multipart, full configuration environment overrides, for array/null responses, typed multipart, explicit configuration references and -check-config,
shared contracts and v0.13 upgrade steps. This example keeps its raw callback, shared contracts and v0.13 upgrade steps. This example keeps its raw callback,
streaming upload and download to exercise those existing extension points. streaming upload and download to exercise those existing extension points.

View File

@@ -1,16 +1,3 @@
# Non-secret settings only. Environment variables override these values.
server: server:
address: '127.0.0.1:8080' address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
enableLogger: true transport: "${GO_OMNI_TRANSPORT:-nethttp}"
slowRequestThreshold: 500ms
shutdownTimeout: 10s
endpoints:
liveness: {enabled: true, path: /health/live, responseMode: app}
readiness: {enabled: true, path: /health/ready, responseMode: app}
openapi: {enabled: true, path: /openapi.json}
log:
service: "example.com/managed"
level: info
format: json
output: stderr
redactKeys: []

View File

@@ -2,7 +2,7 @@ module example.com/managed
go 1.22.0 go 1.22.0
require git.yuntuo.hk/k2safe/go-omni v0.20.0-preview require git.yuntuo.hk/k2safe/go-omni v0.22.0-preview
require ( require (
github.com/gabriel-vasile/mimetype v1.4.3 // indirect github.com/gabriel-vasile/mimetype v1.4.3 // indirect

View File

@@ -3,14 +3,12 @@ package config
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config" import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
// Load is maintained by the generator. Extend Config/Default/Validate in // Load resolves explicit file references, binds onto Default and validates.
// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline. // Ordinary projects only edit Config, Default, Validate and the selected file.
func Load(path string) (Config, error) { func Load(path string) (Config, error) {
loader, err := NewLoader() loader, err := NewLoader()
if err != nil { if err != nil {
return Config{}, err return Config{}, err
} }
return omniConfig.LoadProject(path, Default(), omniConfig.EnvOptions{ return omniConfig.LoadProject(path, Default(), loader)
Prefix: "GO_OMNI_", Empty: "ignore", Aliases: omniConfig.ServerEnvAliases("GO_OMNI_", "server"),
}, loader)
} }

View File

@@ -17,8 +17,15 @@ import (
) )
func main() { func main() {
path := flag.String("config", "etc/config.yaml", "configuration file; empty uses defaults and environment") path := flag.String("config", "etc/config.yaml", "configuration file; empty uses only Default()")
check := flag.Bool("check-config", false, "validate configuration and exit without starting dependencies")
flag.Parse() flag.Parse()
if *check {
if _, err := config.Load(*path); err != nil {
log.Fatal(err)
}
return
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop() defer stop()
if err := run(ctx, *path); err != nil { if err := run(ctx, *path); err != nil {

View File

@@ -7,7 +7,7 @@ go run ./examples/modules
``` ```
默认监听 `127.0.0.1:8080`,使用仅限开发的 `Bearer dev-token`。 默认监听 `127.0.0.1:8080`,使用仅限开发的 `Bearer dev-token`。
配置 `GO_OMNI_TRANSPORT=native` 可直接切换自研网络层,TLS 配置与主框架一致。 默认读取 `examples/modules/config.yaml`,也可用 `-config` 选择一个其他文件。文件显式引用了 `GO_OMNI_TRANSPORT`,设置为 native 可切换自研网络层;TLS 在文件的 `tls.certFile` / `tls.keyFile` 中配置。此示例直接加载框架 Config,没有生成项目的 server 外层。
```sh ```sh
curl -H 'Authorization: Bearer dev-token' http://127.0.0.1:8080/api/catalog curl -H 'Authorization: Bearer dev-token' http://127.0.0.1:8080/api/catalog

View File

@@ -0,0 +1,2 @@
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
transport: "${GO_OMNI_TRANSPORT:-nethttp}"

View File

@@ -5,6 +5,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"flag"
"log" "log"
"os" "os"
"os/signal" "os/signal"
@@ -106,13 +107,11 @@ func newApp(settings omni.Config, token, upstream string) (application *omni.App
} }
func run(ctx context.Context) error { func run(ctx context.Context) error {
settings, err := omni.LoadConfig("") settings, err := omni.LoadConfig(*configPath)
if err != nil { if err != nil {
return err return err
} }
if os.Getenv("GO_OMNI_HTTP_ADDR") == "" {
settings.Address = "127.0.0.1:8080"
}
token := os.Getenv("GO_OMNI_DEV_TOKEN") token := os.Getenv("GO_OMNI_DEV_TOKEN")
if token == "" { if token == "" {
token = "dev-token" token = "dev-token"
@@ -124,7 +123,10 @@ func run(ctx context.Context) error {
return application.RunContext(ctx) return application.RunContext(ctx)
} }
var configPath = flag.String("config", "examples/modules/config.yaml", "configuration file")
func main() { func main() {
flag.Parse()
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop() defer stop()
if err := run(ctx); err != nil { if err := run(ctx); err != nil {

View File

@@ -28,7 +28,7 @@ func TestModulesConfiguredTransports(t *testing.T) {
io.WriteString(w, `{"name":"upstream catalog"}`) io.WriteString(w, `{"name":"upstream catalog"}`)
})) }))
defer upstream.Close() defer upstream.Close()
settings, err := omni.LoadConfig("MODULES_TEST_") settings, err := omni.LoadConfig("")
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -88,16 +88,9 @@ func newApp(cfg omni.Config, d dependencies) (*omni.App, error) {
return app, nil return app, nil
} }
func run(ctx context.Context, configFile, root string) error { func run(ctx context.Context, configFile, root string) error {
var cfg config.Config defaults := config.Default()
var err error defaults.Address = "127.0.0.1:8080"
if configFile != "" { cfg, err := config.LoadProject(configFile, defaults)
cfg, err = config.LoadFile(configFile, "")
} else {
cfg, err = config.LoadFromEnv("")
if os.Getenv("GO_OMNI_HTTP_ADDR") == "" {
cfg.Address = "127.0.0.1:8080"
}
}
if err != nil { if err != nil {
return err return err
} }

View File

@@ -10,10 +10,10 @@
go run ./examples/reference go run ./examples/reference
``` ```
默认监听 `127.0.0.1:8080`,可通过环境变量调整: 默认读取 `examples/reference/config.yaml`,监听 `127.0.0.1:8080`。文件已显式引用地址变量:
```sh ```sh
GO_OMNI_ADDR=127.0.0.1:8081 go run ./examples/reference GO_OMNI_HTTP_ADDR=127.0.0.1:8081 go run ./examples/reference
``` ```
如果终端提示找不到 `go`,需要先把 Go 的 `bin` 目录加入 `PATH`。不要与 `examples/basic` 同时占用同一个端口。 如果终端提示找不到 `go`,需要先把 Go 的 `bin` 目录加入 `PATH`。不要与 `examples/basic` 同时占用同一个端口。
@@ -133,14 +133,10 @@ curl -i http://127.0.0.1:8080/api/v1/users/1
- 列表按 ID 排序、返回总数,空列表使用 `[]`。 - 列表按 ID 排序、返回总数,空列表使用 `[]`。
- 使用传入的 `context.Context` 执行查询,保留创建时间,修改时更新时间。 - 使用传入的 `context.Context` 执行查询,保留创建时间,修改时更新时间。
MySQL 运行配置通过环境变量传入: 此手写示例显式读取 `GO_OMNI_DB_DSN`;驱动固定 mysql,连接池使用 database.DefaultConfig()。项目可将数据库配置放入自己的 Config 并通过 LoadProject 统一加载(见 [配置指南](../../CONFIG.md)),框架不再自动读取数据库环境变量:
```sh ```sh
export GO_OMNI_DB_DRIVER=mysql
export GO_OMNI_DB_DSN='root:password@tcp(127.0.0.1:3306)/go_omni?parseTime=true&loc=UTC' export GO_OMNI_DB_DSN='root:password@tcp(127.0.0.1:3306)/go_omni?parseTime=true&loc=UTC'
export GO_OMNI_DB_MAX_OPEN_CONNS=10
export GO_OMNI_DB_MAX_IDLE_CONNS=5
export GO_OMNI_DB_PING_TIMEOUT=5s
go run ./examples/reference go run ./examples/reference
``` ```

View File

@@ -0,0 +1,9 @@
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
transport: "${GO_OMNI_TRANSPORT:-nethttp}"
tls:
certFile: "${GO_OMNI_TLS_CERT_FILE:-}"
keyFile: "${GO_OMNI_TLS_KEY_FILE:-}"
log:
service: reference-api
environment: local
output: stdout

View File

@@ -2,6 +2,7 @@ package main
import ( import (
"context" "context"
"flag"
"log" "log"
"os" "os"
"os/signal" "os/signal"
@@ -42,6 +43,7 @@ func registerRoutes(application *omni.App, users service.UserRepository, logger
} }
func main() { func main() {
flag.Parse()
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop() defer stop()
if err := runReference(ctx); err != nil { if err := runReference(ctx); err != nil {
@@ -49,29 +51,9 @@ func main() {
} }
} }
func referenceConfig() (omni.Config, error) { var configPath = flag.String("config", "examples/reference/config.yaml", "configuration file")
settings, err := omni.LoadConfig("")
if err != nil { func referenceConfig() (omni.Config, error) { return omni.LoadConfig(*configPath) }
return settings, err
}
// Preserve the example's loopback default and legacy address variable.
if strings.TrimSpace(os.Getenv("GO_OMNI_HTTP_ADDR")) == "" {
settings.Address = strings.TrimSpace(os.Getenv("GO_OMNI_ADDR"))
if settings.Address == "" {
settings.Address = "127.0.0.1:8080"
}
}
if settings.Log.Service == "" {
settings.Log.Service = "reference-api"
}
if settings.Log.Environment == "" {
settings.Log.Environment = "local"
}
if strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT")) == "" {
settings.Log.Output = "stdout"
}
return settings, nil
}
func runReference(ctx context.Context) error { func runReference(ctx context.Context) error {
settings, err := referenceConfig() settings, err := referenceConfig()
@@ -106,10 +88,9 @@ func openUsers(ctx context.Context, logger *logx.Logger) (service.UserRepository
if strings.TrimSpace(os.Getenv("GO_OMNI_DB_DSN")) == "" { if strings.TrimSpace(os.Getenv("GO_OMNI_DB_DSN")) == "" {
return repository.NewMemoryUsers(), func() error { return nil }, "memory", nil return repository.NewMemoryUsers(), func() error { return nil }, "memory", nil
} }
cfg, err := database.ConfigFromEnv("GO_OMNI_DB_") cfg := database.DefaultConfig()
if err != nil { cfg.Driver = "mysql"
return nil, nil, "", err cfg.DSN = os.Getenv("GO_OMNI_DB_DSN")
}
db, err := database.Open(ctx, cfg) db, err := database.Open(ctx, cfg)
if err != nil { if err != nil {
return nil, nil, "", err return nil, nil, "", err

View File

@@ -7,6 +7,8 @@ import (
"io" "io"
"net" "net"
"net/http" "net/http"
"os"
"path/filepath"
"strings" "strings"
"sync" "sync"
"testing" "testing"
@@ -99,7 +101,11 @@ func TestBusinessConfiguredTransports(t *testing.T) {
t.Setenv("REFERENCE_TEST_TLS_CERT_FILE", cert) t.Setenv("REFERENCE_TEST_TLS_CERT_FILE", cert)
t.Setenv("REFERENCE_TEST_TLS_KEY_FILE", key) t.Setenv("REFERENCE_TEST_TLS_KEY_FILE", key)
} }
settings, err := omni.LoadConfig("REFERENCE_TEST_") path := filepath.Join(t.TempDir(), "config.yaml")
if err := os.WriteFile(path, []byte("transport: ${REFERENCE_TEST_TRANSPORT}\nshutdownTimeout: 3s\ntls:\n certFile: \"${REFERENCE_TEST_TLS_CERT_FILE:-}\"\n keyFile: \"${REFERENCE_TEST_TLS_KEY_FILE:-}\""), 0600); err != nil {
t.Fatal(err)
}
settings, err := omni.LoadConfig(path)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -159,24 +165,26 @@ func TestBusinessConfiguredTransports(t *testing.T) {
} }
} }
func TestReferenceAddressCompatibility(t *testing.T) { func TestReferenceExplicitConfig(t *testing.T) {
previous := *configPath
*configPath = "config.yaml"
defer func() { *configPath = previous }()
t.Setenv("GO_OMNI_TRANSPORT", "native") t.Setenv("GO_OMNI_TRANSPORT", "native")
t.Setenv("GO_OMNI_HTTP_ADDR", "")
t.Setenv("GO_OMNI_TLS_CERT_FILE", "") t.Setenv("GO_OMNI_TLS_CERT_FILE", "")
t.Setenv("GO_OMNI_TLS_KEY_FILE", "") t.Setenv("GO_OMNI_TLS_KEY_FILE", "")
t.Setenv("GO_OMNI_HTTP_ADDR", "") c, err := referenceConfig()
t.Setenv("GO_OMNI_ADDR", "") if err != nil || c.Address != "127.0.0.1:8080" || c.Transport != "native" {
settings, err := referenceConfig() t.Fatal(c, err)
if err != nil || settings.Address != "127.0.0.1:8080" || settings.Transport != "native" {
t.Fatalf("default: %+v %v", settings, err)
} }
t.Setenv("GO_OMNI_ADDR", "127.0.0.1:8081") t.Setenv("GO_OMNI_ADDR", "127.0.0.1:1234")
settings, err = referenceConfig() c, err = referenceConfig()
if err != nil || settings.Address != "127.0.0.1:8081" { if err != nil || c.Address != "127.0.0.1:8080" {
t.Fatalf("legacy: %+v %v", settings, err) t.Fatal("implicit alias", err)
} }
t.Setenv("GO_OMNI_HTTP_ADDR", "127.0.0.1:8082") t.Setenv("GO_OMNI_HTTP_ADDR", "127.0.0.1:8082")
settings, err = referenceConfig() c, err = referenceConfig()
if err != nil || settings.Address != "127.0.0.1:8082" { if err != nil || c.Address != "127.0.0.1:8082" {
t.Fatalf("standard: %+v %v", settings, err) t.Fatal(c, err)
} }
} }

View File

@@ -18,13 +18,12 @@ go-omni migrate --dir migrations --apply
go run ./cmd/server go run ./cmd/server
``` ```
源码示例的 go.mod 使用 `replace ... => ../..`。生成代码使用 `v0.8.0-preview` 响应策略入口。 源码示例的 go.mod 使用 `replace ... => ../..`。生成代码使用 `v0.22.0-preview` 显式配置加载入口。
复制到独立仓库时删除 replace,保留同版本固定依赖,再执行 go mod tidy;CLI 也固定到同一版本。 复制到独立仓库时删除 replace,保留同版本固定依赖,再执行 go mod tidy;CLI 也固定到同一版本。
远程版本验证脚本会检查没有本地替换。 远程版本验证脚本会检查没有本地替换。
`GO_OMNI_TRANSPORT=native` 可切换为 native,业务代码无需调整。 `GO_OMNI_TRANSPORT=native` 可切换为 native,业务代码无需调整。
配置是默认值 → etc/config.yaml → 环境变量。日志默认写入 var/log/server.log 并轮转, 配置是 Default() → etc/config.yaml 的显式引用 → 类型绑定与校验。文件里的固定值不会被环境自动覆盖。日志默认写入 var/log/server.log 并轮转;要改用 stderr,在文件明确设置 log.file 为空、log.rotation.enabled 为 false、log.output 为 stderr。MySQL 默认最多 10 条连接。
`GO_OMNI_LOG_OUTPUT=stderr` 可以切换为标准错误输出。MySQL 默认最多 10 条连接。
应用检查 users 和 user_audit 表,缺失时拒绝启动,不自动改表。 应用检查 users 和 user_audit 表,缺失时拒绝启动,不自动改表。
## 接口 ## 接口

View File

@@ -1,6 +1,7 @@
# Non-secret settings only. Environment variables override these values. # Defaults are overridden only by fields in this file.
server: server:
address: '127.0.0.1:8080' transport: "${GO_OMNI_TRANSPORT:-nethttp}"
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
enableLogger: true enableLogger: true
slowRequestThreshold: 500ms slowRequestThreshold: 500ms
shutdownTimeout: 10s shutdownTimeout: 10s
@@ -17,16 +18,19 @@ server:
compress: true compress: true
redactKeys: [] redactKeys: []
storage: storage:
filesToken: "${GO_OMNI_FILES_TOKEN:-}"
signingKey: "${GO_OMNI_STORAGE_SIGNING_KEY}"
provider: local provider: local
maxBytes: 33554432 maxBytes: 33554432
contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf'] contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf']
linkTtl: 5m linkTtl: 5m
requestTimeout: 30s requestTimeout: 30s
root: './var/files' root: './var/files'
publicUrl: 'http://127.0.0.1:8080/storage/object' publicUrl: "${GO_OMNI_STORAGE_PUBLIC_URL:-http://127.0.0.1:8080/storage/object}"
database: database:
dsn: "${GO_OMNI_DB_DSN}"
driver: mysql driver: mysql
maxOpenConns: 10 maxOpenConns: 10
maxIdleConns: 5 maxIdleConns: 5

View File

@@ -3,7 +3,7 @@ module example.com/userService
go 1.22.0 go 1.22.0
require ( require (
git.yuntuo.hk/k2safe/go-omni v0.8.0-preview git.yuntuo.hk/k2safe/go-omni v0.22.0-preview
github.com/go-sql-driver/mysql v1.9.3 github.com/go-sql-driver/mysql v1.9.3
) )

View File

@@ -15,10 +15,9 @@ func TestTransactions(t *testing.T) {
if os.Getenv("USER_SERVICE_TEST_DATABASE") != "1" { if os.Getenv("USER_SERVICE_TEST_DATABASE") != "1" {
t.Skip("isolated MySQL required") t.Skip("isolated MySQL required")
} }
cfg, err := database.ConfigFromEnv("") cfg := database.DefaultConfig()
if err != nil { cfg.Driver = "mysql"
t.Fatal(err) cfg.DSN = os.Getenv("GO_OMNI_DB_DSN")
}
db, err := database.Open(context.Background(), cfg) db, err := database.Open(context.Background(), cfg)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)

View File

@@ -3,14 +3,11 @@ package settings
import ( import (
"errors" "errors"
"fmt"
"git.yuntuo.hk/k2safe/go-omni/config" "git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/database" "git.yuntuo.hk/k2safe/go-omni/database"
"git.yuntuo.hk/k2safe/go-omni/logx" "git.yuntuo.hk/k2safe/go-omni/logx"
"git.yuntuo.hk/k2safe/go-omni/storage" "git.yuntuo.hk/k2safe/go-omni/storage"
"net/url" "net/url"
"os"
"strconv"
"strings" "strings"
"time" "time"
) )
@@ -38,40 +35,7 @@ func Default() Config {
}} }}
} }
// Load uses defaults -> strict YAML/JSON -> nonempty environment variables. func Load(path string) (Config, error) { return config.LoadProject(path, Default()) }
// Pass an empty path to use only defaults and environment.
func Load(path string) (Config, error) {
cfg := Default()
var err error
if path != "" {
cfg, err = config.DecodeFile(path, cfg)
if err != nil {
return Config{}, err
}
}
cfg.Server, err = config.ApplyEnv(cfg.Server, "")
if err != nil {
return Config{}, err
}
// A stream explicitly selected by environment overrides a file profile.
if output := strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT")); output != "" && strings.TrimSpace(os.Getenv("GO_OMNI_LOG_FILE")) == "" {
cfg.Server.Log.File = ""
cfg.Server.Log.Rotation = logx.RotationConfig{}
}
if err = cfg.Storage.applyEnv(); err != nil {
return Config{}, err
}
cfg.Database, err = database.ApplyEnv(cfg.Database, "")
if err != nil {
return Config{}, err
}
if err = cfg.Validate(); err != nil {
return Config{}, err
}
return cfg, nil
}
func (c Config) Validate() error { func (c Config) Validate() error {
if err := c.Database.Validate(); err != nil { if err := c.Database.Validate(); err != nil {
@@ -108,62 +72,17 @@ type StorageConfig struct {
ContentTypes []string `json:"contentTypes"` ContentTypes []string `json:"contentTypes"`
LinkTTL time.Duration `json:"linkTtl"` LinkTTL time.Duration `json:"linkTtl"`
RequestTimeout time.Duration `json:"requestTimeout"` RequestTimeout time.Duration `json:"requestTimeout"`
// Secrets are environment-only and cannot be populated by config files. // The file explicitly selects secret environment references.
FilesToken string `json:"-"` FilesToken string `json:"filesToken"`
SigningKey string `json:"-"` SigningKey string `json:"signingKey"`
AccessKey string `json:"-"` AccessKey string `json:"accessKey"`
SecretKey string `json:"-"` SecretKey string `json:"secretKey"`
SessionToken string `json:"-"` SessionToken string `json:"sessionToken"`
} }
func (s StorageConfig) Policy() storage.UploadPolicy { func (s StorageConfig) Policy() storage.UploadPolicy {
return storage.UploadPolicy{MaxBytes: s.MaxBytes, ContentTypes: s.ContentTypes} return storage.UploadPolicy{MaxBytes: s.MaxBytes, ContentTypes: s.ContentTypes}
} }
func (s *StorageConfig) applyEnv() error {
values := map[string]*string{
"GO_OMNI_STORAGE_ROOT": &s.Root, "GO_OMNI_STORAGE_PUBLIC_URL": &s.PublicURL,
"GO_OMNI_STORAGE_ENDPOINT": &s.Endpoint, "GO_OMNI_STORAGE_REGION": &s.Region,
"GO_OMNI_STORAGE_BUCKET": &s.Bucket, "GO_OMNI_STORAGE_ACCOUNT_ID": &s.AccountID,
"GO_OMNI_STORAGE_SIGNING_KEY": &s.SigningKey, "GO_OMNI_FILES_TOKEN": &s.FilesToken,
"GO_OMNI_STORAGE_ACCESS_KEY": &s.AccessKey, "GO_OMNI_STORAGE_SECRET_KEY": &s.SecretKey,
"GO_OMNI_STORAGE_SESSION_TOKEN": &s.SessionToken,
}
for name, target := range values {
if value := strings.TrimSpace(os.Getenv(name)); value != "" {
*target = value
}
}
for name, target := range map[string]*time.Duration{"GO_OMNI_STORAGE_LINK_TTL": &s.LinkTTL, "GO_OMNI_STORAGE_REQUEST_TIMEOUT": &s.RequestTimeout} {
if value := strings.TrimSpace(os.Getenv(name)); value != "" {
v, err := time.ParseDuration(value)
if err != nil {
return fmt.Errorf("%s must be a duration", name)
}
*target = v
}
}
if value := os.Getenv("GO_OMNI_STORAGE_MAX_BYTES"); value != "" {
v, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return errors.New("GO_OMNI_STORAGE_MAX_BYTES must be an integer")
}
s.MaxBytes = v
}
if value := os.Getenv("GO_OMNI_STORAGE_SECURE"); value != "" {
v, err := strconv.ParseBool(value)
if err != nil {
return errors.New("GO_OMNI_STORAGE_SECURE must be a boolean")
}
s.Secure = v
}
if value := strings.TrimSpace(os.Getenv("GO_OMNI_STORAGE_CONTENT_TYPES")); value != "" {
s.ContentTypes = strings.Split(value, ",")
for i := range s.ContentTypes {
s.ContentTypes[i] = strings.TrimSpace(s.ContentTypes[i])
}
}
return nil
}
func (s StorageConfig) Validate() error { func (s StorageConfig) Validate() error {
if s.Provider != "local" { if s.Provider != "local" {
return errors.New("storage.provider must match the adapter selected at generation") return errors.New("storage.provider must match the adapter selected at generation")
@@ -186,7 +105,7 @@ func (s StorageConfig) Validate() error {
return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object") return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")
} }
if len(s.SigningKey) < 32 { if len(s.SigningKey) < 32 {
return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes") return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")
} }
return nil return nil

153
httpc/README.md Normal file
View File

@@ -0,0 +1,153 @@
# httpc:复用连接的 HTTP 客户端
`httpc` 在标准 `net/http` 上提供独立连接池、网络阶段总超时、保守重试、请求追踪和有界 JSON 调用。普通接口可以直接带认证头调用;上传、下载、特殊协议仍使用标准 Request/Response。没有业务身份模型或全局认证信息。
## 创建一次,业务重复使用
```go
client, err := httpc.New(httpc.Config{
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
Timeout: 3 * time.Second,
DialTimeout: time.Second,
TLSHandshakeTimeout: time.Second,
ResponseHeaderTimeout: time.Second,
MaxConnsPerHost: 32,
MaxIdleConnsPerHost: 16,
MaxIdleConns: 64,
IdleConnTimeout: 60 * time.Second,
MaxResponseBytes: 1 << 20,
})
if err != nil { return err }
// 在 svc 初始化并持有 client,向生命周期登记 client.Close。
// 普通独立程序也可 defer client.Close();不要每次请求重新 New。
err = client.JSON(ctx, http.MethodGet, serviceURL, nil, &result,
httpc.WithHeader("Authorization", "Bearer "+token),
httpc.WithHeaders(http.Header{"Accept": {"application/json"}}),
)
```
Client 可并发使用。每个调用的 input/output 及其读写同步由调用方负责,不要让多个请求同时写同一个 output。
直接运行 [examples/httpClient](../examples/httpClient/README.md),只使用本地临时服务,无需业务 Token 或外部基础设施。
## 代理规则
| 配置 | 行为 |
| --- | --- |
| 不填写 Proxy | 保留被克隆 DefaultTransport 的代理策略;未改动的标准库默认使用环境代理 |
| `Mode: httpc.ProxyEnvironment` | 明确使用 HTTP_PROXY / HTTPS_PROXY / NO_PROXY(及小写形式) |
| `Mode: httpc.ProxyDirect` | 设置 Proxy=nil,明确直连,不受环境代理影响 |
| `Mode: httpc.ProxyURL, URL: "http://proxy.example:8080"` | 固定代理地址,不通过 NO_PROXY 排除 |
支持 http、https、socks5、socks5h;底层行为沿用 Go Transport。固定地址必须是代理 origin,不接受业务路径、查询、fragment 或无效端口。凭证可由 URL userinfo 提供,但配置错误不输出 URL/密码;不要打印整个 Config。环境代理规则由标准库读取并缓存,不提供运行时环境代理热更新。
内部服务要直连,只设置 `proxy.mode: direct`,无需创建 Transport。需要自定义路由、代理认证握手等复杂逻辑时注入自己的 Transport。
## 超时和连接池
除原有字段外,新字段都可选:**0 保留默认行为,正数覆盖,负数拒绝**。需要明确关闭某个阶段超时、关闭 TCP keepalive 或特殊 dialer 时,使用自定义 Transport。
| 字段 | 默认/含义 |
| --- | --- |
| Timeout | 10s;从 Do 开始,覆盖 DNS/建连、等待连接、重试、退避、响应头和读取响应体 |
| DialTimeout | 不填时保留原 dialer;标准库默认 30s;配置后使用标准 net.Dialer |
| DialKeepAlive | 标准 net.Dialer 的 TCP keepalive 探测间隔,标准默认 30s;不是 HTTP 空闲连接超时 |
| TLSHandshakeTimeout | 沿用克隆 Transport;标准库默认 10s |
| ResponseHeaderTimeout | 沿用 Transport,标准默认不单独限制;从完整发送请求后等响应头,总 Timeout 仍生效 |
| ExpectContinueTimeout | 沿用 Transport;标准默认 1s,等待 `100-continue` 的时间 |
| IdleConnTimeout | 沿用 Transport;标准默认 90s,只回收空闲连接 |
| MaxConnsPerHost | 默认 64;每个目标的建连中、活动与空闲连接总上限 |
| MaxIdleConnsPerHost | 默认等于 MaxConnsPerHost,可单独配置 |
| MaxIdleConns | 默认 MaxConnsPerHost×2,可单独配置;所有目标合计的空闲连接上限 |
| MaxResponseBytes | JSON 成功响应默认最多 1 MiB,最大允许配置 1 GiB;应按实际接口保持较小限额 |
每主机上限最多 65536,总空闲连接最多 131072。总空闲上限可以小于各主机上限之和。MaxConnsPerHost 不等于整个进程的连接上限,也不等于 HTTP/2 的请求并发上限。
配置 DialTimeout / DialKeepAlive 会创建标准 net.Dialer;未填的另一个值使用标准默认 30s。不要通过全局修改 DefaultTransport 来塞入项目的特殊拨号逻辑;请显式使用借用的 Transport。若 DefaultTransport 有自定义 TLS dialer,相关阶段覆盖会报错,避免 TLS timeout 被悄悄忽略。
调用者 context 的更早截止时间优先。总超时不能抢占 Go 的 JSON 编解码或不遵守 context 的自定义 RoundTripper;计时从 Do 开始,不包括调用前的 json.Marshal。TLS 校验默认保持开启,没有新增 InsecureSkipVerify 配置。
## JSON 的单次请求选项
```go
err := client.JSON(ctx, "POST", endpoint, input, &output,
httpc.WithHeader("Authorization", "Bearer "+token),
httpc.WithHeader("X-Tenant", tenant),
httpc.WithMaxResponseBytes(64<<10),
)
```
- `WithHeader` 设置一个头;`WithHeaders` 支持多值 http.Header。后出现的选项覆盖同名头,不影响其他头。
- WithHeaders 在创建选项时快照原 map,每次调用复制值。构建后修改原 map 不影响选项,选项可以并发复用;构建时原 map 的读写同步由调用方负责。
- 不提供 Client 全局 Authorization,避免跨用户调用共享凭证。框架不会把请求头写进日志。
- 有 input 时默认 Content-Type 为 application/json;显式请求头可覆盖。请求头名称/值会校验,拒绝 CRLF 等控制字符和歧义的大小写重复键。
- Host、Content-Length、Transfer-Encoding、Trailer 属于请求目标/分帧控制,请用标准 Request 配合 Do;JSON 选项中不允许伪装成普通头。
- `WithMaxResponseBytes` 是本次 JSON 的正数限额,不修改 Client 默认值,不改变 Do 的流式语义。
JSON 自动关闭响应体。仅 2xx 解码;空成功响应(例如 204)保留 output 原值。非空响应使用标准 json.Unmarshal,拒绝多个 JSON 文档,类型错误保留 Go 的错误信息与行为。
`output == nil` 按限额流式丢弃成功响应,不把整个 body 读进内存,也不检查 JSON 语法。响应限额按实际读取的 body 计算;标准库自动 gzip 解压后仍受上限约束。input 会先 json.Marshal 到内存,大文件上传不适合用 JSON helper。
非 2xx 返回 `*httpc.StatusError`,只包含状态码,不包含错误响应体。为连接复用,只尝试读完**声明长度已知且不超过 min(32 KiB, 本次响应限额)** 的小错误体,最多等待 25ms(或更早总截止时间);失败仍返回原 HTTP 状态。未知长度/大错误体立即关闭,优先及时返回。重试过程的错误体另受 32 KiB 大小及整体请求截止时间约束,不无界读取。
## 配置文件与生成项目
在项目 Config 中加一个字段:
```go
Outbound httpc.Config `json:"outbound"`
```
项目 Validate 中调用 `c.Outbound.Validate()`;创建时 New 也会验证。svc 初始化一次并通过生命周期登记 Close。文件例子:
```yaml
outbound:
timeout: 3s
proxy:
mode: direct
dialTimeout: 500ms
tlsHandshakeTimeout: 1s
responseHeaderTimeout: 1s
maxConnsPerHost: 32
maxIdleConnsPerHost: 16
maxIdleConns: 64
idleConnTimeout: 60s
maxResponseBytes: 1048576
```
统一加载入口读取 JSON/YAML/XML。需要环境变量时在文件明确写 `mode: "${HTTP_PROXY_MODE:-direct}"`、`dialTimeout: "${HTTP_DIAL_TIMEOUT:-500ms}"`;固定值不会被环境自动覆盖。无需手写解析。详见 [统一配置](../CONFIG.md)。
## 标准请求、特殊 TLS 与生命周期
```go
req, err := http.NewRequestWithContext(ctx, "GET", downloadURL, nil)
if err != nil { return err }
req.Header.Set("Authorization", "Bearer "+token)
res, err := client.Do(req)
if err != nil { return err }
defer res.Body.Close()
// 自行判断状态,按项目要求限流/限制大小,流式写入目标;不要无界 ReadAll 大文件。
_, err = io.Copy(destination, res.Body)
```
Do 不自动限制 body,允许下载/流式处理,调用方必须关闭。响应体消费/关闭前网络超时持续有效。
`Config.Transport` 可注入任意 http.RoundTripper,适配自定义 dialer、mTLS、根证书、监控包装或测试替身。框架借用它,**不修改、不调用其 CloseIdleConnections**。代理、连接池、TLS/建连阶段配置不能同时填入,避免无效配置;总 Timeout、JSON 限额、框架重试/日志仍有效。借用实现必须遵守标准请求和 body 所有权、并发及 context 约定。
`Close` 幂等:拒绝新调用,关闭自有池的空闲连接,不主动打断已开始的请求。服务关闭应先排空 HTTP/后台调用,再关闭 Client;它不是后台 drain 的替代品。生成项目可通过 svc 的生命周期登记清理。
## 重试、日志与兼容
MaxRetries 默认 0,最多 5。仅可重放的 GET/HEAD/OPTIONS 对网络错误、429/502/503/504 做框架重试。POST/PUT/PATCH/DELETE 不自动做应用层重试。标准库或自定义 Transport 自己的重放规则仍可能生效,不能承诺业务“恰好执行一次”。
Backoff 默认 100ms,MaxBackoff 默认 2s,采用指数退避与抖动。Retry-After 要求等待超过 MaxBackoff 时返回当前响应,不提前请求;总截止时间限制全部尝试。成功或无重试的调用不计算退避。
不跟随重定向,避免自动向其他地址转发认证信息;3xx 由调用者判断。默认传递 context 中的 X-Request-ID,显式请求头优先。日志只包含 method、hostname、status、attempt、收到响应头的耗时等,不打印路径、查询、用户名密码、认证头或响应体;标准 net/http 返回的 URL 错误可能携带地址,项目不要把 URL 中的密钥当作安全日志字段。
v0.21 兼容说明:
- 原来的 JSON 五参数直接调用继续有效;新方法末尾增加 `...httpc.RequestOption`,自行定义的 interface/方法值类型需要同步签名。
- 原默认代理、总超时、重试、池大小和成功响应含义保留;新字段未设置时不改变它们。
- `Transport` 与显式池/代理/阶段参数混用现在报错;此前 MaxConnsPerHost 在自定义 Transport 下实际被忽略。迁移时把参数放到自己的 Transport,或删掉 Transport 改用内置配置。
- 没有移除 Do、JSON、StatusError 或自定义 Transport,没有引入新第三方依赖。

100
httpc/benchmark_test.go Normal file
View File

@@ -0,0 +1,100 @@
package httpc
import (
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
type benchmarkTransport struct {
payload string
status int
}
func (t benchmarkTransport) RoundTrip(r *http.Request) (*http.Response, error) {
if r.Body != nil {
_ = r.Body.Close()
}
return &http.Response{StatusCode: t.status, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(t.payload)), ContentLength: int64(len(t.payload)), Request: r}, nil
}
func BenchmarkHTTPClient(b *testing.B) {
for _, size := range []int{128, 4096, 65536} {
for _, discard := range []bool{false, true} {
label := map[int]string{128: "128B", 4096: "4KiB", 65536: "64KiB"}[size]
if discard {
label += "/discard"
} else {
label += "/decode"
}
b.Run(label, func(b *testing.B) {
c, err := New(Config{Transport: benchmarkTransport{`{"value":"` + strings.Repeat("x", size) + `"}`, 200}, DisableLogging: true})
if err != nil {
b.Fatal(err)
}
defer c.Close()
var output struct {
Value string `json:"value"`
}
var target any = &output
if discard {
target = nil
}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if err := c.JSON(context.Background(), "GET", "http://example.invalid/item", nil, target); err != nil {
b.Fatal(err)
}
}
})
}
}
b.Run("Do", func(b *testing.B) {
c, _ := New(Config{Transport: benchmarkTransport{"", 200}, DisableLogging: true})
defer c.Close()
r, _ := http.NewRequest("GET", "http://example.invalid/item", nil)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
res, err := c.Do(r)
if err != nil {
b.Fatal(err)
}
res.Body.Close()
}
})
}
func BenchmarkHTTPPool(b *testing.B) {
for _, status := range []int{200, 401} {
b.Run(http.StatusText(status), func(b *testing.B) {
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(status); io.WriteString(w, `{"ok":true}`) }))
defer s.Close()
c, _ := New(Config{DisableLogging: true})
defer c.Close()
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
if status == 200 && err != nil {
b.Fatal(err)
}
}
})
}
}
func BenchmarkHTTPHeaders(b *testing.B) {
c, _ := New(Config{Transport: benchmarkTransport{`{"ok":true}`, 200}, DisableLogging: true})
defer c.Close()
option := WithHeaders(http.Header{"Authorization": []string{"Bearer example"}, "X-Request-Id": []string{"example-request"}})
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if err := c.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, option); err != nil {
b.Fatal(err)
}
}
}

View File

@@ -3,9 +3,7 @@
package httpc package httpc
import ( import (
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"fmt" "fmt"
"io" "io"
@@ -22,18 +20,6 @@ import (
var ErrClosed = errors.New("httpc: closed") var ErrClosed = errors.New("httpc: closed")
var ErrResponseTooLarge = errors.New("httpc: response exceeds limit") var ErrResponseTooLarge = errors.New("httpc: response exceeds limit")
type Config struct {
Timeout time.Duration `json:"timeout"`
MaxRetries int `json:"maxRetries"`
Backoff time.Duration `json:"backoff"`
MaxBackoff time.Duration `json:"maxBackoff"`
MaxConnsPerHost int `json:"maxConnsPerHost"`
MaxResponseBytes int64 `json:"maxResponseBytes"`
DisableRequestID bool `json:"disableRequestId"`
DisableLogging bool `json:"disableLogging"`
Transport http.RoundTripper `json:"-"` // Borrowed. The caller owns its lifecycle.
}
type Client struct { type Client struct {
config Config config Config
client *http.Client client *http.Client
@@ -41,43 +27,19 @@ type Client struct {
closed atomic.Bool closed atomic.Bool
} }
// New creates a reusable client. Close it after callers and response streams drain.
func New(cfg Config) (*Client, error) { func New(cfg Config) (*Client, error) {
if cfg.Timeout < 0 || cfg.Backoff < 0 || cfg.MaxBackoff < 0 || cfg.MaxRetries < 0 || cfg.MaxRetries > 5 || cfg.MaxConnsPerHost < 0 || cfg.MaxResponseBytes < 0 { normalized, err := normalizeConfig(cfg)
return nil, errors.New("httpc: invalid configuration") if err != nil {
return nil, err
} }
if cfg.Timeout == 0 { c := &Client{config: normalized}
cfg.Timeout = 10 * time.Second
}
if cfg.Backoff == 0 {
cfg.Backoff = 100 * time.Millisecond
}
if cfg.MaxBackoff == 0 {
cfg.MaxBackoff = 2 * time.Second
}
if cfg.Backoff > cfg.MaxBackoff {
return nil, errors.New("httpc: backoff exceeds maxBackoff")
}
if cfg.MaxConnsPerHost == 0 {
cfg.MaxConnsPerHost = 64
}
if cfg.MaxResponseBytes == 0 {
cfg.MaxResponseBytes = 1 << 20
}
if cfg.MaxResponseBytes > 1<<30 || cfg.MaxConnsPerHost > 65536 {
return nil, errors.New("httpc: response or connection limit exceeds supported capacity")
}
c := &Client{config: cfg}
transport := cfg.Transport transport := cfg.Transport
if transport == nil { if transport == nil {
// Own a private pool; never mutate the process-wide DefaultTransport. c.owned, err = ownedTransport(normalized)
base, ok := http.DefaultTransport.(*http.Transport) if err != nil {
if !ok { return nil, err
return nil, errors.New("httpc: provide Transport when DefaultTransport is replaced")
} }
c.owned = base.Clone()
c.owned.MaxConnsPerHost = cfg.MaxConnsPerHost
c.owned.MaxIdleConnsPerHost = cfg.MaxConnsPerHost
c.owned.MaxIdleConns = cfg.MaxConnsPerHost * 2
transport = c.owned transport = c.owned
} }
c.client = &http.Client{Transport: transport, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }} c.client = &http.Client{Transport: transport, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
@@ -114,7 +76,10 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
} }
replayable := (base.Method == "" || base.Method == "GET" || base.Method == "HEAD" || base.Method == "OPTIONS") && (base.Body == nil || base.Body == http.NoBody || base.GetBody != nil) replayable := (base.Method == "" || base.Method == "GET" || base.Method == "HEAD" || base.Method == "OPTIONS") && (base.Body == nil || base.Body == http.NoBody || base.GetBody != nil)
for attempt := 0; ; attempt++ { for attempt := 0; ; attempt++ {
req := base.Clone(ctx) req := base
if attempt > 0 {
req = base.Clone(ctx)
}
if attempt > 0 && base.Body != nil && base.Body != http.NoBody { if attempt > 0 && base.Body != nil && base.Body != http.NoBody {
body, err := base.GetBody() body, err := base.GetBody()
if err != nil { if err != nil {
@@ -133,23 +98,9 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
logx.InfoContext(ctx, "http.client", "method", req.Method, "host", req.URL.Hostname(), "status", status, "attempt", attempt+1, "durationMs", time.Since(started).Seconds()*1000, "failed", err != nil) logx.InfoContext(ctx, "http.client", "method", req.Method, "host", req.URL.Hostname(), "status", status, "attempt", attempt+1, "durationMs", time.Since(started).Seconds()*1000, "failed", err != nil)
} }
retry := replayable && attempt < c.config.MaxRetries && ctx.Err() == nil && (err != nil || status == 429 || status == 502 || status == 503 || status == 504) retry := replayable && attempt < c.config.MaxRetries && ctx.Err() == nil && (err != nil || status == 429 || status == 502 || status == 503 || status == 504)
delay := c.config.Backoff var delay time.Duration
for i := 0; i < attempt && delay < c.config.MaxBackoff; i++ { if retry {
if delay > c.config.MaxBackoff/2 { delay, retry = c.retryDelay(attempt, response)
delay = c.config.MaxBackoff
} else {
delay *= 2
}
}
if response != nil && response.Header.Get("Retry-After") != "" {
if value, ok := retryAfter(response.Header.Get("Retry-After")); ok {
delay = value
}
} else if delay > 1 {
delay = delay/2 + time.Duration(rand.Int63n(int64(delay/2)+1))
}
if delay > c.config.MaxBackoff {
retry = false
} }
if !retry { if !retry {
if err != nil { if err != nil {
@@ -160,9 +111,7 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
return response, nil return response, nil
} }
if response != nil { if response != nil {
if response.ContentLength >= 0 && response.ContentLength <= 32<<10 { drainSmallBody(response, 32<<10)
_, _ = io.Copy(io.Discard, response.Body)
}
_ = response.Body.Close() _ = response.Body.Close()
} }
timer := time.NewTimer(delay) timer := time.NewTimer(delay)
@@ -204,48 +153,6 @@ type StatusError struct{ StatusCode int }
func (e *StatusError) Error() string { return fmt.Sprintf("httpc: HTTP status %d", e.StatusCode) } func (e *StatusError) Error() string { return fmt.Sprintf("httpc: HTTP status %d", e.StatusCode) }
// JSON bounds response memory, closes the body, and decodes only 2xx responses.
// Error bodies, URLs, credentials and query strings are not included in logs.
func (c *Client) JSON(ctx context.Context, method, url string, input, output any) error {
if ctx == nil {
return errors.New("httpc: nil context")
}
var body io.Reader
if input != nil {
data, err := json.Marshal(input)
if err != nil {
return err
}
body = bytes.NewReader(data)
}
request, err := http.NewRequestWithContext(ctx, method, url, body)
if err != nil {
return err
}
if input != nil {
request.Header.Set("Content-Type", "application/json")
}
response, err := c.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode >= 300 {
return &StatusError{StatusCode: response.StatusCode}
}
data, err := io.ReadAll(io.LimitReader(response.Body, c.config.MaxResponseBytes+1))
if err != nil {
return err
}
if int64(len(data)) > c.config.MaxResponseBytes {
return ErrResponseTooLarge
}
if output == nil || len(bytes.TrimSpace(data)) == 0 {
return nil
}
return json.Unmarshal(data, output)
}
func (c *Client) Close() error { func (c *Client) Close() error {
c.closed.Store(true) c.closed.Store(true)
if c.owned != nil { if c.owned != nil {
@@ -253,3 +160,34 @@ func (c *Client) Close() error {
} }
return nil return nil
} }
// retryDelay is deliberately outside the success path: no jitter/random work
// for calls that cannot or do not need to retry.
func (c *Client) retryDelay(attempt int, response *http.Response) (time.Duration, bool) {
delay := c.config.Backoff
for i := 0; i < attempt && delay < c.config.MaxBackoff; i++ {
if delay > c.config.MaxBackoff/2 {
delay = c.config.MaxBackoff
} else {
delay *= 2
}
}
if response != nil {
if value, ok := retryAfter(response.Header.Get("Retry-After")); ok {
return value, value <= c.config.MaxBackoff
}
}
if delay > 1 {
delay = delay/2 + time.Duration(rand.Int63n(int64(delay/2)+1))
}
return delay, delay <= c.config.MaxBackoff
}
// Drain only a known small error body. Unknown/large bodies are closed promptly;
// never allocate or consume an unbounded error document just to retain a pool slot.
func drainSmallBody(response *http.Response, limit int64) {
limit = min(limit, 32<<10)
if response.Body != nil && response.ContentLength >= 0 && response.ContentLength <= limit {
_, _ = io.Copy(io.Discard, io.LimitReader(response.Body, limit+1))
}
}

205
httpc/config.go Normal file
View File

@@ -0,0 +1,205 @@
package httpc
import (
"errors"
"fmt"
"net"
"net/http"
"net/url"
"reflect"
"strconv"
"strings"
"time"
)
// ProxyMode controls the owned transport's proxy selection. Empty preserves
// DefaultTransport's policy (normally environment proxies).
type ProxyMode string
const (
ProxyEnvironment ProxyMode = "environment"
ProxyDirect ProxyMode = "direct"
ProxyURL ProxyMode = "url"
)
type ProxyConfig struct {
Mode ProxyMode `json:"mode"`
URL string `json:"url"` // May contain credentials; never log the configuration.
}
// Config owns one private connection pool unless Transport is supplied.
// New duration/pool fields are optional: zero retains existing defaults, positive
// values override them. Use a custom Transport for disabling timeouts, custom
// dialers, mTLS and other protocol-specific settings.
type Config struct {
Timeout time.Duration `json:"timeout"`
MaxRetries int `json:"maxRetries"`
Backoff time.Duration `json:"backoff"`
MaxBackoff time.Duration `json:"maxBackoff"`
MaxConnsPerHost int `json:"maxConnsPerHost"`
MaxIdleConns int `json:"maxIdleConns"`
MaxIdleConnsPerHost int `json:"maxIdleConnsPerHost"`
DialTimeout time.Duration `json:"dialTimeout"`
DialKeepAlive time.Duration `json:"dialKeepAlive"`
TLSHandshakeTimeout time.Duration `json:"tlsHandshakeTimeout"`
ResponseHeaderTimeout time.Duration `json:"responseHeaderTimeout"`
ExpectContinueTimeout time.Duration `json:"expectContinueTimeout"`
IdleConnTimeout time.Duration `json:"idleConnTimeout"`
Proxy ProxyConfig `json:"proxy"`
MaxResponseBytes int64 `json:"maxResponseBytes"`
DisableRequestID bool `json:"disableRequestId"`
DisableLogging bool `json:"disableLogging"`
Transport http.RoundTripper `json:"-"` // Borrowed; never modified or closed.
}
const maxResponseBytes int64 = 1 << 30
// Validate checks values without creating pools or making network requests.
func (cfg Config) Validate() error { _, err := normalizeConfig(cfg); return err }
func normalizeConfig(cfg Config) (Config, error) {
for _, field := range []struct {
name string
value time.Duration
}{
{"timeout", cfg.Timeout}, {"backoff", cfg.Backoff}, {"maxBackoff", cfg.MaxBackoff},
{"dialTimeout", cfg.DialTimeout}, {"dialKeepAlive", cfg.DialKeepAlive}, {"tlsHandshakeTimeout", cfg.TLSHandshakeTimeout},
{"responseHeaderTimeout", cfg.ResponseHeaderTimeout}, {"expectContinueTimeout", cfg.ExpectContinueTimeout}, {"idleConnTimeout", cfg.IdleConnTimeout},
} {
if field.value < 0 {
return Config{}, fmt.Errorf("httpc: %s must not be negative", field.name)
}
}
if cfg.MaxRetries < 0 || cfg.MaxRetries > 5 {
return Config{}, errors.New("httpc: maxRetries must be 0..5")
}
if cfg.MaxConnsPerHost < 0 || cfg.MaxConnsPerHost > 65536 || cfg.MaxIdleConnsPerHost < 0 || cfg.MaxIdleConnsPerHost > 65536 || cfg.MaxIdleConns < 0 || cfg.MaxIdleConns > 131072 {
return Config{}, errors.New("httpc: invalid connection pool limits")
}
if cfg.MaxResponseBytes < 0 || cfg.MaxResponseBytes > maxResponseBytes {
return Config{}, errors.New("httpc: maxResponseBytes must be 0..1 GiB")
}
if _, err := proxyFunction(cfg.Proxy); err != nil {
return Config{}, err
}
if cfg.Transport != nil {
value := reflect.ValueOf(cfg.Transport)
switch value.Kind() {
case reflect.Pointer, reflect.Map, reflect.Func, reflect.Slice, reflect.Interface, reflect.Chan:
if value.IsNil() {
return Config{}, errors.New("httpc: nil Transport")
}
}
if cfg.hasTransportOptions() {
return Config{}, errors.New("httpc: Transport cannot be combined with proxy, dial, TLS or connection pool options; configure the borrowed transport directly")
}
}
if cfg.Timeout == 0 {
cfg.Timeout = 10 * time.Second
}
if cfg.Backoff == 0 {
cfg.Backoff = 100 * time.Millisecond
}
if cfg.MaxBackoff == 0 {
cfg.MaxBackoff = 2 * time.Second
}
if cfg.Backoff > cfg.MaxBackoff {
return Config{}, errors.New("httpc: backoff exceeds maxBackoff")
}
if cfg.MaxConnsPerHost == 0 {
cfg.MaxConnsPerHost = 64
}
if cfg.MaxIdleConnsPerHost == 0 {
cfg.MaxIdleConnsPerHost = cfg.MaxConnsPerHost
}
if cfg.MaxIdleConns == 0 {
cfg.MaxIdleConns = cfg.MaxConnsPerHost * 2
}
if cfg.MaxResponseBytes == 0 {
cfg.MaxResponseBytes = 1 << 20
}
return cfg, nil
}
func (c Config) hasTransportOptions() bool {
return c.Proxy.Mode != "" || c.Proxy.URL != "" || c.MaxConnsPerHost != 0 || c.MaxIdleConns != 0 || c.MaxIdleConnsPerHost != 0 || c.DialTimeout != 0 || c.DialKeepAlive != 0 || c.TLSHandshakeTimeout != 0 || c.ResponseHeaderTimeout != 0 || c.ExpectContinueTimeout != 0 || c.IdleConnTimeout != 0
}
func proxyFunction(c ProxyConfig) (func(*http.Request) (*url.URL, error), error) {
if c.Mode != ProxyURL && c.URL != "" {
return nil, errors.New("httpc: proxy URL requires mode url")
}
switch c.Mode {
case "":
return nil, nil
case ProxyDirect:
return nil, nil
case ProxyEnvironment:
return http.ProxyFromEnvironment, nil
case ProxyURL:
u, err := url.Parse(c.URL)
if err != nil || u == nil || u.Hostname() == "" || u.Opaque != "" || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || (u.Path != "" && u.Path != "/") {
return nil, errors.New("httpc: proxy URL must be an absolute proxy origin without path, query or fragment")
}
switch u.Scheme {
case "http", "https", "socks5", "socks5h":
default:
return nil, errors.New("httpc: proxy scheme must be http, https, socks5 or socks5h")
}
if strings.HasSuffix(u.Host, ":") {
return nil, errors.New("httpc: proxy port must be 1..65535")
}
if port := u.Port(); port != "" {
n, e := strconv.Atoi(port)
if e != nil || n < 1 || n > 65535 {
return nil, errors.New("httpc: proxy port must be 1..65535")
}
}
u.Path = ""
u.RawPath = ""
return http.ProxyURL(u), nil
default:
return nil, errors.New("httpc: proxy mode must be environment, direct or url")
}
}
func ownedTransport(cfg Config) (*http.Transport, error) {
base, ok := http.DefaultTransport.(*http.Transport)
if !ok || base == nil {
return nil, errors.New("httpc: provide Transport when DefaultTransport is replaced")
}
if (base.DialTLSContext != nil || base.DialTLS != nil) && (cfg.DialTimeout > 0 || cfg.DialKeepAlive > 0 || cfg.TLSHandshakeTimeout > 0) {
return nil, errors.New("httpc: provide Transport when DefaultTransport has a custom TLS dialer")
}
t := base.Clone()
t.MaxConnsPerHost = cfg.MaxConnsPerHost
t.MaxIdleConnsPerHost = cfg.MaxIdleConnsPerHost
t.MaxIdleConns = cfg.MaxIdleConns
if cfg.Proxy.Mode != "" {
proxy, err := proxyFunction(cfg.Proxy)
if err != nil {
return nil, err
}
t.Proxy = proxy
}
if cfg.DialTimeout > 0 || cfg.DialKeepAlive > 0 {
dialer := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
if cfg.DialTimeout > 0 {
dialer.Timeout = cfg.DialTimeout
}
if cfg.DialKeepAlive > 0 {
dialer.KeepAlive = cfg.DialKeepAlive
}
t.DialContext = dialer.DialContext
t.Dial = nil
}
if cfg.TLSHandshakeTimeout > 0 {
t.TLSHandshakeTimeout = cfg.TLSHandshakeTimeout
}
if cfg.ResponseHeaderTimeout > 0 {
t.ResponseHeaderTimeout = cfg.ResponseHeaderTimeout
}
if cfg.ExpectContinueTimeout > 0 {
t.ExpectContinueTimeout = cfg.ExpectContinueTimeout
}
if cfg.IdleConnTimeout > 0 {
t.IdleConnTimeout = cfg.IdleConnTimeout
}
return t, nil
}

277
httpc/config_test.go Normal file
View File

@@ -0,0 +1,277 @@
package httpc
import (
"context"
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"os/exec"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
func testClient(t *testing.T, cfg Config) *Client {
t.Helper()
cfg.DisableLogging = true
c, err := New(cfg)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { c.Close() })
return c
}
func TestTransportConfigurationAndIsolation(t *testing.T) {
base := http.DefaultTransport.(*http.Transport)
oldConns, oldIdle, oldTimeout := base.MaxConnsPerHost, base.MaxIdleConns, base.ResponseHeaderTimeout
cfg := Config{Proxy: ProxyConfig{Mode: ProxyDirect}, DialTimeout: time.Second, DialKeepAlive: 20 * time.Second, TLSHandshakeTimeout: 2 * time.Second, ResponseHeaderTimeout: 3 * time.Second, ExpectContinueTimeout: 4 * time.Second, IdleConnTimeout: 5 * time.Second, MaxConnsPerHost: 8, MaxIdleConns: 12, MaxIdleConnsPerHost: 3}
c := testClient(t, cfg)
if c.owned == base || c.owned.Proxy != nil || c.owned.DialContext == nil || c.owned.MaxConnsPerHost != 8 || c.owned.MaxIdleConns != 12 || c.owned.MaxIdleConnsPerHost != 3 || c.owned.TLSHandshakeTimeout != 2*time.Second || c.owned.ResponseHeaderTimeout != 3*time.Second || c.owned.ExpectContinueTimeout != 4*time.Second || c.owned.IdleConnTimeout != 5*time.Second {
t.Fatal("transport options not applied")
}
if base.MaxConnsPerHost != oldConns || base.MaxIdleConns != oldIdle || base.ResponseHeaderTimeout != oldTimeout {
t.Fatal("modified shared transport")
}
if c.owned.TLSClientConfig != nil && c.owned.TLSClientConfig.InsecureSkipVerify {
t.Fatal("TLS verification disabled")
}
d := testClient(t, Config{})
if d.owned.MaxConnsPerHost != 64 || d.owned.MaxIdleConnsPerHost != 64 || d.owned.MaxIdleConns != 128 || d.owned.IdleConnTimeout != base.IdleConnTimeout || d.owned.TLSHandshakeTimeout != base.TLSHandshakeTimeout || d.owned.ResponseHeaderTimeout != base.ResponseHeaderTimeout {
t.Fatal("default behavior changed")
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if conn, err := c.owned.DialContext(ctx, "tcp", "127.0.0.1:1"); err == nil {
conn.Close()
t.Fatal("dial ignored context")
}
}
func TestProxyValidation(t *testing.T) {
for _, scheme := range []string{"http", "https", "socks5", "socks5h"} {
c := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyURL, URL: scheme + "://user:private-value@proxy.invalid:8080"}})
req, _ := http.NewRequest("GET", "https://target.invalid", nil)
u, err := c.owned.Proxy(req)
if err != nil || u.Host != "proxy.invalid:8080" || u.Scheme != scheme {
t.Fatal("explicit proxy unavailable")
}
}
bad := []Config{
{Proxy: ProxyConfig{Mode: "private-value"}},
{Proxy: ProxyConfig{URL: "http://private-value"}},
{Proxy: ProxyConfig{Mode: ProxyDirect, URL: "http://private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://user:private-value@proxy.invalid:bad"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "file:///private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid/private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid?token=private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid:65536"}},
{DialTimeout: -1}, {DialKeepAlive: -1}, {TLSHandshakeTimeout: -1}, {ResponseHeaderTimeout: -1}, {ExpectContinueTimeout: -1}, {IdleConnTimeout: -1}, {MaxIdleConns: -1}, {MaxIdleConnsPerHost: 65537},
{Transport: benchmarkTransport{}, Proxy: ProxyConfig{Mode: ProxyDirect}},
{Transport: benchmarkTransport{}, DialTimeout: time.Second}, {Transport: benchmarkTransport{}, MaxConnsPerHost: 2},
}
for i, cfg := range bad {
if _, err := New(cfg); err == nil || strings.Contains(err.Error(), "private-value") {
t.Fatalf("case %d: unsafe or missing error %v", i, err)
}
}
var nilTransport *http.Transport
if _, err := New(Config{Transport: nilTransport}); err == nil {
t.Fatal("typed nil accepted")
}
}
func TestProxyRouting(t *testing.T) {
var proxied, target atomic.Int32
proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxied.Add(1)
if r.URL.Scheme != "http" || r.URL.Host != "target.invalid" {
t.Error("not an absolute proxy request")
}
io.WriteString(w, `{"proxy":true}`)
}))
defer proxy.Close()
origin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { target.Add(1); io.WriteString(w, `{}`) }))
defer origin.Close()
// Isolate the standard library's process-cached environment proxy lookup.
executable, err := os.Executable()
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, executable, "-test.run=^TestProxyEnvironmentProcess$")
for _, value := range os.Environ() {
key := strings.SplitN(value, "=", 2)[0]
switch strings.ToUpper(key) {
case "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "ALL_PROXY", "GO_OMNI_HTTPC_PROXY_CHILD", "GO_OMNI_HTTPC_TARGET":
continue
}
cmd.Env = append(cmd.Env, value)
}
cmd.Env = append(cmd.Env, "GO_OMNI_HTTPC_PROXY_CHILD=1", "GO_OMNI_HTTPC_TARGET="+origin.URL, "HTTP_PROXY="+proxy.URL, "HTTPS_PROXY="+proxy.URL, "NO_PROXY=bypass.invalid")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(err, string(out))
}
if proxied.Load() != 3 || target.Load() != 1 {
t.Fatalf("proxy=%d direct=%d", proxied.Load(), target.Load())
}
}
func TestProxyEnvironmentProcess(t *testing.T) {
if os.Getenv("GO_OMNI_HTTPC_PROXY_CHILD") != "1" {
t.Skip("subprocess only")
}
for _, proxy := range []ProxyConfig{{}, {Mode: ProxyEnvironment}, {Mode: ProxyURL, URL: os.Getenv("HTTP_PROXY")}} {
c := testClient(t, Config{Proxy: proxy})
if err := c.JSON(context.Background(), "GET", "http://target.invalid/path", nil, nil); err != nil {
t.Fatal(err)
}
}
environment := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyEnvironment}})
req, _ := http.NewRequest("GET", "http://bypass.invalid", nil)
u, err := environment.owned.Proxy(req)
if err != nil || u != nil {
t.Fatal("NO_PROXY ignored")
}
direct := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyDirect}})
if direct.owned.Proxy != nil {
t.Fatal("direct still has proxy")
}
if err := direct.JSON(context.Background(), "GET", os.Getenv("GO_OMNI_HTTPC_TARGET"), nil, nil); err != nil {
t.Fatal(err)
}
}
func TestResponseHeaderTimeoutRecovery(t *testing.T) {
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/slow" {
<-r.Context().Done()
return
}
io.WriteString(w, `{}`)
}))
defer s.Close()
c := testClient(t, Config{Timeout: 2 * time.Second, ResponseHeaderTimeout: 30 * time.Millisecond})
start := time.Now()
err := c.JSON(context.Background(), "GET", s.URL+"/slow", nil, nil)
var network net.Error
if !errors.As(err, &network) || !network.Timeout() || time.Since(start) > time.Second {
t.Fatal("header timeout", err)
}
if err = c.JSON(context.Background(), "GET", s.URL+"/ok", nil, nil); err != nil {
t.Fatal("did not recover", err)
}
}
func TestTLSHandshakeTimeout(t *testing.T) {
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer listener.Close()
done := make(chan struct{})
go func() {
defer close(done)
conn, err := listener.Accept()
if err != nil {
return
}
defer conn.Close()
_, _ = io.Copy(io.Discard, conn)
}()
c := testClient(t, Config{Timeout: 2 * time.Second, Proxy: ProxyConfig{Mode: ProxyDirect}, TLSHandshakeTimeout: 30 * time.Millisecond})
start := time.Now()
err = c.JSON(context.Background(), "GET", "https://"+listener.Addr().String(), nil, nil)
var network net.Error
if !errors.As(err, &network) || !network.Timeout() || time.Since(start) > time.Second {
t.Fatal("handshake timeout", err)
}
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("TLS connection leaked")
}
}
func TestPoolLimitsAndIdleExpiry(t *testing.T) {
var connections atomic.Int32
entered := make(chan struct{}, 12)
release := make(chan struct{})
closed := make(chan struct{}, 12)
s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
entered <- struct{}{}
select {
case <-release:
case <-r.Context().Done():
return
}
io.WriteString(w, `{}`)
}))
s.Config.ConnState = func(_ net.Conn, state http.ConnState) {
if state == http.StateNew {
connections.Add(1)
}
if state == http.StateClosed {
closed <- struct{}{}
}
}
s.Start()
defer s.Close()
c := testClient(t, Config{Timeout: 2 * time.Second, MaxConnsPerHost: 2, MaxIdleConnsPerHost: 2, MaxIdleConns: 2, IdleConnTimeout: 40 * time.Millisecond})
var wg sync.WaitGroup
for i := 0; i < 12; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); err != nil {
t.Error(err)
}
}()
}
for i := 0; i < 2; i++ {
select {
case <-entered:
case <-time.After(time.Second):
close(release)
t.Fatal("requests did not start")
}
}
if connections.Load() != 2 {
t.Error("connection cap", connections.Load())
}
close(release)
wg.Wait()
if connections.Load() != 2 {
t.Fatal("pool was not reused", connections.Load())
}
for i := 0; i < 2; i++ {
select {
case <-closed:
case <-time.After(time.Second):
t.Fatal("idle pool was not reclaimed")
}
}
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); err != nil {
t.Fatal(err)
}
if connections.Load() != 3 {
t.Fatal("new connection not established after idle timeout")
}
c.Close()
select {
case <-closed:
case <-time.After(time.Second):
t.Fatal("Close left idle connections open")
}
}
func TestHTTPSVerificationRetained(t *testing.T) {
s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, `{}`) }))
defer s.Close()
c := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyDirect}})
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
var urlError *url.Error
if !errors.As(err, &urlError) {
t.Fatal("self-signed server unexpectedly trusted", err)
}
}

119
httpc/json.go Normal file
View File

@@ -0,0 +1,119 @@
package httpc
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"net/http"
"time"
)
// JSON bounds response memory, closes the body, and decodes only 2xx responses.
// Optional headers/limits belong to this call and never alter the client.
// output=nil discards a bounded stream; it does not buffer or validate JSON.
// Empty successful bodies leave output unchanged. StatusError contains no body.
func (c *Client) JSON(ctx context.Context, method, url string, input, output any, options ...RequestOption) error {
if ctx == nil {
return errors.New("httpc: nil context")
}
if c.closed.Load() {
return ErrClosed
}
if err := ctx.Err(); err != nil {
return err
}
settings := requestOptions{maxResponseBytes: c.config.MaxResponseBytes}
if len(options) > 0 {
var err error
settings, err = jsonOptions(c.config.MaxResponseBytes, options)
if err != nil {
return err
}
}
var body io.Reader
if input != nil {
data, err := json.Marshal(input)
if err != nil {
return err
}
body = bytes.NewReader(data)
}
request, err := http.NewRequestWithContext(ctx, method, url, body)
if err != nil {
return err
}
if settings.headers != nil {
request.Header = settings.headers
}
if input != nil {
if _, exists := request.Header["Content-Type"]; !exists {
request.Header.Set("Content-Type", "application/json")
}
}
response, err := c.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode >= 300 {
drainStatusBody(response, settings.maxResponseBytes)
return &StatusError{StatusCode: response.StatusCode}
}
if response.ContentLength > settings.maxResponseBytes {
return ErrResponseTooLarge
}
limited := io.LimitReader(response.Body, settings.maxResponseBytes+1)
if output == nil {
count, err := io.Copy(io.Discard, limited)
if err != nil {
return err
}
if count > settings.maxResponseBytes {
return ErrResponseTooLarge
}
return nil
}
data, err := io.ReadAll(limited)
if err != nil {
return err
}
if int64(len(data)) > settings.maxResponseBytes {
return ErrResponseTooLarge
}
if len(bytes.TrimSpace(data)) == 0 {
return nil
}
return json.Unmarshal(data, output)
}
// A known small status body may still be slow. Spend at most a short grace
// period trying to retain the connection, then cancel the HTTP read. The status
// remains authoritative and the deferred Close still owns body cleanup.
func drainStatusBody(response *http.Response, limit int64) {
if response.ContentLength < 0 || response.ContentLength > min(limit, 32<<10) {
return
}
body, ok := response.Body.(*responseBody)
if !ok {
return
}
timer := time.AfterFunc(25*time.Millisecond, body.cancel)
defer timer.Stop()
drainSmallBody(response, limit)
}
// Keep the option builder out of the allocation path of existing plain calls.
func jsonOptions(limit int64, options []RequestOption) (requestOptions, error) {
settings := requestOptions{maxResponseBytes: limit}
for _, option := range options {
if option == nil {
return requestOptions{}, errors.New("httpc: nil request option")
}
if err := option(&settings); err != nil {
return requestOptions{}, err
}
}
return settings, nil
}

99
httpc/request.go Normal file
View File

@@ -0,0 +1,99 @@
package httpc
import (
"errors"
"net/http"
"strings"
)
type requestOptions struct {
headers http.Header
maxResponseBytes int64
}
// RequestOption applies only to one JSON call. Returned options can be reused
// concurrently; request headers are independent, with no client-wide credentials.
type RequestOption func(*requestOptions) error
// WithHeader sets a header for this call; later options replace earlier values.
// Framing and Host must be controlled through a standard Request passed to Do.
func WithHeader(name, value string) RequestOption {
name = http.CanonicalHeaderKey(name)
err := checkHeader(name, []string{value})
return func(o *requestOptions) error {
if err != nil {
return err
}
if o.headers == nil {
o.headers = make(http.Header)
}
o.headers[name] = []string{value}
return nil
}
}
// WithHeaders snapshots headers immediately. Editing the original map after
// construction does not change the option; each invocation gets separate slices.
func WithHeaders(headers http.Header) RequestOption {
snapshot := make(http.Header, len(headers))
var err error
for key, values := range headers {
name := http.CanonicalHeaderKey(key)
if err = checkHeader(name, values); err != nil {
break
}
if _, exists := snapshot[name]; exists {
err = errors.New("httpc: duplicate canonical header name")
break
}
snapshot[name] = append([]string(nil), values...)
}
return func(o *requestOptions) error {
if err != nil {
return err
}
if o.headers == nil {
o.headers = make(http.Header, len(snapshot))
}
for name, values := range snapshot {
o.headers[name] = append([]string(nil), values...)
}
return nil
}
}
// WithMaxResponseBytes overrides JSON's bounded response read for one call.
// The size is positive and at most 1 GiB. Do remains a caller-owned stream.
func WithMaxResponseBytes(limit int64) RequestOption {
return func(o *requestOptions) error {
if limit <= 0 || limit > maxResponseBytes {
return errors.New("httpc: response limit must be positive and at most 1 GiB")
}
o.maxResponseBytes = limit
return nil
}
}
func checkHeader(name string, values []string) error {
if name == "" {
return errors.New("httpc: header name must not be empty")
}
for _, c := range []byte(name) {
if c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || strings.ContainsRune("!#$%&'*+-.^_`|~", rune(c)) {
continue
}
return errors.New("httpc: invalid header name")
}
switch name {
case "Host", "Content-Length", "Transfer-Encoding", "Trailer":
return errors.New("httpc: use Do for Host or request framing controls")
}
for _, value := range values {
for i := 0; i < len(value); i++ {
c := value[i]
if c == '\x7f' || c < ' ' && c != '\t' {
return errors.New("httpc: invalid header value")
}
}
}
return nil
}

316
httpc/request_test.go Normal file
View File

@@ -0,0 +1,316 @@
package httpc
import (
"bytes"
"compress/gzip"
"context"
"errors"
"fmt"
"git.yuntuo.hk/k2safe/go-omni/logx"
"io"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
func TestJSONHeadersConcurrentSnapshot(t *testing.T) {
var leaked atomic.Bool
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer "+r.URL.Query().Get("caller") {
leaked.Store(true)
}
if r.Header.Get("X-Shared") != "original" || len(r.Header.Values("X-Tag")) != 2 {
leaked.Store(true)
}
if r.Header.Get("Content-Type") != "application/vnd.example+json" {
leaked.Store(true)
}
io.WriteString(w, `{"ok":true}`)
}))
defer s.Close()
c := testClient(t, Config{})
headers := http.Header{"X-Shared": []string{"original"}, "X-Tag": []string{"one", "two"}}
snapshot := WithHeaders(headers)
headers.Set("X-Shared", "changed")
headers["X-Tag"][0] = "changed"
var wg sync.WaitGroup
for i := 0; i < 24; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
id := fmt.Sprint(i)
var output struct {
OK bool `json:"ok"`
}
if err := c.JSON(context.Background(), "POST", s.URL+"?caller="+id, map[string]bool{"hello": true}, &output, snapshot, WithHeader("Authorization", "Bearer "+id), WithHeader("content-type", "application/vnd.example+json")); err != nil || !output.OK {
t.Errorf("JSON failed: %v", err)
}
}(i)
}
wg.Wait()
if leaked.Load() {
t.Fatal("headers leaked across requests or changed with source map")
}
}
func TestRequestOptionsValidationBeforeNetwork(t *testing.T) {
var calls atomic.Int32
c := testClient(t, Config{Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
calls.Add(1)
return benchmarkTransport{`{}`, 200}.RoundTrip(r)
})})
invalid := []RequestOption{nil, WithHeader("bad name", "private-token"), WithHeader("Authorization", "private-token\r\nInjected: yes"), WithHeader("Host", "example.invalid"), WithHeader("Content-Length", "1"), WithHeader("Transfer-Encoding", "chunked"), WithHeader("Trailer", "X-Signature"), WithHeaders(http.Header{"X-Name": []string{"one"}, "x-name": []string{"two"}}), WithMaxResponseBytes(0), WithMaxResponseBytes(maxResponseBytes + 1)}
for _, option := range invalid {
err := c.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, option)
if err == nil || strings.Contains(err.Error(), "private-token") {
t.Fatal("missing or unsafe error", err)
}
}
if calls.Load() != 0 {
t.Fatal("invalid options reached network")
}
}
func TestJSONResponseLimitsAndStreaming(t *testing.T) {
var calls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
switch r.URL.Path {
case "/empty":
w.WriteHeader(204)
case "/chunked":
w.(http.Flusher).Flush()
io.WriteString(w, strings.Repeat("x", 17))
case "/status-stream":
w.WriteHeader(401)
w.(http.Flusher).Flush()
<-r.Context().Done()
case "/multiple":
io.WriteString(w, `{} {}`)
default:
io.WriteString(w, `{"ok":true}`)
}
}))
defer server.Close()
c := testClient(t, Config{MaxResponseBytes: 8, Timeout: time.Second})
var out struct {
OK bool `json:"ok"`
}
if err := c.JSON(context.Background(), "GET", server.URL, nil, &out, WithMaxResponseBytes(32)); err != nil || !out.OK {
t.Fatal(err)
}
if err := c.JSON(context.Background(), "GET", server.URL, nil, nil); !errors.Is(err, ErrResponseTooLarge) {
t.Fatal("limit changed permanently", err)
}
for _, target := range []any{nil, &out} {
if err := c.JSON(context.Background(), "GET", server.URL+"/chunked", nil, target, WithMaxResponseBytes(16)); !errors.Is(err, ErrResponseTooLarge) {
t.Fatal("unbounded chunked body", err)
}
}
if err := c.JSON(context.Background(), "GET", server.URL+"/empty", nil, &out); err != nil || !out.OK {
t.Fatal("empty success changed output", err)
}
if err := c.JSON(context.Background(), "GET", server.URL+"/multiple", nil, &out); err == nil {
t.Fatal("multiple JSON documents accepted")
}
start := time.Now()
err := c.JSON(context.Background(), "GET", server.URL+"/status-stream", nil, nil)
var status *StatusError
if !errors.As(err, &status) || status.StatusCode != 401 || time.Since(start) > 500*time.Millisecond {
t.Fatal("waited for unknown status body", err)
}
if calls.Load() != 7 {
t.Fatal(calls.Load())
}
}
func TestErrorBodyPoolReuseAndDrainBound(t *testing.T) {
var connections atomic.Int32
s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(401)
io.WriteString(w, "private-error-body")
}))
s.Config.ConnState = func(_ net.Conn, state http.ConnState) {
if state == http.StateNew {
connections.Add(1)
}
}
s.Start()
defer s.Close()
c := testClient(t, Config{})
for i := 0; i < 10; i++ {
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
var status *StatusError
if !errors.As(err, &status) || strings.Contains(err.Error(), "private-error-body") {
t.Fatal(err)
}
}
if connections.Load() != 1 {
t.Fatal("small status response prevented reuse", connections.Load())
}
body := &countedBody{Reader: strings.NewReader(strings.Repeat("x", 1<<20))}
custom := testClient(t, Config{Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
return &http.Response{StatusCode: 401, ContentLength: 1, Body: body, Header: make(http.Header)}, nil
})})
err := custom.JSON(context.Background(), "GET", "http://example.invalid", nil, nil)
var status *StatusError
if !errors.As(err, &status) || !body.closed || body.read > (32<<10)+1 {
t.Fatal("unbounded/missing drain", body.read, body.closed, err)
}
}
type countedBody struct {
io.Reader
read int
closed bool
}
func (b *countedBody) Read(p []byte) (int, error) {
n, err := b.Reader.Read(p)
b.read += n
return n, err
}
func (b *countedBody) Close() error { b.closed = true; return nil }
type roundTripperFunc func(*http.Request) (*http.Response, error)
func (f roundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
type borrowedTransport struct{ closed atomic.Bool }
func (t *borrowedTransport) RoundTrip(r *http.Request) (*http.Response, error) {
return benchmarkTransport{`{}`, 200}.RoundTrip(r)
}
func (t *borrowedTransport) CloseIdleConnections() { t.closed.Store(true) }
func TestBorrowedTransportAndHeaderRetry(t *testing.T) {
owned := &borrowedTransport{}
c := testClient(t, Config{Transport: owned})
c.Close()
if owned.closed.Load() {
t.Fatal("closed caller's transport")
}
var attempts atomic.Int32
retry := testClient(t, Config{MaxRetries: 1, Backoff: time.Millisecond, Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
if r.Header.Get("Authorization") != "Bearer test" {
t.Error("retry lost header")
}
status := 200
if attempts.Add(1) == 1 {
status = 503
}
return benchmarkTransport{`{}`, status}.RoundTrip(r)
})})
if err := retry.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, WithHeader("Authorization", "Bearer test")); err != nil || attempts.Load() != 2 {
t.Fatal(err, attempts.Load())
}
}
func FuzzRequestHeaderOptions(f *testing.F) {
f.Add("Authorization", "Bearer example")
f.Add("X-Name", "one\r\nX-Test: bad")
f.Fuzz(func(t *testing.T, name, value string) {
if len(name)+len(value) > 8192 {
t.Skip()
}
option := WithHeader(name, value)
var settings requestOptions
if err := option(&settings); err != nil {
return
}
req, err := http.NewRequest("GET", "http://example.invalid", nil)
if err != nil {
t.Fatal(err)
}
req.Header = settings.headers
// The standard Request writer must be able to serialize accepted headers.
if err = req.Write(io.Discard); err != nil {
t.Fatal("accepted malformed header", err)
}
})
}
func TestHTTPLogDoesNotContainCredentials(t *testing.T) {
var buffer bytes.Buffer
logger, err := logx.New(logx.Config{Writer: &buffer})
if err != nil {
t.Fatal(err)
}
defer logger.Close()
c, err := New(Config{Transport: benchmarkTransport{"private-response", 401}})
if err != nil {
t.Fatal(err)
}
defer c.Close()
ctx := logx.WithLogger(context.Background(), logger)
err = c.JSON(ctx, "GET", "http://private-user:private-password@example.invalid/private-path?token=private-query", nil, nil, WithHeader("Authorization", "Bearer private-token"))
if err == nil {
t.Fatal("expected status error")
}
for _, secret := range []string{"private-user", "private-password", "private-path", "private-query", "private-token", "private-response"} {
if strings.Contains(buffer.String(), secret) || strings.Contains(err.Error(), secret) {
t.Fatal("sensitive field leaked")
}
}
if !strings.Contains(buffer.String(), "http.client") {
t.Fatal("missing access log")
}
}
func TestCloseAllowsInflightCallToFinish(t *testing.T) {
entered := make(chan struct{})
release := make(chan struct{})
var once sync.Once
releaseNow := func() { once.Do(func() { close(release) }) }
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { close(entered); <-release; io.WriteString(w, `{}`) }))
defer s.Close()
c := testClient(t, Config{Timeout: time.Second})
done := make(chan error, 1)
go func() { done <- c.JSON(context.Background(), "GET", s.URL, nil, nil) }()
defer releaseNow()
select {
case <-entered:
case err := <-done:
t.Fatal("request failed before handler", err)
case <-time.After(2 * time.Second):
t.Fatal("handler not entered")
}
c.Close()
releaseNow()
if err := <-done; err != nil {
t.Fatal("Close interrupted in-flight call", err)
}
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); !errors.Is(err, ErrClosed) {
t.Fatal("accepted new call after Close", err)
}
}
func TestSlowKnownErrorBodyHasShortDrainBudget(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Length", "100")
w.WriteHeader(503)
w.(http.Flusher).Flush()
<-r.Context().Done()
}))
defer server.Close()
c := testClient(t, Config{Timeout: 2 * time.Second})
start := time.Now()
err := c.JSON(context.Background(), "GET", server.URL, nil, nil)
var status *StatusError
if !errors.As(err, &status) || status.StatusCode != 503 || time.Since(start) > 500*time.Millisecond {
t.Fatal("slow error body delayed status", err, time.Since(start))
}
}
func TestJSONLimitAppliesAfterGzipExpansion(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Encoding", "gzip")
writer := gzip.NewWriter(w)
_, _ = io.WriteString(writer, strings.Repeat("x", 4096))
_ = writer.Close()
}))
defer server.Close()
c := testClient(t, Config{MaxResponseBytes: 64})
if err := c.JSON(context.Background(), "GET", server.URL, nil, nil); !errors.Is(err, ErrResponseTooLarge) {
t.Fatal("gzip bypassed response limit", err)
}
}

View File

@@ -76,7 +76,7 @@ func WithLogConfig(settings LogConfig) Option { return config.WithLogConfig(sett
func WithShutdownTimeout(timeout time.Duration) Option { return config.WithShutdownTimeout(timeout) } func WithShutdownTimeout(timeout time.Duration) Option { return config.WithShutdownTimeout(timeout) }
func LoadConfig(prefix string) (Config, error) { return config.LoadFromEnv(prefix) } func LoadConfig(path string) (Config, error) { return config.LoadFile(path) }
func SetupLogger(cfg LogConfig) (func() error, error) { return logx.Setup(cfg) } func SetupLogger(cfg LogConfig) (func() error, error) { return logx.Setup(cfg) }

Some files were not shown because too many files have changed in this diff Show More