Compare commits

...

4 Commits

112 changed files with 3281 additions and 1890 deletions

View File

@@ -11,7 +11,7 @@
| `openapi` | 路由与 OpenAPI 3.0.3 同时注册、DTO schema、响应信封 | 显式声明请求/响应;不自动推断任意 Handler 行为,不附带 Swagger UI |
| `auth` | RS256/EdDSA JWT 验证与公钥轮换、可撤销 Session、Policy 接口 | 登录验证和业务授权规则由接入方提供;不自动获取 token 指定的密钥 |
| `telemetry` | HTTP 入站/出站、Redis、SQL spans,HTTP 指标和 traceId 日志关联 | 注入 OTel exporter/reader 后才外发;SQL Query span 不覆盖后续遍历;不接管全局 provider |
| `config` / `logx` | JSON/YAML 文件、环境变量覆盖、动态日志级别、轮转/脱敏/堆栈/慢请求 | 热更新只开放日志级别;连接池、监听、认证等静态变更要求重启 |
| `config` / `logx` | JSON/YAML/XML 文件、显式环境引用、动态日志级别、轮转/脱敏/堆栈/慢请求 | 热更新只开放日志级别;连接池、监听、认证等静态变更要求重启 |
| `resilience` | 失败阈值、熔断冷却、单探针恢复、上游并发隔离 | 一个实例对应一个依赖;默认内存状态,不做跨进程全局熔断 |
| `middleware/redis` | 跨副本限流、幂等 claim/回放、容量限制 | 专用 Redis namespace;TTL 窗口内保护不等于永久业务去重 |
| `queue` / `queue/redis` | 持久消息接口、Streams 接管、有限重试、失败消息管理 | 至少一次交付,处理器必须幂等;持久性取决于 Redis 部署 |
@@ -102,7 +102,7 @@ err = consumer.Run(serviceContext, func(ctx context.Context, item queue.Delivery
## 配置与日志热更新
`config.LoadFile(path, prefix)` 按默认值→文件→非空环境变量加载并验证。两种文件格式都使用 JSON tag 的 lowerCamelCase,duration 可以写 `"5s"`;文件上限 1 MiB,拒绝未知字段、多文档和非法值。自定义 prefix 要包含期望的尾部下划线,例如 `BILLING_`。空环境变量沿用文件值,不清空已配置字段。
`config.LoadFile(path)` 使用默认值→文件字段显式引用→类型绑定→校验;没有自动环境覆盖。用 `${NAME}` 或 `${NAME:-fallback}` 指定来源,完整规则与迁移见 [CONFIG.md](CONFIG.md)。
```yaml
address: '127.0.0.1:8080'
@@ -113,7 +113,7 @@ log:
format: json
```
先按配置创建 `logx.New(cfg.Log)`,再 `config.NewReloader(path, prefix, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`;后者阻塞,生命周期由应用负责。只有 `log.level` 允许变化,校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。
先按配置创建 `logx.New(cfg.Log)`,再 `config.NewReloader(path, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`;后者阻塞,生命周期由应用负责。只有 `log.level` 允许变化,校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。
## 熔断与观测

View File

@@ -6,3 +6,5 @@
- examples 使用通用场景,避免依赖业务项目、私有数据库或线上服务。必要的外部基础设施配置应明确说明。
- 2026-09-25 用户决定:移除 mappings/mappingFiles 及结构映射代码生成机制,改用可选 copyx 工具包。不得恢复另一套 JSON 或 Go 映射声明语言。
- 深拷贝的自定义转换结果不得与输入共享可变引用;发现共享应报错,不能降级成浅拷贝或静默接受。
- 2026-09-26 用户决定:配置文件显式声明固定值或 `${NAME}` / `${NAME:-fallback}`,旧自动环境覆盖、前缀/别名及 env/envDefault 标签模式移除,不得重新引入隐式二次覆盖。项目维护 Config/Default/Validate/文件,生成器维护 Load/main。

4
API.md
View File

@@ -48,7 +48,7 @@ go run ./cmd/go-omni new --api examples/contracts/user.api \
--log-output file --log-format json ../userService
```
`api generate` 在新目录中也会生成完整项目。默认只监听 localhost;`GO_OMNI_HTTP_ADDR` 修改地址,`GO_OMNI_TRANSPORT=native` 切换网络层。配置顺序为默认值 → `etc/config.yaml` → 环境变量。文件日志支持现有轮转配置。远程试用使用与 CLI 一致的固定版本,见 [发布说明](RELEASE.md)。
`api generate` 在新目录中也会生成完整项目。默认只监听 localhost;`GO_OMNI_HTTP_ADDR` 修改地址,`GO_OMNI_TRANSPORT=native` 切换网络层。配置顺序为 Default() → `etc/config.yaml` 中的显式环境引用 → 类型绑定 → Validate();固定值不再被隐式覆盖。文件日志支持现有轮转配置。远程试用使用与 CLI 一致的固定版本,见 [发布说明](RELEASE.md)。
API 项目现在支持组合数据库、存储和日志:
@@ -63,7 +63,7 @@ go-omni new --api user.api --module example.com/userService \
初次选项记录在 `.goOmni/apiManifest.json`,以后直接 `api generate` 会继承。已有项目通过参数切换选项会报错,因为初始化代码属于业务方;可在新目录生成对照后手动迁移。`--force` 不允许覆盖 API 项目。new 与 api new 统一生成契约项目,不再使用旧式 CRUD 目录。
MySQL 使用官方 `database/mysql` 结构化适配器,密码从 `GO_OMNI_DB_PASSWORD` 覆盖,非秘密池配置位于 YAML,环境变量优先。存储使用原有 `GO_OMNI_STORAGE_*`;API 项目不会额外生成公开的签名发放接口,鉴权和对象权限由业务实现。初始化失败会清理已打开资源,退出清理可重复调用。
MySQL 使用官方 `database/mysql` 结构化适配器,生成文件以 `password: "${GO_OMNI_DB_PASSWORD}"` 显式读取密码;其他字段可写固定值或引用,缺失项保留 Default()。存储配置也只读取文件明确引用的变量;API 项目不会额外生成公开的签名发放接口,鉴权和对象权限由业务实现。初始化失败会清理已打开资源,退出清理可重复调用。
HTTP 请求类型不能推导数据库表结构;迁移和 Repository 由业务自己维护。当前目录的完整示例见 [managedService](examples/managedService/README.md)。MySQL、事务审计和鉴权实现可参考 [旧目录用户管理示例](examples/userService/README.md),接入时按 [项目指南](PROJECT.md) 装配到 config/svc/logic。

View File

@@ -5,7 +5,7 @@
## 实施与验收清单
- [x] 运行期:资源登记、逆序释放、幂等关闭、失败回滚、故障原因保留归框架;生成入口与项目依赖初始化边界明确。多 App 日志隔离、标准 log.Logger 适配、可信请求元数据与依赖中间件示例。
- [x] 配置:项目完整结构体环境覆盖;默认值→文件→环境;嵌套、duration、slice、显式零值、空值策略、必填与脱敏;日志无副作用完整校验。
- [x] 配置:Default()→文件显式环境引用→类型绑定→Validate;嵌套、duration、slice、显式零值、必填/备用值、转义与安全诊断;日志无副作用完整校验。
- [x] 数据库:官方 MySQL 结构化配置/TLS/驱动日志适配,继续复用 OpenConnector 和唯一连接池;独立启动检查、失败清理、生成器接入;不内置版本/表结构/业务错误策略。
- [x] 响应契约:数组、可空对象、成功 null、独立 HTTP 204;Logic/Handler/OpenAPI/漂移检查一致。
- [x] 请求契约:未知 query、JSON UTF-8/charset、无 Body/必填空对象/可选 Body,App 默认与路由覆盖;scalar/any/数组/大小限制一致,普通 JSON 只绑定一次。

View File

@@ -1,5 +1,21 @@
# Changelog
## v0.22.0-preview(2026-09-26)
- 文件显式环境引用与备用值,字符串组合/转义,安全有界展开和类型绑定;变量值原样插入、不递归处理。
- 移除隐式环境覆盖、EnvOptions/前缀/别名及 env/envDefault;数据库统一使用文件加载,不再单独扫描环境。
- 生成 main/Load 新增 -check-config,精简配置文件,保留开发者扩展;错误字段建议和迁移检查。
- README、CONFIG、数据库/HTTP 客户端指南及示例同步新规则;配置中的响应模板需要 $$ 转义,.api 不变。
## v0.21.0-preview(2026-09-26)
- httpc 可配置环境/直连/固定代理、常用连接阶段超时和独立空闲池参数。
- JSON 新增逐请求 Header 快照与响应大小选项,保持有界读取、关闭和认证隔离。
- 成功请求少复制和退避计算,丢弃响应流式处理,错误体有界回收以改善连接复用。
- 增加真实本地代理、超时、连接限制/回收、并发、日志脱敏、gzip、fuzz 和修改前后基准。
- JSON 自定义方法接口签名需同步 variadic 参数;借用 Transport 的参数混配现在报错。详见 httpc/README.md。
## v0.20.0-preview(2026-09-26)
- 生成项目配置加载入口持续由生成器维护;保留业务字段、默认值、校验与 NewLoader 扩展文件。

263
CONFIG.md
View File

@@ -1,178 +1,177 @@
# 统一配置加载
# 配置:文件明确声明值的来源
v0.20 起,新生成服务的 `main.go` 调用生成器维护的 `config.Load`。项目只需要定义业务字段、默认值和校验;日志、HTTP、数据库等配置使用同一流程,配置加载时不打开数据库、监听端口或创建日志文件。
v0.22 起只使用这一条流程:
```text
代码默认值 → envDefault(仅填零值)→ 可选配置文件
→ 完整结构体环境覆盖(含别名)→ 框架规范化 → 项目 Validate
→ 初始化日志及项目依赖 → 启动服务
项目 Default() → 解析选中的一个文件 → 解析字段值中的环境引用
→ 按 Go 类型绑定已提供的字段 → 项目 Validate() → 初始化日志和依赖
```
## 生成项目要维护什么
没有环境前缀、别名表或隐式二次覆盖。业务只维护 `Config`、`Default()`、`Validate()` 和配置文件。项目 Validate 中继续调用嵌套框架配置的 Validate;生成模板已包含这些调用。验证函数只检查值,不应打开资源。
| 文件 | 维护方 | 用途 |
| --- | --- | --- |
| `main.go` | 生成器 | 启动、信号、请求排空、依赖关闭 |
| `internal/config/loadGen.go` | 生成器 | 调用统一加载入口;纳入 `generate --check` |
| `internal/config/config.go` | 项目 | `Config`、`Default()`、`Validate()` |
| `internal/config/loader.go` | 项目 | `NewLoader()`,只有增加文件格式时才需修改 |
| `etc/config.yaml` | 项目 | 部署配置;可改用 JSON/XML 并传 `-config` |
| `internal/svc` | 项目与生成器分文件 | 初始化依赖、登记清理、AppOptions |
比如增加业务的批次数量:在 Config 增加一个字段,在 Validate 加范围校验;需要部署值时改配置文件或环境变量。不修改 main、Load 或 Handler。
## 直接使用
```go
// 添加到项目 Config。
BatchSize int `json:"batchSize" envDefault:"100"`
// 添加到项目 Validate。
if c.BatchSize < 1 || c.BatchSize > 1000 {
return omniConfig.Invalid("batchSize", "must be between 1 and 1000")
}
```
自动支持 `GO_OMNI_BATCH_SIZE`。生成服务的空环境值策略为 `ignore`,0 和 false 是有效覆盖;范围由 Validate 决定。
重新生成保留 config.go、loader.go 和 svc 的手写扩展。
## 手写项目用法
```go
package main
import (
"time"
"git.yuntuo.hk/k2safe/go-omni/config"
)
type Config struct {
Timeout time.Duration `json:"timeout" envDefault:"3s"`
Limit int `json:"limit" envDefault:"100"`
Token string `json:"-" env:"APP_TOKEN,secret"`
Server omniConfig.Config `json:"server"`
BatchSize int `json:"batchSize"`
Enabled bool `json:"enabled"`
Password string `json:"password"`
Timeout time.Duration `json:"timeout"`
}
func Default() Config {
return Config{Server: omniConfig.Default(), BatchSize: 100, Enabled: true,
Timeout: 3*time.Second}
}
func (c Config) Validate() error {
if c.Timeout <= 0 {
return config.Invalid("timeout", "must be positive")
if err := c.Server.Validate(); err != nil {
return omniConfig.AtField("server", err)
}
if c.BatchSize < 0 { return omniConfig.Invalid("batchSize", "must not be negative") }
return nil
}
// cfg, err := config.LoadProject("etc/config.yaml", Config{},
// config.EnvOptions{Prefix: "APP_", Empty: "reject"})
```
`LoadProject` 接收结构体默认值,可实现值接收器或指针接收器的 `Validate() error`。空路径表示不读文件。
`DecodeFile` 只做文件覆盖,不做 envDefault、环境覆盖或业务 Validate;旧的两参数调用仍有效,可用结构体或结构体指针。低层 `ApplyDefaults`、`OverlayEnv`、`LoadFile` 继续存在。
嵌套校验用 `config.AtField("client", err)` 加路径前缀。公共原因通过 `config.Invalid` 提供,原因中不要包含密码、DSN、Token 或输入原值。
## JSON / YAML / XML
按文件扩展名选择解析器:`.json`、`.yaml`、`.yml`、`.xml`,大小写不敏感。所有格式使用 Go 结构体的 **json 字段名**,无需为 XML 复制一份结构或标签。
以下配置等价:
```json
{"client":{"timeout":"2s"},"enabled":false,"ports":[8080,8081],"optional":null}
// 手写项目;生成项目的 Load 已自动调用此入口。
cfg, err := omniConfig.LoadProject("etc/config.yaml", Default())
```
```yaml
client:
timeout: 2s
server:
address: "${HTTP_ADDR:-127.0.0.1:8080}"
password: "${DB_PASSWORD}"
timeout: "${REQUEST_TIMEOUT:-3s}"
enabled: false
ports: [8080, 8081]
optional: null
```
文件未写 batchSize,保留 100;enabled 明确写 false,就使用 false。变量名字完全由文件决定,不需要对应 Go 字段名。配置文件可只列出部署需要变动的字段;不会自动导出/打印包含凭据的完整配置。
## 引用语法
| 文件里的值 | 含义 |
| --- | --- |
| `host: 127.0.0.1` | 固定值,不读取任何环境变量 |
| `password: "${DB_PASSWORD}"` | 未设置或空字符串报错,不回退到 Default() |
| `port: "${DB_PORT:-3306}"` | 未设置或为空时使用 3306 |
| `sessionToken: "${SESSION_TOKEN:-}"` | 未设置或为空时使用空字符串,是否合法由目标类型和 Validate 决定 |
| `url: "https://${HOST}:${PORT:-443}/v1"` | 普通字符串中可出现多个引用 |
| `literal: "$${DB_PASSWORD}"` | 得到字面量 `${DB_PASSWORD}`,不读取变量 |
| `literal: "$$cost"` | `$$` 转义一个 `$`,得到 `$cost` |
变量名遵循 `[A-Za-z_][A-Za-z0-9_]*`。只支持 `${NAME}`、`${NAME:-fallback}` 和 `$$` 转义,不支持 `${NAME-default}`、嵌套引用或 shell 表达式。普通单独的 `$` 保留。fallback 到第一个 `}` 结束,可包含冒号、`:-`、引号等文件格式允许的字符;需要包含 `}` 的值可使用固定值或环境变量。
解析只扫描文件的**已解码字段值**,不替换 YAML/JSON/XML 源文本,不扩展字段名,也不扫描 Default() 的字符串。插入的环境值和 fallback 不再展开。密码 `DDSFLKJlsf:saf:-123`、引号、换行、`${OTHER}` 均原样保留,不破坏文档结构。字符串不统一 TrimSpace;`"0"`、`"false"` 和全空格字符串均属于已提供。
同一变量每次加载最多读取一次;下一次加载会重新读取。没有全局变量值缓存。并发加载各自维护结果,不能把调用者正在修改的默认配置当作共享可变输入。
## 类型与覆盖
目标 Go 类型决定引用结果的转换:整数、浮点数、布尔值、`time.Duration`、字符串、指针及已有自定义 JSON/Text 反序列化。数值范围按目标位宽检查,整数不经过 float64。自定义反序列化仍由自定义类型解释文本,不猜测其业务含义。
- 文件缺失字段保留初始值;显式 0、false、空字符串正常覆盖,不用零值判断字段是否存在。
- `${MISSING}` 失败就报错;需要备用值必须写 `${MISSING:-value}`。
- 字符串编号 `"001"` 保持 `001`;绑定到 int 才转整数。环境里的 `null` 不自动变成空指针,`[1,2]` 不自动变成数组。
- 集合直接在文件里声明,可在每个标量元素中使用引用,例如 `ports: ["${PORT_A:-8080}", "${PORT_B:-8081}"]`。
- 数组整体替换;空 slice 保持非 nil;固定长度 Go 数组要求长度一致。map 合并键,已提供键的值整体替换。
- 固定文件 null 沿用 JSON 语义:清空指针/map/slice/interface,普通值字段保持初始值。需要区分 null 时使用指针。
- `any` 的引用结果保持字符串,不猜测动态类型。固定 JSON/YAML 数字使用 json.Number 保留精度。
- 引用成功后的值不会再覆盖一次。旧 `GO_OMNI_*` 名称只有出现在文件引用中才有作用;不隐式重置日志 file/rotation。
普通固定字符串仍按原有严格绑定规则处理;不会为了支持变量,把所有 `"123"` 都自动转换成数字。
## YAML、JSON、XML 一致
使用 Go 的 `json` 字段名,无需维护三份 tag。以下三份文件等价:
```yaml
port: "${PORT:-3306}"
password: "${PASSWORD}"
```
```json
{"port":"${PORT:-3306}","password":"${PASSWORD}"}
```
```xml
<config>
<client><timeout>2s</timeout></client>
<enabled>false</enabled>
<ports type="array"><item>8080</item><item>8081</item></ports>
<optional type="null"/>
</config>
<config><port>${PORT:-3306}</port><password>${PASSWORD}</password></config>
```
XML 必须使用一个 `<config>` 根;普通叶节点按目标 Go 类型转换。数组显式 `type="array"` 加 `<item>`,不根据重复元素猜测类型。`type="array"` 空元素表示空数组,`type="object"` 表示空对象,`type="null"` 表示 null。`any` 中的 XML 叶节点默认是字符串,需要动态数字/布尔时标注 `type="number"` / `type="bool"`;也可用 `type="string"`。
XML 使用 `<config>` 根,标量占位符写在普通叶元素或 `type="string"` 元素中,由 Go 字段决定类型;显式 `type="number"` / `type="bool"` 表示文件里已经有合法数字/布尔常量。XML 数组用 `type="array"` 加 `<item>`;空对象 `type="object"`、null 用 `type="null"`。不从重复元素猜测数组。
类型和覆盖规则:
`.api` 不经过配置加载器,响应模板 `${code}` 不变。**如果把响应模板写在 YAML/JSON/XML 配置文件里,须转义:**
- 缺失字段保留默认值;显式 0、false、空字符串正常覆盖。
- null 清空指针、map、slice、interface;普通值字段保持默认值,沿用 Go JSON 解码语义。需要区分 null 时用指针。
- 数组整体替换;空 slice 保持非 nil;固定长度 Go 数组要求长度一致,避免丢弃多余配置。
- map 合并键,已提供的键整体覆盖对应值;map 中未提供的键保留。
- duration 支持 `250ms`、`3s`;整数保持精度,动态 JSON 数字使用 `json.Number`,不先转 float64。
- 自定义 `json.Unmarshaler` / `encoding.TextUnmarshaler` 仍可接入。这些回调是可信项目代码,必须自己保证并发和副作用安全;不要修改共享外部状态。
- JSON `,string` 字段、嵌入字段遵循 JSON 字段选择;未知字段、重复绑定、冲突均拒绝。
内置解析器限制:单文件最多 1 MiB,解析最多 64 层、100000 个节点;绑定和默认值复制也有深度限制。只接受普通文件,不读取设备或命名管道。不支持多文档、重复字段、无效 UTF-8;YAML 拒绝 alias/merge/custom tag,XML 拒绝 DTD/外部实体、命名空间、业务属性和混合内容。XML 仅支持配置所需的子集,不是通用 XML 文档映射器。
这些限制针对启动配置,避免意外超大文件、别名扩展和模糊覆盖。自定义值解码器的内部字段校验由该解码器负责;例如现有错误目录使用其专有解码规则。
## 环境覆盖
统一入口的 EnvOptions 控制前缀、空值策略、别名和 Lookup。完整 JSON 路径自动转换为 UPPER_SNAKE_CASE:`client.timeout` → `APP_CLIENT_TIMEOUT`。
```go
config.EnvOptions{
Prefix: "GO_OMNI_", Empty: "ignore",
Aliases: config.ServerEnvAliases("GO_OMNI_", "server"),
}
```yaml
server:
response:
body:
code: "$${code}"
msg: "$${message}"
data: "$${data}"
```
生成项目同时接受完整的 `GO_OMNI_SERVER_HTTP_READ_TIMEOUT` 与短别名 `GO_OMNI_HTTP_READ_TIMEOUT`。完整名存在时优先,即便其空值被 ignore,也不回退别名。不会再二次调用 ApplyEnv 覆盖业务字段。
加载后恢复为 `${code}` 等响应占位符,仍由响应编码器处理。
显式 `env:"CUSTOM_NAME"` 是绝对名称;`env:"-"` 跳过子树。`json:"-"` 只在声明显式 env 名时参与,可让密钥只能从环境加载。`secret` 标签表达用途,解析错误对所有字段都不打印值。slice/map 的环境值采用 JSON。`required` 检查字符串非空、指针/集合非 nil;复杂范围由 Validate 负责。详见 [完整业务环境覆盖](DEVELOPMENT.md#完整业务配置的环境覆盖)。
## 生成项目与检查
显式环境选择日志 stdout/stderr 会清除文件 profile 的 File/Rotation,除非同时提供非空 File 覆盖。该兼容规则由框架处理。
| 文件 | 维护方 |
| --- | --- |
| `main.go`、`internal/config/loadGen.go` | 生成器维护,纳入 `generate --check` |
| `internal/config/config.go` | 项目维护 Config、Default、Validate |
| `etc/config.yaml` 等选定文件 | 项目维护,重新生成不覆盖 |
| `internal/config/loader.go` | 初次生成即可使用,仅添加自定义格式时才修改 |
## 扩展解析器
```sh
go run . -config etc/config.test.yaml -check-config
go run . -config etc/config.local.yaml
go build .
go-omni api generate --check
```
内置格式直接可用。新增格式只在 `internal/config/loader.go` 的 `NewLoader` 注册:
`-check-config` 使用与启动完全相同的 Load,通过时退出 0、失败时非零;不打印配置,不创建日志文件、数据库、后台组件或 HTTP 监听。它只校验配置,不检测真实数据库连接或证书握手。
`-config` 只选一个文件,不自动叠加 local/test/default。指定文件不存在时报错;`-config ''` 只使用 Default() 后校验,不读取任何环境变量,也不寻找其他文件。示例:[examples/configuration](examples/configuration/README.md)。
## 安全诊断与限制
```text
config "etc/config.yaml":7:9 env "DB_PORT" field "database.port": expected a valid int
```
`*config.Error` 包含 File/Field/Env/Line/Column/Reason。引用错误同时保留文件与环境变量名;组合引用失败可列出多个名字。行列定位原文件字段,不把环境值伪装成源文件文本。无法取得位置时仍提供可用的文件/字段信息。
用 `config.Invalid(field, publicReason)` 或 `config.AtField(prefix, err)` 报业务校验错误。未知字段给出唯一且接近的已声明字段建议,不自动修改。错误默认不打印原值;自定义 Reason 不应拼入秘密值。原始 cause 仍支持 errors.Is/As,不要把 Unwrap 后的错误或整个配置写到日志。
文件最多 1 MiB;单个环境值/展开字段最多 1 MiB,展开字符串合计/变量缓存分别最多 4 MiB;解析/展开限制 64 层、100000 个节点与引用计数。拒绝无效 UTF-8、多文档、重复字段、未知字段、非普通文件;YAML 不接受 alias/merge/custom tag,XML 不接受 DTD/外部实体。不执行 shell,不自动加载 `.env`。
## 可选解析器与测试来源
内置格式无需项目手写加载器。需要额外格式时在已有 NewLoader 注册 Decoder,后续仍自动经过同一引用/绑定/校验流程:
```go
func NewLoader() (*omniConfig.Loader, error) {
return omniConfig.NewLoader(
omniConfig.WithDecoder("custom", omniConfig.DecoderFunc(
func(data []byte) (omniConfig.Document, error) {
// 演示注册扩展名;实际可在这里调用项目选定的 TOML 等解析库。
return (omniConfig.JSONDecoder{}).Decode(data)
},
)),
omniConfig.WithDecoder("custom", omniConfig.JSONDecoder{}),
)
}
```
Decoder 返回 Document:Value 是 map[string]any 根,子值为 string/bool/json.Number/nil/[]any/map[string]any;无自身类型的叶节点可返回 config.Text。Positions 可选,用 `client.timeout` 或 `servers[0].timeout` 标记 1-based 行列。不提供位置时仍有文件、字段、原因。
Decoder 返回 Document(map/string/bool/json.Number/nil/[]any,XML 类标量可用 Text),可提供 Positions。实例注册无全局 init;插件必须可并发调用、不修改共享输入,并且是可信 Go 代码。
自己的语法错误可返回 `&config.Error{Field: "client.timeout", Line: 3, Column: 5, Reason: "expected a duration"}`。Reason 必须安全;普通第三方错误仅公开 `invalid configuration syntax`,内部 cause 通过 errors.Is/As 保留。
测试或嵌入宿主可以用 `WithLookupEnv(func(string)(string,bool))` 替换引用查找函数。它仍然只读取文件明确引用的名称,不启用任何自动覆盖。普通业务不需要这个选项。
注册表属于 Loader 实例,无全局 init 注册;不能覆盖已有扩展名。Loader 可共享,但 Decoder 必须支持并发且不保留/修改输入。解析插件是可信 Go 代码,框架不是插件沙箱。新格式依赖由选择它的项目显式引入,核心不强制引入 TOML 等库。
## v0.22 迁移:旧模式已移除
## 定位配置错误
这是不兼容调整,不保留两套加载逻辑:
```text
config "etc/config.yaml":4:12 field "client.timeout": expected a duration such as 5s
config env "APP_CLIENT_TIMEOUT" field "client.timeout": expected a valid time.Duration
```
- 删除 `EnvOptions`、`LoadFromEnv`、`ApplyEnv`、`OverlayEnv`、`ApplyDefaults`、`ServerEnvAliases` 和 `env`/`envDefault` 标签处理;检出遗留标签会提示迁移。
- 删除 `database.ConfigFromEnv` / `database.ApplyEnv`;数据库配置作为项目 Config 的字段绑定。DSN 字段可在文件里使用 `dsn: "${DB_DSN}"`,序列化 database.Config 不输出 DSN 原值。
- `LoadProject(path, defaults[, loader])` 不再接收 EnvOptions;`LoadFile(path)`、`NewReloader(path, logger)` 不再接收前缀。
- `omni.LoadConfig(path)` 的参数现在是文件路径;空路径只用框架默认值。
`errors.As(err, &diagnostic)` 可获取 `*config.Error` 的 File/Field/Env/Line/Column/Reason;原 `DecodeError` 名称保留为别名。`errors.Is` 可判断文件不存在等 I/O 原因。
迁移顺序:
解析失败显示文件和可获取的行列;Validate 使用 Invalid/AtField 才能定位业务字段。环境覆盖导致的错误优先标环境变量,不错误地归因到文件;数组元素验证也会追溯到覆盖它的集合环境变量。默认值未出现在文件里时没有行列。YAML 语法错误通常只有行号,列显示 1。
1. 将 envDefault 默认值搬入 Default(),移除 env/envDefault 标签。需要文件绑定的旧 `json:"-"` 密钥字段改成明确 json 名称;不打印配置对象。
2. 按实际部署需要,在选定文件写环境引用。固定值保持固定,不按原前缀猜测。原先依赖环境切换日志输出的项目,应在文件明确配置 file/output/rotation。
3. 配置文件中的其他 `${...}` 模板改成 `$${...}` 字面量;`.api` 无需修改。
4. 手写项目改为 LoadProject。生成项目保留 Config/Default/Validate/NewLoader,删除旧手写 Load 后重新生成;main/loadGen 由生成器更新,业务配置文件不会被静默改写。
5. 升级匹配的 CLI 与框架,执行 generate、generate --check、go test、go build 和 `-check-config`,再使用真实部署配置启动。
不打印底层错误原文或配置值;任意普通 `errors.New(...)` 校验失败的公开原因是 `validation failed`,内部原因仍在链中。项目应改用 Invalid 给出安全的原因。第三方子配置校验若没有字段元数据,只定位到对应子树,框架不会从任意错误文案猜字段。
不要把 Unwrap 后的原始错误、整个配置对象写入公开日志/HTTP;自定义解析器与校验器传入的 Reason 由项目负责脱敏。
## v0.19 及更早项目迁移
先提交现有代码,再升级匹配的 CLI 与框架。旧项目的 `internal/config/config.go` 含手写 Load,新生成器会明确阻止 generate/check/dry-run,不会直接覆盖它。
1. 保留 Config、Default、Validate、业务配置结构及辅助函数。
2. 删除旧 Load。普通的 defaults/file/env/Validate 流程已由 LoadProject 接管;移除由此不再使用的 import。将自定义默认值和校验分别迁入 Default / Validate,自定义文件格式迁入 NewLoader。
3. 框架只在 `loader.go` 不存在时创建 NewLoader;已有项目文件同名时需要保证它提供同签名函数。无需手写 loadGen.go。
4. 重新生成,执行 `generate --check`、`go test ./...`、`go build .`,再用真实配置验证启动。
保留业务需要的显式 env 标签/规则;如果原 Load 有解密或其他项目特殊步骤,不要直接删除,应先明确其归属,必要时在自定义 Decoder 中处理。生成器不自动搬动或猜测手写行为。`api upgrade` 仍可预览较早版本的基础设施差异,但已修改的配置必须按上述步骤迁移。
不会移除低层配置 API;本次移除的是**新项目里重复手写的加载流程**,不删除旧项目的业务文件。
完整可运行例子:[examples/configuration](examples/configuration/README.md)。
低层 `DecodeFile(path, defaults[, loader])` 也解析显式引用,但不调用 Validate;默认值和原始 Document 都不会因失败而被部分修改。日志 Reloader 仍只允许 log.level 变化,其他静态字段变更要求重启。

View File

@@ -1,4 +1,4 @@
# 通用业务开发指南(v0.20.0-preview)
# 通用业务开发指南(v0.22.0-preview)
从 [README](README.md) 创建服务后,业务主要维护 `.api`、配置、`svc.Init`、Logic、Repository 和项目中间件。本页说明通用接入能力;字段、错误码、权限、表结构和事务边界都由项目定义。
@@ -100,44 +100,13 @@ defer reader.Close()
不声明 multipart 字段时,保留原来的 `body *httpx.Body` 流式入口,不创建临时文件。该方式适合特殊协议;项目应在不可撤销副作用之前完成校验并调用 `body.Complete()`。下载和原始回调继续使用 `@produces` / `@body(mode: raw)`。
## 完整业务配置的环境覆盖
## 文件声明配置来源
生成器维护 `internal/config/loadGen.go`;项目维护 Config/Default/Validate 和可选 NewLoader。三种格式、可扩展解析器、字段错误定位及旧 Load 迁移见 [统一配置指南](CONFIG.md)。
项目维护 Config、Default()、Validate() 和配置文件;生成器维护 Load/main。流程为默认值→选定文件→显式 `${NAME}` / `${NAME:-fallback}` 引用→类型绑定→校验。没有前缀、别名或 env/envDefault 隐式覆盖。
生成项目默认执行:代码与 `envDefault` 默认值 → JSON/YAML/XML 文件 → 全结构体环境覆盖(含别名)→ 规范化 → `Validate`。增加业务字段不需要另外写 `os.Getenv`。
固定值保持固定,缺失字段保留 Default(),0/false/空字符串正常覆盖;引用失败不回退初始化默认值。`$$` 转义 `$`,字符串凭据保留空格与特殊字符。
```go
type ClientConfig struct {
Endpoint string `json:"endpoint" env:"CLIENT_ENDPOINT,required"`
Token string `json:"-" env:"CLIENT_TOKEN,required,secret,empty=reject"`
Timeout time.Duration `json:"timeout" envDefault:"3s"`
Retries []int `json:"retries"`
Enabled bool `json:"enabled"`
}
```
把字段加入项目 `Config` 后,默认环境名取 `GO_OMNI_` 加 JSON 路径的 UPPER_SNAKE_CASE,如 `GO_OMNI_CLIENT_TIMEOUT`。显式 env 名是完整名称,可以使用自己的业务前缀。`env:"-"` 排除整个子树;`json:"-"` 字段只有显式 env 标签才参与。
duration 使用 `250ms`、`3s`;slice/map 使用 JSON,如 `[1,3,5]`、`{"retry":"2s"}`。支持嵌套、零/false、空数组、自定义 TextUnmarshaler。空环境值默认允许;可选择 `empty=ignore` 或 `empty=reject`。required 要求字符串非空、指针/集合非 nil;0 和 false 可以是合法业务值,范围由项目 Validate 决定。
手写项目可使用 `config.LoadProject(path, defaults, config.EnvOptions{Prefix:"MY_APP_"})`,或分别调用 `ApplyDefaults/DecodeFile/OverlayEnv`。读取失败不修改输入默认值;诊断不会打印输入值。不要主动打印含密钥的整个配置,或把底层解析错误展开到公开响应。Validate 不打开日志文件、数据库或存储连接;证书的可读性和握手信任在依赖初始化阶段验证。
### 一次覆盖完整配置与 HTTP/日志短名称
```go
cfg, err = config.OverlayEnv(cfg, config.EnvOptions{
Prefix: "MY_APP_",
Empty: "reject", // allow / ignore / reject,项目自行选择
Aliases: config.ServerEnvAliases("MY_APP_", "server"),
// Lookup 可注入测试或项目配置源;默认 os.LookupEnv。
})
```
不要再紧接着调用 `ApplyEnv(cfg.Server, ...)`。上面的调用同时处理完整配置和 HTTP_ADDR、SERVICE、LOG_LEVEL 等框架短名称。例如 `MY_APP_SERVER_ADDRESS` 与 `MY_APP_HTTP_ADDR` 指向同一字段,**完整名称优先**;未设置时才读取短名称。完整名称显式设空仍优先:allow 交给该字段解析(string 可清空;数字等仍要求合法值),ignore 保留文件/默认值,reject 报错。不会因 ignore 再退回别名。字段 `env` 标签的 empty 规则优先于全局规则。
自定义别名用 `EnvOptions.Aliases`,键是完整环境名、值是按优先级排列的别名数组。两种名称都使用同一 Lookup、空值策略和类型转换;布尔 false、数值 0 与未设置不同。集合始终使用 JSON,包括 `MY_APP_TRUSTED_PROXIES='["127.0.0.1"]'`。
新项目模板一次调用 OverlayEnv,默认 `Empty: "ignore"`。旧 `ApplyEnv` 保留原有忽略空值、修剪空白及 TRUSTED_PROXIES 逗号列表行为。已有项目的 `internal/config` 属于开发者文件,升级不覆盖它;按上例合并原来的两次覆盖调用后,保留项目自己的 Validate。日志输出的显式完整名称也优先于短名称。
`go run . -config etc/config.test.yaml -check-config` 只校验,不打开日志、数据库或端口。生成/check/dry-run 保留业务文件;新的模板要求 v0.22。完整格式、错误定位、解析器与移除清单见 [CONFIG.md](CONFIG.md)。
## 依赖、日志与中间件
@@ -280,7 +249,7 @@ go-omni workspace check --file omni.workspace.json
## v0.13 项目升级
先使用与目标框架匹配的 CLI;CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.20.0-preview(或显式引用当前源码)。
先使用与目标框架匹配的 CLI;CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.22.0-preview(或显式引用当前源码)。
```sh
go-omni api upgrade --dir . --dry-run

View File

@@ -70,30 +70,26 @@ application.OnShutdown(client.Close)
```go
client, err := httpc.New(httpc.Config{
Timeout: 3*time.Second, MaxRetries: 2, MaxConnsPerHost: 32,
MaxResponseBytes: 1 << 20,
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
Timeout: 3*time.Second, DialTimeout: time.Second,
TLSHandshakeTimeout: time.Second, ResponseHeaderTimeout: time.Second,
MaxConnsPerHost: 32, MaxIdleConnsPerHost: 16, MaxIdleConns: 64,
IdleConnTimeout: 60*time.Second, MaxResponseBytes: 1<<20,
})
if err != nil { return err }
application.OnShutdown(client.Close)
err = client.JSON(ctx, "GET", serviceURL, nil, &result)
err = client.JSON(ctx, "GET", serviceURL, nil, &result,
httpc.WithHeader("Authorization", "Bearer "+token),
)
```
客户端应在服务启动时创建并复用。默认使用独立的 `net/http.Transport` 连接池,每个目标主机
有连接上限;该上限不是整个进程的全局连接数上限。也可以注入自有 `http.RoundTripper`,
其连接限制和生命周期由调用方负责。这样出站实现同样可以替换。
启动时创建并复用 Client。代理可选环境、直连、固定地址;不配置时保留标准库原代理策略。JSON 支持单次请求头和响应限额,不保存 Client 全局 Token。新阶段超时与空闲连接参数可选,0 沿用默认。
总超时覆盖建连、重试、等待和读取响应体。`Do` 返回后调用方必须读取/关闭 `response.Body`;
`JSON` 自动关闭,并限制解码前读取的字节数。`Do` 允许流式读取,本身不限制响应体大小。
`Close` 拒绝新请求、关闭自有池的空闲连接;在途请求应先排空。
成功响应读取有界;output=nil 流式丢弃,不整体缓存。小错误体只在大小/时间预算内尝试读完以复用连接;未知或大错误体及时关闭。Do 保留标准流式 Request/Response,由调用者限量读取并关闭 body。Close 拒绝新调用并关闭自有池空闲连接,项目先排空在途工作。
`MaxRetries` 默认 0,最多 5。封装只对可重放的 GET/HEAD/OPTIONS 重试网络错误及
429/502/503/504;POST/PUT/PATCH/DELETE 不做应用层自动重试。标准库 Transport 和借用的
RoundTripper 自身的重放规则仍生效,不能据此承诺“只执行一次”。退避带抖动,尊重
`Retry-After`;服务器要求等待超过 `MaxBackoff` 时直接返回该响应,不提前重试。
重定向交给业务处理,3xx 不会自动跟随。
自定义 Transport 借用、不修改、不关闭;不能同时指定池/代理/阶段选项。总超时持续覆盖 HTTP 等待、重试和读取 body,但不抢占 JSON 编解码或不遵守 context 的自定义代码。MaxRetries 默认 0,只对可重放 GET/HEAD/OPTIONS 重试;3xx 不跟随。标准库和借用 Transport 自身仍可能重放,不能据此承诺业务只执行一次。
默认从 context 传递 `X-Request-ID`,日志只包含方法、主机、状态、尝试次数和收到响应头的耗时,
不记录完整 URL、查询参数、凭证和响应体。非 2xx 的 JSON 调用返回只带状态码的 `StatusError`。
完整参数、NO_PROXY、TLS、安全日志、重试、签名兼容说明见 [httpc 指南](httpc/README.md),可运行 [HTTP Client 示例](examples/httpClient/README.md)。
## 请求保护:middleware

View File

@@ -1,6 +1,6 @@
# 生成项目与文件维护边界
v0.20.0-preview 提供受管理的生成项目。安装同版远程 CLI,生成的项目直接引用该版本,无需本地 replace。根目录入口支持 `go run .` 和 `go build .`;框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。
v0.22.0-preview 提供受管理的生成项目。安装同版远程 CLI,生成的项目直接引用该版本,无需本地 replace。根目录入口支持 `go run .` 和 `go build .`;框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。
## 创建和日常开发
@@ -143,7 +143,7 @@ server:
`responseMode: raw` 返回 `{"status":"ready"}` 等健康数据,有依赖时包含检查名称、结果与耗时;不包装公共体。存活检查不执行依赖探针;服务开始退出后,就绪检查返回 503,存活仍可返回 200,直到监听关闭。OpenAPI 始终输出规范文档本身,不使用业务响应包装;关闭 HTTP 文档端点不影响生成 `docs/openapi.json`。
环境变量覆盖配置文件,例如 `GO_OMNI_ENDPOINTS_READINESS_PATH=/readyz`、`GO_OMNI_ENDPOINTS_READINESS_RESPONSE_MODE=raw`、`GO_OMNI_ENDPOINTS_OPENAPI_ENABLED=false`;liveness/readiness/openapi 都支持对应的 ENABLED 和 PATH。Config/AppOptions 的最终有效配置决定是否注册。
内置路由配置由项目文件明确设置,也可以使用 `${NAME:-fallback}` 引用;不再按字段名自动读取环境变量。Config/AppOptions 的最终有效配置决定是否注册。
关闭或改名后,旧路径不再由框架保留,项目可以在 `.api` 自己声明。启用的内置 GET 路由与业务路由相同或被业务参数/通配路径覆盖时,在创建 App、开始监听之前报错;不会覆盖业务路由。内置接口仍经过全局中间件;项目在鉴权中间件中明确决定这些路径的访问规则,框架不自动绕过鉴权。
@@ -215,7 +215,7 @@ get /download (DownloadReq)
## 已有服务接入步骤
每个独立服务生成一个项目,module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.20.0-preview:
每个独立服务生成一个项目,module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.22.0-preview:
1. 先保留旧服务,整理实际路由及现有响应、鉴权和超时要求。在新目录执行 `go-omni new --api /path/to/service.api --module example.com/service ./service`;不对旧入口强制覆盖。
2. 在 `.api` 中选择 `@app(defaultResponse: ...)`,显式保留原项目的字段、成功码、错误码和 `dataMode`。需要独立回执的路由单独声明 raw/produces。普通路由、404/405 和提前返回的中间件一起验收。

View File

@@ -2,7 +2,7 @@
GoOmni 是面向多个业务项目复用的 Go 后端框架:用 `.api` 定义接口、请求和返回类型,生成项目目录、HTTP 接入代码及 OpenAPI,业务代码负责 Logic 和 Repository。也可以在已有项目中直接使用 `omni.New()`,或单独引入日志、存储等模块。
**从这份 README 开始接入。** 当前版本 **`v0.20.0-preview`**,新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入;文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md),版本变化与验收见 [发布说明](RELEASE.md)。
**从这份 README 开始接入。** 当前版本 **`v0.22.0-preview`**,新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入;文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md),版本变化与验收见 [发布说明](RELEASE.md)。
**v0.19 改动**:新增可选 `copyx` 工具,用普通 Go 调用完成浅拷贝、深拷贝和显式类型转换;深拷贝会拒绝自定义转换中残留的源对象可变引用。**移除了 `mappings` / `mappingFiles` 和映射代码生成**,已有使用者须按 [迁移步骤](COPY.md#从旧-mapping-迁移) 调整,不能直接删除仍在调用的旧文件。
@@ -12,9 +12,9 @@ Go **1.22+**。框架不依赖 Gin,默认使用标准库网络层,提供可
响应字段、业务码、登录方式、权限规则和数据库表结构由接入项目决定。文中的 `code/msg/data`、`200/9999` 都是示例配置,可以替换。
契约支持 `returns([]string/[]int64/[]bool/[]any)`、OpenAPI 公共引用、workspace 一次导出共享 Go 错误目录,以及统一环境变量别名覆盖。详细配置与升级步骤见 [业务开发指南](DEVELOPMENT.md)。
契约支持 `returns([]string/[]int64/[]bool/[]any)`、OpenAPI 公共引用、workspace 一次导出共享 Go 错误目录,以及文件显式环境引用与配置检查。详细配置与升级步骤见 [业务开发指南](DEVELOPMENT.md)。
已有通用开发能力见 [业务开发指南](DEVELOPMENT.md):数组/null、严格请求策略、声明式上传、完整配置环境覆盖、资源生命周期、共享契约和升级工具。MySQL 直接使用 [官方适配器](database/mysql/README.md)。
已有通用开发能力见 [业务开发指南](DEVELOPMENT.md):数组/null、严格请求策略、声明式上传、文件显式配置引用、资源生命周期、共享契约和升级工具。MySQL 直接使用 [官方适配器](database/mysql/README.md)。
后台组件可在 svc 登记 [分阶段关闭](lifecycle/README.md#后台组件分阶段关闭),先停止接收工作并等待收尾,再关闭共享资源;无需绑定队列或 ORM。
@@ -38,13 +38,13 @@ Go **1.22+**。框架不依赖 Gin,默认使用标准库网络层,提供可
| 错误目录与动态提示 | `WithErrors(catalog)` 注册一次;`ByCode(...).WithParams(...).WithCause(err)` | [错误目录](errors/README.md)、[模板参数示例](examples/contracts/errorParams.api) |
| OpenAPI 与错误公共引用 | `.api` 自动生成;手写接口可使用 `openapi` 注册器;Go 错误目录显式导出 | [契约工具](DEVELOPMENT.md)、[注册示例](examples/platform/main.go) |
| 结构化业务日志 | `logx.Info/Error`、`logx.FromContext(ctx)`、`logx.New`;支持轮转、脱敏、慢请求和级别更新 | [下方日志示例](#日志)、[完整配置](logx/README.md) |
| 文件配置与环境覆盖 | 默认值→JSON/YAML/XML→环境→校验;完整业务结构用 `config.LoadProject[T]` / 环境别名 | [统一配置指南](CONFIG.md)、[完整可运行示例](examples/configuration/README.md)、[动态级别](ADVANCED.md#配置与日志热更新) |
| 文件配置与环境引用 | Default()→JSON/YAML/XML 字段引用→类型绑定→校验;无隐式覆盖,支持 `-check-config` | [统一配置指南](CONFIG.md)、[完整可运行示例](examples/configuration/README.md)、[动态级别](ADVANCED.md#配置与日志热更新) |
| 生命周期与健康检查 | svc 登记 OnClose/OnStop/OnDrain,RequestShutdown 通知故障;健康路径/开关可配 | [生命周期用法](lifecycle/README.md)、[健康检查](health/README.md)、[生成项目扩展](PROJECT.md) |
| 数据库、连接池与事务 | `database.Open` / `OpenConnector`;事务与检查由项目显式调用 | [通用数据库](database/README.md)、[MySQL TLS/Logger](database/mysql/README.md) |
| 可选 GORM 与迁移 | gormx 复用已有连接池;`go-omni migrate` 预览,显式 apply | [GORM 示例](database/gormx/README.md)、[迁移用法](ADVANCED.md#迁移审计与分页) |
| 认证与权限接口 | 注入 Authenticator / Policy;JWT 或 Session 只是可选实现 | [认证说明](auth/README.md)、[鉴权完整例子](examples/platform/README.md) |
| 缓存 | `cache.NewMemory` 或 Redis 实现;业务面向 Store 的 Get/Set/Delete | [缓存配置/代码](MODULES.md)、[可运行模块服务](examples/modules/README.md) |
| 出站 HTTP、熔断 | `httpc.New` 复用连接池与截止时间;`resilience.New` 隔离故障依赖 | [HTTP 用法](MODULES.md)、[熔断/观测](ADVANCED.md#熔断与观测) |
| 出站 HTTP、熔断 | `httpc.New` 配置代理、连接池和阶段超时;JSON 支持单次请求头/限额;`resilience.New` 隔离故障 | [HTTP 完整用法](httpc/README.md)、[可运行示例](examples/httpClient/README.md)、[熔断/观测](ADVANCED.md#熔断与观测) |
| 请求追踪、访问日志、恢复、限流、幂等、超时 | `app.Use(...)` 按需组合 middleware;跨副本选择 Redis 实现 | [模块说明与限制](MODULES.md#请求追踪与基础指标)、[分布式实现](ADVANCED.md#分布式限流与幂等) |
| 本地后台任务与调度 | `task.New`、Submit、Every;有界队列、排空关闭 | [task 示例与生命周期](MODULES.md#后台执行task) |
| 持久队列 | 业务使用 Publisher / Consumer;Redis Streams 提供至少一次交付 | [队列初始化/消费示例](ADVANCED.md#可靠队列) |
@@ -54,28 +54,57 @@ Go **1.22+**。框架不依赖 Gin,默认使用标准库网络层,提供可
| 可选结构转换与深拷贝 | `copyx.Copy` / `Clone` / `New`;Go converter 处理差异 | [下方示例](#可选结构转换与深拷贝)、[完整 copyx 指南](COPY.md)、[可运行代码](examples/copying) |
| 自定义适配器验收 | `transporttest.Run`、`storagetest.Run` / `RunUploads` 验证扩展契约 | [网络契约测试入口](transport/transporttest/router.go)、[存储契约测试入口](storage/storagetest/contract.go) |
## 统一配置加载
## 带认证的出站 HTTP
新生成项目的加载流程由框架管理。业务在 `internal/config/config.go` 增加字段、默认值和 `Validate`,无需改 main 或复制环境变量解析。JSON、YAML、XML 共用字段和校验;扩展格式在 `loader.go` 注册。
HTTP Client 在 svc 初始化一次,业务重复使用。普通内部请求可直接选择直连和连接参数,无需手写 Transport:
```go
// 项目 Config 中增加字段,自动支持 GO_OMNI_BATCH_SIZE:
BatchSize int `json:"batchSize" envDefault:"100"`
// 项目 Validate 中增加业务范围检查:
if c.BatchSize < 1 {
return omniConfig.Invalid("batchSize", "must be positive")
}
client, err := httpc.New(httpc.Config{
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
Timeout: 3*time.Second, DialTimeout: time.Second,
TLSHandshakeTimeout: time.Second, ResponseHeaderTimeout: time.Second,
MaxConnsPerHost: 32, MaxIdleConnsPerHost: 16, MaxIdleConns: 64,
})
if err != nil { return err }
// 在 svc 生命周期登记 client.Close;关闭前先排空使用它的工作。
err = client.JSON(ctx, "GET", endpoint, nil, &result,
httpc.WithHeader("Authorization", "Bearer "+token),
httpc.WithMaxResponseBytes(64<<10),
)
```
错误包含文件、字段、可用的行列或环境变量名;输入原值不自动写入错误。
请求头只作用于本次调用,不把用户 Token 留在共享 Client。默认代理/参数保持原行为;也可选择环境代理或固定代理。JSON 自动关闭并限制响应,Do 保留下载/流式读取扩展。完整配置、重试边界和已有自定义 Transport 的迁移见 [httpc](httpc/README.md)。
```sh
go run ./examples/httpClient
```
## 统一配置加载
业务只维护 Config、Default()、Validate() 和配置文件。生成器维护 Load/main;文件缺失字段保留默认值,显式 0/false/空字符串正常覆盖,固定值不会被同名环境变量再次覆盖。
```go
BatchSize int `json:"batchSize"` // 加入 Config;在 Default() 中设置 BatchSize: 100。
```
```yaml
server:
address: "${HTTP_ADDR:-127.0.0.1:8080}"
batchSize: "${BATCH_SIZE:-100}"
password: "${DB_PASSWORD}"
sessionToken: "${SESSION_TOKEN:-}"
```
`${NAME}` 未设置或为空就报错;`${NAME:-value}` 明确指定备用值。`$$` 转义 `$`;密码中的冒号、换行、引号和 `${...}` 原样插入,不二次解释。JSON/YAML/XML 共用解析后的字段处理和 Go 类型绑定,错误同时定位文件、字段、变量及可用行列,不打印原值。
```sh
go run ./examples/configuration -format xml
DEMO_CLIENT_TIMEOUT=500ms go run ./examples/configuration
# 在生成项目中,仅检查选中的文件,不初始化依赖:
go run . -config etc/config.test.yaml -check-config
```
详见 [统一加载、解析器、迁移](CONFIG.md) 和 [完整示例](examples/configuration/README.md)。**旧项目须先迁移手写 Load,再重新生成**;生成器会保留业务配置并拒绝冲突,不静默覆盖。
完整 [语义与迁移](CONFIG.md)、[可运行示例](examples/configuration/README.md)。**v0.22 移除旧自动环境覆盖、前缀/别名及 env/envDefault 标签**,生成项目先迁移开发者配置再重新生成;文件中的响应模板 `${code}` 写成 `$${code}`,`.api` 保持原样。
## 选择接入方式
@@ -97,7 +126,7 @@ module 地址是 `git.yuntuo.hk/k2safe/go-omni`。先确保本机 Go 和 Git 可
```sh
export GOPRIVATE="$(go env GOPRIVATE),git.yuntuo.hk"
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.20.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview
export PATH="$(go env GOPATH)/bin:$PATH"
go-omni --help
```
@@ -254,7 +283,7 @@ go-omni new --api catalog.api \
在业务项目的 **go.mod 所在目录**执行:
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.20.0-preview
go get git.yuntuo.hk/k2safe/go-omni@v0.22.0-preview
```
已有项目可直接复用现有目录、Service、Repository 和数据库连接,先选择一个接口接入。也可以只用 `logx`、`storage` 等包,暂时保留原来的 HTTP 框架。希望入口和 HTTP 接入都交给生成器的项目,按 [已有服务接入步骤](PROJECT.md#已有服务接入步骤) 在新目录生成后迁入业务代码。
@@ -498,20 +527,22 @@ API 项目的数据库迁移需要单独维护并显式执行;启动不会自
## 配置与运行
生成项目从 `etc/config.yaml` 读取配置,优先级为 **默认值 → 配置文件 → 环境变量**。密钥通过环境变量提供。手写入口需显式调用配置加载函数并传给 `omni.New(omni.WithConfig(settings))`;单独调用 `omni.New()` 不会自动读取环境变量。
生成项目使用 **Default() → 选定文件的显式引用 → 类型绑定 → Validate()**。固定值不会被环境变量覆盖;配置规则、转义及迁移见 [配置指南](CONFIG.md)。手写入口显式加载配置,再传入 `omni.New(omni.WithConfig(settings))`。
| 环境变量 | 用途 |
| 文件字段 | 用途 |
| --- | --- |
| `GO_OMNI_HTTP_ADDR` | 监听地址;生成项目默认 `127.0.0.1:8080` |
| `GO_OMNI_TRANSPORT` | `nethttp`(默认)、`native`、`http2` |
| `GO_OMNI_TLS_CERT_FILE` / `GO_OMNI_TLS_KEY_FILE` | 成对设置 TLS 证书和私钥路径 |
| `GO_OMNI_HTTP_READ_HEADER_TIMEOUT` / `GO_OMNI_HTTP_READ_TIMEOUT` | 请求头 / 请求读取超时,如 `5s` / `30s` |
| `GO_OMNI_HTTP_WRITE_TIMEOUT` / `GO_OMNI_HTTP_IDLE_TIMEOUT` | 响应写入 / Keep-Alive 空闲超时 |
| `GO_OMNI_HTTP_MAX_HEADER_BYTES` | 请求头大小限制,字节数 |
| `GO_OMNI_SHUTDOWN_TIMEOUT` | 优雅退出等待时间,默认 `10s` |
| `GO_OMNI_ENABLE_LOGGER` / `GO_OMNI_LOG_LEVEL` | 访问日志开关、日志级别 |
| `GO_OMNI_TRUSTED_PROXIES` | 可解析转发地址头的可信代理 IP/CIDR,逗号分隔 |
| `GO_OMNI_DB_HOST/USER/PASSWORD/DATABASE` | 官方 MySQL 适配器的结构化连接配置 |
| `server.address` | 监听地址,生成项目默认 127.0.0.1:8080 |
| `server.transport` | nethttp(默认)、native、http2 |
| `server.tls.certFile/keyFile` | 成对配置 TLS 证书和私钥路径 |
| `server.http.readHeaderTimeout/readTimeout` | 请求头 / 请求读取超时 |
| `server.http.writeTimeout/idleTimeout` | 响应写入 / Keep-Alive 空闲超时 |
| `server.http.maxHeaderBytes` | 请求头大小限制,字节数 |
| `server.shutdownTimeout` | 优雅退出等待时间,默认 10s |
| `server.enableLogger` / `server.log.level` | 访问日志开关、日志级别 |
| `server.trustedProxies` | 可信代理 IP/CIDR,文件数组 |
| `database.host/user/password/database` | 官方 MySQL 结构化连接配置 |
初始生成文件已为地址、网络层和选中模块的必要配置声明环境引用;其他字段按需要添加固定值或引用。
例如在生成项目中切换 native:
@@ -556,7 +587,7 @@ go test ./...
`main.go`、`Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理,手动修改会报告冲突。Logic、Repository、中间件实现、配置和依赖装配属于项目,重复生成保留手写内容。契约变化会提示受影响的 Logic 文件和 DTO 字段,明确的签名错误会给出原签名与期望签名;生成后 `--check` 仍能检出未修复的问题。将 `.api`、`omni.gen.json`、生成文件和 `.goOmni/apiManifest.json` 一起提交,CI 同时运行 `generate --check` 与 `go test ./...`。详见 [契约变更提示与检查边界](PROJECT.md#契约变更提示)。
CLI 与业务 module 固定使用 `v0.20.0-preview`。v0.14–v0.19 项目先按 [配置入口迁移](CONFIG.md#v019-及更早项目迁移) 整理旧 Load,再升级依赖与 CLI 并重新生成,框架维护的 main/runtimeGen 自动更新;v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理,不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移),旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。
CLI 与业务 module 固定使用 `v0.22.0-preview`。v0.21 及更早项目先按 [配置迁移](CONFIG.md#v022-迁移旧模式已移除) 整理旧标签、文件与 Load,再升级依赖与 CLI 并重新生成,框架维护的 main/runtimeGen 自动更新;v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理,不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移),旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。
## 按需深入

View File

@@ -1,4 +1,50 @@
# v0.20.0-preview
# v0.22.0-preview
配置文件明确决定每个值的来源,项目只维护 Config、Default()、Validate() 和文件。
- `${NAME}` 必填,`${NAME:-fallback}` 在未设置/空字符串时使用备用值,`$$` 转义 `$`。支持字符串组合;环境值不二次展开,凭据空格和特殊字符原样保留。
- 在 JSON/YAML/XML 已解析的字段值上展开,再按目标 Go 类型绑定;缺失字段保留默认值,0/false/空字符串可显式覆盖。字段错误定位文件、环境名和可用行列,不回显值。
- 生成 Load/main 自动接入;新增 `-check-config`,选定一个文件检查,成功退出,不启动日志/依赖/监听。精简生成配置,重新生成保留业务 Config、Default、Validate 和文件。
- **不兼容变更:旧自动环境覆盖、前缀/别名、env/envDefault、database.ConfigFromEnv/ApplyEnv 已移除。** LoadFile/NewReloader 不再接收 prefix,omni.LoadConfig 参数改为文件路径。发现旧标签提示迁移。
- 配置文件中的响应模板用 `$${code}` 等转义;`.api` 中的 `${code}` 不变。日志 file/output/rotation 均由显式文件配置决定,不再隐式清除。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.22.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview
# 按 CONFIG.md 迁移开发者配置后:
go-omni api generate
go-omni api generate --check
go run . -config etc/config.yaml -check-config
```
[完整规则与迁移](CONFIG.md)、[可运行示例](examples/configuration/README.md)、[场景与性能验收](reports/explicitConfigurationAcceptance.md) 已同步。48 个测试包全量 race、实际生成消费者、Go 1.22、vet/build 和约 28 万次引用 fuzz 通过;小配置固定值/带引用加载约 25 µs,属于本机启动阶段基准。当前仍为 preview。
---
# v0.21.0-preview(历史版本)
让带认证的 HTTP 调用和内部直连可以直接使用 httpc,减少手写 Transport 与 JSON 响应处理。
- Proxy 支持环境、直连、固定代理;默认保留既有策略。
- 建连、TCP keepalive、TLS 握手、响应头、100-continue、空闲超时和独立空闲连接数量可选配置;不修改全局 Transport,不关闭借用实例。
- JSON 支持 WithHeader / WithHeaders / WithMaxResponseBytes;认证信息逐请求隔离,自动关闭和有界响应处理保留。
- 成功路径减少请求复制和无用退避;output=nil 按限额流式丢弃;小错误体在大小和时间预算内尝试复用连接。
- 不新增依赖、不增加业务规则;Do 保留标准流式扩展。CLI 最新版本与模板最低版本分开,本轮模板兼容门槛仍是 v0.20。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.21.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.21.0-preview
```
**兼容注意**:JSON 旧的直接调用仍有效,自定义 interface/函数值需要同步新增的 variadic 参数。自定义 Transport 与显式连接/代理/阶段参数混用现在报错,避免参数被忽略;迁移时在自己的 Transport 配置,或改用内置字段。没有删除原调用入口。
[参数与使用说明](httpc/README.md)、[可运行示例](examples/httpClient/README.md)、[场景与性能验收](reports/httpcAcceptance.md) 已同步到 README。48 个测试包全量 race、Go 1.22 兼容、vet/build、约 110 万次请求头 fuzz 通过。性能来自本机同条件基准,不能代替项目实际流量验证;仍为 preview。
空缓存远程安装、下载模块的 httpc race 与示例、8 组生成消费者验收通过,无本地 replace。标签固定在 `96e7d9629b5c8b78e9cfefbcb05da0ab702c242b`,见 [远程记录](reports/httpcRemote.json)。
---
# v0.20.0-preview(历史版本)
统一配置加载,减少生成项目重复维护的启动代码。
@@ -18,7 +64,7 @@ go test ./...
go build .
```
完整用法、格式、脱敏边界和迁移见 [统一配置指南](CONFIG.md),可运行例子见 [examples/configuration](examples/configuration/README.md)。回归、模糊测试与启动加载性能见 [验收记录](reports/configurationAcceptance.md)。仍为 preview;项目的实际配置和业务依赖应自行验收。
完整用法、格式、脱敏边界和迁移见 [统一配置指南](CONFIG.md),可运行例子见 [examples/configuration](examples/configuration/README.md)。回归、模糊测试与启动加载性能见 [验收记录](reports/configurationAcceptance.md)。47 个测试包的全量 race、Go 1.22 兼容、vet/build、30 秒 fuzz 与 5 个基准场景三轮测量通过。空缓存远程安装和 8 组生成消费者回归通过,无本地 replace;标签固定在 `c60841f901a3bcc3e9582eece6cfb2cad008fb9e`,见 [远程记录](reports/configurationRemote.json)。仍为 preview;项目的实际配置和业务依赖应自行验收。
---
@@ -116,7 +162,7 @@ go-omni api generate --check
go test ./...
```
旧项目 config/Logic/svc 继续由开发者维护,不会被重新生成覆盖。已有环境配置需按 [一次覆盖完整配置](DEVELOPMENT.md#一次覆盖完整配置与-http日志短名称) 合并 OverlayEnv 与 ApplyEnv;共享错误目录接入见 [workspace 配置](DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。已有手写权限 Handler 切换自动生成前,先迁出实现并显式移除 Manual 文件,生成器不会静默删除。
旧项目 config/Logic/svc 继续由开发者维护,不会被重新生成覆盖。已有环境配置需按 本版当时的统一覆盖方式合并 OverlayEnv 与 ApplyEnv(该模式已在 v0.22 移除,当前升级见 [配置迁移](CONFIG.md#v022-迁移旧模式已移除));共享错误目录接入见 [workspace 配置](DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。已有手写权限 Handler 切换自动生成前,先迁出实现并显式移除 Manual 文件,生成器不会静默删除。
全量 race(含实际生成消费者)、vet、build、根入口/退出/再生成验证通过。eTax 只读契约验收:API 文档 20,751,149 → 847,171 字节,ID 文档 1,008,210 → 392,481 字节;70 个操作展开引用后与原文档完全一致。空缓存远程 CLI / 固定版本消费者验收也已通过,没有本地 replace;标签固定在 `64e3bd6cfced225476d58e95037249cd0a4fe695`。详见 [验收记录](reports/contractFollowupAcceptance.md)。这不代表已完成业务项目迁移。

View File

@@ -14,7 +14,7 @@ v0.14 新增 `api format [--check]`、`api upgrade [--dry-run|--preview <空目
```sh
# 仓库读取权限与 PATH 配置见根目录 README
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.20.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview
go-omni new --module example.com/hello ./hello
# 从自己的契约生成;service.api 及其相对 import 文件需要在本地存在
@@ -49,14 +49,14 @@ main、Gen.go、OpenAPI 和 manifest 由生成器维护;业务文件按字节
## 版本与迁移
CLI 与框架依赖固定使用 v0.20.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod,不添加本地 replace;`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换:
CLI 与框架依赖固定使用 v0.22.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod,不添加本地 replace;`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换:
```sh
# 在 go-omni 源码仓库中执行
go run ./cmd/go-omni new --module example.com/hello --framework-dir . ../hello
```
当前模板与依赖以 v0.20.0-preview 为基线,继续使用 manifest v2;GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级):原始模板可自动替换,含业务修改的基础文件按预览合并;通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务,不使用旧 `add` 命令的手动路由装配。
当前模板与依赖以 v0.22.0-preview 为基线,继续使用 manifest v2;GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级):原始模板可自动替换,含业务修改的基础文件按预览合并;通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务,不使用旧 `add` 命令的手动路由装配。
## 数据库迁移命令
@@ -70,6 +70,8 @@ go-omni migrate --dir migrations --apply
SQL 命名示例 `0001_createUsers.up.sql`;预览只显示版本和 SHA-256,不连接数据库。失败保留 dirty 状态,不自动 down/force。细节见 [扩展模块](../../ADVANCED.md)。
### v0.20 配置入口
### v0.22 文件显式配置
新生成 `internal/config/loadGen.go` 调用统一加载,纳入生成检查;项目 config.go 只维护字段、默认值和校验,loader.go 用于注册额外格式。JSON/YAML/XML 内置。旧手写 Load 会阻止生成,请先按 [迁移步骤](../../CONFIG.md#v019-及更早项目迁移) 整理,不会自动覆盖业务代码。
新生成 `internal/config/loadGen.go` 调用统一加载,纳入生成检查;项目 config.go 只维护字段、默认值和校验,loader.go 用于注册额外格式。JSON/YAML/XML 内置。旧手写 Load 会阻止生成,请先按 [迁移步骤](../../CONFIG.md#v022-迁移旧模式已移除) 整理,不会自动覆盖业务代码。
`go run . -config etc/config.test.yaml -check-config` 只做配置解析和校验,不初始化日志、数据库或监听。文件中的固定值不受环境自动覆盖;`${NAME}` 为必填引用,`${NAME:-fallback}` 提供备用值。

View File

@@ -26,8 +26,8 @@ func TestGeneratedConfigurationConsumer(t *testing.T) {
// The business only changes fields/defaults/validation, not the loading flow.
writeFile(t, filepath.Join(dir, "internal/config/config.go"), `package config
import("git.yuntuo.hk/k2safe/go-omni/config";"git.yuntuo.hk/k2safe/go-omni/logx")
type Config struct { Server config.Config `+"`json:\"server\"`"+`;Limit int `+"`json:\"limit\" envDefault:\"10\"`"+` }
func Default()Config{ c:=config.Default();c.Log.File="./var/log/app.log";c.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:10};return Config{Server:c} }
type Config struct { Server config.Config `+"`json:\"server\"`"+`;Limit int `+"`json:\"limit\"`"+` }
func Default()Config{ c:=config.Default();c.Log.File="./var/log/app.log";c.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:10};return Config{Server:c,Limit:10} }
func(c Config)Validate()error{if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)};if c.Limit<0{return config.Invalid("limit","must not be negative")};return nil}
`)
writeFile(t, filepath.Join(dir, "internal/config/loader.go"), `package config
@@ -111,3 +111,26 @@ func TestConfigLoadMigrationGuard(t *testing.T) {
}
}
}
func TestRemovedEnvironmentTagsRejectBeforeGeneration(t *testing.T) {
dir := filepath.Join(t.TempDir(), "app")
o := Options{Dir: dir, Module: "example.com/migrate", FrameworkDir: frameworkRoot(t)}
if _, err := New(o); err != nil {
t.Fatal(err)
}
o.API = filepath.Join(dir, "api/service.api")
path := filepath.Join(dir, "internal/config/config.go")
source := readFile(t, path)
writeFile(t, path, strings.Replace(source, "json:\"server\"", "json:\"server\" env:\"OLD_SERVER\"", 1))
before := snapshot(t, dir)
for _, mode := range []string{"dry", "generate", "check"} {
o.DryRun = mode == "dry"
o.Check = mode == "check"
if _, err := Generate(o); err == nil || !strings.Contains(err.Error(), "env/envDefault") {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("migration rejection wrote files")
}
}
}

View File

@@ -19,7 +19,10 @@ import (
)
const frameworkModule = "git.yuntuo.hk/k2safe/go-omni"
const minimumVersion = "v0.20.0-preview"
// Runtime-only releases need not raise the managed template compatibility floor.
const minimumVersion = "v0.22.0-preview"
const currentVersion = "v0.22.0-preview"
const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n"
type Options struct {
@@ -155,7 +158,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
if !filepath.IsAbs(path) {
path = filepath.Join(directory, path)
}
if source, err := os.ReadFile(filepath.Join(path, "config", "loader.go")); err == nil && strings.Contains(string(source), "func NewLoader(") {
if source, err := os.ReadFile(filepath.Join(path, "config", "references.go")); err == nil && strings.Contains(string(source), "func WithLookupEnv(") {
localCurrent = true
}
}
@@ -265,7 +268,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
return Output{}, errors.New("generation configuration changed during generation; retry")
}
out, err := apply(directory, o.Module, entry, generated, seeds, o.Check, o.DryRun, o.profile(), snapshotContract(c.Spec), o.lockHeld)
expectedVersion := minimumVersion
expectedVersion := currentVersion
if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" && semver.IsValid(info.Main.Version) {
expectedVersion = info.Main.Version
}
@@ -304,7 +307,7 @@ func moduleFile(o Options) ([]byte, error) {
if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" {
version = info.Main.Version
} else {
version = minimumVersion
version = currentVersion
}
}
if !semver.IsValid(version) || semver.Major(version) != "v0" && semver.Major(version) != "v1" {

View File

@@ -63,18 +63,23 @@ func TestAPIInfrastructureConsumers(t *testing.T) {
t.Fatal(err)
}
test := `package config
import("testing";"os")
import("testing";"os";"strings")
func TestConfiguration(t *testing.T){
t.Setenv("GO_OMNI_DB_DSN","test:test@tcp(127.0.0.1:1)/test")
t.Setenv("GO_OMNI_DB_DATABASE","test");t.Setenv("GO_OMNI_DB_USER","test");t.Setenv("GO_OMNI_DB_PASSWORD","test")
t.Setenv("GO_OMNI_STORAGE_SIGNING_KEY","01234567890123456789012345678901")
t.Setenv("GO_OMNI_STORAGE_ACCESS_KEY","test");t.Setenv("GO_OMNI_STORAGE_SECRET_KEY","test")
t.Setenv("GO_OMNI_STORAGE_BUCKET","test");t.Setenv("GO_OMNI_STORAGE_ENDPOINT","ENDPOINT")
t.Setenv("GO_OMNI_STORAGE_ACCOUNT_ID","01234567890123456789012345678901")
cfg,err:=Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
if cfg.Server.Log.File==""{t.Fatal("file logger absent")}
t.Setenv("GO_OMNI_LOG_OUTPUT","stdout");cfg,err=Load("");if err!=nil||cfg.Server.Log.File!=""{t.Fatalf("override: %v",err)}
t.Setenv("GO_OMNI_SERVER_LOG_FILE","./explicit.log");cfg,err=Load("");if err!=nil||cfg.Server.Log.File!="./explicit.log"{t.Fatalf("canonical log file: %v",err)}
t.Setenv("GO_OMNI_SERVER_LOG_OUTPUT","stderr");cfg,err=Load("");if err!=nil||cfg.Server.Log.Output!="stderr"{t.Fatalf("canonical output: %v",err)}
t.Setenv("GO_OMNI_LOG_OUTPUT","stdout");t.Setenv("GO_OMNI_SERVER_LOG_FILE","./implicit.log")
cfg,err=Load("../../etc/config.yaml");if err!=nil||cfg.Server.Log.File!=Default().Server.Log.File{t.Fatalf("implicit environment override: %v",err)}
raw,err:=os.ReadFile("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
// Add explicit fields while keeping storage and DB requirements.
raw=[]byte(strings.Replace(string(raw),"server:\n","server:\n log:\n output: '${OUTPUT:-stdout}'\n file: ''\n rotation:\n enabled: false\n",1))
path:="explicit.yaml";if err=os.WriteFile(path,raw,0600);err!=nil{t.Fatal(err)};defer os.Remove(path)
cfg,err=Load(path);if err!=nil||cfg.Server.Log.Output!="stdout"||cfg.Server.Log.File!=""||cfg.Server.Log.Rotation.Enabled{t.Fatalf("explicit file override: %v",err)}
_=os.Getenv("unused")
}`
endpoint := ""

View File

@@ -19,8 +19,10 @@ import (
"MODULE/internal/svc"
)
func main(){
path:=flag.String("config","etc/config.yaml","configuration file; empty uses defaults and environment")
path:=flag.String("config","etc/config.yaml","configuration file; empty uses only Default()")
check:=flag.Bool("check-config",false,"validate configuration and exit without starting dependencies")
flag.Parse()
if *check {if _,err:=config.Load(*path);err!=nil{log.Fatal(err)};return}
ctx,stop:=signal.NotifyContext(context.Background(),os.Interrupt,syscall.SIGTERM)
defer stop()
if err:=run(ctx,*path);err!=nil{log.Fatal(err)}
@@ -58,7 +60,7 @@ Unimplemented/nil factories fail startup and clean registered dependencies.
Existing projects retain middleware: handlers until explicitly migrated.
Ordinary Logic uses the generated Request/Response types directly. Where an
independent business type is needed, use ordinary Go or the optional copyx package.
Configuration loading is generator-managed in internal/config/loadGen.go. Extend fields/defaults/Validate in config.go; additional decoders belong in loader.go/NewLoader. JSON/YAML/XML use one pipeline. See the framework CONFIG.md.
Configuration loading is generator-managed in internal/config/loadGen.go. Extend fields/defaults/Validate in config.go; additional decoders belong in loader.go/NewLoader. JSON/YAML/XML use explicit ${NAME} / ${NAME:-fallback} references; fixed values are never overridden by inferred environment names. Use -check-config to validate without opening dependencies. See the framework CONFIG.md.
There are no mappings/mappingFiles settings or generated mapping files in v0.19.
See COPY.md in the framework repository for copy policies and migration steps,
@@ -77,7 +79,7 @@ Configure server.endpoints to disable or move each built-in route. Health
responseMode is app by default (project format), or raw for plain health data.
Probe HTTP statuses remain 200/503 even when business errors use HTTP 200.
OpenAPI remains an unwrapped document; disabled endpoints reserve no paths.
Set GO_OMNI_TRANSPORT=native to select the native transport without changing services.
The default file explicitly references GO_OMNI_TRANSPORT; set it to native or edit server.transport.
The greeting starter works immediately; other new operation stubs return 501 until
their business logic is implemented. Authentication hooks also fail closed with 501.
@@ -139,7 +141,7 @@ Full reference: https://git.yuntuo.hk/k2safe/go-omni/src/branch/main/DEVELOPMENT
api/ # editable contract and imported types
main.go # GENERATED startup, lifecycle and route registration
etc/config.yaml # lowerCamelCase configuration
internal/config/ # defaults -> file -> environment
internal/config/ # defaults -> file references -> typed binding -> validation
internal/svc/ # dependency initialization and cleanup
internal/types/*Gen.go # DTOs split by source API, one types package
internal/svc/runtimeGen.go # GENERATED lifecycle and infrastructure startup
@@ -240,13 +242,16 @@ func infrastructureREADME(o Options) string {
Selected database: %s; storage: %s; logging: %s/%s.
MySQL uses database/sql through deps.DB, including InTx and a readiness probe.
Use structured database/mysql configuration and GO_OMNI_DB_PASSWORD. Other connection/pool/TLS settings live in etc/config.yaml and GO_OMNI_DB_*.
Use structured database/mysql configuration. The generated etc/config.yaml explicitly
references GO_OMNI_DB_PASSWORD and other required connection values; add fixed
values or explicit references for pool/TLS fields. No implicit environment overlay.
No tables are invented or migrated automatically from request/response DTOs.
The none/memory selection opens no database; implement repositories per business.
With storage enabled, services receive deps.Files, Uploads, Downloads and
UploadLinks capability interfaces. Local signed GET/HEAD/PUT uses /storage/object;
set GO_OMNI_STORAGE_SIGNING_KEY (32+ bytes) and GO_OMNI_STORAGE_PUBLIC_URL.
the generated file references GO_OMNI_STORAGE_SIGNING_KEY (32+ bytes) and
GO_OMNI_STORAGE_PUBLIC_URL. File values may instead be fixed explicitly.
Cloud credentials use GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY; set bucket and endpoint
or R2 accountId as appropriate. Signed URLs grant temporary access: authenticate
and authorize before issuing them. No public upload-link route is added.

View File

@@ -6,28 +6,26 @@ import (
"example.com/configuration/internal/config"
omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
func TestGeneratedConfiguration(t *testing.T) {
for ext, source := range map[string]string{
"json": `{"limit":7}`, "yaml": "limit: 7\n", "xml": "<config><limit>7</limit></config>", "custom": `{"limit":7}`,
} {
for ext, source := range map[string]string{"json": `{"limit":"${LIMIT:-7}"}`, "yaml": "limit: '${LIMIT:-7}'\n", "xml": "<config><limit>${LIMIT:-7}</limit></config>", "custom": `{"limit":"${LIMIT:-7}"}`} {
t.Run(ext, func(t *testing.T) {
t.Setenv("LIMIT", "")
t.Setenv("GO_OMNI_LIMIT", "999")
path := filepath.Join(t.TempDir(), "app."+ext)
if err := os.WriteFile(path, []byte(source), 0600); err != nil {
t.Fatal(err)
}
cfg, err := config.Load(path)
if err != nil {
t.Fatal(err)
if err != nil || cfg.Limit != 7 {
t.Fatal(cfg.Limit, err)
}
if cfg.Limit != 7 {
t.Fatal(cfg.Limit)
}
t.Setenv("GO_OMNI_LIMIT", "0")
t.Setenv("LIMIT", "0")
cfg, err = config.Load(path)
if err != nil || cfg.Limit != 0 {
t.Fatal(cfg.Limit, err)
@@ -36,24 +34,60 @@ func TestGeneratedConfiguration(t *testing.T) {
}
t.Setenv("GO_OMNI_LOG_OUTPUT", "stderr")
cfg, err := config.Load("")
if err != nil {
t.Fatal(err)
}
if cfg.Limit != 10 || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled {
t.Fatal("defaults/log normalization", cfg, err)
if err != nil || cfg.Limit != 10 || cfg.Server.Log.File != "./var/log/app.log" || !cfg.Server.Log.Rotation.Enabled {
t.Fatal("implicit overlay", err)
}
path := filepath.Join(t.TempDir(), "bad.yaml")
os.WriteFile(path, []byte("limit: -1\n"), 0600)
// Validation must run before svc.Initialize; the project initializer panics
// below if reached, so this also detects accidentally starting dependencies.
os.WriteFile(path, []byte("limit: '${LIMIT}'\n"), 0600)
t.Setenv("LIMIT", "-1")
// svc.Init panics if reached. Both invalid startup and successful check must
// stop before opening the configured file logger or project dependencies.
err = run(context.Background(), path)
var diagnostic *omniConfig.Error
if !errors.As(err, &diagnostic) || diagnostic.Field != "limit" || diagnostic.File != path || diagnostic.Line != 1 {
var d *omniConfig.Error
if !errors.As(err, &d) || d.Field != "limit" || d.File != path || d.Line != 1 || d.Env != "LIMIT" {
t.Fatal(err)
}
t.Setenv("GO_OMNI_LIMIT", "private-not-a-number")
t.Setenv("LIMIT", "private-invalid")
_, err = config.Load(path)
if !errors.As(err, &diagnostic) || diagnostic.Env != "GO_OMNI_LIMIT" || strings.Contains(err.Error(), "private-not-a-number") {
if !errors.As(err, &d) || d.Env != "LIMIT" || strings.Contains(err.Error(), "private-invalid") {
t.Fatal(err)
}
}
func TestConfigurationCheckProcess(t *testing.T) {
exe := filepath.Join(t.TempDir(), "server")
cmd := exec.Command("go", "build", "-o", exe, ".")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(err, string(out))
}
dir := t.TempDir()
path := filepath.Join(dir, "selected.yaml")
if err := os.WriteFile(path, []byte("limit: '${LIMIT:-0}'\nserver:\n address: 127.0.0.1:1\n"), 0600); err != nil {
t.Fatal(err)
}
// A broken neighboring default/local/test file must never be layered in.
for _, name := range []string{"config.yaml", "config.local.yaml", "config.test.yaml"} {
os.WriteFile(filepath.Join(dir, name), []byte("broken: ["), 0600)
}
cmd = exec.Command(exe, "-config", path, "-check-config")
cmd.Dir = dir
if out, err := cmd.CombinedOutput(); err != nil || len(out) != 0 {
t.Fatal(err, string(out))
}
if _, err := os.Stat(filepath.Join(dir, "var")); !os.IsNotExist(err) {
t.Fatal("check opened logger/dependencies")
}
os.WriteFile(path, []byte("limit: '${MISSING_REQUIRED}'"), 0600)
cmd = exec.Command(exe, "-config", path, "-check-config")
cmd.Dir = dir
cmd.Env = append(os.Environ(), "MISSING_REQUIRED=")
out, err := cmd.CombinedOutput()
if err == nil || !strings.Contains(string(out), "MISSING_REQUIRED") || strings.Contains(string(out), "panic") {
t.Fatal(err, string(out))
}
cmd = exec.Command(exe, "-config", path)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "MISSING_REQUIRED=")
if out, err := cmd.CombinedOutput(); err == nil || strings.Contains(string(out), "panic") {
t.Fatal(err, string(out))
}
}

View File

@@ -123,7 +123,7 @@ func TestGeneratedTransfers(t *testing.T) {
func configFile(t *testing.T, address string, fail bool) string {
t.Helper()
path := filepath.Join(t.TempDir(), "config.json")
data, _ := json.Marshal(map[string]any{"server": map[string]any{"address": address}, "failInit": fail})
data, _ := json.Marshal(map[string]any{"server": map[string]any{"address": address, "transport": "${GO_OMNI_TRANSPORT:-nethttp}"}, "failInit": fail})
if err := os.WriteFile(path, data, 0600); err != nil {
t.Fatal(err)
}

View File

@@ -24,7 +24,7 @@ func phasedConfig(t *testing.T, address, mode string, fail bool) string {
if mode == "stuck" {
drainTimeout = "50ms"
}
data, _ := json.Marshal(map[string]any{"backgroundMode": mode, "failInit": fail, "server": map[string]any{"address": address, "lifecycle": map[string]string{"stopTimeout": "1s", "drainTimeout": drainTimeout, "cancelTimeout": "50ms"}}})
data, _ := json.Marshal(map[string]any{"backgroundMode": mode, "failInit": fail, "server": map[string]any{"address": address, "transport": "${GO_OMNI_TRANSPORT:-nethttp}", "lifecycle": map[string]string{"stopTimeout": "1s", "drainTimeout": drainTimeout, "cancelTimeout": "50ms"}}})
if err := os.WriteFile(path, data, 0600); err != nil {
t.Fatal(err)
}

View File

@@ -14,7 +14,9 @@ import (
"golang.org/x/mod/semver"
"os"
"path/filepath"
"reflect"
"sort"
"strconv"
"strings"
)
@@ -54,6 +56,22 @@ func checkConfigLoadExtension(root string) error {
if err != nil {
return fmt.Errorf("parse %s: %w", rel, err)
}
var removed bool
ast.Inspect(f, func(n ast.Node) bool {
field, ok := n.(*ast.Field)
if ok && field.Tag != nil {
if raw, e := strconv.Unquote(field.Tag.Value); e == nil {
tag := reflect.StructTag(raw)
_, a := tag.Lookup("env")
_, b := tag.Lookup("envDefault")
removed = removed || a || b
}
}
return true
})
if removed {
return fmt.Errorf("%s: env/envDefault tags were removed; migrate defaults and explicit file references before generating; see CONFIG.md", rel)
}
for _, decl := range f.Decls {
if fn, ok := decl.(*ast.FuncDecl); ok && fn.Recv == nil && fn.Name.Name == "Load" {
return fmt.Errorf("%s: migrate the project-owned Load function before generating internal/config/loadGen.go; retain Config/Default/Validate and move decoder options to NewLoader; see CONFIG.md", rel)

View File

@@ -39,7 +39,7 @@ func oldInfrastructureProject(t *testing.T) string {
raw, _ := json.Marshal(m)
writeFile(t, filepath.Join(dir, manifestPath), string(raw))
path := filepath.Join(dir, "go.mod")
writeFile(t, path, strings.ReplaceAll(readFile(t, path), minimumVersion, "v0.13.0-preview"))
writeFile(t, path, strings.ReplaceAll(readFile(t, path), currentVersion, "v0.13.0-preview"))
return dir
}
@@ -74,7 +74,7 @@ func TestUpgradeV014PreservesExtensions(t *testing.T) {
raw, _ := json.Marshal(m)
writeFile(t, filepath.Join(dir, manifestPath), string(raw))
mod := filepath.Join(dir, "go.mod")
writeFile(t, mod, strings.ReplaceAll(readFile(t, mod), minimumVersion, "v0.14.1-preview"))
writeFile(t, mod, strings.ReplaceAll(readFile(t, mod), currentVersion, "v0.14.1-preview"))
owned := map[string]string{}
for _, name := range []string{"internal/svc/serviceContext.go", "internal/config/config.go", "etc/config.yaml"} {
path := filepath.Join(dir, name)

View File

@@ -8,7 +8,7 @@ import (
)
func TestNewWithRemoteFrameworkVersion(t *testing.T) {
for _, flag := range [][]string{{"--framework-version", "v0.20.0-preview"}, {"--framework-version=v0.20.0-preview"}} {
for _, flag := range [][]string{{"--framework-version", "v0.22.0-preview"}, {"--framework-version=v0.22.0-preview"}} {
project := filepath.Join(t.TempDir(), "billing")
args := append([]string{"--module", "example.com/billing"}, flag...)
if err := runNew(append(args, project)); err != nil {
@@ -18,7 +18,7 @@ func TestNewWithRemoteFrameworkVersion(t *testing.T) {
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(content), "go-omni v0.20.0-preview") || strings.Contains(string(content), "replace ") {
if !strings.Contains(string(content), "go-omni v0.22.0-preview") || strings.Contains(string(content), "replace ") {
t.Fatalf("CLI did not pin remote dependency: %s", content)
}
}

View File

@@ -66,9 +66,13 @@ func TestGenerateModuleProfiles(t *testing.T) {
t.Fatal("replacement does not resolve", err)
}
cfg, _ := os.ReadFile(filepath.Join(project, "etc/config.yaml"))
if strings.Contains(string(cfg), "secretKey:") || strings.Contains(string(cfg), "filesToken:") {
t.Fatal("secret defaults emitted")
for _, line := range strings.Split(string(cfg), "\n") {
key, value, ok := strings.Cut(strings.TrimSpace(line), ":")
if ok && (key == "secretKey" || key == "filesToken" || key == "signingKey") && !strings.Contains(value, "${GO_OMNI_") {
t.Fatal("secret must be an explicit reference")
}
}
})
}
}
@@ -76,6 +80,7 @@ func TestModuleOptionsRejectBeforeWriting(t *testing.T) {
for _, options := range []Options{
{LogOutput: "invalid"}, {LogFormat: "xml"}, {Storage: "ftp"},
{Storage: "local", FrameworkVersion: "v0.6.0-preview"},
{FrameworkVersion: "v0.21.0-preview"},
{Module: "example.com/x\nrequire malicious v1.0.0"},
} {
options.Dir = filepath.Join(t.TempDir(), "project")

View File

@@ -34,9 +34,13 @@ type Options struct {
}
var files = map[string]string{
"internal/settings/config.go": "templates/modules-settings.go.tmpl",
"internal/settings/config_test.go": "templates/modules-settings_test.go.tmpl",
"etc/config.yaml": "templates/modules-config.yaml.tmpl",
"go.mod": "templates/go.mod.tmpl",
"README.md": "templates/README.md.tmpl",
"cmd/server/main.go": "templates/cmd-server-main.go.tmpl",
"cmd/server/main.go": "templates/modules-main.go.tmpl",
"internal/model/user.go": "templates/internal-model-user.go.tmpl",
"internal/model/errors.go": "templates/internal-model-errors.go.tmpl",
"internal/repository/users.go": "templates/internal-repository-users.go.tmpl",
@@ -109,7 +113,7 @@ func Generate(options Options) error {
if options.FrameworkDir != "" || options.FrameworkVersion == "" {
replace = frameworkReplace(directory, options.FrameworkDir)
}
if modules && replace == "" && semver.Compare(version, modulesFrameworkVersion) < 0 && version != installedFrameworkVersion() {
if replace == "" && semver.Compare(version, modulesFrameworkVersion) < 0 && version != installedFrameworkVersion() {
return fmt.Errorf("module templates require %s or newer; use --framework-dir for unpublished source", modulesFrameworkVersion)
}
data := templateData{

View File

@@ -39,7 +39,7 @@ func TestGenerateCreatesServiceLayout(t *testing.T) {
if err != nil {
t.Fatalf("read generated server: %v", err)
}
for _, expected := range []string{"config.LoadFromEnv(\"\")", "logx.SetDefault(logger)", "omni.WithConfig(settings)", "servicecontext.New(ctx, logger)", "services.Health.ReadinessHandler()", "application.RunContext(ctx, settings.Address)"} {
for _, expected := range []string{"settings.Load(path)", "logx.SetDefault(logger)", "omni.WithConfig(cfg.Server)", "servicecontext.New(ctx, logger, cfg)", "services.Health.ReadinessHandler()", "application.RunContext(ctx, cfg.Server.Address)"} {
if !strings.Contains(string(mainContent), expected) {
t.Fatalf("generated server missing %q: %s", expected, mainContent)
}
@@ -110,7 +110,7 @@ func TestFrameworkReplaceUsesCanonicalPaths(t *testing.T) {
}
func TestGenerateRemoteProjectPinsVersion(t *testing.T) {
for _, version := range []string{"v0.1.0", "v1.2.3-rc.1", "v0.0.0-20260920140624-8b19aa79eea0"} {
for _, version := range []string{"v0.22.0-preview", "v1.2.3-rc.1", "v0.22.1-0.20260926000000-8b19aa79eea0"} {
t.Run(version, func(t *testing.T) {
project := filepath.Join(t.TempDir(), "billing")
if err := Generate(Options{Dir: project, Module: "example.com/billing", FrameworkVersion: version}); err != nil {

View File

@@ -32,46 +32,44 @@ go mod tidy
go run ./cmd/server
```
运行时配置从 `GO_OMNI_*` 环境变量加载,例如:
配置入口是 `internal/settings.Config`、`Default()`、`Validate()` 和 `etc/config.yaml`。
只有文件里的 `${NAME}` / `${NAME:-fallback}` 会读取环境变量,固定值不受进程环境覆盖。生成的文件已声明地址、网络层,以及所选数据库/存储的必要引用;其他参数直接写进文件。
```sh
GO_OMNI_ENABLE_LOGGER=true GO_OMNI_LOG_LEVEL=debug go run ./cmd/server
```yaml
server:
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
transport: "${GO_OMNI_TRANSPORT:-nethttp}"
log:
level: debug
shutdownTimeout: 10s
```
`GO_OMNI_HTTP_ADDR` 可以修改监听地址。生成项目默认提供 `/health/live` 和
`/health/ready`,日志、请求 ID 和优雅退出等基础能力由 GoOmni 统一初始化;业务依赖集中放在项目自己的 `internal/servicecontext`。
服务使用 `signal.NotifyContext` 处理 `SIGINT` / `SIGTERM`。部署时构建并运行服务二进制,
使信号直接到达服务进程:
用 `-config` 选择一个 YAML/JSON/XML 文件;`-config ''` 只使用 Default();`-check-config` 仅加载和校验,不创建依赖或监听:
```sh
go run ./cmd/server -config etc/config.yaml -check-config
go build -o server ./cmd/server
./server
./server -config etc/config.yaml
```
`GO_OMNI_SHUTDOWN_TIMEOUT` 控制等待请求完成的时间,默认 `10s`。
服务通过 `signal.NotifyContext` 处理 `SIGINT` / `SIGTERM`,先排空 HTTP 请求,再关闭依赖和日志。生成项目默认提供 `/health/live` 和 `/health/ready`,业务依赖集中在 `internal/servicecontext`。
使用支持统一网络配置的框架版本时,设置 `GO_OMNI_TRANSPORT=native` 即可切换为自研 HTTP/1.1;
默认 `nethttp`。`GO_OMNI_TLS_CERT_FILE` 和 `GO_OMNI_TLS_KEY_FILE` 配对启用 HTTPS。
实验版 HTTP/2 使用 `GO_OMNI_TRANSPORT=http2`,必须同时配置 TLS 证书和私钥。
业务 Handler / Service / Repository 不需要修改;三种适配器共用上述退出等待时间。
通过文件中的引用设置 `GO_OMNI_TRANSPORT=native` 可切换自研 HTTP/1.1,默认 nethttp。HTTPS 在文件的 `server.tls.certFile` / `server.tls.keyFile` 成对配置;实验 HTTP/2 选择 `http2` 且必须配置 TLS。不会自动读取未在文件声明的 TLS 环境变量。
{{if eq .Database "mysql"}}
`/health/ready` 已注册数据库探针:连接失败返回 `503`,数据库恢复后重新返回 `200`。
`/health/live` 始终只检查服务存活。`GO_OMNI_DB_PING_TIMEOUT` 限制初始化检查和数据库探针的等待时间。
退出时先排空 HTTP 请求,再关闭数据库连接池,最后关闭日志。
`/health/ready` 已注册数据库探针,连接失败返回 503,恢复后返回 200;`/health/live` 只检查进程存活。连接池参数直接写入 `database`,缺失字段保留 Default():
数据库连接池参数:
```text
GO_OMNI_DB_MAX_OPEN_CONNS=10
GO_OMNI_DB_MAX_IDLE_CONNS=5
GO_OMNI_DB_CONN_MAX_LIFETIME=30m
GO_OMNI_DB_CONN_MAX_IDLE_TIME=5m
GO_OMNI_DB_PING_TIMEOUT=5s
```yaml
database:
driver: mysql
dsn: "${GO_OMNI_DB_DSN}"
maxOpenConns: 10
maxIdleConns: 5
connMaxLifetime: 30m
connMaxIdleTime: 5m
pingTimeout: 5s
```
`MAX_IDLE_CONNS=0` 禁用空闲连接,`MAX_OPEN_CONNS=0` 表示不限制连接数。
项目会启用 MySQL `parseTime` 并按 UTC 读取 DATETIME 字段,JSON 字段保持 lowerCamelCase。
`maxIdleConns: 0` 禁用空闲连接;此项目要求 maxOpenConns 为正数。项目启用 MySQL parseTime,并按 UTC 读取 DATETIME;JSON 字段保持 lowerCamelCase。
事务内复用 Repository(在业务代码中导入 `database/sql` 和项目的 `internal/repository`):

View File

@@ -11,8 +11,8 @@ import (
)
type Config struct {
Server config.Config `json:"server"`
{{if eq .Database "mysql"}}Database mysqladapter.Config `json:"database" env:"GO_OMNI_DB"`{{end}}
{{if ne .Storage "none"}}Storage StorageConfig `json:"storage" env:"GO_OMNI_STORAGE"`{{end}}
{{if eq .Database "mysql"}}Database mysqladapter.Config `json:"database"`{{end}}
{{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}}
}
func Default() Config {
server:=config.Default()
@@ -54,12 +54,12 @@ type StorageConfig struct {
ContentTypes []string `json:"contentTypes"`
LinkTTL time.Duration `json:"linkTtl"`
RequestTimeout time.Duration `json:"requestTimeout"`
// Secrets are environment-only and cannot be populated by config files.
FilesToken string `json:"-" env:"GO_OMNI_FILES_TOKEN,secret"`
SigningKey string `json:"-" env:"GO_OMNI_STORAGE_SIGNING_KEY,secret"`
AccessKey string `json:"-" env:"GO_OMNI_STORAGE_ACCESS_KEY,secret"`
SecretKey string `json:"-" env:"GO_OMNI_STORAGE_SECRET_KEY,secret"`
SessionToken string `json:"-" env:"GO_OMNI_STORAGE_SESSION_TOKEN,secret"`
// File values may explicitly reference environment variables. Never log this struct.
FilesToken string `json:"filesToken"`
SigningKey string `json:"signingKey"`
AccessKey string `json:"accessKey"`
SecretKey string `json:"secretKey"`
SessionToken string `json:"sessionToken"`
}
func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}}
func (s StorageConfig) Validate()error {
@@ -67,14 +67,14 @@ func (s StorageConfig) Validate()error {
if err:=s.Policy().Validate();err!=nil{return err}
if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")}
if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")}
{{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("set GO_OMNI_FILES_TOKEN to a token of at least 32 bytes without whitespace")}{{end}}
{{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("storage.filesToken requires at least 32 bytes without whitespace")}{{end}}
{{if eq .Storage "local"}}
if s.Root=="" {return errors.New("storage.root is required")}
u,err:=url.Parse(s.PublicURL)
if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")}
if len(s.SigningKey)<32 {return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")}
if len(s.SigningKey)<32 {return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")}
{{else}}
if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region and GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY are required")}
if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region, accessKey and secretKey are required")}
{{if eq .Storage "r2"}}
if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")}
if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")}

View File

@@ -1,13 +1,8 @@
package config
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
// Load is maintained by the generator. Extend Config/Default/Validate in
// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline.
func Load(path string) (Config, error) {
loader,err:=NewLoader()
if err!=nil{return Config{},err}
return omniConfig.LoadProject(path,Default(),omniConfig.EnvOptions{
Prefix:"GO_OMNI_",Empty:"ignore",Aliases:omniConfig.ServerEnvAliases("GO_OMNI_","server"),
},loader)
// Load resolves explicit file references, binds onto Default and validates.
// Ordinary projects only edit Config, Default, Validate and the selected file.
func Load(path string)(Config,error){
loader,err:=NewLoader();if err!=nil{return Config{},err}
return omniConfig.LoadProject(path,Default(),loader)
}

View File

@@ -1,57 +0,0 @@
package main
import (
"context"
"log"
"os"
"os/signal"
"strings"
"syscall"
omni "git.yuntuo.hk/k2safe/go-omni"
"git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/logx"
"{{.Module}}/internal/handler"
"{{.Module}}/internal/service"
"{{.Module}}/internal/servicecontext"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
settings, err := config.LoadFromEnv("")
if err != nil {
return err
}
if strings.TrimSpace(settings.Log.Service) == "" {
settings.Log.Service = "{{.Module}}"
}
logger, err := logx.New(settings.Log)
if err != nil {
return err
}
defer logger.Close()
// Access logs and business logs share the same configured sink.
logx.SetDefault(logger)
services, err := servicecontext.New(ctx, logger)
if err != nil { return err }
defer services.Close()
userService := service.NewUsers(services)
userHandler := handler.NewUsers(userService)
application := omni.New(omni.WithConfig(settings))
application.OnShutdown(logger.Close)
application.OnShutdown(services.Close)
application.GET("/health/live", services.Health.LivenessHandler())
application.GET("/health/ready", services.Health.ReadinessHandler())
userHandler.RegisterRoutes(application)
return application.RunContext(ctx, settings.Address)
}

View File

@@ -13,6 +13,7 @@ import (
"git.yuntuo.hk/k2safe/go-omni/database"
"git.yuntuo.hk/k2safe/go-omni/logx"
"{{.Module}}/internal/model"
"{{.Module}}/internal/settings"
"{{.Module}}/internal/repository"
"{{.Module}}/internal/servicecontext"
)
@@ -23,15 +24,12 @@ func databaseContext(t *testing.T) *servicecontext.Context {
t.Helper()
dsn := os.Getenv("GO_OMNI_TEST_MYSQL_DSN")
if dsn == "" { t.Skip("set GO_OMNI_TEST_MYSQL_DSN to a dedicated test database") }
for _, key := range []string{"MAX_OPEN_CONNS", "MAX_IDLE_CONNS", "CONN_MAX_LIFETIME", "CONN_MAX_IDLE_TIME", "PING_TIMEOUT"} {
t.Setenv("GO_OMNI_DB_"+key, "")
}
t.Setenv("GO_OMNI_DB_DSN", dsn)
t.Setenv("GO_OMNI_DB_DRIVER", "mysql")
logger, err := logx.New(logx.Config{Writer: io.Discard})
if err != nil { t.Fatal(err) }
t.Cleanup(func() { _ = logger.Close() })
services, err := servicecontext.New(context.Background(), logger)
cfg,err:=settings.Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
services, err := servicecontext.New(context.Background(), logger,cfg)
if err != nil { t.Fatal(err) }
t.Cleanup(func() { _ = services.Close() })
return services

View File

@@ -2,6 +2,7 @@ package servicecontext
import (
"context"
"{{.Module}}/internal/settings"
{{if and .Modules (ne .Storage "none")}}"errors"
"net/http"
"sync"
@@ -38,16 +39,16 @@ type Context struct {
// New initializes the project's dependencies before the HTTP listener opens.
// The caller owns Logger; Close releases only resources created here.
func New(ctx context.Context, logger *logx.Logger) (*Context, error) {
func New(ctx context.Context, logger *logx.Logger, project settings.Config) (*Context, error) {
if ctx == nil { ctx = context.Background() }
if err := ctx.Err(); err != nil { return nil, err }
checker, err := health.New()
if err != nil { return nil, err }
services := &Context{Logger: logger, Health: checker}
{{if eq .Database "mysql"}}
cfg, err := database.ConfigFromEnv("")
if err != nil { return nil, err }
if cfg.Driver != "mysql" { return nil, fmt.Errorf("this project requires GO_OMNI_DB_DRIVER=mysql") }
cfg:=project.Database
if err:=cfg.Validate();err!=nil{return nil,err}
if cfg.Driver != "mysql" { return nil, fmt.Errorf("this project requires database.driver=mysql") }
dsn, err := mysql.ParseDSN(cfg.DSN)
if err != nil { return nil, fmt.Errorf("parse mysql connection settings: %w", err) }
if dsn.DBName == "" { return nil, fmt.Errorf("mysql connection settings require a database name") }

View File

@@ -2,7 +2,7 @@
本项目选用日志输出 `{{.LogOutput}}`、格式 `{{.LogFormat}}`、对象存储 `{{.Storage}}`。数据库选择 `{{.Database}}` 独立于对象存储。
配置入口为 `etc/config.yaml`,通过 `go run ./cmd/server -config /path/to/config.yaml` 指定其他 JSON/YAML 文件;`-config ''` 只用代码默认值和环境变量。优先级:默认值 → 文件 → 非空环境变量。未知字段、无效参数会在监听前拒绝。
配置入口为 `etc/config.yaml`,通过 `go run ./cmd/server -config /path/to/config.yaml` 指定其他 JSON/YAML/XML 文件;`-config ''` 只用代码默认值。文件用 `${NAME}` 或 `${NAME:-fallback}` 显式引用变量,固定值不受隐式环境覆盖;`-check-config` 只检查、不初始化依赖。未知字段、无效参数会在监听前拒绝。
```sh
go mod tidy
@@ -24,7 +24,7 @@ go run ./cmd/server
{{if eq .Database "mysql"}}还需按 README 配置独立 MySQL 数据库并应用 schema.sql;应用不自动迁移。{{end}}
默认仅监听 `127.0.0.1:8080`。`GO_OMNI_HTTP_ADDR` 覆盖地址;本地签名 URL 的外部地址通过 `GO_OMNI_STORAGE_PUBLIC_URL` 单独配置,始终以 `/storage/object` 结尾,不使用请求 Host 自动拼接可信链接。
默认仅监听 `127.0.0.1:8080`。文件中的 `${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}` 设置地址;本地签名 URL 的外部地址通过 `GO_OMNI_STORAGE_PUBLIC_URL` 单独配置,始终以 `/storage/object` 结尾,不使用请求 Host 自动拼接可信链接。
## 日志
@@ -35,7 +35,7 @@ curl -H 'Content-Type: application/json' -H 'X-Request-ID: example-request' \
-d '{"name":"Example","email":"example@example.com"}' http://127.0.0.1:8080/users
```
`GO_OMNI_LOG_LEVEL`、`GO_OMNI_LOG_FORMAT`、`GO_OMNI_LOG_FILE` 覆盖对应项;显式设置 `GO_OMNI_LOG_OUTPUT=stdout` 且未设置 LOG_FILE 时,切换到控制台并关闭文件轮转。慢请求阈值为 500ms,可通过 `GO_OMNI_SLOW_REQUEST_THRESHOLD` 调整。字段脱敏不扫描 msg/error 中的自由文本;不要把凭据或签名 URL 拼到日志中。
日志参数直接写在 `server.log`,例如 `level: debug`;也可显式写 `level: "${LOG_LEVEL:-info}"`。切换文件日志为控制台时,明确设置 `file: ""`、`output: stdout` 和 `rotation.enabled: false`,不会隐式重置其他字段。慢请求阈值使用 `server.slowRequestThreshold`,默认 500ms。字段脱敏不扫描 msg/error 自由文本;不要把凭据或签名 URL 拼到日志中。
{{if ne .Storage "none"}}## 文件示例
@@ -66,18 +66,18 @@ curl -H "Authorization: Bearer $GO_OMNI_FILES_TOKEN" -H 'Content-Type: applicati
响应包括随机 `data.key` 和 `data.request`(method/url/headers/expiresAt)。按返回的 method 和 headers 用原始文件字节上传到 URL;不能额外包 FormData。浏览器自动填写 Content-Length,其余签名 Header 原样发送;云桶需配置允许业务域名的 CORS。云端直传只绑定声明类型,实际内容需上传后验证;本地和 API 中转上传会检查允许类型及前缀嗅探结果。未完成内容验证前不要把暂存文件当作可信文件公开。
| 参数 | 默认值 / 环境变量 |
| 文件字段 | 默认值 / 含义 |
| --- | --- |
| 大小 | 32 MiB / GO_OMNI_STORAGE_MAX_BYTES |
| 类型 | text/plain、image/png、image/jpeg、application/pdf / GO_OMNI_STORAGE_CONTENT_TYPES(逗号分隔) |
| 链接有效期 | 5m / GO_OMNI_STORAGE_LINK_TTL(1s..1h) |
| 请求超时 | 30s / GO_OMNI_STORAGE_REQUEST_TIMEOUT(>0 且 <=5m) |
| 本地目录 | var/files / GO_OMNI_STORAGE_ROOT |
| 本地外部 URL | http://127.0.0.1:8080/storage/object / GO_OMNI_STORAGE_PUBLIC_URL |
| 云配置 | GO_OMNI_STORAGE_ENDPOINT、REGION、BUCKET、ACCOUNT_ID、SECURE |
| 环境密钥 | GO_OMNI_FILES_TOKEN、GO_OMNI_STORAGE_SIGNING_KEY、GO_OMNI_STORAGE_ACCESS_KEY、SECRET_KEY、SESSION_TOKEN |
| `storage.maxBytes` | 32 MiB |
| `storage.contentTypes` | text/plain、image/png、image/jpeg、application/pdf;文件数组 |
| `storage.linkTtl` | 5m,允许 1s..1h |
| `storage.requestTimeout` | 30s,大于 0 且不超过 5m |
| `storage.root` | var/files |
| `storage.publicUrl` | http://127.0.0.1:8080/storage/object |
| `storage.endpoint/region/bucket/accountId/secure` | 对应平台连接配置 |
| `storage.filesToken/signingKey/accessKey/secretKey/sessionToken` | 文件显式引用环境变量提供凭据 |
全部 JSON/YAML 字段使用 lowerCamelCase。密钥字段仅允许环境变量,配置文件中写入会报未知字段;不要把运行环境文件提交到仓库。S3 的 endpoint 为 host[:port],OSS 的可选 endpoint 为 HTTPS origin;R2 从 accountId 派生官方 endpoint,region 为 auto。仅 S3/MinIO 的本地测试可设置 SECURE=false。
全部 JSON/YAML/XML 字段使用 lowerCamelCase。文件可使用固定值或显式环境引用,不按字段名自动推导环境变量;凭据建议使用 `${NAME}`,不把含秘密的配置提交到仓库。S3 endpoint 为 host[:port],OSS 可选 endpoint 为 HTTPS origin;R2 从 accountId 派生官方 endpoint,region 为 auto。仅 S3/MinIO 本地测试可设置 `storage.secure: false`。
错误响应:未授权 401、无 Content-Length 411、超限 413、类型不允许 415、长度不符/非法 key 400、对象不存在 404、存储超时 504,其余内部错误统一隐藏细节。底层签名入口遵循存储协议返回 201/403 等状态,流式传输且访问日志保留实际状态。

View File

@@ -1,68 +1,29 @@
# Non-secret settings only. Environment variables override these values.
# Only fields listed here override Default(). Environment names are explicit.
server:
address: '127.0.0.1:8080'
enableLogger: true
slowRequestThreshold: 500ms
shutdownTimeout: 10s
{{if .API}} lifecycle:
stopTimeout: 5s
drainTimeout: 10s
cancelTimeout: 5s
endpoints:
liveness:
enabled: true
path: /health/live
responseMode: app
readiness:
enabled: true
path: /health/ready
responseMode: app
openapi:
enabled: true
path: /openapi.json
{{end}}
log:
service: {{printf "%q" .Module}}
level: info
format: {{.LogFormat}}
{{if eq .LogOutput "file"}} file: './var/log/server.log'
rotation:
enabled: true
maxSizeMb: 100
maxBackups: 10
maxAgeDays: 7
compress: true
{{else}} output: {{.LogOutput}}
{{end}} redactKeys: []
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
transport: "${GO_OMNI_TRANSPORT:-nethttp}"
{{if ne .Storage "none"}}storage:
provider: {{.Storage}}
maxBytes: 33554432
contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf']
linkTtl: 5m
requestTimeout: 30s
{{if eq .Storage "local"}} root: './var/files'
publicUrl: 'http://127.0.0.1:8080/storage/object'
{{else}} bucket: '' # set GO_OMNI_STORAGE_BUCKET
secure: true
{{if eq .Storage "r2"}} accountId: '' # set GO_OMNI_STORAGE_ACCOUNT_ID
region: auto
{{else if eq .Storage "s3"}} endpoint: '' # set GO_OMNI_STORAGE_ENDPOINT (host[:port])
region: us-east-1
{{else}} region: cn-hangzhou
endpoint: '' # empty uses the official regional endpoint
{{if not .API}} filesToken: "${GO_OMNI_FILES_TOKEN}"
{{end}}{{if eq .Storage "local"}} root: "${GO_OMNI_STORAGE_ROOT:-var/files}"
publicUrl: "${GO_OMNI_STORAGE_PUBLIC_URL:-http://127.0.0.1:8080/storage/object}"
signingKey: "${GO_OMNI_STORAGE_SIGNING_KEY}"
{{else}} bucket: "${GO_OMNI_STORAGE_BUCKET}"
accessKey: "${GO_OMNI_STORAGE_ACCESS_KEY}"
secretKey: "${GO_OMNI_STORAGE_SECRET_KEY}"
sessionToken: "${GO_OMNI_STORAGE_SESSION_TOKEN:-}"
{{if eq .Storage "r2"}} accountId: "${GO_OMNI_STORAGE_ACCOUNT_ID}"
{{else if eq .Storage "s3"}} endpoint: "${GO_OMNI_STORAGE_ENDPOINT}"
region: "${GO_OMNI_STORAGE_REGION:-us-east-1}"
{{else}} region: "${GO_OMNI_STORAGE_REGION:-cn-hangzhou}"
{{end}}{{end}}{{end}}
{{if and .API (eq .Database "mysql")}}database:
host: 127.0.0.1
port: 3306
database: '' # GO_OMNI_DB_DATABASE
user: '' # GO_OMNI_DB_USER; password via GO_OMNI_DB_PASSWORD
tls:
mode: required
pool:
maxOpenConns: 10
maxIdleConns: 5
connMaxLifetime: 30m
connMaxIdleTime: 5m
pingTimeout: 5s
host: "${GO_OMNI_DB_HOST:-127.0.0.1}"
port: "${GO_OMNI_DB_PORT:-3306}"
database: "${GO_OMNI_DB_DATABASE}"
user: "${GO_OMNI_DB_USER}"
password: "${GO_OMNI_DB_PASSWORD}"
{{end}}
{{if and (not .API) (eq .Database "mysql")}}database:
driver: mysql
dsn: "${GO_OMNI_DB_DSN}"
{{end}}

View File

@@ -18,8 +18,10 @@ import (
)
func main(){
path:=flag.String("config","etc/config.yaml","JSON/YAML settings; empty uses defaults and environment")
path:=flag.String("config","etc/config.yaml","configuration file; empty uses only Default()")
check:=flag.Bool("check-config",false,"validate without starting dependencies")
flag.Parse()
if *check {if _,err:=settings.Load(*path);err!=nil{log.Fatal(err)};return}
if err:=run(*path);err!=nil{log.Fatal(err)}
}
func run(path string)(runErr error){
@@ -31,7 +33,7 @@ func run(path string)(runErr error){
if err!=nil{return err}
defer func(){runErr=errors.Join(runErr,logger.Close())}()
logx.SetDefault(logger)
services,err:=servicecontext.New(ctx,logger)
services,err:=servicecontext.New(ctx,logger,cfg)
if err!=nil{return err}
defer func(){runErr=errors.Join(runErr,services.Close())}()
{{if ne .Storage "none"}}if err=services.InitStorage(ctx,cfg.Storage);err!=nil{return err}{{end}}

View File

@@ -3,21 +3,19 @@ package settings
import (
"errors"
"os"
"strings"
"time"
{{if ne .Storage "none"}}"fmt"
{{if ne .Storage "none"}}
{{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}}
"strconv"
"git.yuntuo.hk/k2safe/go-omni/storage"{{end}}
{{if and .API (eq .Database "mysql")}}"git.yuntuo.hk/k2safe/go-omni/database"{{end}}
{{if eq .Database "mysql"}}"git.yuntuo.hk/k2safe/go-omni/database"{{end}}
"git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/logx"
{{if eq .LogOutput "file"}}"git.yuntuo.hk/k2safe/go-omni/logx"{{end}}
)
type Config struct {
Server config.Config `json:"server"`
{{if and .API (eq .Database "mysql")}}Database database.Config `json:"database"`{{end}}
{{if eq .Database "mysql"}}Database database.Config `json:"database"`{{end}}
{{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}}
}
@@ -31,7 +29,8 @@ func Default() Config {
{{if eq .LogOutput "file"}}server.Log.File="./var/log/server.log"
server.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:100,MaxBackups:10,MaxAgeDays:7,Compress:true}
{{else}}server.Log.Output={{printf "%q" .LogOutput}}{{end}}
return Config{Server:server{{if and .API (eq .Database "mysql")}},Database:database.DefaultConfig(){{end}}{{if ne .Storage "none"}},Storage:StorageConfig{
{{if eq .Database "mysql"}}db:=database.DefaultConfig();db.Driver="mysql"{{end}}
return Config{Server:server{{if eq .Database "mysql"}},Database:db{{end}}{{if ne .Storage "none"}},Storage:StorageConfig{
Provider:{{printf "%q" .Storage}}, Root:"./var/files", PublicURL:"http://127.0.0.1:8080/storage/object",
Secure:true, MaxBytes:32<<20, ContentTypes:[]string{"text/plain","image/png","image/jpeg","application/pdf"},
LinkTTL:5*time.Minute, RequestTimeout:30*time.Second,
@@ -41,27 +40,10 @@ func Default() Config {
}{{end}}}
}
// Load uses defaults -> strict YAML/JSON -> nonempty environment variables.
// Pass an empty path to use only defaults and environment.
func Load(path string)(Config,error) {
cfg:=Default()
var err error
if path!="" { cfg,err=config.DecodeFile(path,cfg);if err!=nil{return Config{},err} }
cfg.Server,err=config.ApplyEnv(cfg.Server,"")
if err!=nil{return Config{},err}
// A stream explicitly selected by environment overrides a file profile.
if output:=strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT"));output!="" && strings.TrimSpace(os.Getenv("GO_OMNI_LOG_FILE"))=="" {
cfg.Server.Log.File="";cfg.Server.Log.Rotation=logx.RotationConfig{}
}
{{if ne .Storage "none"}}
if err=cfg.Storage.applyEnv();err!=nil{return Config{},err}
{{end}}
{{if and .API (eq .Database "mysql")}}cfg.Database,err=database.ApplyEnv(cfg.Database,"");if err!=nil{return Config{},err}{{end}}
if err=cfg.Validate();err!=nil{return Config{},err}
return cfg,nil
}
// Load uses Default(), the selected file's explicit references, and Validate.
func Load(path string)(Config,error){return config.LoadProject(path,Default())}
func (c Config) Validate()error {
{{if and .API (eq .Database "mysql")}}
{{if eq .Database "mysql"}}
if err:=c.Database.Validate();err!=nil{return err}
if c.Database.Driver!="mysql" || c.Database.MaxOpenConns<1 {return errors.New("database requires driver mysql and a positive maxOpenConns")}
{{end}}
@@ -85,45 +67,27 @@ type StorageConfig struct {
ContentTypes []string `json:"contentTypes"`
LinkTTL time.Duration `json:"linkTtl"`
RequestTimeout time.Duration `json:"requestTimeout"`
// Secrets are environment-only and cannot be populated by config files.
FilesToken string `json:"-"`
SigningKey string `json:"-"`
AccessKey string `json:"-"`
SecretKey string `json:"-"`
SessionToken string `json:"-"`
// Use explicit environment references in the configuration file for secrets.
FilesToken string `json:"filesToken"`
SigningKey string `json:"signingKey"`
AccessKey string `json:"accessKey"`
SecretKey string `json:"secretKey"`
SessionToken string `json:"sessionToken"`
}
func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}}
func (s *StorageConfig) applyEnv()error {
values:=map[string]*string{
"GO_OMNI_STORAGE_ROOT":&s.Root,"GO_OMNI_STORAGE_PUBLIC_URL":&s.PublicURL,
"GO_OMNI_STORAGE_ENDPOINT":&s.Endpoint,"GO_OMNI_STORAGE_REGION":&s.Region,
"GO_OMNI_STORAGE_BUCKET":&s.Bucket,"GO_OMNI_STORAGE_ACCOUNT_ID":&s.AccountID,
"GO_OMNI_STORAGE_SIGNING_KEY":&s.SigningKey,"GO_OMNI_FILES_TOKEN":&s.FilesToken,
"GO_OMNI_STORAGE_ACCESS_KEY":&s.AccessKey,"GO_OMNI_STORAGE_SECRET_KEY":&s.SecretKey,
"GO_OMNI_STORAGE_SESSION_TOKEN":&s.SessionToken,
}
for name,target:=range values {if value:=strings.TrimSpace(os.Getenv(name));value!=""{*target=value}}
for name,target:=range map[string]*time.Duration{"GO_OMNI_STORAGE_LINK_TTL":&s.LinkTTL,"GO_OMNI_STORAGE_REQUEST_TIMEOUT":&s.RequestTimeout} {
if value:=strings.TrimSpace(os.Getenv(name));value!="" {v,err:=time.ParseDuration(value);if err!=nil{return fmt.Errorf("%s must be a duration",name)};*target=v}
}
if value:=os.Getenv("GO_OMNI_STORAGE_MAX_BYTES");value!=""{v,err:=strconv.ParseInt(value,10,64);if err!=nil{return errors.New("GO_OMNI_STORAGE_MAX_BYTES must be an integer")};s.MaxBytes=v}
if value:=os.Getenv("GO_OMNI_STORAGE_SECURE");value!=""{v,err:=strconv.ParseBool(value);if err!=nil{return errors.New("GO_OMNI_STORAGE_SECURE must be a boolean")};s.Secure=v}
if value:=strings.TrimSpace(os.Getenv("GO_OMNI_STORAGE_CONTENT_TYPES"));value!="" {s.ContentTypes=strings.Split(value,",");for i:=range s.ContentTypes{s.ContentTypes[i]=strings.TrimSpace(s.ContentTypes[i])}}
return nil
}
func (s StorageConfig) Validate()error {
if s.Provider!={{printf "%q" .Storage}} {return errors.New("storage.provider must match the adapter selected at generation")}
if err:=s.Policy().Validate();err!=nil{return err}
if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")}
if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")}
{{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("set GO_OMNI_FILES_TOKEN to a token of at least 32 bytes without whitespace")}{{end}}
{{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("storage.filesToken requires at least 32 bytes without whitespace")}{{end}}
{{if eq .Storage "local"}}
if s.Root=="" {return errors.New("storage.root is required")}
u,err:=url.Parse(s.PublicURL)
if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")}
if len(s.SigningKey)<32 {return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")}
if len(s.SigningKey)<32 {return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")}
{{else}}
if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region and GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY are required")}
if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region, accessKey and secretKey are required")}
{{if eq .Storage "r2"}}
if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")}
if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")}

View File

@@ -21,18 +21,18 @@ func environment(t *testing.T){
}
func TestSettingsPrecedenceAndStrictness(t *testing.T){
environment(t)
{{if eq .Database "mysql"}}t.Setenv("GO_OMNI_DB_DSN","test:secret@tcp(localhost)/demo"){{end}}
source,err:=os.ReadFile("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
path:=filepath.Join(t.TempDir(),"config.yaml")
if err:=os.WriteFile(path,[]byte("server:\n address: '127.0.0.1:8123'\n log:\n level: debug\n"),0600);err!=nil{t.Fatal(err)}
source=[]byte(strings.Replace(string(source),"${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}","127.0.0.1:8123",1))
if err=os.WriteFile(path,source,0600);err!=nil{t.Fatal(err)}
t.Setenv("GO_OMNI_HTTP_ADDR","127.0.0.1:9123")
t.Setenv("GO_OMNI_LOG_OUTPUT","stdout")
cfg,err:=Load(path);if err!=nil{t.Fatal(err)}
if cfg.Server.Address!="127.0.0.1:9123" || cfg.Server.Log.Level!="debug" || cfg.Server.Log.Output!="stdout" || cfg.Server.Log.File!="" || cfg.Server.Log.Rotation.Enabled {t.Fatal("configuration precedence failed")}
for _,body:=range []string{"unexpected: true","server:\n slowRequestThreshold: -1s","server:\n log:\n format: invalid"{{if ne .Storage "none"}},"storage:\n filesToken: forbidden","storage:\n maxBytes: -1","storage:\n provider: unsupported"{{end}}}{
if err:=os.WriteFile(path,[]byte(body),0600);err!=nil{t.Fatal(err)}
if _,err:=Load(path);err==nil{t.Fatal("invalid configuration accepted")}
initial:=Default()
if cfg.Server.Address!="127.0.0.1:8123"||cfg.Server.Log.Output!=initial.Server.Log.Output||cfg.Server.Log.File!=initial.Server.Log.File {t.Fatal("implicit overwrite")}
for _,body:=range []string{"unexpected: true","server:\n slowRequestThreshold: -1s","server:\n log:\n format: invalid"}{
if err=os.WriteFile(path,[]byte(body),0600);err!=nil{t.Fatal(err)}
if _,err=Load(path);err==nil{t.Fatal("invalid configuration accepted")}
}
{{if ne .Storage "none"}}
t.Setenv("GO_OMNI_FILES_TOKEN","")
if _,err:=Load("");err==nil{t.Fatal("missing file token accepted")}
{{end}}
}

View File

@@ -10,11 +10,11 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni"
// Source builds can carry an unpublished VCS version (including +dirty).
// Keep this fallback in sync with the next release; installed CLIs use their
// downloaded module version, which also has a module checksum.
const defaultFrameworkVersion = "v0.20.0-preview"
const defaultFrameworkVersion = "v0.22.0-preview"
// This target is not a claim that the release already exists. Source users
// select a local framework checkout until the module templates are published.
const modulesFrameworkVersion = "v0.7.0-preview"
const modulesFrameworkVersion = "v0.22.0-preview"
func installedFrameworkVersion() string {
info, _ := debug.ReadBuildInfo()

View File

@@ -1,66 +0,0 @@
package config
import (
"strings"
"testing"
"time"
)
func TestUnifiedEnvAliases(t *testing.T) {
type project struct {
Server Config `json:"server"`
Label string `json:"label"`
}
base := project{Server: Default(), Label: "base"}
for _, policy := range []string{"allow", "ignore", "reject"} {
t.Run(policy, func(t *testing.T) {
env := map[string]string{"TEST_HTTP_ADDR": "127.0.0.1:8099", "TEST_LOG_SERVICE": "", "TEST_SERVICE": "", "TEST_LABEL": ""}
opts := EnvOptions{Prefix: "TEST_", Empty: policy, Aliases: ServerEnvAliases("TEST_", "server"), Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}
got, err := OverlayEnv(base, opts)
if policy == "reject" {
if err == nil || !strings.Contains(err.Error(), "empty") {
t.Fatal(err)
}
return
}
if err != nil {
t.Fatal(err)
}
if got.Server.Address != "127.0.0.1:8099" {
t.Fatal("alias ignored")
}
if policy == "allow" && got.Label != "" || policy == "ignore" && got.Label != "base" {
t.Fatal("empty policy differs")
}
env["TEST_SERVER_ADDRESS"] = "127.0.0.1:9001"
env["TEST_HTTP_READ_TIMEOUT"] = "3s"
env["TEST_ENABLE_LOGGER"] = "false"
env["TEST_TRUSTED_PROXIES"] = `["127.0.0.1"]`
got, err = OverlayEnv(base, opts)
if err != nil || got.Server.Address != "127.0.0.1:9001" || got.Server.HTTP.ReadTimeout != 3*time.Second || got.Server.EnableLogger || len(got.Server.TrustedProxies) != 1 {
t.Fatal(got, err)
}
env["TEST_SERVER_ADDRESS"] = ""
got, err = OverlayEnv(base, opts)
if err != nil {
t.Fatal(err)
}
expected := ""
if policy == "ignore" {
expected = base.Server.Address
}
if got.Server.Address != expected {
t.Fatal("empty canonical unexpectedly fell through to alias")
}
})
}
if base.Server.Address != ":8080" || base.Label != "base" {
t.Fatal("defaults mutated")
}
}
func TestAliasTypeErrorsDoNotExposeValue(t *testing.T) {
opts := EnvOptions{Prefix: "X_", Empty: "reject", Aliases: ServerEnvAliases("X_", ""), Lookup: func(k string) (string, bool) { return "private-invalid", k == "X_HTTP_READ_TIMEOUT" }}
if _, err := OverlayEnv(Default(), opts); err == nil || strings.Contains(err.Error(), "private-invalid") {
t.Fatal(err)
}
}

View File

@@ -59,6 +59,20 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int,
if err := b.step(path, depth); err != nil {
return err
}
if text, ok := v.(resolvedText); ok {
if dst.Kind() != reflect.Pointer && dst.Type() != durationType && dst.CanAddr() && !dst.Addr().Type().Implements(jsonUnmarshalerType) && !dst.Addr().Type().Implements(textType) {
switch dst.Kind() {
case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64:
if err := parseTextValue(dst, string(text)); err != nil {
return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err)
}
return nil
}
}
if dst.Kind() != reflect.Pointer {
v = string(text)
}
}
if v == nil {
return b.decode(nil, dst, path)
} // Preserve encoding/json null semantics.
@@ -78,7 +92,7 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int,
default:
return b.decode(v, dst, path)
}
if err := parseEnvValue(dst, strings.TrimSpace(raw)); err != nil {
if err := parseTextValue(dst, strings.TrimSpace(raw)); err != nil {
return b.doc.diagnostic(path, "expected a duration such as 5s", err)
}
return nil
@@ -102,7 +116,7 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int,
}
switch dst.Kind() {
case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64:
if err := parseEnvValue(dst, strings.TrimSpace(string(text))); err != nil {
if err := parseTextValue(dst, strings.TrimSpace(string(text))); err != nil {
return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err)
}
return nil
@@ -143,7 +157,7 @@ func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int,
}
}
if !ok {
return b.doc.diagnostic(child, "unknown field", nil)
return b.doc.diagnostic(child, unknownFieldReason(name, fields), nil)
}
if used[canonical] {
return b.doc.diagnostic(child, "duplicate field binding", nil)
@@ -237,6 +251,8 @@ func (b *documentBinder) dynamic(v any, path string, depth int) (any, error) {
switch x := v.(type) {
case Text:
return string(x), nil
case resolvedText:
return string(x), nil
case map[string]any:
out := make(map[string]any, len(x))
for _, k := range sortedKeys(x) {

View File

@@ -1,132 +1,6 @@
package config
import (
"strings"
"testing"
"time"
)
func TestLoadFromEnvUsesDefaults(t *testing.T) {
for _, key := range []string{
"GO_OMNI_HTTP_ADDR", "GO_OMNI_REQUEST_ID_HEADER", "GO_OMNI_ENABLE_LOGGER",
"GO_OMNI_TRUSTED_PROXIES",
"GO_OMNI_TRANSPORT", "GO_OMNI_TLS_CERT_FILE", "GO_OMNI_TLS_KEY_FILE",
"GO_OMNI_LOG_LEVEL", "GO_OMNI_LOG_FORMAT", "GO_OMNI_LOG_OUTPUT",
"GO_OMNI_LOG_FILE", "GO_OMNI_SERVICE", "GO_OMNI_ENVIRONMENT",
"GO_OMNI_LOG_ADD_SOURCE", "GO_OMNI_SHUTDOWN_TIMEOUT",
"GO_OMNI_SLOW_REQUEST_THRESHOLD",
} {
t.Setenv(key, "")
}
cfg, err := LoadFromEnv("")
if err != nil {
t.Fatal(err)
}
if cfg.Address != ":8080" || cfg.Transport != TransportNetHTTP || cfg.TLS != (TLSConfig{}) || cfg.RequestIDHeader != "X-Request-ID" || cfg.ShutdownTimeout != 10*time.Second {
t.Fatalf("unexpected defaults: %+v", cfg)
}
if cfg.EnableLogger || cfg.Log.Level != "info" || cfg.Log.Format != "json" || cfg.Log.Output != "stderr" {
t.Fatalf("unexpected logger defaults: %+v", cfg.Log)
}
}
func TestTransportEnvironment(t *testing.T) {
for _, test := range []struct {
name, transport, cert, key, errorField string
}{
{"default", "", "", "", ""},
{"native", "native", "", "", ""},
{"nativeTLS", "native", "server.crt", "server.key", ""},
{"http2", "http2", "server.crt", "server.key", ""},
{"unknown", "typo", "", "", "transport"},
{"certOnly", "nethttp", "server.crt", "", "tls"},
{"keyOnly", "native", "", "server.key", "tls"},
{"http2WithoutTLS", "http2", "", "", "tls"},
} {
t.Run(test.name, func(t *testing.T) {
t.Setenv("TRANSPORT_TEST_TRANSPORT", test.transport)
t.Setenv("TRANSPORT_TEST_TLS_CERT_FILE", test.cert)
t.Setenv("TRANSPORT_TEST_TLS_KEY_FILE", test.key)
cfg, err := LoadFromEnv("TRANSPORT_TEST_")
if test.errorField != "" {
if err == nil || !strings.Contains(err.Error(), test.errorField) {
t.Fatalf("unexpected error: %v", err)
}
return
}
if err != nil {
t.Fatal(err)
}
want := test.transport
if want == "" {
want = TransportNetHTTP
}
if cfg.Transport != want || cfg.TLS.CertFile != test.cert || cfg.TLS.KeyFile != test.key {
t.Fatalf("settings: %+v", cfg)
}
})
}
}
func TestLoadFromEnvOverridesSettings(t *testing.T) {
t.Setenv("GO_OMNI_HTTP_ADDR", "127.0.0.1:9090")
t.Setenv("GO_OMNI_REQUEST_ID_HEADER", "X-Correlation-ID")
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "127.0.0.1, 10.0.0.0/8")
t.Setenv("GO_OMNI_ENABLE_LOGGER", "true")
t.Setenv("GO_OMNI_LOG_LEVEL", "debug")
t.Setenv("GO_OMNI_LOG_FORMAT", "text")
t.Setenv("GO_OMNI_LOG_OUTPUT", "stdout")
t.Setenv("GO_OMNI_SERVICE", "billing")
t.Setenv("GO_OMNI_ENVIRONMENT", "test")
t.Setenv("GO_OMNI_LOG_ADD_SOURCE", "true")
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "3s")
t.Setenv("GO_OMNI_SLOW_REQUEST_THRESHOLD", "500ms")
cfg, err := LoadFromEnv("")
if err != nil {
t.Fatal(err)
}
if cfg.Address != "127.0.0.1:9090" || cfg.RequestIDHeader != "X-Correlation-ID" || !cfg.EnableLogger || len(cfg.TrustedProxies) != 2 || cfg.TrustedProxies[1] != "10.0.0.0/8" {
t.Fatalf("unexpected application settings: %+v", cfg)
}
if cfg.ShutdownTimeout != 3*time.Second || cfg.Log.Level != "debug" || cfg.Log.Format != "text" || cfg.Log.Output != "stdout" || cfg.Log.Service != "billing" || cfg.Log.Environment != "test" || !cfg.Log.AddSource {
t.Fatalf("unexpected logger settings: %+v", cfg.Log)
}
if cfg.SlowRequestThreshold != 500*time.Millisecond {
t.Fatal(cfg.SlowRequestThreshold)
}
}
func TestLoadFromEnvRejectsInvalidValues(t *testing.T) {
t.Setenv("GO_OMNI_ENABLE_LOGGER", "sometimes")
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "ENABLE_LOGGER") {
t.Fatalf("expected invalid logger flag error, got %v", err)
}
t.Setenv("GO_OMNI_ENABLE_LOGGER", "")
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "later")
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "SHUTDOWN_TIMEOUT") {
t.Fatalf("expected invalid timeout error, got %v", err)
}
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "1s")
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "proxy.example.com")
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "trustedProxies") {
t.Fatalf("expected invalid trusted proxy error, got %v", err)
}
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "")
cfg := Default()
cfg.Address = ""
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "address") {
t.Fatalf("expected invalid address error, got %v", err)
}
cfg = Default()
cfg.RequestIDHeader = " "
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "requestIdHeader") {
t.Fatalf("expected invalid request header error, got %v", err)
}
}
import "testing"
func TestWithConfigAndWithLogConfig(t *testing.T) {
settings := Default()
@@ -147,23 +21,3 @@ func TestWithConfigAndWithLogConfig(t *testing.T) {
t.Fatalf("log config was not replaced: %+v", cfg.Log)
}
}
func TestApplyEnvPreservesProjectDefaults(t *testing.T) {
cfg := Default()
cfg.Address = "127.0.0.1:8123"
cfg.Log.RedactKeys = []string{"bankAccount"}
cfg.TrustedProxies = []string{"127.0.0.1"}
t.Setenv("OVERLAY_TEST_LOG_LEVEL", "debug")
got, err := ApplyEnv(cfg, "OVERLAY_TEST_")
if err != nil {
t.Fatal(err)
}
if got.Address != cfg.Address || got.Log.Level != "debug" {
t.Fatal(got)
}
got.Log.RedactKeys[0] = "changed"
got.TrustedProxies[0] = "192.0.2.1"
if cfg.Log.RedactKeys[0] != "bankAccount" || cfg.TrustedProxies[0] != "127.0.0.1" {
t.Fatal("caller-owned defaults mutated")
}
}

View File

@@ -82,8 +82,9 @@ type Position struct{ Line, Column int }
// string, bool, json.Number, nil, []any, map[string]any. Text is available for
// formats such as XML whose scalar tokens have no inherent JSON type.
type Document struct {
Value any
Positions map[string]Position
Value any
Positions map[string]Position
environments map[string][]string
}
type Text string

View File

@@ -18,14 +18,15 @@ func TestEndpointConfiguration(t *testing.T) {
if cfg.Endpoints.Liveness.Enabled || cfg.Endpoints.OpenAPI.Enabled || !cfg.Endpoints.Readiness.Enabled || cfg.Endpoints.Readiness.ResponseMode != "raw" {
t.Fatal(cfg.Endpoints)
}
refs := configFile(t, "yaml", "endpoints:\n readiness:\n path: ${TEST_ENDPOINTS_READINESS_PATH}\n openapi:\n enabled: ${TEST_ENDPOINTS_OPENAPI_ENABLED}\n")
t.Setenv("TEST_ENDPOINTS_READINESS_PATH", "/internal/ready")
t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "true")
cfg, err = ApplyEnv(cfg, "TEST_")
cfg, err = LoadProject(refs, cfg)
if err != nil || cfg.Endpoints.Readiness.Path != "/internal/ready" || !cfg.Endpoints.OpenAPI.Enabled {
t.Fatal(cfg.Endpoints, err)
}
t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "invalid")
if _, err = ApplyEnv(cfg, "TEST_"); err == nil {
if _, err = LoadProject(refs, cfg); err == nil {
t.Fatal("invalid bool accepted")
}
}

View File

@@ -1,83 +0,0 @@
package config
import (
"encoding/json"
"os"
"strings"
)
const defaultEnvPrefix = "GO_OMNI_"
// ServerEnvAliases returns the documented short names for Config fields.
// field is the JSON path of Config in a project (usually "server"), or ""
// for Config itself. Canonical names win over aliases. Both use the same
// EnvOptions.Empty/Lookup and typed parser; collections use JSON syntax.
func ServerEnvAliases(prefix, field string) map[string][]string {
root := prefix
if field != "" {
for _, part := range strings.Split(field, ".") {
root += envName(part) + "_"
}
}
names := map[string]string{
"ADDRESS": "HTTP_ADDR", "TRANSPORT": "TRANSPORT", "TLS_CERT_FILE": "TLS_CERT_FILE", "TLS_KEY_FILE": "TLS_KEY_FILE",
"REQUEST_ID_HEADER": "REQUEST_ID_HEADER", "ENABLE_LOGGER": "ENABLE_LOGGER", "SLOW_REQUEST_THRESHOLD": "SLOW_REQUEST_THRESHOLD",
"TRUSTED_PROXIES": "TRUSTED_PROXIES", "SHUTDOWN_TIMEOUT": "SHUTDOWN_TIMEOUT",
"LOG_LEVEL": "LOG_LEVEL", "LOG_FORMAT": "LOG_FORMAT", "LOG_OUTPUT": "LOG_OUTPUT", "LOG_FILE": "LOG_FILE", "LOG_SERVICE": "SERVICE", "LOG_ENVIRONMENT": "ENVIRONMENT", "LOG_ADD_SOURCE": "LOG_ADD_SOURCE",
"HTTP_READ_HEADER_TIMEOUT": "HTTP_READ_HEADER_TIMEOUT", "HTTP_READ_TIMEOUT": "HTTP_READ_TIMEOUT", "HTTP_WRITE_TIMEOUT": "HTTP_WRITE_TIMEOUT", "HTTP_IDLE_TIMEOUT": "HTTP_IDLE_TIMEOUT", "HTTP_MAX_HEADER_BYTES": "HTTP_MAX_HEADER_BYTES",
"LIFECYCLE_STOP_TIMEOUT": "LIFECYCLE_STOP_TIMEOUT", "LIFECYCLE_DRAIN_TIMEOUT": "LIFECYCLE_DRAIN_TIMEOUT", "LIFECYCLE_CANCEL_TIMEOUT": "LIFECYCLE_CANCEL_TIMEOUT",
}
for _, endpoint := range []string{"LIVENESS", "READINESS", "OPENAPI"} {
for _, part := range []string{"ENABLED", "PATH", "RESPONSE_MODE"} {
if endpoint == "OPENAPI" && part == "RESPONSE_MODE" {
continue
}
name := "ENDPOINTS_" + endpoint + "_" + part
names[name] = name
}
}
aliases := map[string][]string{}
for key, alias := range names {
if root+key != prefix+alias {
aliases[root+key] = []string{prefix + alias}
}
}
return aliases
}
// LoadFromEnv loads defaults using the legacy short-name conventions.
func LoadFromEnv(prefix string) (Config, error) { return ApplyEnv(Default(), prefix) }
// ApplyEnv retains legacy whitespace trimming, ignored empty variables and
// comma-separated TRUSTED_PROXIES. New projects should use OverlayEnv with
// ServerEnvAliases for one explicit empty-value policy across the whole config.
func ApplyEnv(defaults Config, prefix string) (Config, error) {
prefix = strings.TrimSpace(prefix)
if prefix == "" {
prefix = defaultEnvPrefix
}
lookup := func(key string) (string, bool) {
raw, ok := os.LookupEnv(key)
raw = strings.TrimSpace(raw)
if ok && raw != "" && key == prefix+"TRUSTED_PROXIES" && !strings.HasPrefix(raw, "[") {
values := strings.Split(raw, ",")
for i := range values {
values[i] = strings.TrimSpace(values[i])
}
data, _ := json.Marshal(values)
raw = string(data)
}
return raw, ok
}
defaults.Errors = defaults.Errors.Clone()
defaults.TrustedProxies = append([]string(nil), defaults.TrustedProxies...)
defaults.Log.RedactKeys = append([]string(nil), defaults.Log.RedactKeys...)
cfg, err := OverlayEnv(defaults, EnvOptions{Prefix: prefix, Empty: "ignore", Lookup: lookup, Aliases: ServerEnvAliases(prefix, "")})
if err != nil {
return Config{}, err
}
if err = cfg.Validate(); err != nil {
return Config{}, err
}
return cfg, nil
}

View File

@@ -21,7 +21,7 @@ func TestCatalogConfigFiles(t *testing.T) {
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err)
}
cfg, err := config.LoadFile(path, "CATALOG_FIXTURE_")
cfg, err := config.LoadFile(path)
if err != nil {
t.Fatal(err)
}
@@ -40,7 +40,7 @@ func TestCatalogConfigFiles(t *testing.T) {
if err := os.WriteFile(path, []byte(body), 0600); err != nil {
t.Fatal(err)
}
if _, err := config.LoadFile(path, "CATALOG_FIXTURE_"); err == nil {
if _, err := config.LoadFile(path); err == nil {
t.Fatal("accepted invalid catalog", body)
}
}
@@ -57,7 +57,7 @@ func TestCatalogReloadSnapshotIsolation(t *testing.T) {
t.Fatal(err)
}
defer logger.Close()
reloader, err := config.NewReloader(path, "CATALOG_RELOAD_", logger)
reloader, err := config.NewReloader(path, logger)
if err != nil {
t.Fatal(err)
}

View File

@@ -20,7 +20,7 @@ func TestTemplateConfigFiles(t *testing.T) {
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err)
}
cfg, err := config.LoadFile(path, "TEMPLATE_TEST_")
cfg, err := config.LoadFile(path)
if err != nil {
t.Fatal(err)
}

View File

@@ -4,103 +4,38 @@ import (
"context"
"errors"
"reflect"
"strings"
"sync"
"time"
"git.yuntuo.hk/k2safe/go-omni/logx"
)
func durations(v any, t reflect.Type) (any, error) {
if t == nil {
return v, nil
}
for t.Kind() == reflect.Pointer {
t = t.Elem()
}
if t == reflect.TypeOf(time.Duration(0)) {
if s, ok := v.(string); ok {
d, err := time.ParseDuration(s)
return int64(d), err
}
return v, nil
}
if t.Kind() == reflect.Struct {
if m, ok := v.(map[string]any); ok {
for n := 0; n < t.NumField(); n++ {
field := t.Field(n)
name := strings.Split(field.Tag.Get("json"), ",")[0]
if name == "-" {
continue
}
if name == "" {
name = field.Name
}
if item, exists := m[name]; exists {
converted, err := durations(item, field.Type)
if err != nil {
return nil, err
}
m[name] = converted
}
}
}
}
if t.Kind() == reflect.Map && t.Key().Kind() == reflect.String {
if values, ok := v.(map[string]any); ok {
for key, value := range values {
converted, err := durations(value, t.Elem())
if err != nil {
return nil, err
}
values[key] = converted
}
}
}
if t.Kind() == reflect.Slice {
if items, ok := v.([]any); ok {
for n, item := range items {
converted, err := durations(item, t.Elem())
if err != nil {
return nil, err
}
items[n] = converted
}
}
}
return v, nil
}
func LoadFile(path, prefix string) (Config, error) {
cfg, err := DecodeFile(path, Default())
if err != nil {
return Config{}, err
}
return ApplyEnv(cfg, prefix)
}
// LoadFile loads and validates framework configuration using explicit file references.
func LoadFile(path string) (Config, error) { return LoadProject(path, Default()) }
// Reloader only accepts log.level changes. Static changes and invalid files
// leave the current snapshot and logger untouched. Watch does not own a goroutine.
type Reloader struct {
mu sync.Mutex
path, prefix string
current Config
logger *logx.Logger
mu sync.Mutex
path string
current Config
logger *logx.Logger
}
func NewReloader(path, prefix string, logger *logx.Logger) (*Reloader, error) {
cfg, err := LoadFile(path, prefix)
func NewReloader(path string, logger *logx.Logger) (*Reloader, error) {
cfg, err := LoadFile(path)
if err != nil {
return nil, err
}
if err = logger.SetLevel(cfg.Log.Level); err != nil {
return nil, err
}
return &Reloader{path: path, prefix: prefix, current: cfg, logger: logger}, nil
return &Reloader{path: path, current: cfg, logger: logger}, nil
}
func (r *Reloader) Reload() error {
r.mu.Lock()
defer r.mu.Unlock()
next, err := LoadFile(r.path, r.prefix)
next, err := LoadFile(r.path)
if err != nil {
return err
}

View File

@@ -20,8 +20,8 @@ func TestFileMergeAndSafeReload(t *testing.T) {
}
write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: info\n")
t.Setenv("TEST_MODULE_HTTP_ADDR", ":9123")
cfg, err := LoadFile(path, "TEST_MODULE_")
if err != nil || cfg.Address != ":9123" || cfg.ShutdownTimeout != 7*time.Second {
cfg, err := LoadFile(path)
if err != nil || cfg.Address != ":8123" || cfg.ShutdownTimeout != 7*time.Second {
t.Fatal(cfg, err)
}
var output bytes.Buffer
@@ -33,7 +33,7 @@ func TestFileMergeAndSafeReload(t *testing.T) {
}
defer l.Close()
child := l.With("scope", "test")
r, err := NewReloader(path, "TEST_MODULE_", l)
r, err := NewReloader(path, l)
if err != nil {
t.Fatal(err)
}

View File

@@ -22,7 +22,7 @@ func TestHTTPFileAndEnvironment(t *testing.T) {
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err)
}
cfg, err := config.LoadFile(path, "HTTP_TEST_")
cfg, err := config.LoadFile(path)
if err != nil {
t.Fatal(err)
}
@@ -35,7 +35,10 @@ func TestHTTPFileAndEnvironment(t *testing.T) {
t.Setenv("HTTP_TEST_HTTP_WRITE_TIMEOUT", "8s")
t.Setenv("HTTP_TEST_HTTP_IDLE_TIMEOUT", "9s")
t.Setenv("HTTP_TEST_HTTP_MAX_HEADER_BYTES", "16384")
got, err := config.ApplyEnv(cfg, "HTTP_TEST_")
if err := os.WriteFile(path, []byte(`{"http":{"readHeaderTimeout":"${HTTP_TEST_HTTP_READ_HEADER_TIMEOUT}","readTimeout":"${HTTP_TEST_HTTP_READ_TIMEOUT}","writeTimeout":"${HTTP_TEST_HTTP_WRITE_TIMEOUT}","idleTimeout":"${HTTP_TEST_HTTP_IDLE_TIMEOUT}","maxHeaderBytes":"${HTTP_TEST_HTTP_MAX_HEADER_BYTES}"}}`), 0600); err != nil {
t.Fatal(err)
}
got, err := config.LoadProject(path, cfg)
if err != nil {
t.Fatal(err)
}
@@ -55,7 +58,12 @@ func TestHTTPInvalidAndUnsupportedSettings(t *testing.T) {
value = "-1s"
}
t.Setenv("HTTP_INVALID_HTTP_"+key, value)
if _, err := config.LoadFromEnv("HTTP_INVALID_"); err == nil {
names := map[string]string{"READ_HEADER_TIMEOUT": "readHeaderTimeout", "READ_TIMEOUT": "readTimeout", "WRITE_TIMEOUT": "writeTimeout", "IDLE_TIMEOUT": "idleTimeout", "MAX_HEADER_BYTES": "maxHeaderBytes"}
path := filepath.Join(t.TempDir(), "config.json")
if err := os.WriteFile(path, []byte(`{"http":{"`+names[key]+`":"${HTTP_INVALID_HTTP_`+key+`}"}}`), 0600); err != nil {
t.Fatal(err)
}
if _, err := config.LoadFile(path); err == nil {
t.Fatal("invalid HTTP value accepted")
}
})

View File

@@ -14,7 +14,7 @@ func TestLifecycleConfiguration(t *testing.T) {
if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: 2s\n drainTimeout: 3s\n cancelTimeout: 4s\n"), 0600); err != nil {
t.Fatal(err)
}
cfg, err := config.LoadFile(path, "PHASE_TEST_")
cfg, err := config.LoadFile(path)
if err != nil {
t.Fatal(err)
}
@@ -24,13 +24,16 @@ func TestLifecycleConfiguration(t *testing.T) {
for _, key := range []string{"STOP_TIMEOUT", "DRAIN_TIMEOUT", "CANCEL_TIMEOUT"} {
t.Setenv("PHASE_TEST_LIFECYCLE_"+key, "6s")
}
cfg, err = config.ApplyEnv(cfg, "PHASE_TEST_")
if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: ${PHASE_TEST_LIFECYCLE_STOP_TIMEOUT}\n drainTimeout: ${PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT}\n cancelTimeout: ${PHASE_TEST_LIFECYCLE_CANCEL_TIMEOUT}\n"), 0600); err != nil {
t.Fatal(err)
}
cfg, err = config.LoadProject(path, cfg)
if err != nil || cfg.Lifecycle.StopTimeout != 6*time.Second || cfg.Lifecycle.DrainTimeout != 6*time.Second || cfg.Lifecycle.CancelTimeout != 6*time.Second {
t.Fatal(cfg.Lifecycle, err)
}
for _, v := range []string{"bad", "-1s"} {
t.Setenv("PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT", v)
if _, err := config.ApplyEnv(cfg, "PHASE_TEST_"); err == nil {
if _, err := config.LoadProject(path, cfg); err == nil {
t.Fatal("invalid duration accepted")
}
}

View File

@@ -26,6 +26,7 @@ func (f DecoderFunc) Decode(data []byte) (Document, error) { return f(data) }
type Loader struct {
decoders map[string]Decoder
sealed bool
lookup func(string) (string, bool)
}
type LoaderOption func(*Loader) error
@@ -55,7 +56,7 @@ func WithDecoder(extension string, decoder Decoder) LoaderOption {
}
func NewLoader(options ...LoaderOption) (*Loader, error) {
l := &Loader{decoders: map[string]Decoder{"json": JSONDecoder{}, "yaml": YAMLDecoder{}, "yml": YAMLDecoder{}, "xml": XMLDecoder{}}}
l := &Loader{lookup: os.LookupEnv, decoders: map[string]Decoder{"json": JSONDecoder{}, "yaml": YAMLDecoder{}, "yml": YAMLDecoder{}, "xml": XMLDecoder{}}}
for _, o := range options {
if o == nil {
return nil, errors.New("config: nil loader option")
@@ -135,7 +136,7 @@ func (l *Loader) read(path string) (doc Document, err error) {
if _, ok := doc.Value.(map[string]any); !ok {
return doc, doc.diagnostic("", "root must be an object", nil)
}
return doc, nil
return resolveDocument(doc, l.lookup)
}
func decodeSafely(decoder Decoder, data []byte) (doc Document, err error) {
defer func() {
@@ -165,11 +166,12 @@ func DecodeFile[T any](path string, defaults T, loaders ...*Loader) (T, error) {
if err != nil {
return zero, err
}
if err := rejectEnvironmentTags(reflect.TypeOf(defaults), map[reflect.Type]bool{}); err != nil {
return zero, err
}
result, err := bindDocument(doc, defaults)
if err != nil {
d := validationError(err)
d.File = path
return zero, d
return zero, locateError(err, path, doc)
}
return result, nil
}

View File

@@ -16,7 +16,7 @@ import (
type loaderExample struct {
Server Config `json:"server"`
Upload struct {
MaxBytes int64 `json:"maxBytes" envDefault:"1024"`
MaxBytes int64 `json:"maxBytes"`
} `json:"upload"`
Values []int64 `json:"values"`
Optional *string `json:"optional"`
@@ -45,19 +45,19 @@ func configFile(t *testing.T, ext, raw string) string {
func TestUnifiedFormatAndPrecedence(t *testing.T) {
formats := map[string]string{
"json": `{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":7},"values":[9007199254740993],"optional":null,"enabled":true,"dynamic":{"id":9007199254740993}}`,
"yaml": "server:\n http:\n readTimeout: 2s\nupload:\n maxBytes: 7\nvalues: [9007199254740993]\noptional: null\nenabled: true\ndynamic:\n id: 9007199254740993\n",
"xml": `<config><server><http><readTimeout>2s</readTimeout></http></server><upload><maxBytes>7</maxBytes></upload><values type="array"><item>9007199254740993</item></values><optional type="null"/><enabled>true</enabled><dynamic><id type="number">9007199254740993</id></dynamic></config>`,
"json": `{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":"${APP_UPLOAD_MAX_BYTES}"},"values":[9007199254740993],"optional":null,"enabled":"${APP_ENABLED}","dynamic":{"id":9007199254740993}}`,
"yaml": "server:\n http:\n readTimeout: 2s\nupload:\n maxBytes: \"${APP_UPLOAD_MAX_BYTES}\"\nvalues: [9007199254740993]\noptional: null\nenabled: \"${APP_ENABLED}\"\ndynamic:\n id: 9007199254740993\n",
"xml": `<config><server><http><readTimeout>2s</readTimeout></http></server><upload><maxBytes>${APP_UPLOAD_MAX_BYTES}</maxBytes></upload><values type="array"><item>9007199254740993</item></values><optional type="null"/><enabled>${APP_ENABLED}</enabled><dynamic><id type="number">9007199254740993</id></dynamic></config>`,
}
var reference loaderExample
for _, ext := range []string{"json", "yaml", "xml"} {
t.Run(ext, func(t *testing.T) {
old := "old"
base := loaderExample{Server: Default(), Optional: &old}
cfg, err := LoadProject(configFile(t, ext, formats[ext]), base, EnvOptions{Prefix: "APP_", Lookup: func(key string) (string, bool) {
cfg, err := LoadProject(configFile(t, ext, formats[ext]), base, testLoader(func(key string) (string, bool) {
v, ok := map[string]string{"APP_UPLOAD_MAX_BYTES": "0", "APP_ENABLED": "false"}[key]
return v, ok
}})
}))
if err != nil {
t.Fatal(err)
}
@@ -77,8 +77,8 @@ func TestUnifiedFormatAndPrecedence(t *testing.T) {
}
})
}
cfg, err := LoadProject("", loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment})
if err != nil || cfg.Upload.MaxBytes != 1024 {
cfg, err := LoadProject("", loaderExample{Server: Default()}, testLoader(noEnvironment))
if err != nil || cfg.Upload.MaxBytes != 0 {
t.Fatal(cfg, err)
}
}
@@ -97,7 +97,7 @@ func TestConfigDiagnosticsAndCauses(t *testing.T) {
} {
t.Run(tc.ext+tc.field, func(t *testing.T) {
path := configFile(t, tc.ext, tc.body)
_, err := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment})
_, err := LoadProject(path, loaderExample{Server: Default()}, testLoader(noEnvironment))
var d *Error
if !errors.As(err, &d) || d.File != path || d.Field != tc.field || d.Reason == "" || tc.line && d.Line < 1 {
t.Fatalf("missing location: %#v %v", d, err)
@@ -108,7 +108,7 @@ func TestConfigDiagnosticsAndCauses(t *testing.T) {
})
}
path := filepath.Join(t.TempDir(), "missing.json")
_, err := LoadProject(path, loaderExample{}, EnvOptions{})
_, err := LoadProject(path, loaderExample{})
if !errors.Is(err, os.ErrNotExist) {
t.Fatal("I/O cause discarded", err)
}
@@ -119,11 +119,12 @@ func TestConfigDiagnosticsAndCauses(t *testing.T) {
if !errors.As(err, &typed) {
t.Fatal("type cause discarded", err)
}
_, err = LoadProject(configFile(t, "yaml", "upload:\n maxBytes: 7"), loaderExample{Server: Default()}, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { return "-1", k == "APP_UPLOAD_MAX_BYTES" }})
_, err = LoadProject(configFile(t, "yaml", "upload:\n maxBytes: \"${APP_UPLOAD_MAX_BYTES}\""), loaderExample{Server: Default()}, testLoader(func(k string) (string, bool) { return "-1", k == "APP_UPLOAD_MAX_BYTES" }))
var d *Error
if !errors.As(err, &d) || d.Env != "APP_UPLOAD_MAX_BYTES" || d.Field != "upload.maxBytes" || d.File != "" || d.Line != 0 {
t.Fatal("environment validation wrongly blamed file", err)
if !errors.As(err, &d) || d.Env != "APP_UPLOAD_MAX_BYTES" || d.Field != "upload.maxBytes" || d.File == "" || d.Line != 2 {
t.Fatal("missing file and variable provenance", err)
}
}
func TestMalformedFormatsAndLimits(t *testing.T) {
@@ -134,7 +135,7 @@ func TestMalformedFormatsAndLimits(t *testing.T) {
} {
for i, raw := range bodies {
t.Run(fmt.Sprintf("%s%d", ext, i), func(t *testing.T) {
_, err := LoadProject(configFile(t, ext, raw), loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment})
_, err := LoadProject(configFile(t, ext, raw), loaderExample{Server: Default()}, testLoader(noEnvironment))
if err == nil || strings.Contains(err.Error(), "private-value") {
t.Fatal(err)
}
@@ -163,7 +164,7 @@ func TestDecoderRegistryAndConcurrentUse(t *testing.T) {
wg.Add(1)
go func() {
defer wg.Done()
cfg, e := LoadProject(path, loaderExample{Server: Default()}, EnvOptions{Lookup: noEnvironment}, l)
cfg, e := LoadProject(path, loaderExample{Server: Default()}, l)
if e != nil || cfg.Upload.MaxBytes != 42 {
t.Error(e)
}
@@ -191,25 +192,6 @@ func TestDecoderRegistryAndConcurrentUse(t *testing.T) {
}
}
func TestUnifiedLoggerNormalization(t *testing.T) {
type project struct {
Server Config `json:"server"`
}
base := project{Server: Default()}
base.Server.Log.File = "old.log"
base.Server.Log.Rotation.Enabled = true
for _, full := range []bool{false, true} {
name := "GO_OMNI_LOG_OUTPUT"
if full {
name = "GO_OMNI_SERVER_LOG_OUTPUT"
}
cfg, err := LoadProject("", base, EnvOptions{Prefix: "GO_OMNI_", Empty: "ignore", Aliases: ServerEnvAliases("GO_OMNI_", "server"), Lookup: func(k string) (string, bool) { return "stdout", k == name }})
if err != nil || cfg.Server.Log.File != "" || cfg.Server.Log.Rotation.Enabled || base.Server.Log.File != "old.log" {
t.Fatal(cfg, err)
}
}
}
func FuzzConfigurationDecoders(f *testing.F) {
f.Add(uint8(0), []byte(`{"count":1}`))
f.Add(uint8(1), []byte("count: 1"))
@@ -332,18 +314,21 @@ func TestConfigDefaultsIsolationAndEmptyValues(t *testing.T) {
func TestDefaultStageDoesNotReadEnvironment(t *testing.T) {
reads := 0
type C struct {
Value int `json:"value" envDefault:"3"`
Value int `json:"value"`
}
got, err := LoadProject("", C{}, EnvOptions{Lookup: func(string) (string, bool) { reads++; return "", false }})
if err != nil || got.Value != 3 || reads != 1 {
got, err := LoadProject("", C{Value: 3}, testLoader(func(string) (string, bool) { reads++; return "9", true }))
if err != nil || got.Value != 3 || reads != 0 {
t.Fatal(got, err, reads)
}
d := locateError(Invalid("servers[0].timeout", "must be positive"), "a.json", Document{}, map[string]string{"servers": "APP_SERVERS"})
if d.Env != "APP_SERVERS" || d.File != "" {
t.Fatal(d)
}
}
func testLoader(lookup func(string) (string, bool)) *Loader {
l, err := NewLoader(WithLookupEnv(lookup))
if err != nil {
panic(err)
}
return l
}
func TestDecoderRegistryFrozen(t *testing.T) {
l, err := NewLoader()
if err != nil {
@@ -358,17 +343,12 @@ func BenchmarkConfigurationStartup(b *testing.B) {
if err := os.WriteFile(path, []byte(`{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":4096}}`), 0600); err != nil {
b.Fatal(err)
}
options := EnvOptions{Prefix: "BENCH_", Lookup: func(name string) (string, bool) {
if name == "BENCH_UPLOAD_MAX_BYTES" {
return "8192", true
}
return "", false
}}
defaults := loaderExample{Server: Default()}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := LoadProject(path, defaults, options); err != nil {
if _, err := LoadProject(path, defaults); err != nil {
b.Fatal(err)
}
}

View File

@@ -1,318 +1,105 @@
package config
import (
"bytes"
"encoding"
"encoding/json"
"errors"
"fmt"
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
"git.yuntuo.hk/k2safe/go-omni/logx"
"math"
"os"
"reflect"
"strconv"
"strings"
"time"
"unicode"
)
// EnvOptions controls typed project configuration overlays. Explicit env tags
// are absolute names; otherwise JSON names become PREFIX_NESTED_FIELD names.
// Empty is allow (default), ignore or reject. No input values appear in errors.
type EnvOptions struct {
Prefix string
Empty string
Lookup func(string) (string, bool)
// Aliases maps canonical absolute environment names to ordered fallback names.
// A present canonical value wins, including an explicit empty value.
Aliases map[string][]string
}
// ApplyDefaults applies envDefault tags to zero fields before file overlays.
func ApplyDefaults[T any](base T) (T, error) { return projectOverlay(base, EnvOptions{}, true) }
// LoadProject applies code/tag defaults, a strict optional file, environment,
// then the project's Validate method, before any logger or dependency is opened.
// envDefault is only applied to zero-valued defaults before reading the file.
func LoadProject[T any](path string, defaults T, options EnvOptions, loaders ...*Loader) (T, error) {
// LoadProject clones defaults, resolves only explicit file references, binds the
// provided fields and validates. Empty path uses only defaults. No env tags,
// prefixes, aliases or implicit environment overlays are applied.
func LoadProject[T any](path string, defaults T, loaders ...*Loader) (T, error) {
var zero T
loader, err := selectLoader(loaders)
if err != nil {
return zero, err
}
cfg, err := projectOverlay(defaults, options, true)
if err != nil {
if err = rejectEnvironmentTags(reflect.TypeOf(defaults), map[reflect.Type]bool{}); err != nil {
return zero, err
}
var doc Document
doc := Document{Value: map[string]any{}}
if path != "" {
doc, err = loader.read(path)
if err != nil {
return zero, err
}
cfg, err = bindDocument(doc, cfg)
if err != nil {
return zero, locateError(err, path, doc, nil)
}
} else {
value, e := cloneConfiguration(reflect.ValueOf(&cfg).Elem(), 0)
if e != nil {
return zero, &Error{Reason: "cannot clone defaults", cause: e}
}
cfg = value.Interface().(T)
}
origins := map[string]string{}
cfg, err = projectOverlay(cfg, options, false, origins)
cfg, err := bindDocument(doc, defaults)
if err != nil {
return zero, err
return zero, locateError(err, path, doc)
}
normalizeProject(reflect.ValueOf(&cfg).Elem(), "", origins, 0)
if valid, ok := any(&cfg).(interface{ Validate() error }); ok {
if err = valid.Validate(); err != nil {
return zero, locateError(err, path, doc, origins)
return zero, locateError(err, path, doc)
}
} else if valid, ok := any(cfg).(interface{ Validate() error }); ok {
if err = valid.Validate(); err != nil {
return zero, locateError(err, path, doc)
}
}
return cfg, nil
}
func locateError(err error, path string, doc Document, origins map[string]string) *Error {
func locateError(err error, path string, doc Document) *Error {
d := validationError(err)
field := d.Field
for field != "" && origins[field] == "" {
index := strings.LastIndexAny(field, ".[")
if index < 0 {
field = ""
} else {
field = field[:index]
}
}
if env := origins[field]; env != "" {
d.Env = env
d.File = ""
d.Line = 0
d.Column = 0
return d
}
d.File = path
if d.Line == 0 {
p := doc.Positions[d.Field]
d.Line = p.Line
d.Column = p.Column
}
for field := d.Field; field != ""; {
if names := doc.environments[field]; len(names) > 0 {
d.Env = strings.Join(names, ", ")
break
}
i := strings.LastIndexAny(field, ".[")
if i < 0 {
break
}
field = field[:i]
}
return d
}
// Framework-owned normalization stays inside the common pipeline. Explicit
// environment stream selection clears a file profile unless a file override
// was also provided; project code never needs to repeat this rule.
func normalizeProject(v reflect.Value, path string, origins map[string]string, depth int) {
if depth > 32 {
return
// Reject removed declarations rather than silently ignoring deployment rules.
func rejectEnvironmentTags(t reflect.Type, seen map[reflect.Type]bool) error {
if t == nil || seen[t] {
return nil
}
if v.Kind() == reflect.Pointer {
if v.IsNil() {
return
seen[t] = true
switch t.Kind() {
case reflect.Pointer, reflect.Slice, reflect.Array, reflect.Map:
return rejectEnvironmentTags(t.Elem(), seen)
case reflect.Struct:
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
if f.PkgPath != "" {
continue
}
_, env := f.Tag.Lookup("env")
_, def := f.Tag.Lookup("envDefault")
if env || def {
return &Error{Field: f.Name, Reason: "env/envDefault tags were removed; use Default() and explicit file references; see CONFIG.md"}
}
if f.Tag.Get("json") != "-" {
if err := rejectEnvironmentTags(f.Type, seen); err != nil {
return err
}
}
}
v = v.Elem()
}
if v.Kind() != reflect.Struct {
return
}
if v.Type() == reflect.TypeOf(Config{}) && v.CanAddr() {
c := v.Addr().Interface().(*Config)
if origins[joinField(path, "log.output")] != "" && c.Log.Output != "" && (origins[joinField(path, "log.file")] == "" || c.Log.File == "") {
c.Log.File = ""
c.Log.Rotation = logx.RotationConfig{}
}
return
}
for i := 0; i < v.NumField(); i++ {
f := v.Type().Field(i)
if f.PkgPath != "" {
continue
}
name := strings.Split(f.Tag.Get("json"), ",")[0]
if name == "-" {
continue
}
if name == "" {
name = f.Name
}
normalizeProject(v.Field(i), joinField(path, name), origins, depth+1)
}
}
// OverlayEnv overlays the entire project struct without mutating the input.
// Tags: env:"CUSTOM_NAME,required,secret,empty=reject"; env:"-" excludes a
// subtree. required means nonempty string/non-nil pointer or collection; false
// and zero are valid scalars. Use project Validate for semantic constraints.
// Slices/maps use JSON syntax, avoiding ambiguous comma and empty semantics.
func OverlayEnv[T any](base T, options EnvOptions) (T, error) {
return projectOverlay(base, options, false)
return nil
}
var textType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()
var durationType = reflect.TypeOf(time.Duration(0))
func projectOverlay[T any](base T, options EnvOptions, defaults bool, sourceMaps ...map[string]string) (T, error) {
var zero T
if options.Lookup == nil {
options.Lookup = os.LookupEnv
}
if options.Empty == "" {
options.Empty = "allow"
}
if options.Empty != "allow" && options.Empty != "ignore" && options.Empty != "reject" {
return zero, errors.New("config: empty policy must be allow, ignore or reject")
}
v := reflect.ValueOf(&base).Elem()
if v.Kind() != reflect.Struct {
return zero, errors.New("config: project configuration must be a struct")
}
result := reflect.New(v.Type()).Elem()
result.Set(v)
seen := map[string]bool{}
var walk func(reflect.Value, string, string, int) error
walk = func(value reflect.Value, prefix, path string, depth int) error {
if depth > 32 {
return errors.New("config: nesting exceeds 32")
}
for i := 0; i < value.NumField(); i++ {
field := value.Type().Field(i)
dst := value.Field(i)
if !dst.CanSet() || field.PkgPath != "" {
continue
}
tag := strings.Split(field.Tag.Get("env"), ",")
if tag[0] == "-" {
continue
}
jsonName := strings.Split(field.Tag.Get("json"), ",")[0]
if jsonName == "-" && tag[0] == "" {
continue
}
if jsonName == "" {
jsonName = field.Name
}
fieldPath := joinField(path, jsonName)
key := prefix + envName(jsonName)
if tag[0] != "" {
key = tag[0]
}
empty, required := options.Empty, false
for _, opt := range tag[1:] {
switch opt {
case "required":
required = true
case "secret", "":
case "empty=allow":
empty = "allow"
case "empty=ignore":
empty = "ignore"
case "empty=reject":
empty = "reject"
default:
return fmt.Errorf("config: invalid env tag for %s", key)
}
}
if dst.Kind() == reflect.Struct && !dst.Addr().Type().Implements(textType) {
if err := walk(dst, key+"_", fieldPath, depth+1); err != nil {
return err
}
continue
}
if dst.Kind() == reflect.Pointer && dst.Type().Elem().Kind() == reflect.Struct && !dst.Type().Implements(textType) {
// Clone before traversing so a failed overlay never edits defaults.
child := reflect.New(dst.Type().Elem())
if !dst.IsNil() {
child.Elem().Set(dst.Elem())
}
if err := walk(child.Elem(), key+"_", fieldPath, depth+1); err != nil {
return err
}
if !dst.IsNil() || !child.Elem().IsZero() {
dst.Set(child)
}
if required && !defaults && dst.IsNil() {
return &Error{Env: key, Field: fieldPath, Reason: "is required"}
}
continue
}
// Unexported runtime capabilities are skipped unless explicitly tagged.
if dst.Kind() == reflect.Func || dst.Kind() == reflect.Chan || dst.Kind() == reflect.Interface {
if tag[0] != "" {
return fmt.Errorf("config: %s has unsupported environment type", key)
}
continue
}
if seen[key] {
return fmt.Errorf("config: duplicate environment key %s", key)
}
seen[key] = true
var raw string
var present bool
if !defaults {
raw, present = options.Lookup(key)
}
selectedKey := key
if !present && !defaults {
for _, alias := range options.Aliases[key] {
if raw, present = options.Lookup(alias); present {
selectedKey = alias
break
}
}
}
if defaults {
raw, present = field.Tag.Lookup("envDefault")
present = present && dst.IsZero()
}
if present && raw == "" && !defaults {
if empty == "reject" {
return &Error{Env: selectedKey, Field: fieldPath, Reason: "must not be empty"}
}
if empty == "ignore" {
present = false
}
}
if present {
parsed := reflect.New(dst.Type()).Elem()
if err := parseEnvValue(parsed, raw); err != nil {
origin := selectedKey
if defaults {
origin = ""
}
return &Error{Env: origin, Field: fieldPath, Reason: "expected a valid " + dst.Type().String(), cause: err}
}
dst.Set(parsed)
if !defaults && len(sourceMaps) > 0 {
sourceMaps[0][fieldPath] = selectedKey
}
}
if required && !defaults && emptyRequired(dst) {
return &Error{Env: key, Field: fieldPath, Reason: "is required"}
}
}
return nil
}
if err := walk(result, options.Prefix, "", 0); err != nil {
return zero, err
}
return result.Interface().(T), nil
}
func emptyRequired(v reflect.Value) bool {
switch v.Kind() {
case reflect.String:
return v.Len() == 0
case reflect.Pointer, reflect.Map, reflect.Slice:
return v.IsNil()
}
return false
}
func parseEnvValue(v reflect.Value, raw string) (err error) {
func parseTextValue(v reflect.Value, raw string) (err error) {
defer func() {
if recover() != nil {
err = errors.New("invalid custom value")
@@ -320,7 +107,7 @@ func parseEnvValue(v reflect.Value, raw string) (err error) {
}()
if v.Kind() == reflect.Pointer {
v.Set(reflect.New(v.Type().Elem()))
return parseEnvValue(v.Elem(), raw)
return parseTextValue(v.Elem(), raw)
}
if v.CanAddr() && v.Addr().Type().Implements(textType) {
return v.Addr().Interface().(encoding.TextUnmarshaler).UnmarshalText([]byte(raw))
@@ -363,37 +150,6 @@ func parseEnvValue(v reflect.Value, raw string) (err error) {
v.SetFloat(x)
}
return err
case reflect.Slice, reflect.Map:
var value any
if err := apierrors.DecodeCatalogJSON([]byte(raw), &value); err != nil {
return err
}
value, err := durations(value, v.Type())
if err != nil {
return err
}
encoded, err := json.Marshal(value)
if err != nil {
return err
}
decoder := json.NewDecoder(bytes.NewReader(encoded))
decoder.DisallowUnknownFields()
return decoder.Decode(v.Addr().Interface())
}
return errors.New("unsupported environment value")
}
func envName(s string) string {
r := []rune(s)
var out strings.Builder
for i, c := range r {
if i > 0 && unicode.IsUpper(c) && (unicode.IsLower(r[i-1]) || unicode.IsDigit(r[i-1]) || (i+1 < len(r) && unicode.IsLower(r[i+1]))) {
out.WriteByte('_')
}
if c == '-' {
c = '_'
}
out.WriteRune(unicode.ToUpper(c))
}
return out.String()
return errors.New("expected scalar text")
}

View File

@@ -42,20 +42,3 @@ func TestConfigFilePreservesDefaultsWithoutSharingMutableData(t *testing.T) {
t.Fatal("decode error leaked contents", err)
}
}
func TestEnvCollectionPrecisionAndDurationMaps(t *testing.T) {
type cfg struct {
IDs []int64 `json:"ids"`
Timers map[string]time.Duration `json:"timers"`
}
lookup := func(key string) (string, bool) {
v, ok := map[string]string{"IDS": "[9007199254740993]", "TIMERS": `{"backoff":"250ms"}`}[key]
return v, ok
}
c, err := OverlayEnv(cfg{}, EnvOptions{Lookup: lookup})
if err != nil || c.IDs[0] != 9007199254740993 || c.Timers["backoff"] != 250*time.Millisecond {
t.Fatal(c, err)
}
if _, err = OverlayEnv(cfg{}, EnvOptions{Lookup: func(key string) (string, bool) { return `{"x":"1s","x":"2s"}`, key == "TIMERS" }}); err == nil {
t.Fatal("duplicate map key accepted")
}
}

View File

@@ -1,124 +0,0 @@
package config
import (
"errors"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
type projectNested struct {
ReadTimeout time.Duration `json:"readTimeout"`
Names []string `json:"names"`
}
type projectConfig struct {
Nested *projectNested `json:"nested"`
Enabled bool `json:"enabled"`
Count int `json:"count" envDefault:"12"`
Password string `json:"-" env:"ETAX_PASSWORD,secret,required,empty=reject"`
Label string `json:"label" env:",empty=ignore"`
}
func (c projectConfig) Validate() error {
if c.Count < 0 {
return errors.New("count must be nonnegative")
}
return nil
}
func TestProjectConfigurationOrderAndValues(t *testing.T) {
f := filepath.Join(t.TempDir(), "config.yaml")
if err := os.WriteFile(f, []byte("count: 3\nenabled: true\nnested:\n readTimeout: 2s\n names: [file]\nlabel: keep\n"), 0600); err != nil {
t.Fatal(err)
}
env := map[string]string{"ETAX_PASSWORD": " password ", "APP_ENABLED": "false", "APP_COUNT": "0", "APP_NESTED_READ_TIMEOUT": "7s", "APP_NESTED_NAMES": "[]", "APP_LABEL": ""}
options := EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}
cfg, err := LoadProject(f, projectConfig{}, options)
if err != nil {
t.Fatal(err)
}
if cfg.Enabled || cfg.Count != 0 || cfg.Password != " password " || cfg.Nested.ReadTimeout != 7*time.Second || cfg.Nested.Names == nil || len(cfg.Nested.Names) != 0 || cfg.Label != "keep" {
t.Fatal("overlay mismatch")
}
delete(env, "APP_COUNT")
cfg, err = LoadProject("", projectConfig{}, options)
if err != nil || cfg.Count != 12 {
t.Fatal(cfg.Count, err)
}
env["APP_COUNT"] = "-1"
if _, err = LoadProject(f, projectConfig{}, options); err == nil {
t.Fatal("project Validate not called")
}
}
func TestProjectOverlayIsTransactionalAndRedacted(t *testing.T) {
base := projectConfig{Nested: &projectNested{ReadTimeout: time.Second, Names: []string{"old"}}, Password: "original"}
env := map[string]string{"APP_NESTED_READ_TIMEOUT": "2s", "APP_NESTED_NAMES": "[\"new\"]", "APP_COUNT": "secret-value"}
_, err := OverlayEnv(base, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }})
if err == nil || strings.Contains(err.Error(), "secret-value") {
t.Fatal(err)
}
if base.Nested.ReadTimeout != time.Second || !reflect.DeepEqual(base.Nested.Names, []string{"old"}) {
t.Fatal("default pointer modified")
}
delete(env, "APP_COUNT")
env["ETAX_PASSWORD"] = ""
if _, err = OverlayEnv(base, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}); err == nil {
t.Fatal("empty secret accepted")
}
type required struct {
Token string `env:"TOKEN,required"`
}
if _, err = OverlayEnv(required{}, EnvOptions{Lookup: func(string) (string, bool) { return "", false }}); err == nil {
t.Fatal("required absent")
}
type duplicate struct {
A string `env:"SAME"`
B string `env:"SAME"`
}
if _, err = OverlayEnv(duplicate{}, EnvOptions{}); err == nil {
t.Fatal("duplicate env binding accepted")
}
}
func TestEnvironmentCollectionAndEmptySemantics(t *testing.T) {
type sample struct {
Values []time.Duration
Flag bool
Text string
Optional *int
}
cases := []struct {
raw string
valid bool
}{{`["1s","2s"]`, true}, {`["invalid-secret"]`, false}}
for _, tc := range cases {
cfg, err := OverlayEnv(sample{}, EnvOptions{Lookup: func(k string) (string, bool) {
if k == "VALUES" {
return tc.raw, true
}
return "", false
}})
if (err == nil) != tc.valid {
t.Fatal(err)
}
if tc.valid && !reflect.DeepEqual(cfg.Values, []time.Duration{time.Second, 2 * time.Second}) {
t.Fatal(cfg)
}
if err != nil && strings.Contains(err.Error(), "secret") {
t.Fatal(err)
}
}
for _, mode := range []string{"allow", "ignore", "reject"} {
cfg, err := OverlayEnv(sample{Text: "old"}, EnvOptions{Empty: mode, Lookup: func(k string) (string, bool) { return "", k == "TEXT" }})
if mode == "reject" {
if err == nil {
t.Fatal("empty not rejected")
}
} else if err != nil || (mode == "allow" && cfg.Text != "") || (mode == "ignore" && cfg.Text != "old") {
t.Fatal(cfg, err)
}
}
}

210
config/references.go Normal file
View File

@@ -0,0 +1,210 @@
package config
import (
"errors"
"fmt"
"strings"
"unicode/utf8"
)
// WithLookupEnv supplies an explicit-reference lookup for tests or an embedding
// host. Ordinary projects need no loader options. The function must support
// concurrent loads; each load caches each referenced name independently.
func WithLookupEnv(lookup func(string) (string, bool)) LoaderOption {
return func(l *Loader) error {
if l == nil || l.sealed || lookup == nil {
return errors.New("config: expected a non-nil environment lookup on a new loader")
}
l.lookup = lookup
return nil
}
}
// resolvedText is textual input bound according to the target scalar type. It
// never becomes an object, collection or null by interpreting environment data.
type resolvedText string
const maxExpandedBytes = 4 << 20
type environmentValue struct {
value string
present bool
}
type referenceResolver struct {
lookup func(string) (string, bool)
cache map[string]environmentValue
doc Document
nodes, bytes, cacheBytes int
}
func resolveDocument(doc Document, lookup func(string) (string, bool)) (Document, error) {
r := referenceResolver{lookup: lookup, cache: map[string]environmentValue{}, doc: Document{Positions: doc.Positions, environments: map[string][]string{}}}
v, err := r.walk(doc.Value, "", 0)
if err != nil {
return Document{}, err
}
r.doc.Value = v
return r.doc, nil
}
func (r *referenceResolver) walk(v any, path string, depth int) (any, error) {
r.nodes++
if depth > maxConfigDepth || r.nodes > maxConfigNodes {
return nil, r.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil)
}
switch x := v.(type) {
case string:
return r.text(x, path, false)
case Text:
return r.text(string(x), path, true)
case map[string]any:
out := make(map[string]any, len(x))
for _, key := range sortedKeys(x) {
value, err := r.walk(x[key], joinField(path, key), depth+1)
if err != nil {
return nil, err
}
out[key] = value
}
return out, nil
case []any:
out := make([]any, len(x))
for i, value := range x {
var err error
out[i], err = r.walk(value, fmt.Sprintf("%s[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
}
return out, nil
default:
return v, nil
}
}
func (r *referenceResolver) fail(path, name, reason string) error {
d := r.doc.diagnostic(path, reason, nil)
d.Env = name
return d
}
func validEnvironmentName(name string) bool {
if len(name) == 0 {
return false
}
for i := 0; i < len(name); i++ {
c := name[i]
if c == '_' || c >= 'A' && c <= 'Z' || c >= 'a' && c <= 'z' || i > 0 && c >= '0' && c <= '9' {
continue
}
return false
}
return true
}
func (r *referenceResolver) lookupValue(name, path string) (environmentValue, error) {
if v, ok := r.cache[name]; ok {
return v, nil
}
value, present := r.lookup(name)
if len(value) > maxConfigBytes || r.cacheBytes+len(value) > maxExpandedBytes {
return environmentValue{}, r.fail(path, name, "environment value exceeds configuration size limit")
}
if !utf8.ValidString(value) {
return environmentValue{}, r.fail(path, name, "environment value is not valid UTF-8")
}
r.cacheBytes += len(value)
v := environmentValue{value, present}
r.cache[name] = v
return v, nil
}
func (r *referenceResolver) text(raw, path string, xmlText bool) (any, error) {
var out strings.Builder
var names []string
seen := map[string]bool{}
if !strings.Contains(raw, "$") {
if len(raw) > maxConfigBytes || r.bytes+len(raw) > maxExpandedBytes {
return nil, r.fail(path, "", "expanded configuration exceeds size limit")
}
r.bytes += len(raw)
if xmlText {
return Text(raw), nil
}
return raw, nil
}
appendText := func(s string) error {
if out.Len()+len(s) > maxConfigBytes || r.bytes+len(s) > maxExpandedBytes {
return r.fail(path, strings.Join(names, ", "), "expanded configuration exceeds size limit")
}
out.WriteString(s)
r.bytes += len(s)
return nil
}
// Scan only the file's decoded scalar. Inserted environment values are never
// scanned again, including dollars, quotes, separators or newline characters.
for i := 0; i < len(raw); {
j := strings.IndexByte(raw[i:], '$')
if j < 0 {
if err := appendText(raw[i:]); err != nil {
return nil, err
}
break
}
j += i
if err := appendText(raw[i:j]); err != nil {
return nil, err
}
switch {
case j+1 < len(raw) && raw[j+1] == '$':
if err := appendText("$"); err != nil {
return nil, err
}
i = j + 2
case j+1 < len(raw) && raw[j+1] == '{':
end := strings.IndexByte(raw[j+2:], '}')
if end < 0 {
return nil, r.fail(path, "", "unclosed environment reference; use ${NAME}, ${NAME:-fallback} or $${...} for a literal")
}
end += j + 2
expression := raw[j+2 : end]
name, fallback, hasDefault := strings.Cut(expression, ":-")
if !validEnvironmentName(name) || strings.Contains(fallback, "${") {
return nil, r.fail(path, "", "invalid environment reference; use ${NAME} or ${NAME:-fallback}; nested references are not supported")
}
r.nodes++
if r.nodes > maxConfigNodes {
return nil, r.fail(path, name, "environment reference count exceeds configuration limit")
}
if !seen[name] {
names = append(names, name)
seen[name] = true
}
value, err := r.lookupValue(name, path)
if err != nil {
return nil, err
}
selected := value.value
if !value.present || selected == "" {
if !hasDefault {
return nil, r.fail(path, name, "required environment variable is unset or empty")
}
selected = fallback
}
if err := appendText(selected); err != nil {
return nil, err
}
i = end + 1
default:
if err := appendText("$"); err != nil {
return nil, err
}
i = j + 1
}
}
if len(names) > 0 {
r.doc.environments[path] = names
return resolvedText(out.String()), nil
}
if xmlText {
return Text(out.String()), nil
}
return out.String(), nil
}

230
config/references_test.go Normal file
View File

@@ -0,0 +1,230 @@
package config
import (
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"sync"
"testing"
"time"
)
type referenceConfig struct {
Port int `json:"port"`
BatchSize int `json:"batchSize"`
Enabled bool `json:"enabled"`
Password string `json:"password"`
Address string `json:"address"`
Timeout time.Duration `json:"timeout"`
Optional *int `json:"optional"`
Labels map[string]string `json:"labels"`
Numbers []int64 `json:"numbers"`
Dynamic any `json:"dynamic"`
}
func (c referenceConfig) Validate() error {
if c.Port < 0 {
return Invalid("port", "must not be negative")
}
return nil
}
func TestExplicitReferencesAcrossFormats(t *testing.T) {
// Delimiters, quotes and newlines belong to the credential, never file syntax.
secret := " DDSFLKJlsf:saf:-123\"\n${DO_NOT_READ}&<>\\ "
sources := map[string]string{
"json": `{"port":"${PORT:-3306}","enabled":"${ENABLED:-true}","password":"${PASSWORD}","address":"https://${HOST}:${PORT:-3306}/v1","timeout":"${TIMEOUT:-3s}","optional":"${ZERO}","numbers":["${BIG}"],"dynamic":"${ZERO}","labels":{"literal":"$${NOT_READ}","empty":"${OPTIONAL:-}","dollars":"$$cost"}}`,
"yaml": "port: \"${PORT:-3306}\"\nenabled: \"${ENABLED:-true}\"\npassword: \"${PASSWORD}\"\naddress: \"https://${HOST}:${PORT:-3306}/v1\"\ntimeout: \"${TIMEOUT:-3s}\"\noptional: \"${ZERO}\"\nnumbers: [\"${BIG}\"]\ndynamic: \"${ZERO}\"\nlabels:\n literal: \"$${NOT_READ}\"\n empty: \"${OPTIONAL:-}\"\n dollars: \"$$cost\"\n",
"xml": `<config><port>${PORT:-3306}</port><enabled>${ENABLED:-true}</enabled><password>${PASSWORD}</password><address>https://${HOST}:${PORT:-3306}/v1</address><timeout>${TIMEOUT:-3s}</timeout><optional>${ZERO}</optional><numbers type="array"><item>${BIG}</item></numbers><dynamic>${ZERO}</dynamic><labels><literal>$${NOT_READ}</literal><empty>${OPTIONAL:-}</empty><dollars>$$cost</dollars></labels></config>`,
}
for ext, source := range sources {
t.Run(ext, func(t *testing.T) {
calls := map[string]int{}
values := map[string]string{"PORT": "0", "ENABLED": "false", "PASSWORD": secret, "HOST": "example.test", "ZERO": "0", "BIG": "9007199254740993"}
l := testLoader(func(name string) (string, bool) { calls[name]++; v, ok := values[name]; return v, ok })
got, err := LoadProject(configFile(t, ext, source), referenceConfig{Port: 77, BatchSize: 100, Enabled: true, Password: "old"}, l)
if err != nil {
t.Fatal(err)
}
if got.Port != 0 || got.BatchSize != 100 || got.Enabled || got.Password != secret || got.Address != "https://example.test:0/v1" || got.Timeout != 3*time.Second || got.Optional == nil || *got.Optional != 0 || got.Numbers[0] != 9007199254740993 || got.Dynamic != "0" {
t.Fatal("binding changed values")
}
if got.Labels["literal"] != "${NOT_READ}" || got.Labels["empty"] != "" || got.Labels["dollars"] != "$cost" {
t.Fatal(got.Labels)
}
for name, n := range calls {
if n != 1 {
t.Fatal("not cached", name, n)
}
}
if calls["DO_NOT_READ"] != 0 || calls["NOT_READ"] != 0 {
t.Fatal("recursive expansion")
}
})
}
}
func TestReferenceFallbackAndFixedValues(t *testing.T) {
for _, present := range []bool{false, true} {
l := testLoader(func(string) (string, bool) { return "", present })
got, err := LoadProject(configFile(t, "yaml", "port: \"${PORT:-3306}\"\npassword: \"${PASSWORD:-DDSFLKJlsf:saf:-123}\""), referenceConfig{}, l)
if err != nil || got.Port != 3306 || got.Password != "DDSFLKJlsf:saf:-123" {
t.Fatal(got, err)
}
for _, ext := range []string{"json", "yaml", "xml"} {
raw := `{"password":"${PASSWORD}"}`
if ext == "yaml" {
raw = "password: '${PASSWORD}'"
}
if ext == "xml" {
raw = "<config><password>${PASSWORD}</password></config>"
}
path := configFile(t, ext, raw)
_, err = LoadProject(path, referenceConfig{Password: "initial"}, l)
var d *Error
if !errors.As(err, &d) || d.Env != "PASSWORD" || d.File != path || d.Field != "password" || d.Line < 1 {
t.Fatal("missing required-reference diagnostic", err)
}
}
}
calls := 0
l := testLoader(func(string) (string, bool) { calls++; return "999", true })
got, err := LoadProject(configFile(t, "json", `{"port":0,"enabled":false,"password":""}`), referenceConfig{Port: 77, BatchSize: 100, Enabled: true, Password: "old"}, l)
if err != nil || got.Port != 0 || got.BatchSize != 100 || got.Enabled || got.Password != "" || calls != 0 {
t.Fatal(got, err, calls)
}
got, err = LoadProject(configFile(t, "yaml", "password: '${VALUE}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return " ", true }))
if err != nil || got.Password != " " {
t.Fatal("trimmed credential", err)
}
}
func TestReferenceDiagnosticsAndScalarLimits(t *testing.T) {
for _, value := range []string{"private-not-int", "92233720368547758080", "false", "null", "-1"} {
path := configFile(t, "yaml", "port: '${PORT}'")
_, err := LoadProject(path, referenceConfig{}, testLoader(func(string) (string, bool) { return value, true }))
var d *Error
if !errors.As(err, &d) || d.File != path || d.Env != "PORT" || d.Field != "port" || d.Line != 1 || d.Column < 1 || strings.Contains(err.Error(), value) {
t.Fatalf("invalid diagnostic: %v", err)
}
}
for _, raw := range []string{"${BAD-123}", "${}", "${X", "${1BAD}", "${BAD NAME}", "${X:-${Y}}"} {
encoded, _ := json.Marshal(map[string]string{"password": raw})
if _, err := LoadProject(configFile(t, "json", string(encoded)), referenceConfig{}, testLoader(noEnvironment)); err == nil {
t.Fatal("invalid syntax accepted", raw)
}
}
for _, value := range []string{string([]byte{0xff}), strings.Repeat("x", maxConfigBytes+1)} {
if _, err := LoadProject(configFile(t, "yaml", "password: '${SECRET}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return value, true })); err == nil {
t.Fatal("unbounded/invalid environment")
}
}
path := configFile(t, "yaml", "labels:\n a: '${BIG}'\n b: '${BIG}'\n c: '${BIG}'\n d: '${BIG}'\n e: '${BIG}'")
if _, err := LoadProject(path, referenceConfig{}, testLoader(func(string) (string, bool) { return strings.Repeat("x", maxConfigBytes), true })); err == nil {
t.Fatal("expansion budget bypassed")
}
// An environment value cannot inject an array/object or coerce dynamic null.
for _, raw := range []string{"[1,2]", "{\"x\":1}", "null"} {
_, err := LoadProject(configFile(t, "yaml", "numbers: '${VALUE}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return raw, true }))
if err == nil {
t.Fatal("structure injected")
}
}
}
func TestReferenceCacheIsolationAndRemovedRules(t *testing.T) {
path := configFile(t, "yaml", "password: '${TOKEN}'")
for i := 0; i < 24; i++ {
t.Run(fmt.Sprint(i), func(t *testing.T) {
t.Parallel()
v := fmt.Sprint(i)
l := testLoader(func(string) (string, bool) { return v, true })
for n := 0; n < 4; n++ {
got, err := LoadProject(path, referenceConfig{}, l)
if err != nil || got.Password != v {
t.Fatal("load leaked", err)
}
}
})
}
}
func TestReferenceLookupPerLoadAndConcurrentLoader(t *testing.T) {
var mu sync.Mutex
reads := 0
l := testLoader(func(string) (string, bool) { mu.Lock(); defer mu.Unlock(); reads++; return "secret", true })
path := configFile(t, "yaml", "password: '${TOKEN}${TOKEN}'")
var wg sync.WaitGroup
for i := 0; i < 24; i++ {
wg.Add(1)
go func() {
defer wg.Done()
got, err := LoadProject(path, referenceConfig{}, l)
if err != nil || got.Password != "secretsecret" {
t.Error(err)
}
}()
}
wg.Wait()
if reads != 24 {
t.Fatal(reads)
}
}
func TestRemovedEnvTagsAndFieldSuggestion(t *testing.T) {
type old struct {
Value int `json:"value" envDefault:"9"`
}
if _, err := LoadProject("", old{}); err == nil || !strings.Contains(err.Error(), "removed") {
t.Fatal(err)
}
type oldSecret struct {
Token string `json:"-" env:"TOKEN"`
}
if _, err := DecodeFile(configFile(t, "json", `{}`), oldSecret{}); err == nil {
t.Fatal("old env tag ignored")
}
_, err := LoadProject(configFile(t, "yaml", "batchSiz: 10"), referenceConfig{})
if err == nil || !strings.Contains(err.Error(), `did you mean "batchSize"`) {
t.Fatal(err)
}
t.Setenv("GO_OMNI_BATCH_SIZE", "999")
got, err := LoadProject("", referenceConfig{BatchSize: 100})
if err != nil || got.BatchSize != 100 {
t.Fatal("implicit overlay", err)
}
}
func FuzzConfigurationReferences(f *testing.F) {
for _, s := range []string{"${X}", "$${X}", "${X:-DDS:abc:-123}", "${X:-}", "https://${X}/${X}", "${X:-${Y}}"} {
f.Add(s, " secret${NO}\n\" ")
}
f.Fuzz(func(t *testing.T, raw, value string) {
if len(raw) > 8192 || len(value) > 8192 {
return
}
doc := Document{Value: map[string]any{"password": raw}}
resolved, err := resolveDocument(doc, func(string) (string, bool) { return value, true })
if err != nil {
return
}
_, _ = bindDocument(resolved, referenceConfig{})
})
}
func BenchmarkExplicitConfiguration(b *testing.B) {
for _, refs := range []bool{false, true} {
b.Run(fmt.Sprint(refs), func(b *testing.B) {
source := `{"port":3306,"enabled":false,"password":"fixed","batchSize":0}`
if refs {
source = `{"port":"${PORT:-3306}","enabled":"${ENABLED:-false}","password":"${PASSWORD}","batchSize":0}`
}
path := b.TempDir() + "/config.json"
if err := os.WriteFile(path, []byte(source), 0600); err != nil {
b.Fatal(err)
}
l := testLoader(func(k string) (string, bool) { return "private", k == "PASSWORD" })
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := LoadProject(path, referenceConfig{BatchSize: 100}, l); err != nil {
b.Fatal(err)
}
}
})
}
}

51
config/suggestions.go Normal file
View File

@@ -0,0 +1,51 @@
package config
import (
"reflect"
"strconv"
"strings"
)
// Suggest only a unique, nearby declared field. Input values are never used.
func unknownFieldReason(name string, fields map[string]reflect.StructField) string {
if len(name) < 3 || len(name) > 128 {
return "unknown field"
}
best, match, tied := 3, "", false
for candidate := range fields {
if len(candidate) > 128 {
continue
}
d := fieldDistance(strings.ToLower(name), strings.ToLower(candidate))
if d < best {
best, match, tied = d, candidate, false
} else if d == best {
tied = true
}
}
if match != "" && !tied && best <= 2 {
return "unknown field; did you mean " + strconv.Quote(match) + "?"
}
return "unknown field"
}
func fieldDistance(a, b string) int {
prev := make([]int, len(b)+1)
for i := range prev {
prev[i] = i
}
for i := 0; i < len(a); i++ {
left, diagonal := i+1, prev[0]
prev[0] = left
for j := 0; j < len(b); j++ {
cost := 0
if a[i] != b[j] {
cost = 1
}
old := prev[j+1]
left = min(left+1, old+1, diagonal+cost)
prev[j+1] = left
diagonal = old
}
}
return prev[len(b)]
}

View File

@@ -6,7 +6,7 @@
`database` 包提供基于标准库 `database/sql` 的连接池生命周期,不绑定 ORM 或驱动:
- `Config` / `ConfigFromEnv`:连接池参数和 `GO_OMNI_DB_*` 环境变量。
- `Config`:驱动、DSN 与连接池参数;嵌入项目配置,通过文件显式引用变量。
- `Open` / `OpenConnector`:通过 DSN 或标准 Connector 创建连接池、设置连接上限并在返回前执行 `PingContext`。
- `Health`:用于就绪检查,等待连接和 Ping 均受 `PingTimeout` 限制。
- `InTx`:统一事务提交和回滚;工作函数返回错误或 panic 时回滚。
@@ -17,11 +17,12 @@
```go
import (
"context"
"git.yuntuo.hk/k2safe/go-omni/config"
_ "github.com/go-sql-driver/mysql"
"git.yuntuo.hk/k2safe/go-omni/database"
)
cfg, err := database.ConfigFromEnv("GO_OMNI_DB_")
cfg, err := config.LoadProject("etc/database.yaml", database.DefaultConfig())
if err != nil {
return err
}
@@ -32,19 +33,15 @@ if err != nil {
defer db.Close()
```
连接池相关环境变量:
对应文件(框架不规定变量名):
```text
GO_OMNI_DB_DRIVER=mysql
GO_OMNI_DB_DSN=user:password@tcp(127.0.0.1:3306)/app?parseTime=true
GO_OMNI_DB_MAX_OPEN_CONNS=10
GO_OMNI_DB_MAX_IDLE_CONNS=5
GO_OMNI_DB_CONN_MAX_LIFETIME=30m
GO_OMNI_DB_CONN_MAX_IDLE_TIME=5m
GO_OMNI_DB_PING_TIMEOUT=5s
```yaml
driver: mysql
dsn: "${DB_DSN}"
maxOpenConns: "${DB_MAX_OPEN:-10}"
```
使用 `ConfigFromEnv` / `Open` 时 `DSN` 必填,环境变量入口的驱动默认是 `mysql`。数据库建库、迁移、表结构和具体 Repository 属于业务项目或可选模块,不会被核心包自动执行。
Open 使用的 Driver、DSN 必填。建库、迁移、表结构和业务 Repository 由项目负责。v0.22 删除 ConfigFromEnv/ApplyEnv;不再隐式扫描数据库变量。
`MaxIdleConns=0` 会禁用空闲连接。初始化 Ping 失败会关闭已创建的连接池。
@@ -62,12 +59,9 @@ python3 scripts/verifyDatabase.py --version v0.2.0
结束时清理容器、匿名数据卷和临时项目。常规 `go test ./...` 不依赖 Docker。
## 项目配置叠加
## 项目配置
`database.ApplyEnv(cfg, prefix)` 在已有文件/默认配置上覆盖非空环境变量,保留未覆盖的池参数。
`Config` 的 JSON/YAML 字段使用 lowerCamelCase,`DSN` 使用 `json:"-"`,连接密钥从环境变量加载。
API 脚手架 `--database mysql` 使用此流程,并要求正的 `maxOpenConns`,提供数据库 readiness 与幂等清理。
具体表结构、迁移和 Repository 留在业务项目,参考 `examples/userService`。
配置结构作为项目 Config 的字段,Default() 初始化连接池,文件按需覆盖。DSN 可使用 `dsn: "${DB_DSN}"`;database.Config 的 JSON 序列化不输出 DSN 原值。不要打印原始配置对象。当前 API 脚手架使用结构化 MySQL 适配器,由 svc 统一初始化与关闭。
## Connector 接入(v0.9.1-preview)
@@ -103,7 +97,7 @@ func Open(ctx context.Context, address, user, password, name string,
}
```
调用时以 `database.DefaultPoolConfig()` 为起点,按项目需要调整。已有 `database.Config` 可调用 `.PoolConfig()` 提取池设置;`ConfigFromEnv` / `ApplyEnv` 仍是 DSN 入口,不用为 Connector 填假 DSN。`PoolConfig` 支持通过 `config.DecodeFile` 加载项目的 JSON/YAML 嵌套配置,环境覆盖由项目按自己的驱动配置组织。
调用时以 `database.DefaultPoolConfig()` 为起点,按项目需要调整。已有 `database.Config` 可调用 `.PoolConfig()` 提取池设置;Connector 不需要填写 DSN。`PoolConfig` 支持通过 `config.DecodeFile` 加载项目的 JSON/YAML 嵌套配置,环境引用由项目在文件里显式声明。
| PoolConfig 字段 | 默认值 / 含义 |
| --- | --- |

View File

@@ -2,19 +2,27 @@ package database
import (
"encoding/json"
"git.yuntuo.hk/k2safe/go-omni/config"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestApplyEnvPreservesFilePoolAndHidesDSN(t *testing.T) {
func TestExplicitConfigPreservesPoolAndHidesDSN(t *testing.T) {
cfg := DefaultConfig()
cfg.Driver = "mysql"
path := filepath.Join(t.TempDir(), "db.yaml")
if err := os.WriteFile(path, []byte("dsn: \"${OMNI_TEST_DSN}\"\nmaxIdleConns: \"${OMNI_TEST_MAX_IDLE_CONNS}\"\nmaxOpenConns: \"${OMNI_TEST_MAX_OPEN_CONNS:-3}\""), 0600); err != nil {
t.Fatal(err)
}
cfg.MaxOpenConns = 3
cfg.MaxIdleConns = 1
cfg.PingTimeout = time.Second
t.Setenv("OMNI_TEST_DSN", "user:secret@tcp(localhost:3306)/test")
t.Setenv("OMNI_TEST_MAX_IDLE_CONNS", "0")
got, err := ApplyEnv(cfg, "OMNI_TEST_")
got, err := config.LoadProject(path, cfg)
if err != nil {
t.Fatal(err)
}
@@ -29,7 +37,7 @@ func TestApplyEnvPreservesFilePoolAndHidesDSN(t *testing.T) {
t.Fatalf("unsafe config encoding: %s", data)
}
t.Setenv("OMNI_TEST_MAX_OPEN_CONNS", "bad")
if _, err = ApplyEnv(cfg, "OMNI_TEST_"); err == nil {
if _, err = config.LoadProject(path, cfg); err == nil {
t.Fatal("invalid override accepted")
}
}

View File

@@ -5,17 +5,16 @@ package database
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"os"
"strconv"
"strings"
"time"
)
type Config struct {
Driver string `json:"driver"`
DSN string `json:"-"`
DSN string `json:"dsn"`
MaxOpenConns int `json:"maxOpenConns"`
MaxIdleConns int `json:"maxIdleConns"`
ConnMaxLifetime time.Duration `json:"connMaxLifetime"`
@@ -23,6 +22,14 @@ type Config struct {
PingTimeout time.Duration `json:"pingTimeout"`
}
// MarshalJSON omits the connection credential from serialized diagnostics.
func (c Config) MarshalJSON() ([]byte, error) {
type plain Config
safe := plain(c)
safe.DSN = ""
return json.Marshal(safe)
}
func DefaultConfig() Config { return DefaultPoolConfig().config() }
// PoolConfig returns driver-independent connection pool settings.
@@ -34,49 +41,6 @@ func (c Config) PoolConfig() PoolConfig {
}
}
// ConfigFromEnv reads GO_OMNI_DB_* style settings. DSN is required.
func ConfigFromEnv(prefix string) (Config, error) {
return ApplyEnv(DefaultConfig(), prefix)
}
// ApplyEnv overlays nonempty environment settings on file/default configuration.
// DSN is environment-only when Config is decoded from JSON or YAML.
func ApplyEnv(cfg Config, prefix string) (Config, error) {
prefix = strings.TrimSpace(prefix)
if prefix == "" {
prefix = "GO_OMNI_DB_"
}
if value := strings.TrimSpace(os.Getenv(prefix + "DRIVER")); value != "" {
cfg.Driver = value
}
if cfg.Driver == "" {
cfg.Driver = "mysql"
}
if value := strings.TrimSpace(os.Getenv(prefix + "DSN")); value != "" {
cfg.DSN = value
}
if cfg.DSN == "" {
return Config{}, fmt.Errorf("%sDSN is required", prefix)
}
var err error
if cfg.MaxOpenConns, err = envInt(prefix+"MAX_OPEN_CONNS", cfg.MaxOpenConns); err != nil {
return Config{}, err
}
if cfg.MaxIdleConns, err = envInt(prefix+"MAX_IDLE_CONNS", cfg.MaxIdleConns); err != nil {
return Config{}, err
}
if cfg.ConnMaxLifetime, err = envDuration(prefix+"CONN_MAX_LIFETIME", cfg.ConnMaxLifetime); err != nil {
return Config{}, err
}
if cfg.ConnMaxIdleTime, err = envDuration(prefix+"CONN_MAX_IDLE_TIME", cfg.ConnMaxIdleTime); err != nil {
return Config{}, err
}
if cfg.PingTimeout, err = envDuration(prefix+"PING_TIMEOUT", cfg.PingTimeout); err != nil {
return Config{}, err
}
return cfg, cfg.Validate()
}
func (c Config) Validate() error {
if strings.TrimSpace(c.Driver) == "" {
return errors.New("database driver is required")
@@ -165,27 +129,3 @@ func (db *DB) InTx(ctx context.Context, options *sql.TxOptions, work func(*sql.T
}
return nil
}
func envInt(key string, fallback int) (int, error) {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback, nil
}
parsed, err := strconv.Atoi(value)
if err != nil {
return 0, fmt.Errorf("%s must be an integer: %w", key, err)
}
return parsed, nil
}
func envDuration(key string, fallback time.Duration) (time.Duration, error) {
value := strings.TrimSpace(os.Getenv(key))
if value == "" {
return fallback, nil
}
parsed, err := time.ParseDuration(value)
if err != nil {
return 0, fmt.Errorf("%s must be a duration: %w", key, err)
}
return parsed, nil
}

View File

@@ -2,54 +2,10 @@ package database
import (
"context"
"os"
"strings"
"testing"
"time"
)
func TestConfigFromEnv(t *testing.T) {
keys := []string{
"GO_OMNI_DB_DRIVER", "GO_OMNI_DB_DSN", "GO_OMNI_DB_MAX_OPEN_CONNS",
"GO_OMNI_DB_MAX_IDLE_CONNS", "GO_OMNI_DB_CONN_MAX_LIFETIME",
"GO_OMNI_DB_CONN_MAX_IDLE_TIME", "GO_OMNI_DB_PING_TIMEOUT",
}
for _, key := range keys {
t.Setenv(key, "")
_ = os.Unsetenv(key)
}
t.Setenv("GO_OMNI_DB_DSN", "root:secret@tcp(127.0.0.1:3306)/demo")
t.Setenv("GO_OMNI_DB_MAX_OPEN_CONNS", "20")
t.Setenv("GO_OMNI_DB_MAX_IDLE_CONNS", "8")
t.Setenv("GO_OMNI_DB_CONN_MAX_LIFETIME", "2m")
t.Setenv("GO_OMNI_DB_PING_TIMEOUT", "750ms")
cfg, err := ConfigFromEnv("")
if err != nil {
t.Fatal(err)
}
if cfg.Driver != "mysql" || cfg.DSN == "" || cfg.MaxOpenConns != 20 || cfg.MaxIdleConns != 8 || cfg.ConnMaxLifetime != 2*time.Minute || cfg.PingTimeout != 750*time.Millisecond {
t.Fatalf("unexpected config: %+v", cfg)
}
}
func TestConfigFromEnvRejectsInvalidValues(t *testing.T) {
t.Setenv("GO_OMNI_DB_DSN", "demo")
t.Setenv("GO_OMNI_DB_MAX_OPEN_CONNS", "bad")
if _, err := ConfigFromEnv(""); err == nil || !strings.Contains(err.Error(), "MAX_OPEN_CONNS") {
t.Fatalf("expected invalid pool limit error, got %v", err)
}
t.Setenv("GO_OMNI_DB_MAX_OPEN_CONNS", "2")
t.Setenv("GO_OMNI_DB_MAX_IDLE_CONNS", "3")
if _, err := ConfigFromEnv(""); err == nil || !strings.Contains(err.Error(), "max idle") {
t.Fatalf("expected pool relationship error, got %v", err)
}
t.Setenv("GO_OMNI_DB_MAX_IDLE_CONNS", "1")
t.Setenv("GO_OMNI_DB_PING_TIMEOUT", "0s")
if _, err := ConfigFromEnv(""); err == nil || !strings.Contains(err.Error(), "timeouts") {
t.Fatalf("expected timeout error, got %v", err)
}
}
func TestOpenValidatesBeforeLoadingDriver(t *testing.T) {
_, err := Open(context.Background(), Config{Driver: "", DSN: "demo"})
if err == nil || !strings.Contains(err.Error(), "driver") {

View File

@@ -10,13 +10,14 @@
go-omni new --database mysql --module example.com/billing ./billing
```
`etc/config.yaml` 使用结构化配置,环境覆盖优先:
`etc/config.yaml` 使用结构化配置;需要环境变量时在字段值明确写引用,固定值不被覆盖:
```yaml
database:
host: db.example.com
port: 3306
user: app
password: "${DB_PASSWORD}"
database: billing
parseTime: true
location: UTC
@@ -37,7 +38,7 @@ database:
pingTimeout: 5s
```
通过 `GO_OMNI_DB_PASSWORD` 传密码;还有 HOST、PORT、USER、DATABASE、TLS_CA_FILE、TLS_CERT_FILE、TLS_KEY_FILE、POOL_MAX_OPEN_CONNS 等。业务也可自定义 env 标签。新生成服务不再读取 DSN;独立数据库迁移 CLI 和手写 `database.Open/OpenConnector` 的原接口保留。
示例通过文件里的 `${DB_PASSWORD}` 读取密码;HOST、PORT、TLS 和池参数同样按需在文件写引用,不自动扫描变量、不使用 env 标签。新生成服务使用结构化连接参数;独立迁移 CLI 的显式 DSN 参数及 Open/OpenConnector 接口保留。
默认启用 TLS 并校验系统信任链与主机名;私有 CA 用 caFile,双向 TLS 额外设置 certFile/keyFile。只有明确设置 `tls.mode: disabled` 才使用明文。Unix socket 用 socket,若在 socket 上使用 TLS,必须提供 serverName。不会注册全局 TLS 配置或全局驱动 Logger。

View File

@@ -28,7 +28,7 @@ type Config struct {
Port int `json:"port"`
Socket string `json:"socket"`
User string `json:"user"`
Password string `json:"password" env:",secret"`
Password string `json:"password"`
Database string `json:"database"`
ParseTime bool `json:"parseTime"`
Location string `json:"location"`

View File

@@ -10,11 +10,13 @@ DEMO_CLIENT_TIMEOUT=500ms DEMO_MAX_ITEMS=0 go run ./examples/configuration
DEMO_MAX_ITEMS=wrong go run ./examples/configuration
```
前三次都输出 `timeout=2s maxItems=100 allowed=[json xml]`。环境覆盖允许显式零值。
前三次都输出 `timeout=2s maxItems=100 allowed=[json xml]`。文件显式引用允许零值。
最后一条输出定位 `maxItems` / `DEMO_MAX_ITEMS` 的安全错误,不打印输入原值。
也可 `-config /path/to/app.yaml` 读取自己的文件。示例没有建立网络连接。
`Config` 定义字段,`envDefault` 定义缺省值,`Validate` 用 `config.Invalid` 返回字段与公开原因。
`Config` 定义字段,`Default()` 定义初始值,`Validate` 用 `config.Invalid` 返回字段与公开原因。
文件中的 `${NAME}` 必填,`${NAME:-value}` 支持空/未设置的备用值,`$${NAME}` 保留字面量;字符串不裁剪,也不二次展开。没有出现在文件引用中的变量不会被读取。
`newLoader` 额外注册 `.custom`(演示复用 JSON decoder);同一个 `load` 可读取所有格式。
JSON/YAML/XML 直接可用,无需注册。新增格式只修改解析器注册,不改业务结构或校验。

View File

@@ -1,5 +1 @@
{
"client": {"endpoint": "https://example.com", "timeout": "2s"},
"maxItems": 100,
"allowed": ["json", "xml"]
}
{"client":{"endpoint":"https://${DEMO_HOST:-example.com}","timeout":"${DEMO_CLIENT_TIMEOUT:-2s}","token":"${DEMO_CLIENT_TOKEN:-}"},"maxItems":"${DEMO_MAX_ITEMS:-100}","allowed":["json","xml"]}

View File

@@ -1,8 +1 @@
<config>
<client>
<endpoint>https://example.com</endpoint>
<timeout>2s</timeout>
</client>
<maxItems>100</maxItems>
<allowed type="array"><item>json</item><item>xml</item></allowed>
</config>
<config><client><endpoint>https://${DEMO_HOST:-example.com}</endpoint><timeout>${DEMO_CLIENT_TIMEOUT:-2s}</timeout><token>${DEMO_CLIENT_TOKEN:-}</token></client><maxItems>${DEMO_MAX_ITEMS:-100}</maxItems><allowed type="array"><item>json</item><item>xml</item></allowed></config>

View File

@@ -1,5 +1,6 @@
client:
endpoint: https://example.com
timeout: 2s
maxItems: 100
endpoint: "https://${DEMO_HOST:-example.com}"
timeout: "${DEMO_CLIENT_TIMEOUT:-2s}"
token: "${DEMO_CLIENT_TOKEN:-}"
maxItems: "${DEMO_MAX_ITEMS:-100}"
allowed: [json, xml]

View File

@@ -17,16 +17,18 @@ var examples embed.FS
type ClientConfig struct {
Endpoint string `json:"endpoint"`
Timeout time.Duration `json:"timeout" envDefault:"3s"`
// This field is deliberately unavailable to configuration files.
Token string `json:"-" env:"DEMO_CLIENT_TOKEN,secret"`
Timeout time.Duration `json:"timeout"`
// The file chooses a fixed value or an explicit environment reference.
Token string `json:"token"`
}
type Config struct {
Client ClientConfig `json:"client"`
MaxItems int `json:"maxItems" envDefault:"20"`
MaxItems int `json:"maxItems"`
Allowed []string `json:"allowed"`
}
func Default() Config { return Config{Client: ClientConfig{Timeout: 3 * time.Second}, MaxItems: 20} }
func (c Config) Validate() error {
if c.Client.Timeout <= 0 {
return config.Invalid("client.timeout", "must be positive")
@@ -48,7 +50,7 @@ func load(path string) (Config, error) {
if err != nil {
return Config{}, err
}
return config.LoadProject(path, Config{}, config.EnvOptions{Prefix: "DEMO_", Empty: "reject"}, loader)
return config.LoadProject(path, Default(), loader)
}
func run() error {
path := flag.String("config", "", "external configuration file")

View File

@@ -0,0 +1,12 @@
# HTTP Client 示例
```sh
go run ./examples/httpClient
```
只创建本机临时 HTTP 服务,不访问业务系统、数据库或公网;结束后关闭连接和监听器。
示例创建并复用一个 Client,显式直连,配置连接池/阶段超时,通过 JSON 选项传入 Authorization 和 Accept,并按单次请求收紧响应上限。
输出 `authenticated request succeeded`。示例 token 是固定演示字符串,实际项目从配置或请求上下文取得认证信息。
完整参数、代理规则、流式调用、生命周期、重试边界和兼容说明见 [httpc](../../httpc/README.md)。

View File

@@ -0,0 +1,57 @@
// Run with go run ./examples/httpClient. All HTTP traffic stays on loopback.
package main
import (
"context"
"encoding/json"
"fmt"
"net/http"
"net/http/httptest"
"os"
"time"
"git.yuntuo.hk/k2safe/go-omni/httpc"
)
type result struct {
Message string `json:"message"`
}
func run() error {
upstream := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer example-token" {
w.WriteHeader(http.StatusUnauthorized)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(result{Message: "authenticated request succeeded"})
}))
defer upstream.Close()
client, err := httpc.New(httpc.Config{
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
Timeout: 3 * time.Second, DialTimeout: time.Second, TLSHandshakeTimeout: time.Second,
ResponseHeaderTimeout: time.Second, IdleConnTimeout: 30 * time.Second,
MaxConnsPerHost: 16, MaxIdleConnsPerHost: 8, MaxIdleConns: 32,
MaxResponseBytes: 1 << 20, DisableLogging: true,
})
if err != nil {
return err
}
defer client.Close()
var output result
if err = client.JSON(context.Background(), http.MethodGet, upstream.URL, nil, &output,
httpc.WithHeader("Authorization", "Bearer example-token"),
httpc.WithHeaders(http.Header{"Accept": []string{"application/json"}}),
httpc.WithMaxResponseBytes(64<<10),
); err != nil {
return err
}
fmt.Println(output.Message)
return nil
}
func main() {
if err := run(); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
}

View File

@@ -0,0 +1,9 @@
package main
import "testing"
func TestExample(t *testing.T) {
if err := run(); err != nil {
t.Fatal(err)
}
}

View File

@@ -1,6 +1,6 @@
{
"generatorConfig": true,
"templateVersion": "v0.20.0-preview",
"templateVersion": "v0.22.0-preview",
"profile": {
"database": "none",
"storage": "none",
@@ -12,7 +12,7 @@
"entry": "api/managed.api",
"generated": {
"docs/openapi.json": "0370fd6bd6a387e2ef21f6b42b009ce66749846b8bea3d481009a34f3af039a5",
"internal/config/loadGen.go": "5d630b894abdbbd66ddfa7d7d30f7573fc4ff995020a32a9f9157cd4af3a5655",
"internal/config/loadGen.go": "d85e3123e4c96c95cb0886e60e4760230426efca4d2bbbfc981822da08b90d13",
"internal/handler/appGen.go": "f5251b4dc33e2312b0bdb71ead70d2171977d851265e124f640458d705a2cc9d",
"internal/handler/demo/handlersGen.go": "3c76d85dbe89eb5127d9d134dc625203ecf9719dd11d174c1f335691b99e1375",
"internal/handler/openapiGen.go": "f24cab3abbb8557a3fc4dd93fba23cc0b093f3e0c944d3ba1bd5b0eeff2d96f3",
@@ -22,7 +22,7 @@
"internal/svc/runtimeGen.go": "29f6afb6614881935099c1845ecb3f7902ee773caa077940f566cb8dfdbcdbb1",
"internal/types/dtoManagedd28645a2af1cGen.go": "7586e86fa5d67051283178cc578c890f6c02fd2d5c383170a71a39c2e77ce344",
"internal/types/typesGen.go": "372d67f23a313b3e77d2a3b7a236315db9721bc63350bf1f71db1293e57570d1",
"main.go": "64e26d97de7e7c6c17f47186785edc73d43c23d6c20b50bfc7fff6ef2e914f17"
"main.go": "10cb6a84b2eb1b68dc10c2dca949c9dd7c68a9f78a8e6d8f56ce9336d4b2d125"
},
"contract": {
"routes": {

View File

@@ -46,6 +46,6 @@ Configuration, API contracts, svc fields/Init/AppOptions, Logic, Repository and
middleware implementations remain developer-owned. Regeneration preserves them.
Use `go-omni api format` to format local contracts, then `api generate` and
`api generate --check`. See the [framework development guide](../../DEVELOPMENT.md)
for array/null responses, typed multipart, full configuration environment overrides,
for array/null responses, typed multipart, explicit configuration references and -check-config,
shared contracts and v0.13 upgrade steps. This example keeps its raw callback,
streaming upload and download to exercise those existing extension points.

View File

@@ -1,16 +1,3 @@
# Non-secret settings only. Environment variables override these values.
server:
address: '127.0.0.1:8080'
enableLogger: true
slowRequestThreshold: 500ms
shutdownTimeout: 10s
endpoints:
liveness: {enabled: true, path: /health/live, responseMode: app}
readiness: {enabled: true, path: /health/ready, responseMode: app}
openapi: {enabled: true, path: /openapi.json}
log:
service: "example.com/managed"
level: info
format: json
output: stderr
redactKeys: []
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
transport: "${GO_OMNI_TRANSPORT:-nethttp}"

View File

@@ -2,7 +2,7 @@ module example.com/managed
go 1.22.0
require git.yuntuo.hk/k2safe/go-omni v0.20.0-preview
require git.yuntuo.hk/k2safe/go-omni v0.22.0-preview
require (
github.com/gabriel-vasile/mimetype v1.4.3 // indirect

View File

@@ -3,14 +3,12 @@ package config
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
// Load is maintained by the generator. Extend Config/Default/Validate in
// config.go, or NewLoader in loader.go; do not duplicate the loading pipeline.
// Load resolves explicit file references, binds onto Default and validates.
// Ordinary projects only edit Config, Default, Validate and the selected file.
func Load(path string) (Config, error) {
loader, err := NewLoader()
if err != nil {
return Config{}, err
}
return omniConfig.LoadProject(path, Default(), omniConfig.EnvOptions{
Prefix: "GO_OMNI_", Empty: "ignore", Aliases: omniConfig.ServerEnvAliases("GO_OMNI_", "server"),
}, loader)
return omniConfig.LoadProject(path, Default(), loader)
}

View File

@@ -17,8 +17,15 @@ import (
)
func main() {
path := flag.String("config", "etc/config.yaml", "configuration file; empty uses defaults and environment")
path := flag.String("config", "etc/config.yaml", "configuration file; empty uses only Default()")
check := flag.Bool("check-config", false, "validate configuration and exit without starting dependencies")
flag.Parse()
if *check {
if _, err := config.Load(*path); err != nil {
log.Fatal(err)
}
return
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := run(ctx, *path); err != nil {

View File

@@ -7,7 +7,7 @@ go run ./examples/modules
```
默认监听 `127.0.0.1:8080`,使用仅限开发的 `Bearer dev-token`。
配置 `GO_OMNI_TRANSPORT=native` 可直接切换自研网络层,TLS 配置与主框架一致。
默认读取 `examples/modules/config.yaml`,也可用 `-config` 选择一个其他文件。文件显式引用了 `GO_OMNI_TRANSPORT`,设置为 native 可切换自研网络层;TLS 在文件的 `tls.certFile` / `tls.keyFile` 中配置。此示例直接加载框架 Config,没有生成项目的 server 外层。
```sh
curl -H 'Authorization: Bearer dev-token' http://127.0.0.1:8080/api/catalog

View File

@@ -0,0 +1,2 @@
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
transport: "${GO_OMNI_TRANSPORT:-nethttp}"

View File

@@ -5,6 +5,7 @@ import (
"context"
"encoding/json"
"errors"
"flag"
"log"
"os"
"os/signal"
@@ -106,13 +107,11 @@ func newApp(settings omni.Config, token, upstream string) (application *omni.App
}
func run(ctx context.Context) error {
settings, err := omni.LoadConfig("")
settings, err := omni.LoadConfig(*configPath)
if err != nil {
return err
}
if os.Getenv("GO_OMNI_HTTP_ADDR") == "" {
settings.Address = "127.0.0.1:8080"
}
token := os.Getenv("GO_OMNI_DEV_TOKEN")
if token == "" {
token = "dev-token"
@@ -124,7 +123,10 @@ func run(ctx context.Context) error {
return application.RunContext(ctx)
}
var configPath = flag.String("config", "examples/modules/config.yaml", "configuration file")
func main() {
flag.Parse()
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := run(ctx); err != nil {

View File

@@ -28,7 +28,7 @@ func TestModulesConfiguredTransports(t *testing.T) {
io.WriteString(w, `{"name":"upstream catalog"}`)
}))
defer upstream.Close()
settings, err := omni.LoadConfig("MODULES_TEST_")
settings, err := omni.LoadConfig("")
if err != nil {
t.Fatal(err)
}

View File

@@ -88,16 +88,9 @@ func newApp(cfg omni.Config, d dependencies) (*omni.App, error) {
return app, nil
}
func run(ctx context.Context, configFile, root string) error {
var cfg config.Config
var err error
if configFile != "" {
cfg, err = config.LoadFile(configFile, "")
} else {
cfg, err = config.LoadFromEnv("")
if os.Getenv("GO_OMNI_HTTP_ADDR") == "" {
cfg.Address = "127.0.0.1:8080"
}
}
defaults := config.Default()
defaults.Address = "127.0.0.1:8080"
cfg, err := config.LoadProject(configFile, defaults)
if err != nil {
return err
}

View File

@@ -10,10 +10,10 @@
go run ./examples/reference
```
默认监听 `127.0.0.1:8080`,可通过环境变量调整:
默认读取 `examples/reference/config.yaml`,监听 `127.0.0.1:8080`。文件已显式引用地址变量:
```sh
GO_OMNI_ADDR=127.0.0.1:8081 go run ./examples/reference
GO_OMNI_HTTP_ADDR=127.0.0.1:8081 go run ./examples/reference
```
如果终端提示找不到 `go`,需要先把 Go 的 `bin` 目录加入 `PATH`。不要与 `examples/basic` 同时占用同一个端口。
@@ -133,14 +133,10 @@ curl -i http://127.0.0.1:8080/api/v1/users/1
- 列表按 ID 排序、返回总数,空列表使用 `[]`。
- 使用传入的 `context.Context` 执行查询,保留创建时间,修改时更新时间。
MySQL 运行配置通过环境变量传入:
此手写示例显式读取 `GO_OMNI_DB_DSN`;驱动固定 mysql,连接池使用 database.DefaultConfig()。项目可将数据库配置放入自己的 Config 并通过 LoadProject 统一加载(见 [配置指南](../../CONFIG.md)),框架不再自动读取数据库环境变量:
```sh
export GO_OMNI_DB_DRIVER=mysql
export GO_OMNI_DB_DSN='root:password@tcp(127.0.0.1:3306)/go_omni?parseTime=true&loc=UTC'
export GO_OMNI_DB_MAX_OPEN_CONNS=10
export GO_OMNI_DB_MAX_IDLE_CONNS=5
export GO_OMNI_DB_PING_TIMEOUT=5s
go run ./examples/reference
```

View File

@@ -0,0 +1,9 @@
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
transport: "${GO_OMNI_TRANSPORT:-nethttp}"
tls:
certFile: "${GO_OMNI_TLS_CERT_FILE:-}"
keyFile: "${GO_OMNI_TLS_KEY_FILE:-}"
log:
service: reference-api
environment: local
output: stdout

View File

@@ -2,6 +2,7 @@ package main
import (
"context"
"flag"
"log"
"os"
"os/signal"
@@ -42,6 +43,7 @@ func registerRoutes(application *omni.App, users service.UserRepository, logger
}
func main() {
flag.Parse()
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
if err := runReference(ctx); err != nil {
@@ -49,29 +51,9 @@ func main() {
}
}
func referenceConfig() (omni.Config, error) {
settings, err := omni.LoadConfig("")
if err != nil {
return settings, err
}
// Preserve the example's loopback default and legacy address variable.
if strings.TrimSpace(os.Getenv("GO_OMNI_HTTP_ADDR")) == "" {
settings.Address = strings.TrimSpace(os.Getenv("GO_OMNI_ADDR"))
if settings.Address == "" {
settings.Address = "127.0.0.1:8080"
}
}
if settings.Log.Service == "" {
settings.Log.Service = "reference-api"
}
if settings.Log.Environment == "" {
settings.Log.Environment = "local"
}
if strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT")) == "" {
settings.Log.Output = "stdout"
}
return settings, nil
}
var configPath = flag.String("config", "examples/reference/config.yaml", "configuration file")
func referenceConfig() (omni.Config, error) { return omni.LoadConfig(*configPath) }
func runReference(ctx context.Context) error {
settings, err := referenceConfig()
@@ -106,10 +88,9 @@ func openUsers(ctx context.Context, logger *logx.Logger) (service.UserRepository
if strings.TrimSpace(os.Getenv("GO_OMNI_DB_DSN")) == "" {
return repository.NewMemoryUsers(), func() error { return nil }, "memory", nil
}
cfg, err := database.ConfigFromEnv("GO_OMNI_DB_")
if err != nil {
return nil, nil, "", err
}
cfg := database.DefaultConfig()
cfg.Driver = "mysql"
cfg.DSN = os.Getenv("GO_OMNI_DB_DSN")
db, err := database.Open(ctx, cfg)
if err != nil {
return nil, nil, "", err

View File

@@ -7,6 +7,8 @@ import (
"io"
"net"
"net/http"
"os"
"path/filepath"
"strings"
"sync"
"testing"
@@ -99,7 +101,11 @@ func TestBusinessConfiguredTransports(t *testing.T) {
t.Setenv("REFERENCE_TEST_TLS_CERT_FILE", cert)
t.Setenv("REFERENCE_TEST_TLS_KEY_FILE", key)
}
settings, err := omni.LoadConfig("REFERENCE_TEST_")
path := filepath.Join(t.TempDir(), "config.yaml")
if err := os.WriteFile(path, []byte("transport: ${REFERENCE_TEST_TRANSPORT}\nshutdownTimeout: 3s\ntls:\n certFile: \"${REFERENCE_TEST_TLS_CERT_FILE:-}\"\n keyFile: \"${REFERENCE_TEST_TLS_KEY_FILE:-}\""), 0600); err != nil {
t.Fatal(err)
}
settings, err := omni.LoadConfig(path)
if err != nil {
t.Fatal(err)
}
@@ -159,24 +165,26 @@ func TestBusinessConfiguredTransports(t *testing.T) {
}
}
func TestReferenceAddressCompatibility(t *testing.T) {
func TestReferenceExplicitConfig(t *testing.T) {
previous := *configPath
*configPath = "config.yaml"
defer func() { *configPath = previous }()
t.Setenv("GO_OMNI_TRANSPORT", "native")
t.Setenv("GO_OMNI_HTTP_ADDR", "")
t.Setenv("GO_OMNI_TLS_CERT_FILE", "")
t.Setenv("GO_OMNI_TLS_KEY_FILE", "")
t.Setenv("GO_OMNI_HTTP_ADDR", "")
t.Setenv("GO_OMNI_ADDR", "")
settings, err := referenceConfig()
if err != nil || settings.Address != "127.0.0.1:8080" || settings.Transport != "native" {
t.Fatalf("default: %+v %v", settings, err)
c, err := referenceConfig()
if err != nil || c.Address != "127.0.0.1:8080" || c.Transport != "native" {
t.Fatal(c, err)
}
t.Setenv("GO_OMNI_ADDR", "127.0.0.1:8081")
settings, err = referenceConfig()
if err != nil || settings.Address != "127.0.0.1:8081" {
t.Fatalf("legacy: %+v %v", settings, err)
t.Setenv("GO_OMNI_ADDR", "127.0.0.1:1234")
c, err = referenceConfig()
if err != nil || c.Address != "127.0.0.1:8080" {
t.Fatal("implicit alias", err)
}
t.Setenv("GO_OMNI_HTTP_ADDR", "127.0.0.1:8082")
settings, err = referenceConfig()
if err != nil || settings.Address != "127.0.0.1:8082" {
t.Fatalf("standard: %+v %v", settings, err)
c, err = referenceConfig()
if err != nil || c.Address != "127.0.0.1:8082" {
t.Fatal(c, err)
}
}

View File

@@ -18,13 +18,12 @@ go-omni migrate --dir migrations --apply
go run ./cmd/server
```
源码示例的 go.mod 使用 `replace ... => ../..`。生成代码使用 `v0.8.0-preview` 响应策略入口。
源码示例的 go.mod 使用 `replace ... => ../..`。生成代码使用 `v0.22.0-preview` 显式配置加载入口。
复制到独立仓库时删除 replace,保留同版本固定依赖,再执行 go mod tidy;CLI 也固定到同一版本。
远程版本验证脚本会检查没有本地替换。
`GO_OMNI_TRANSPORT=native` 可切换为 native,业务代码无需调整。
配置是默认值 → etc/config.yaml → 环境变量。日志默认写入 var/log/server.log 并轮转,
`GO_OMNI_LOG_OUTPUT=stderr` 可以切换为标准错误输出。MySQL 默认最多 10 条连接。
配置是 Default() → etc/config.yaml 的显式引用 → 类型绑定与校验。文件里的固定值不会被环境自动覆盖。日志默认写入 var/log/server.log 并轮转;要改用 stderr,在文件明确设置 log.file 为空、log.rotation.enabled 为 false、log.output 为 stderr。MySQL 默认最多 10 条连接。
应用检查 users 和 user_audit 表,缺失时拒绝启动,不自动改表。
## 接口

View File

@@ -1,6 +1,7 @@
# Non-secret settings only. Environment variables override these values.
# Defaults are overridden only by fields in this file.
server:
address: '127.0.0.1:8080'
transport: "${GO_OMNI_TRANSPORT:-nethttp}"
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
enableLogger: true
slowRequestThreshold: 500ms
shutdownTimeout: 10s
@@ -17,16 +18,19 @@ server:
compress: true
redactKeys: []
storage:
filesToken: "${GO_OMNI_FILES_TOKEN:-}"
signingKey: "${GO_OMNI_STORAGE_SIGNING_KEY}"
provider: local
maxBytes: 33554432
contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf']
linkTtl: 5m
requestTimeout: 30s
root: './var/files'
publicUrl: 'http://127.0.0.1:8080/storage/object'
publicUrl: "${GO_OMNI_STORAGE_PUBLIC_URL:-http://127.0.0.1:8080/storage/object}"
database:
dsn: "${GO_OMNI_DB_DSN}"
driver: mysql
maxOpenConns: 10
maxIdleConns: 5

View File

@@ -3,7 +3,7 @@ module example.com/userService
go 1.22.0
require (
git.yuntuo.hk/k2safe/go-omni v0.8.0-preview
git.yuntuo.hk/k2safe/go-omni v0.22.0-preview
github.com/go-sql-driver/mysql v1.9.3
)

View File

@@ -15,10 +15,9 @@ func TestTransactions(t *testing.T) {
if os.Getenv("USER_SERVICE_TEST_DATABASE") != "1" {
t.Skip("isolated MySQL required")
}
cfg, err := database.ConfigFromEnv("")
if err != nil {
t.Fatal(err)
}
cfg := database.DefaultConfig()
cfg.Driver = "mysql"
cfg.DSN = os.Getenv("GO_OMNI_DB_DSN")
db, err := database.Open(context.Background(), cfg)
if err != nil {
t.Fatal(err)

View File

@@ -3,14 +3,11 @@ package settings
import (
"errors"
"fmt"
"git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/database"
"git.yuntuo.hk/k2safe/go-omni/logx"
"git.yuntuo.hk/k2safe/go-omni/storage"
"net/url"
"os"
"strconv"
"strings"
"time"
)
@@ -38,40 +35,7 @@ func Default() Config {
}}
}
// Load uses defaults -> strict YAML/JSON -> nonempty environment variables.
// Pass an empty path to use only defaults and environment.
func Load(path string) (Config, error) {
cfg := Default()
var err error
if path != "" {
cfg, err = config.DecodeFile(path, cfg)
if err != nil {
return Config{}, err
}
}
cfg.Server, err = config.ApplyEnv(cfg.Server, "")
if err != nil {
return Config{}, err
}
// A stream explicitly selected by environment overrides a file profile.
if output := strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT")); output != "" && strings.TrimSpace(os.Getenv("GO_OMNI_LOG_FILE")) == "" {
cfg.Server.Log.File = ""
cfg.Server.Log.Rotation = logx.RotationConfig{}
}
if err = cfg.Storage.applyEnv(); err != nil {
return Config{}, err
}
cfg.Database, err = database.ApplyEnv(cfg.Database, "")
if err != nil {
return Config{}, err
}
if err = cfg.Validate(); err != nil {
return Config{}, err
}
return cfg, nil
}
func Load(path string) (Config, error) { return config.LoadProject(path, Default()) }
func (c Config) Validate() error {
if err := c.Database.Validate(); err != nil {
@@ -108,62 +72,17 @@ type StorageConfig struct {
ContentTypes []string `json:"contentTypes"`
LinkTTL time.Duration `json:"linkTtl"`
RequestTimeout time.Duration `json:"requestTimeout"`
// Secrets are environment-only and cannot be populated by config files.
FilesToken string `json:"-"`
SigningKey string `json:"-"`
AccessKey string `json:"-"`
SecretKey string `json:"-"`
SessionToken string `json:"-"`
// The file explicitly selects secret environment references.
FilesToken string `json:"filesToken"`
SigningKey string `json:"signingKey"`
AccessKey string `json:"accessKey"`
SecretKey string `json:"secretKey"`
SessionToken string `json:"sessionToken"`
}
func (s StorageConfig) Policy() storage.UploadPolicy {
return storage.UploadPolicy{MaxBytes: s.MaxBytes, ContentTypes: s.ContentTypes}
}
func (s *StorageConfig) applyEnv() error {
values := map[string]*string{
"GO_OMNI_STORAGE_ROOT": &s.Root, "GO_OMNI_STORAGE_PUBLIC_URL": &s.PublicURL,
"GO_OMNI_STORAGE_ENDPOINT": &s.Endpoint, "GO_OMNI_STORAGE_REGION": &s.Region,
"GO_OMNI_STORAGE_BUCKET": &s.Bucket, "GO_OMNI_STORAGE_ACCOUNT_ID": &s.AccountID,
"GO_OMNI_STORAGE_SIGNING_KEY": &s.SigningKey, "GO_OMNI_FILES_TOKEN": &s.FilesToken,
"GO_OMNI_STORAGE_ACCESS_KEY": &s.AccessKey, "GO_OMNI_STORAGE_SECRET_KEY": &s.SecretKey,
"GO_OMNI_STORAGE_SESSION_TOKEN": &s.SessionToken,
}
for name, target := range values {
if value := strings.TrimSpace(os.Getenv(name)); value != "" {
*target = value
}
}
for name, target := range map[string]*time.Duration{"GO_OMNI_STORAGE_LINK_TTL": &s.LinkTTL, "GO_OMNI_STORAGE_REQUEST_TIMEOUT": &s.RequestTimeout} {
if value := strings.TrimSpace(os.Getenv(name)); value != "" {
v, err := time.ParseDuration(value)
if err != nil {
return fmt.Errorf("%s must be a duration", name)
}
*target = v
}
}
if value := os.Getenv("GO_OMNI_STORAGE_MAX_BYTES"); value != "" {
v, err := strconv.ParseInt(value, 10, 64)
if err != nil {
return errors.New("GO_OMNI_STORAGE_MAX_BYTES must be an integer")
}
s.MaxBytes = v
}
if value := os.Getenv("GO_OMNI_STORAGE_SECURE"); value != "" {
v, err := strconv.ParseBool(value)
if err != nil {
return errors.New("GO_OMNI_STORAGE_SECURE must be a boolean")
}
s.Secure = v
}
if value := strings.TrimSpace(os.Getenv("GO_OMNI_STORAGE_CONTENT_TYPES")); value != "" {
s.ContentTypes = strings.Split(value, ",")
for i := range s.ContentTypes {
s.ContentTypes[i] = strings.TrimSpace(s.ContentTypes[i])
}
}
return nil
}
func (s StorageConfig) Validate() error {
if s.Provider != "local" {
return errors.New("storage.provider must match the adapter selected at generation")
@@ -186,7 +105,7 @@ func (s StorageConfig) Validate() error {
return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")
}
if len(s.SigningKey) < 32 {
return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")
return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")
}
return nil

153
httpc/README.md Normal file
View File

@@ -0,0 +1,153 @@
# httpc:复用连接的 HTTP 客户端
`httpc` 在标准 `net/http` 上提供独立连接池、网络阶段总超时、保守重试、请求追踪和有界 JSON 调用。普通接口可以直接带认证头调用;上传、下载、特殊协议仍使用标准 Request/Response。没有业务身份模型或全局认证信息。
## 创建一次,业务重复使用
```go
client, err := httpc.New(httpc.Config{
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
Timeout: 3 * time.Second,
DialTimeout: time.Second,
TLSHandshakeTimeout: time.Second,
ResponseHeaderTimeout: time.Second,
MaxConnsPerHost: 32,
MaxIdleConnsPerHost: 16,
MaxIdleConns: 64,
IdleConnTimeout: 60 * time.Second,
MaxResponseBytes: 1 << 20,
})
if err != nil { return err }
// 在 svc 初始化并持有 client,向生命周期登记 client.Close。
// 普通独立程序也可 defer client.Close();不要每次请求重新 New。
err = client.JSON(ctx, http.MethodGet, serviceURL, nil, &result,
httpc.WithHeader("Authorization", "Bearer "+token),
httpc.WithHeaders(http.Header{"Accept": {"application/json"}}),
)
```
Client 可并发使用。每个调用的 input/output 及其读写同步由调用方负责,不要让多个请求同时写同一个 output。
直接运行 [examples/httpClient](../examples/httpClient/README.md),只使用本地临时服务,无需业务 Token 或外部基础设施。
## 代理规则
| 配置 | 行为 |
| --- | --- |
| 不填写 Proxy | 保留被克隆 DefaultTransport 的代理策略;未改动的标准库默认使用环境代理 |
| `Mode: httpc.ProxyEnvironment` | 明确使用 HTTP_PROXY / HTTPS_PROXY / NO_PROXY(及小写形式) |
| `Mode: httpc.ProxyDirect` | 设置 Proxy=nil,明确直连,不受环境代理影响 |
| `Mode: httpc.ProxyURL, URL: "http://proxy.example:8080"` | 固定代理地址,不通过 NO_PROXY 排除 |
支持 http、https、socks5、socks5h;底层行为沿用 Go Transport。固定地址必须是代理 origin,不接受业务路径、查询、fragment 或无效端口。凭证可由 URL userinfo 提供,但配置错误不输出 URL/密码;不要打印整个 Config。环境代理规则由标准库读取并缓存,不提供运行时环境代理热更新。
内部服务要直连,只设置 `proxy.mode: direct`,无需创建 Transport。需要自定义路由、代理认证握手等复杂逻辑时注入自己的 Transport。
## 超时和连接池
除原有字段外,新字段都可选:**0 保留默认行为,正数覆盖,负数拒绝**。需要明确关闭某个阶段超时、关闭 TCP keepalive 或特殊 dialer 时,使用自定义 Transport。
| 字段 | 默认/含义 |
| --- | --- |
| Timeout | 10s;从 Do 开始,覆盖 DNS/建连、等待连接、重试、退避、响应头和读取响应体 |
| DialTimeout | 不填时保留原 dialer;标准库默认 30s;配置后使用标准 net.Dialer |
| DialKeepAlive | 标准 net.Dialer 的 TCP keepalive 探测间隔,标准默认 30s;不是 HTTP 空闲连接超时 |
| TLSHandshakeTimeout | 沿用克隆 Transport;标准库默认 10s |
| ResponseHeaderTimeout | 沿用 Transport,标准默认不单独限制;从完整发送请求后等响应头,总 Timeout 仍生效 |
| ExpectContinueTimeout | 沿用 Transport;标准默认 1s,等待 `100-continue` 的时间 |
| IdleConnTimeout | 沿用 Transport;标准默认 90s,只回收空闲连接 |
| MaxConnsPerHost | 默认 64;每个目标的建连中、活动与空闲连接总上限 |
| MaxIdleConnsPerHost | 默认等于 MaxConnsPerHost,可单独配置 |
| MaxIdleConns | 默认 MaxConnsPerHost×2,可单独配置;所有目标合计的空闲连接上限 |
| MaxResponseBytes | JSON 成功响应默认最多 1 MiB,最大允许配置 1 GiB;应按实际接口保持较小限额 |
每主机上限最多 65536,总空闲连接最多 131072。总空闲上限可以小于各主机上限之和。MaxConnsPerHost 不等于整个进程的连接上限,也不等于 HTTP/2 的请求并发上限。
配置 DialTimeout / DialKeepAlive 会创建标准 net.Dialer;未填的另一个值使用标准默认 30s。不要通过全局修改 DefaultTransport 来塞入项目的特殊拨号逻辑;请显式使用借用的 Transport。若 DefaultTransport 有自定义 TLS dialer,相关阶段覆盖会报错,避免 TLS timeout 被悄悄忽略。
调用者 context 的更早截止时间优先。总超时不能抢占 Go 的 JSON 编解码或不遵守 context 的自定义 RoundTripper;计时从 Do 开始,不包括调用前的 json.Marshal。TLS 校验默认保持开启,没有新增 InsecureSkipVerify 配置。
## JSON 的单次请求选项
```go
err := client.JSON(ctx, "POST", endpoint, input, &output,
httpc.WithHeader("Authorization", "Bearer "+token),
httpc.WithHeader("X-Tenant", tenant),
httpc.WithMaxResponseBytes(64<<10),
)
```
- `WithHeader` 设置一个头;`WithHeaders` 支持多值 http.Header。后出现的选项覆盖同名头,不影响其他头。
- WithHeaders 在创建选项时快照原 map,每次调用复制值。构建后修改原 map 不影响选项,选项可以并发复用;构建时原 map 的读写同步由调用方负责。
- 不提供 Client 全局 Authorization,避免跨用户调用共享凭证。框架不会把请求头写进日志。
- 有 input 时默认 Content-Type 为 application/json;显式请求头可覆盖。请求头名称/值会校验,拒绝 CRLF 等控制字符和歧义的大小写重复键。
- Host、Content-Length、Transfer-Encoding、Trailer 属于请求目标/分帧控制,请用标准 Request 配合 Do;JSON 选项中不允许伪装成普通头。
- `WithMaxResponseBytes` 是本次 JSON 的正数限额,不修改 Client 默认值,不改变 Do 的流式语义。
JSON 自动关闭响应体。仅 2xx 解码;空成功响应(例如 204)保留 output 原值。非空响应使用标准 json.Unmarshal,拒绝多个 JSON 文档,类型错误保留 Go 的错误信息与行为。
`output == nil` 按限额流式丢弃成功响应,不把整个 body 读进内存,也不检查 JSON 语法。响应限额按实际读取的 body 计算;标准库自动 gzip 解压后仍受上限约束。input 会先 json.Marshal 到内存,大文件上传不适合用 JSON helper。
非 2xx 返回 `*httpc.StatusError`,只包含状态码,不包含错误响应体。为连接复用,只尝试读完**声明长度已知且不超过 min(32 KiB, 本次响应限额)** 的小错误体,最多等待 25ms(或更早总截止时间);失败仍返回原 HTTP 状态。未知长度/大错误体立即关闭,优先及时返回。重试过程的错误体另受 32 KiB 大小及整体请求截止时间约束,不无界读取。
## 配置文件与生成项目
在项目 Config 中加一个字段:
```go
Outbound httpc.Config `json:"outbound"`
```
项目 Validate 中调用 `c.Outbound.Validate()`;创建时 New 也会验证。svc 初始化一次并通过生命周期登记 Close。文件例子:
```yaml
outbound:
timeout: 3s
proxy:
mode: direct
dialTimeout: 500ms
tlsHandshakeTimeout: 1s
responseHeaderTimeout: 1s
maxConnsPerHost: 32
maxIdleConnsPerHost: 16
maxIdleConns: 64
idleConnTimeout: 60s
maxResponseBytes: 1048576
```
统一加载入口读取 JSON/YAML/XML。需要环境变量时在文件明确写 `mode: "${HTTP_PROXY_MODE:-direct}"`、`dialTimeout: "${HTTP_DIAL_TIMEOUT:-500ms}"`;固定值不会被环境自动覆盖。无需手写解析。详见 [统一配置](../CONFIG.md)。
## 标准请求、特殊 TLS 与生命周期
```go
req, err := http.NewRequestWithContext(ctx, "GET", downloadURL, nil)
if err != nil { return err }
req.Header.Set("Authorization", "Bearer "+token)
res, err := client.Do(req)
if err != nil { return err }
defer res.Body.Close()
// 自行判断状态,按项目要求限流/限制大小,流式写入目标;不要无界 ReadAll 大文件。
_, err = io.Copy(destination, res.Body)
```
Do 不自动限制 body,允许下载/流式处理,调用方必须关闭。响应体消费/关闭前网络超时持续有效。
`Config.Transport` 可注入任意 http.RoundTripper,适配自定义 dialer、mTLS、根证书、监控包装或测试替身。框架借用它,**不修改、不调用其 CloseIdleConnections**。代理、连接池、TLS/建连阶段配置不能同时填入,避免无效配置;总 Timeout、JSON 限额、框架重试/日志仍有效。借用实现必须遵守标准请求和 body 所有权、并发及 context 约定。
`Close` 幂等:拒绝新调用,关闭自有池的空闲连接,不主动打断已开始的请求。服务关闭应先排空 HTTP/后台调用,再关闭 Client;它不是后台 drain 的替代品。生成项目可通过 svc 的生命周期登记清理。
## 重试、日志与兼容
MaxRetries 默认 0,最多 5。仅可重放的 GET/HEAD/OPTIONS 对网络错误、429/502/503/504 做框架重试。POST/PUT/PATCH/DELETE 不自动做应用层重试。标准库或自定义 Transport 自己的重放规则仍可能生效,不能承诺业务“恰好执行一次”。
Backoff 默认 100ms,MaxBackoff 默认 2s,采用指数退避与抖动。Retry-After 要求等待超过 MaxBackoff 时返回当前响应,不提前请求;总截止时间限制全部尝试。成功或无重试的调用不计算退避。
不跟随重定向,避免自动向其他地址转发认证信息;3xx 由调用者判断。默认传递 context 中的 X-Request-ID,显式请求头优先。日志只包含 method、hostname、status、attempt、收到响应头的耗时等,不打印路径、查询、用户名密码、认证头或响应体;标准 net/http 返回的 URL 错误可能携带地址,项目不要把 URL 中的密钥当作安全日志字段。
v0.21 兼容说明:
- 原来的 JSON 五参数直接调用继续有效;新方法末尾增加 `...httpc.RequestOption`,自行定义的 interface/方法值类型需要同步签名。
- 原默认代理、总超时、重试、池大小和成功响应含义保留;新字段未设置时不改变它们。
- `Transport` 与显式池/代理/阶段参数混用现在报错;此前 MaxConnsPerHost 在自定义 Transport 下实际被忽略。迁移时把参数放到自己的 Transport,或删掉 Transport 改用内置配置。
- 没有移除 Do、JSON、StatusError 或自定义 Transport,没有引入新第三方依赖。

100
httpc/benchmark_test.go Normal file
View File

@@ -0,0 +1,100 @@
package httpc
import (
"context"
"io"
"net/http"
"net/http/httptest"
"strings"
"testing"
)
type benchmarkTransport struct {
payload string
status int
}
func (t benchmarkTransport) RoundTrip(r *http.Request) (*http.Response, error) {
if r.Body != nil {
_ = r.Body.Close()
}
return &http.Response{StatusCode: t.status, Header: make(http.Header), Body: io.NopCloser(strings.NewReader(t.payload)), ContentLength: int64(len(t.payload)), Request: r}, nil
}
func BenchmarkHTTPClient(b *testing.B) {
for _, size := range []int{128, 4096, 65536} {
for _, discard := range []bool{false, true} {
label := map[int]string{128: "128B", 4096: "4KiB", 65536: "64KiB"}[size]
if discard {
label += "/discard"
} else {
label += "/decode"
}
b.Run(label, func(b *testing.B) {
c, err := New(Config{Transport: benchmarkTransport{`{"value":"` + strings.Repeat("x", size) + `"}`, 200}, DisableLogging: true})
if err != nil {
b.Fatal(err)
}
defer c.Close()
var output struct {
Value string `json:"value"`
}
var target any = &output
if discard {
target = nil
}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if err := c.JSON(context.Background(), "GET", "http://example.invalid/item", nil, target); err != nil {
b.Fatal(err)
}
}
})
}
}
b.Run("Do", func(b *testing.B) {
c, _ := New(Config{Transport: benchmarkTransport{"", 200}, DisableLogging: true})
defer c.Close()
r, _ := http.NewRequest("GET", "http://example.invalid/item", nil)
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
res, err := c.Do(r)
if err != nil {
b.Fatal(err)
}
res.Body.Close()
}
})
}
func BenchmarkHTTPPool(b *testing.B) {
for _, status := range []int{200, 401} {
b.Run(http.StatusText(status), func(b *testing.B) {
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(status); io.WriteString(w, `{"ok":true}`) }))
defer s.Close()
c, _ := New(Config{DisableLogging: true})
defer c.Close()
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
if status == 200 && err != nil {
b.Fatal(err)
}
}
})
}
}
func BenchmarkHTTPHeaders(b *testing.B) {
c, _ := New(Config{Transport: benchmarkTransport{`{"ok":true}`, 200}, DisableLogging: true})
defer c.Close()
option := WithHeaders(http.Header{"Authorization": []string{"Bearer example"}, "X-Request-Id": []string{"example-request"}})
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if err := c.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, option); err != nil {
b.Fatal(err)
}
}
}

View File

@@ -3,9 +3,7 @@
package httpc
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
@@ -22,18 +20,6 @@ import (
var ErrClosed = errors.New("httpc: closed")
var ErrResponseTooLarge = errors.New("httpc: response exceeds limit")
type Config struct {
Timeout time.Duration `json:"timeout"`
MaxRetries int `json:"maxRetries"`
Backoff time.Duration `json:"backoff"`
MaxBackoff time.Duration `json:"maxBackoff"`
MaxConnsPerHost int `json:"maxConnsPerHost"`
MaxResponseBytes int64 `json:"maxResponseBytes"`
DisableRequestID bool `json:"disableRequestId"`
DisableLogging bool `json:"disableLogging"`
Transport http.RoundTripper `json:"-"` // Borrowed. The caller owns its lifecycle.
}
type Client struct {
config Config
client *http.Client
@@ -41,43 +27,19 @@ type Client struct {
closed atomic.Bool
}
// New creates a reusable client. Close it after callers and response streams drain.
func New(cfg Config) (*Client, error) {
if cfg.Timeout < 0 || cfg.Backoff < 0 || cfg.MaxBackoff < 0 || cfg.MaxRetries < 0 || cfg.MaxRetries > 5 || cfg.MaxConnsPerHost < 0 || cfg.MaxResponseBytes < 0 {
return nil, errors.New("httpc: invalid configuration")
normalized, err := normalizeConfig(cfg)
if err != nil {
return nil, err
}
if cfg.Timeout == 0 {
cfg.Timeout = 10 * time.Second
}
if cfg.Backoff == 0 {
cfg.Backoff = 100 * time.Millisecond
}
if cfg.MaxBackoff == 0 {
cfg.MaxBackoff = 2 * time.Second
}
if cfg.Backoff > cfg.MaxBackoff {
return nil, errors.New("httpc: backoff exceeds maxBackoff")
}
if cfg.MaxConnsPerHost == 0 {
cfg.MaxConnsPerHost = 64
}
if cfg.MaxResponseBytes == 0 {
cfg.MaxResponseBytes = 1 << 20
}
if cfg.MaxResponseBytes > 1<<30 || cfg.MaxConnsPerHost > 65536 {
return nil, errors.New("httpc: response or connection limit exceeds supported capacity")
}
c := &Client{config: cfg}
c := &Client{config: normalized}
transport := cfg.Transport
if transport == nil {
// Own a private pool; never mutate the process-wide DefaultTransport.
base, ok := http.DefaultTransport.(*http.Transport)
if !ok {
return nil, errors.New("httpc: provide Transport when DefaultTransport is replaced")
c.owned, err = ownedTransport(normalized)
if err != nil {
return nil, err
}
c.owned = base.Clone()
c.owned.MaxConnsPerHost = cfg.MaxConnsPerHost
c.owned.MaxIdleConnsPerHost = cfg.MaxConnsPerHost
c.owned.MaxIdleConns = cfg.MaxConnsPerHost * 2
transport = c.owned
}
c.client = &http.Client{Transport: transport, CheckRedirect: func(*http.Request, []*http.Request) error { return http.ErrUseLastResponse }}
@@ -114,7 +76,10 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
}
replayable := (base.Method == "" || base.Method == "GET" || base.Method == "HEAD" || base.Method == "OPTIONS") && (base.Body == nil || base.Body == http.NoBody || base.GetBody != nil)
for attempt := 0; ; attempt++ {
req := base.Clone(ctx)
req := base
if attempt > 0 {
req = base.Clone(ctx)
}
if attempt > 0 && base.Body != nil && base.Body != http.NoBody {
body, err := base.GetBody()
if err != nil {
@@ -133,23 +98,9 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
logx.InfoContext(ctx, "http.client", "method", req.Method, "host", req.URL.Hostname(), "status", status, "attempt", attempt+1, "durationMs", time.Since(started).Seconds()*1000, "failed", err != nil)
}
retry := replayable && attempt < c.config.MaxRetries && ctx.Err() == nil && (err != nil || status == 429 || status == 502 || status == 503 || status == 504)
delay := c.config.Backoff
for i := 0; i < attempt && delay < c.config.MaxBackoff; i++ {
if delay > c.config.MaxBackoff/2 {
delay = c.config.MaxBackoff
} else {
delay *= 2
}
}
if response != nil && response.Header.Get("Retry-After") != "" {
if value, ok := retryAfter(response.Header.Get("Retry-After")); ok {
delay = value
}
} else if delay > 1 {
delay = delay/2 + time.Duration(rand.Int63n(int64(delay/2)+1))
}
if delay > c.config.MaxBackoff {
retry = false
var delay time.Duration
if retry {
delay, retry = c.retryDelay(attempt, response)
}
if !retry {
if err != nil {
@@ -160,9 +111,7 @@ func (c *Client) Do(request *http.Request) (*http.Response, error) {
return response, nil
}
if response != nil {
if response.ContentLength >= 0 && response.ContentLength <= 32<<10 {
_, _ = io.Copy(io.Discard, response.Body)
}
drainSmallBody(response, 32<<10)
_ = response.Body.Close()
}
timer := time.NewTimer(delay)
@@ -204,48 +153,6 @@ type StatusError struct{ StatusCode int }
func (e *StatusError) Error() string { return fmt.Sprintf("httpc: HTTP status %d", e.StatusCode) }
// JSON bounds response memory, closes the body, and decodes only 2xx responses.
// Error bodies, URLs, credentials and query strings are not included in logs.
func (c *Client) JSON(ctx context.Context, method, url string, input, output any) error {
if ctx == nil {
return errors.New("httpc: nil context")
}
var body io.Reader
if input != nil {
data, err := json.Marshal(input)
if err != nil {
return err
}
body = bytes.NewReader(data)
}
request, err := http.NewRequestWithContext(ctx, method, url, body)
if err != nil {
return err
}
if input != nil {
request.Header.Set("Content-Type", "application/json")
}
response, err := c.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode >= 300 {
return &StatusError{StatusCode: response.StatusCode}
}
data, err := io.ReadAll(io.LimitReader(response.Body, c.config.MaxResponseBytes+1))
if err != nil {
return err
}
if int64(len(data)) > c.config.MaxResponseBytes {
return ErrResponseTooLarge
}
if output == nil || len(bytes.TrimSpace(data)) == 0 {
return nil
}
return json.Unmarshal(data, output)
}
func (c *Client) Close() error {
c.closed.Store(true)
if c.owned != nil {
@@ -253,3 +160,34 @@ func (c *Client) Close() error {
}
return nil
}
// retryDelay is deliberately outside the success path: no jitter/random work
// for calls that cannot or do not need to retry.
func (c *Client) retryDelay(attempt int, response *http.Response) (time.Duration, bool) {
delay := c.config.Backoff
for i := 0; i < attempt && delay < c.config.MaxBackoff; i++ {
if delay > c.config.MaxBackoff/2 {
delay = c.config.MaxBackoff
} else {
delay *= 2
}
}
if response != nil {
if value, ok := retryAfter(response.Header.Get("Retry-After")); ok {
return value, value <= c.config.MaxBackoff
}
}
if delay > 1 {
delay = delay/2 + time.Duration(rand.Int63n(int64(delay/2)+1))
}
return delay, delay <= c.config.MaxBackoff
}
// Drain only a known small error body. Unknown/large bodies are closed promptly;
// never allocate or consume an unbounded error document just to retain a pool slot.
func drainSmallBody(response *http.Response, limit int64) {
limit = min(limit, 32<<10)
if response.Body != nil && response.ContentLength >= 0 && response.ContentLength <= limit {
_, _ = io.Copy(io.Discard, io.LimitReader(response.Body, limit+1))
}
}

205
httpc/config.go Normal file
View File

@@ -0,0 +1,205 @@
package httpc
import (
"errors"
"fmt"
"net"
"net/http"
"net/url"
"reflect"
"strconv"
"strings"
"time"
)
// ProxyMode controls the owned transport's proxy selection. Empty preserves
// DefaultTransport's policy (normally environment proxies).
type ProxyMode string
const (
ProxyEnvironment ProxyMode = "environment"
ProxyDirect ProxyMode = "direct"
ProxyURL ProxyMode = "url"
)
type ProxyConfig struct {
Mode ProxyMode `json:"mode"`
URL string `json:"url"` // May contain credentials; never log the configuration.
}
// Config owns one private connection pool unless Transport is supplied.
// New duration/pool fields are optional: zero retains existing defaults, positive
// values override them. Use a custom Transport for disabling timeouts, custom
// dialers, mTLS and other protocol-specific settings.
type Config struct {
Timeout time.Duration `json:"timeout"`
MaxRetries int `json:"maxRetries"`
Backoff time.Duration `json:"backoff"`
MaxBackoff time.Duration `json:"maxBackoff"`
MaxConnsPerHost int `json:"maxConnsPerHost"`
MaxIdleConns int `json:"maxIdleConns"`
MaxIdleConnsPerHost int `json:"maxIdleConnsPerHost"`
DialTimeout time.Duration `json:"dialTimeout"`
DialKeepAlive time.Duration `json:"dialKeepAlive"`
TLSHandshakeTimeout time.Duration `json:"tlsHandshakeTimeout"`
ResponseHeaderTimeout time.Duration `json:"responseHeaderTimeout"`
ExpectContinueTimeout time.Duration `json:"expectContinueTimeout"`
IdleConnTimeout time.Duration `json:"idleConnTimeout"`
Proxy ProxyConfig `json:"proxy"`
MaxResponseBytes int64 `json:"maxResponseBytes"`
DisableRequestID bool `json:"disableRequestId"`
DisableLogging bool `json:"disableLogging"`
Transport http.RoundTripper `json:"-"` // Borrowed; never modified or closed.
}
const maxResponseBytes int64 = 1 << 30
// Validate checks values without creating pools or making network requests.
func (cfg Config) Validate() error { _, err := normalizeConfig(cfg); return err }
func normalizeConfig(cfg Config) (Config, error) {
for _, field := range []struct {
name string
value time.Duration
}{
{"timeout", cfg.Timeout}, {"backoff", cfg.Backoff}, {"maxBackoff", cfg.MaxBackoff},
{"dialTimeout", cfg.DialTimeout}, {"dialKeepAlive", cfg.DialKeepAlive}, {"tlsHandshakeTimeout", cfg.TLSHandshakeTimeout},
{"responseHeaderTimeout", cfg.ResponseHeaderTimeout}, {"expectContinueTimeout", cfg.ExpectContinueTimeout}, {"idleConnTimeout", cfg.IdleConnTimeout},
} {
if field.value < 0 {
return Config{}, fmt.Errorf("httpc: %s must not be negative", field.name)
}
}
if cfg.MaxRetries < 0 || cfg.MaxRetries > 5 {
return Config{}, errors.New("httpc: maxRetries must be 0..5")
}
if cfg.MaxConnsPerHost < 0 || cfg.MaxConnsPerHost > 65536 || cfg.MaxIdleConnsPerHost < 0 || cfg.MaxIdleConnsPerHost > 65536 || cfg.MaxIdleConns < 0 || cfg.MaxIdleConns > 131072 {
return Config{}, errors.New("httpc: invalid connection pool limits")
}
if cfg.MaxResponseBytes < 0 || cfg.MaxResponseBytes > maxResponseBytes {
return Config{}, errors.New("httpc: maxResponseBytes must be 0..1 GiB")
}
if _, err := proxyFunction(cfg.Proxy); err != nil {
return Config{}, err
}
if cfg.Transport != nil {
value := reflect.ValueOf(cfg.Transport)
switch value.Kind() {
case reflect.Pointer, reflect.Map, reflect.Func, reflect.Slice, reflect.Interface, reflect.Chan:
if value.IsNil() {
return Config{}, errors.New("httpc: nil Transport")
}
}
if cfg.hasTransportOptions() {
return Config{}, errors.New("httpc: Transport cannot be combined with proxy, dial, TLS or connection pool options; configure the borrowed transport directly")
}
}
if cfg.Timeout == 0 {
cfg.Timeout = 10 * time.Second
}
if cfg.Backoff == 0 {
cfg.Backoff = 100 * time.Millisecond
}
if cfg.MaxBackoff == 0 {
cfg.MaxBackoff = 2 * time.Second
}
if cfg.Backoff > cfg.MaxBackoff {
return Config{}, errors.New("httpc: backoff exceeds maxBackoff")
}
if cfg.MaxConnsPerHost == 0 {
cfg.MaxConnsPerHost = 64
}
if cfg.MaxIdleConnsPerHost == 0 {
cfg.MaxIdleConnsPerHost = cfg.MaxConnsPerHost
}
if cfg.MaxIdleConns == 0 {
cfg.MaxIdleConns = cfg.MaxConnsPerHost * 2
}
if cfg.MaxResponseBytes == 0 {
cfg.MaxResponseBytes = 1 << 20
}
return cfg, nil
}
func (c Config) hasTransportOptions() bool {
return c.Proxy.Mode != "" || c.Proxy.URL != "" || c.MaxConnsPerHost != 0 || c.MaxIdleConns != 0 || c.MaxIdleConnsPerHost != 0 || c.DialTimeout != 0 || c.DialKeepAlive != 0 || c.TLSHandshakeTimeout != 0 || c.ResponseHeaderTimeout != 0 || c.ExpectContinueTimeout != 0 || c.IdleConnTimeout != 0
}
func proxyFunction(c ProxyConfig) (func(*http.Request) (*url.URL, error), error) {
if c.Mode != ProxyURL && c.URL != "" {
return nil, errors.New("httpc: proxy URL requires mode url")
}
switch c.Mode {
case "":
return nil, nil
case ProxyDirect:
return nil, nil
case ProxyEnvironment:
return http.ProxyFromEnvironment, nil
case ProxyURL:
u, err := url.Parse(c.URL)
if err != nil || u == nil || u.Hostname() == "" || u.Opaque != "" || u.RawQuery != "" || u.ForceQuery || u.Fragment != "" || (u.Path != "" && u.Path != "/") {
return nil, errors.New("httpc: proxy URL must be an absolute proxy origin without path, query or fragment")
}
switch u.Scheme {
case "http", "https", "socks5", "socks5h":
default:
return nil, errors.New("httpc: proxy scheme must be http, https, socks5 or socks5h")
}
if strings.HasSuffix(u.Host, ":") {
return nil, errors.New("httpc: proxy port must be 1..65535")
}
if port := u.Port(); port != "" {
n, e := strconv.Atoi(port)
if e != nil || n < 1 || n > 65535 {
return nil, errors.New("httpc: proxy port must be 1..65535")
}
}
u.Path = ""
u.RawPath = ""
return http.ProxyURL(u), nil
default:
return nil, errors.New("httpc: proxy mode must be environment, direct or url")
}
}
func ownedTransport(cfg Config) (*http.Transport, error) {
base, ok := http.DefaultTransport.(*http.Transport)
if !ok || base == nil {
return nil, errors.New("httpc: provide Transport when DefaultTransport is replaced")
}
if (base.DialTLSContext != nil || base.DialTLS != nil) && (cfg.DialTimeout > 0 || cfg.DialKeepAlive > 0 || cfg.TLSHandshakeTimeout > 0) {
return nil, errors.New("httpc: provide Transport when DefaultTransport has a custom TLS dialer")
}
t := base.Clone()
t.MaxConnsPerHost = cfg.MaxConnsPerHost
t.MaxIdleConnsPerHost = cfg.MaxIdleConnsPerHost
t.MaxIdleConns = cfg.MaxIdleConns
if cfg.Proxy.Mode != "" {
proxy, err := proxyFunction(cfg.Proxy)
if err != nil {
return nil, err
}
t.Proxy = proxy
}
if cfg.DialTimeout > 0 || cfg.DialKeepAlive > 0 {
dialer := &net.Dialer{Timeout: 30 * time.Second, KeepAlive: 30 * time.Second}
if cfg.DialTimeout > 0 {
dialer.Timeout = cfg.DialTimeout
}
if cfg.DialKeepAlive > 0 {
dialer.KeepAlive = cfg.DialKeepAlive
}
t.DialContext = dialer.DialContext
t.Dial = nil
}
if cfg.TLSHandshakeTimeout > 0 {
t.TLSHandshakeTimeout = cfg.TLSHandshakeTimeout
}
if cfg.ResponseHeaderTimeout > 0 {
t.ResponseHeaderTimeout = cfg.ResponseHeaderTimeout
}
if cfg.ExpectContinueTimeout > 0 {
t.ExpectContinueTimeout = cfg.ExpectContinueTimeout
}
if cfg.IdleConnTimeout > 0 {
t.IdleConnTimeout = cfg.IdleConnTimeout
}
return t, nil
}

277
httpc/config_test.go Normal file
View File

@@ -0,0 +1,277 @@
package httpc
import (
"context"
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"net/url"
"os"
"os/exec"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
func testClient(t *testing.T, cfg Config) *Client {
t.Helper()
cfg.DisableLogging = true
c, err := New(cfg)
if err != nil {
t.Fatal(err)
}
t.Cleanup(func() { c.Close() })
return c
}
func TestTransportConfigurationAndIsolation(t *testing.T) {
base := http.DefaultTransport.(*http.Transport)
oldConns, oldIdle, oldTimeout := base.MaxConnsPerHost, base.MaxIdleConns, base.ResponseHeaderTimeout
cfg := Config{Proxy: ProxyConfig{Mode: ProxyDirect}, DialTimeout: time.Second, DialKeepAlive: 20 * time.Second, TLSHandshakeTimeout: 2 * time.Second, ResponseHeaderTimeout: 3 * time.Second, ExpectContinueTimeout: 4 * time.Second, IdleConnTimeout: 5 * time.Second, MaxConnsPerHost: 8, MaxIdleConns: 12, MaxIdleConnsPerHost: 3}
c := testClient(t, cfg)
if c.owned == base || c.owned.Proxy != nil || c.owned.DialContext == nil || c.owned.MaxConnsPerHost != 8 || c.owned.MaxIdleConns != 12 || c.owned.MaxIdleConnsPerHost != 3 || c.owned.TLSHandshakeTimeout != 2*time.Second || c.owned.ResponseHeaderTimeout != 3*time.Second || c.owned.ExpectContinueTimeout != 4*time.Second || c.owned.IdleConnTimeout != 5*time.Second {
t.Fatal("transport options not applied")
}
if base.MaxConnsPerHost != oldConns || base.MaxIdleConns != oldIdle || base.ResponseHeaderTimeout != oldTimeout {
t.Fatal("modified shared transport")
}
if c.owned.TLSClientConfig != nil && c.owned.TLSClientConfig.InsecureSkipVerify {
t.Fatal("TLS verification disabled")
}
d := testClient(t, Config{})
if d.owned.MaxConnsPerHost != 64 || d.owned.MaxIdleConnsPerHost != 64 || d.owned.MaxIdleConns != 128 || d.owned.IdleConnTimeout != base.IdleConnTimeout || d.owned.TLSHandshakeTimeout != base.TLSHandshakeTimeout || d.owned.ResponseHeaderTimeout != base.ResponseHeaderTimeout {
t.Fatal("default behavior changed")
}
ctx, cancel := context.WithCancel(context.Background())
cancel()
if conn, err := c.owned.DialContext(ctx, "tcp", "127.0.0.1:1"); err == nil {
conn.Close()
t.Fatal("dial ignored context")
}
}
func TestProxyValidation(t *testing.T) {
for _, scheme := range []string{"http", "https", "socks5", "socks5h"} {
c := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyURL, URL: scheme + "://user:private-value@proxy.invalid:8080"}})
req, _ := http.NewRequest("GET", "https://target.invalid", nil)
u, err := c.owned.Proxy(req)
if err != nil || u.Host != "proxy.invalid:8080" || u.Scheme != scheme {
t.Fatal("explicit proxy unavailable")
}
}
bad := []Config{
{Proxy: ProxyConfig{Mode: "private-value"}},
{Proxy: ProxyConfig{URL: "http://private-value"}},
{Proxy: ProxyConfig{Mode: ProxyDirect, URL: "http://private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://user:private-value@proxy.invalid:bad"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "file:///private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid/private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid?token=private-value"}},
{Proxy: ProxyConfig{Mode: ProxyURL, URL: "http://proxy.invalid:65536"}},
{DialTimeout: -1}, {DialKeepAlive: -1}, {TLSHandshakeTimeout: -1}, {ResponseHeaderTimeout: -1}, {ExpectContinueTimeout: -1}, {IdleConnTimeout: -1}, {MaxIdleConns: -1}, {MaxIdleConnsPerHost: 65537},
{Transport: benchmarkTransport{}, Proxy: ProxyConfig{Mode: ProxyDirect}},
{Transport: benchmarkTransport{}, DialTimeout: time.Second}, {Transport: benchmarkTransport{}, MaxConnsPerHost: 2},
}
for i, cfg := range bad {
if _, err := New(cfg); err == nil || strings.Contains(err.Error(), "private-value") {
t.Fatalf("case %d: unsafe or missing error %v", i, err)
}
}
var nilTransport *http.Transport
if _, err := New(Config{Transport: nilTransport}); err == nil {
t.Fatal("typed nil accepted")
}
}
func TestProxyRouting(t *testing.T) {
var proxied, target atomic.Int32
proxy := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
proxied.Add(1)
if r.URL.Scheme != "http" || r.URL.Host != "target.invalid" {
t.Error("not an absolute proxy request")
}
io.WriteString(w, `{"proxy":true}`)
}))
defer proxy.Close()
origin := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { target.Add(1); io.WriteString(w, `{}`) }))
defer origin.Close()
// Isolate the standard library's process-cached environment proxy lookup.
executable, err := os.Executable()
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, executable, "-test.run=^TestProxyEnvironmentProcess$")
for _, value := range os.Environ() {
key := strings.SplitN(value, "=", 2)[0]
switch strings.ToUpper(key) {
case "HTTP_PROXY", "HTTPS_PROXY", "NO_PROXY", "ALL_PROXY", "GO_OMNI_HTTPC_PROXY_CHILD", "GO_OMNI_HTTPC_TARGET":
continue
}
cmd.Env = append(cmd.Env, value)
}
cmd.Env = append(cmd.Env, "GO_OMNI_HTTPC_PROXY_CHILD=1", "GO_OMNI_HTTPC_TARGET="+origin.URL, "HTTP_PROXY="+proxy.URL, "HTTPS_PROXY="+proxy.URL, "NO_PROXY=bypass.invalid")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(err, string(out))
}
if proxied.Load() != 3 || target.Load() != 1 {
t.Fatalf("proxy=%d direct=%d", proxied.Load(), target.Load())
}
}
func TestProxyEnvironmentProcess(t *testing.T) {
if os.Getenv("GO_OMNI_HTTPC_PROXY_CHILD") != "1" {
t.Skip("subprocess only")
}
for _, proxy := range []ProxyConfig{{}, {Mode: ProxyEnvironment}, {Mode: ProxyURL, URL: os.Getenv("HTTP_PROXY")}} {
c := testClient(t, Config{Proxy: proxy})
if err := c.JSON(context.Background(), "GET", "http://target.invalid/path", nil, nil); err != nil {
t.Fatal(err)
}
}
environment := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyEnvironment}})
req, _ := http.NewRequest("GET", "http://bypass.invalid", nil)
u, err := environment.owned.Proxy(req)
if err != nil || u != nil {
t.Fatal("NO_PROXY ignored")
}
direct := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyDirect}})
if direct.owned.Proxy != nil {
t.Fatal("direct still has proxy")
}
if err := direct.JSON(context.Background(), "GET", os.Getenv("GO_OMNI_HTTPC_TARGET"), nil, nil); err != nil {
t.Fatal(err)
}
}
func TestResponseHeaderTimeoutRecovery(t *testing.T) {
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/slow" {
<-r.Context().Done()
return
}
io.WriteString(w, `{}`)
}))
defer s.Close()
c := testClient(t, Config{Timeout: 2 * time.Second, ResponseHeaderTimeout: 30 * time.Millisecond})
start := time.Now()
err := c.JSON(context.Background(), "GET", s.URL+"/slow", nil, nil)
var network net.Error
if !errors.As(err, &network) || !network.Timeout() || time.Since(start) > time.Second {
t.Fatal("header timeout", err)
}
if err = c.JSON(context.Background(), "GET", s.URL+"/ok", nil, nil); err != nil {
t.Fatal("did not recover", err)
}
}
func TestTLSHandshakeTimeout(t *testing.T) {
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
defer listener.Close()
done := make(chan struct{})
go func() {
defer close(done)
conn, err := listener.Accept()
if err != nil {
return
}
defer conn.Close()
_, _ = io.Copy(io.Discard, conn)
}()
c := testClient(t, Config{Timeout: 2 * time.Second, Proxy: ProxyConfig{Mode: ProxyDirect}, TLSHandshakeTimeout: 30 * time.Millisecond})
start := time.Now()
err = c.JSON(context.Background(), "GET", "https://"+listener.Addr().String(), nil, nil)
var network net.Error
if !errors.As(err, &network) || !network.Timeout() || time.Since(start) > time.Second {
t.Fatal("handshake timeout", err)
}
select {
case <-done:
case <-time.After(time.Second):
t.Fatal("TLS connection leaked")
}
}
func TestPoolLimitsAndIdleExpiry(t *testing.T) {
var connections atomic.Int32
entered := make(chan struct{}, 12)
release := make(chan struct{})
closed := make(chan struct{}, 12)
s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
entered <- struct{}{}
select {
case <-release:
case <-r.Context().Done():
return
}
io.WriteString(w, `{}`)
}))
s.Config.ConnState = func(_ net.Conn, state http.ConnState) {
if state == http.StateNew {
connections.Add(1)
}
if state == http.StateClosed {
closed <- struct{}{}
}
}
s.Start()
defer s.Close()
c := testClient(t, Config{Timeout: 2 * time.Second, MaxConnsPerHost: 2, MaxIdleConnsPerHost: 2, MaxIdleConns: 2, IdleConnTimeout: 40 * time.Millisecond})
var wg sync.WaitGroup
for i := 0; i < 12; i++ {
wg.Add(1)
go func() {
defer wg.Done()
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); err != nil {
t.Error(err)
}
}()
}
for i := 0; i < 2; i++ {
select {
case <-entered:
case <-time.After(time.Second):
close(release)
t.Fatal("requests did not start")
}
}
if connections.Load() != 2 {
t.Error("connection cap", connections.Load())
}
close(release)
wg.Wait()
if connections.Load() != 2 {
t.Fatal("pool was not reused", connections.Load())
}
for i := 0; i < 2; i++ {
select {
case <-closed:
case <-time.After(time.Second):
t.Fatal("idle pool was not reclaimed")
}
}
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); err != nil {
t.Fatal(err)
}
if connections.Load() != 3 {
t.Fatal("new connection not established after idle timeout")
}
c.Close()
select {
case <-closed:
case <-time.After(time.Second):
t.Fatal("Close left idle connections open")
}
}
func TestHTTPSVerificationRetained(t *testing.T) {
s := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { io.WriteString(w, `{}`) }))
defer s.Close()
c := testClient(t, Config{Proxy: ProxyConfig{Mode: ProxyDirect}})
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
var urlError *url.Error
if !errors.As(err, &urlError) {
t.Fatal("self-signed server unexpectedly trusted", err)
}
}

119
httpc/json.go Normal file
View File

@@ -0,0 +1,119 @@
package httpc
import (
"bytes"
"context"
"encoding/json"
"errors"
"io"
"net/http"
"time"
)
// JSON bounds response memory, closes the body, and decodes only 2xx responses.
// Optional headers/limits belong to this call and never alter the client.
// output=nil discards a bounded stream; it does not buffer or validate JSON.
// Empty successful bodies leave output unchanged. StatusError contains no body.
func (c *Client) JSON(ctx context.Context, method, url string, input, output any, options ...RequestOption) error {
if ctx == nil {
return errors.New("httpc: nil context")
}
if c.closed.Load() {
return ErrClosed
}
if err := ctx.Err(); err != nil {
return err
}
settings := requestOptions{maxResponseBytes: c.config.MaxResponseBytes}
if len(options) > 0 {
var err error
settings, err = jsonOptions(c.config.MaxResponseBytes, options)
if err != nil {
return err
}
}
var body io.Reader
if input != nil {
data, err := json.Marshal(input)
if err != nil {
return err
}
body = bytes.NewReader(data)
}
request, err := http.NewRequestWithContext(ctx, method, url, body)
if err != nil {
return err
}
if settings.headers != nil {
request.Header = settings.headers
}
if input != nil {
if _, exists := request.Header["Content-Type"]; !exists {
request.Header.Set("Content-Type", "application/json")
}
}
response, err := c.Do(request)
if err != nil {
return err
}
defer response.Body.Close()
if response.StatusCode < 200 || response.StatusCode >= 300 {
drainStatusBody(response, settings.maxResponseBytes)
return &StatusError{StatusCode: response.StatusCode}
}
if response.ContentLength > settings.maxResponseBytes {
return ErrResponseTooLarge
}
limited := io.LimitReader(response.Body, settings.maxResponseBytes+1)
if output == nil {
count, err := io.Copy(io.Discard, limited)
if err != nil {
return err
}
if count > settings.maxResponseBytes {
return ErrResponseTooLarge
}
return nil
}
data, err := io.ReadAll(limited)
if err != nil {
return err
}
if int64(len(data)) > settings.maxResponseBytes {
return ErrResponseTooLarge
}
if len(bytes.TrimSpace(data)) == 0 {
return nil
}
return json.Unmarshal(data, output)
}
// A known small status body may still be slow. Spend at most a short grace
// period trying to retain the connection, then cancel the HTTP read. The status
// remains authoritative and the deferred Close still owns body cleanup.
func drainStatusBody(response *http.Response, limit int64) {
if response.ContentLength < 0 || response.ContentLength > min(limit, 32<<10) {
return
}
body, ok := response.Body.(*responseBody)
if !ok {
return
}
timer := time.AfterFunc(25*time.Millisecond, body.cancel)
defer timer.Stop()
drainSmallBody(response, limit)
}
// Keep the option builder out of the allocation path of existing plain calls.
func jsonOptions(limit int64, options []RequestOption) (requestOptions, error) {
settings := requestOptions{maxResponseBytes: limit}
for _, option := range options {
if option == nil {
return requestOptions{}, errors.New("httpc: nil request option")
}
if err := option(&settings); err != nil {
return requestOptions{}, err
}
}
return settings, nil
}

99
httpc/request.go Normal file
View File

@@ -0,0 +1,99 @@
package httpc
import (
"errors"
"net/http"
"strings"
)
type requestOptions struct {
headers http.Header
maxResponseBytes int64
}
// RequestOption applies only to one JSON call. Returned options can be reused
// concurrently; request headers are independent, with no client-wide credentials.
type RequestOption func(*requestOptions) error
// WithHeader sets a header for this call; later options replace earlier values.
// Framing and Host must be controlled through a standard Request passed to Do.
func WithHeader(name, value string) RequestOption {
name = http.CanonicalHeaderKey(name)
err := checkHeader(name, []string{value})
return func(o *requestOptions) error {
if err != nil {
return err
}
if o.headers == nil {
o.headers = make(http.Header)
}
o.headers[name] = []string{value}
return nil
}
}
// WithHeaders snapshots headers immediately. Editing the original map after
// construction does not change the option; each invocation gets separate slices.
func WithHeaders(headers http.Header) RequestOption {
snapshot := make(http.Header, len(headers))
var err error
for key, values := range headers {
name := http.CanonicalHeaderKey(key)
if err = checkHeader(name, values); err != nil {
break
}
if _, exists := snapshot[name]; exists {
err = errors.New("httpc: duplicate canonical header name")
break
}
snapshot[name] = append([]string(nil), values...)
}
return func(o *requestOptions) error {
if err != nil {
return err
}
if o.headers == nil {
o.headers = make(http.Header, len(snapshot))
}
for name, values := range snapshot {
o.headers[name] = append([]string(nil), values...)
}
return nil
}
}
// WithMaxResponseBytes overrides JSON's bounded response read for one call.
// The size is positive and at most 1 GiB. Do remains a caller-owned stream.
func WithMaxResponseBytes(limit int64) RequestOption {
return func(o *requestOptions) error {
if limit <= 0 || limit > maxResponseBytes {
return errors.New("httpc: response limit must be positive and at most 1 GiB")
}
o.maxResponseBytes = limit
return nil
}
}
func checkHeader(name string, values []string) error {
if name == "" {
return errors.New("httpc: header name must not be empty")
}
for _, c := range []byte(name) {
if c >= 'a' && c <= 'z' || c >= 'A' && c <= 'Z' || c >= '0' && c <= '9' || strings.ContainsRune("!#$%&'*+-.^_`|~", rune(c)) {
continue
}
return errors.New("httpc: invalid header name")
}
switch name {
case "Host", "Content-Length", "Transfer-Encoding", "Trailer":
return errors.New("httpc: use Do for Host or request framing controls")
}
for _, value := range values {
for i := 0; i < len(value); i++ {
c := value[i]
if c == '\x7f' || c < ' ' && c != '\t' {
return errors.New("httpc: invalid header value")
}
}
}
return nil
}

316
httpc/request_test.go Normal file
View File

@@ -0,0 +1,316 @@
package httpc
import (
"bytes"
"compress/gzip"
"context"
"errors"
"fmt"
"git.yuntuo.hk/k2safe/go-omni/logx"
"io"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
)
func TestJSONHeadersConcurrentSnapshot(t *testing.T) {
var leaked atomic.Bool
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.Header.Get("Authorization") != "Bearer "+r.URL.Query().Get("caller") {
leaked.Store(true)
}
if r.Header.Get("X-Shared") != "original" || len(r.Header.Values("X-Tag")) != 2 {
leaked.Store(true)
}
if r.Header.Get("Content-Type") != "application/vnd.example+json" {
leaked.Store(true)
}
io.WriteString(w, `{"ok":true}`)
}))
defer s.Close()
c := testClient(t, Config{})
headers := http.Header{"X-Shared": []string{"original"}, "X-Tag": []string{"one", "two"}}
snapshot := WithHeaders(headers)
headers.Set("X-Shared", "changed")
headers["X-Tag"][0] = "changed"
var wg sync.WaitGroup
for i := 0; i < 24; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
id := fmt.Sprint(i)
var output struct {
OK bool `json:"ok"`
}
if err := c.JSON(context.Background(), "POST", s.URL+"?caller="+id, map[string]bool{"hello": true}, &output, snapshot, WithHeader("Authorization", "Bearer "+id), WithHeader("content-type", "application/vnd.example+json")); err != nil || !output.OK {
t.Errorf("JSON failed: %v", err)
}
}(i)
}
wg.Wait()
if leaked.Load() {
t.Fatal("headers leaked across requests or changed with source map")
}
}
func TestRequestOptionsValidationBeforeNetwork(t *testing.T) {
var calls atomic.Int32
c := testClient(t, Config{Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
calls.Add(1)
return benchmarkTransport{`{}`, 200}.RoundTrip(r)
})})
invalid := []RequestOption{nil, WithHeader("bad name", "private-token"), WithHeader("Authorization", "private-token\r\nInjected: yes"), WithHeader("Host", "example.invalid"), WithHeader("Content-Length", "1"), WithHeader("Transfer-Encoding", "chunked"), WithHeader("Trailer", "X-Signature"), WithHeaders(http.Header{"X-Name": []string{"one"}, "x-name": []string{"two"}}), WithMaxResponseBytes(0), WithMaxResponseBytes(maxResponseBytes + 1)}
for _, option := range invalid {
err := c.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, option)
if err == nil || strings.Contains(err.Error(), "private-token") {
t.Fatal("missing or unsafe error", err)
}
}
if calls.Load() != 0 {
t.Fatal("invalid options reached network")
}
}
func TestJSONResponseLimitsAndStreaming(t *testing.T) {
var calls atomic.Int32
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
calls.Add(1)
switch r.URL.Path {
case "/empty":
w.WriteHeader(204)
case "/chunked":
w.(http.Flusher).Flush()
io.WriteString(w, strings.Repeat("x", 17))
case "/status-stream":
w.WriteHeader(401)
w.(http.Flusher).Flush()
<-r.Context().Done()
case "/multiple":
io.WriteString(w, `{} {}`)
default:
io.WriteString(w, `{"ok":true}`)
}
}))
defer server.Close()
c := testClient(t, Config{MaxResponseBytes: 8, Timeout: time.Second})
var out struct {
OK bool `json:"ok"`
}
if err := c.JSON(context.Background(), "GET", server.URL, nil, &out, WithMaxResponseBytes(32)); err != nil || !out.OK {
t.Fatal(err)
}
if err := c.JSON(context.Background(), "GET", server.URL, nil, nil); !errors.Is(err, ErrResponseTooLarge) {
t.Fatal("limit changed permanently", err)
}
for _, target := range []any{nil, &out} {
if err := c.JSON(context.Background(), "GET", server.URL+"/chunked", nil, target, WithMaxResponseBytes(16)); !errors.Is(err, ErrResponseTooLarge) {
t.Fatal("unbounded chunked body", err)
}
}
if err := c.JSON(context.Background(), "GET", server.URL+"/empty", nil, &out); err != nil || !out.OK {
t.Fatal("empty success changed output", err)
}
if err := c.JSON(context.Background(), "GET", server.URL+"/multiple", nil, &out); err == nil {
t.Fatal("multiple JSON documents accepted")
}
start := time.Now()
err := c.JSON(context.Background(), "GET", server.URL+"/status-stream", nil, nil)
var status *StatusError
if !errors.As(err, &status) || status.StatusCode != 401 || time.Since(start) > 500*time.Millisecond {
t.Fatal("waited for unknown status body", err)
}
if calls.Load() != 7 {
t.Fatal(calls.Load())
}
}
func TestErrorBodyPoolReuseAndDrainBound(t *testing.T) {
var connections atomic.Int32
s := httptest.NewUnstartedServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(401)
io.WriteString(w, "private-error-body")
}))
s.Config.ConnState = func(_ net.Conn, state http.ConnState) {
if state == http.StateNew {
connections.Add(1)
}
}
s.Start()
defer s.Close()
c := testClient(t, Config{})
for i := 0; i < 10; i++ {
err := c.JSON(context.Background(), "GET", s.URL, nil, nil)
var status *StatusError
if !errors.As(err, &status) || strings.Contains(err.Error(), "private-error-body") {
t.Fatal(err)
}
}
if connections.Load() != 1 {
t.Fatal("small status response prevented reuse", connections.Load())
}
body := &countedBody{Reader: strings.NewReader(strings.Repeat("x", 1<<20))}
custom := testClient(t, Config{Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
return &http.Response{StatusCode: 401, ContentLength: 1, Body: body, Header: make(http.Header)}, nil
})})
err := custom.JSON(context.Background(), "GET", "http://example.invalid", nil, nil)
var status *StatusError
if !errors.As(err, &status) || !body.closed || body.read > (32<<10)+1 {
t.Fatal("unbounded/missing drain", body.read, body.closed, err)
}
}
type countedBody struct {
io.Reader
read int
closed bool
}
func (b *countedBody) Read(p []byte) (int, error) {
n, err := b.Reader.Read(p)
b.read += n
return n, err
}
func (b *countedBody) Close() error { b.closed = true; return nil }
type roundTripperFunc func(*http.Request) (*http.Response, error)
func (f roundTripperFunc) RoundTrip(r *http.Request) (*http.Response, error) { return f(r) }
type borrowedTransport struct{ closed atomic.Bool }
func (t *borrowedTransport) RoundTrip(r *http.Request) (*http.Response, error) {
return benchmarkTransport{`{}`, 200}.RoundTrip(r)
}
func (t *borrowedTransport) CloseIdleConnections() { t.closed.Store(true) }
func TestBorrowedTransportAndHeaderRetry(t *testing.T) {
owned := &borrowedTransport{}
c := testClient(t, Config{Transport: owned})
c.Close()
if owned.closed.Load() {
t.Fatal("closed caller's transport")
}
var attempts atomic.Int32
retry := testClient(t, Config{MaxRetries: 1, Backoff: time.Millisecond, Transport: roundTripperFunc(func(r *http.Request) (*http.Response, error) {
if r.Header.Get("Authorization") != "Bearer test" {
t.Error("retry lost header")
}
status := 200
if attempts.Add(1) == 1 {
status = 503
}
return benchmarkTransport{`{}`, status}.RoundTrip(r)
})})
if err := retry.JSON(context.Background(), "GET", "http://example.invalid", nil, nil, WithHeader("Authorization", "Bearer test")); err != nil || attempts.Load() != 2 {
t.Fatal(err, attempts.Load())
}
}
func FuzzRequestHeaderOptions(f *testing.F) {
f.Add("Authorization", "Bearer example")
f.Add("X-Name", "one\r\nX-Test: bad")
f.Fuzz(func(t *testing.T, name, value string) {
if len(name)+len(value) > 8192 {
t.Skip()
}
option := WithHeader(name, value)
var settings requestOptions
if err := option(&settings); err != nil {
return
}
req, err := http.NewRequest("GET", "http://example.invalid", nil)
if err != nil {
t.Fatal(err)
}
req.Header = settings.headers
// The standard Request writer must be able to serialize accepted headers.
if err = req.Write(io.Discard); err != nil {
t.Fatal("accepted malformed header", err)
}
})
}
func TestHTTPLogDoesNotContainCredentials(t *testing.T) {
var buffer bytes.Buffer
logger, err := logx.New(logx.Config{Writer: &buffer})
if err != nil {
t.Fatal(err)
}
defer logger.Close()
c, err := New(Config{Transport: benchmarkTransport{"private-response", 401}})
if err != nil {
t.Fatal(err)
}
defer c.Close()
ctx := logx.WithLogger(context.Background(), logger)
err = c.JSON(ctx, "GET", "http://private-user:private-password@example.invalid/private-path?token=private-query", nil, nil, WithHeader("Authorization", "Bearer private-token"))
if err == nil {
t.Fatal("expected status error")
}
for _, secret := range []string{"private-user", "private-password", "private-path", "private-query", "private-token", "private-response"} {
if strings.Contains(buffer.String(), secret) || strings.Contains(err.Error(), secret) {
t.Fatal("sensitive field leaked")
}
}
if !strings.Contains(buffer.String(), "http.client") {
t.Fatal("missing access log")
}
}
func TestCloseAllowsInflightCallToFinish(t *testing.T) {
entered := make(chan struct{})
release := make(chan struct{})
var once sync.Once
releaseNow := func() { once.Do(func() { close(release) }) }
s := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { close(entered); <-release; io.WriteString(w, `{}`) }))
defer s.Close()
c := testClient(t, Config{Timeout: time.Second})
done := make(chan error, 1)
go func() { done <- c.JSON(context.Background(), "GET", s.URL, nil, nil) }()
defer releaseNow()
select {
case <-entered:
case err := <-done:
t.Fatal("request failed before handler", err)
case <-time.After(2 * time.Second):
t.Fatal("handler not entered")
}
c.Close()
releaseNow()
if err := <-done; err != nil {
t.Fatal("Close interrupted in-flight call", err)
}
if err := c.JSON(context.Background(), "GET", s.URL, nil, nil); !errors.Is(err, ErrClosed) {
t.Fatal("accepted new call after Close", err)
}
}
func TestSlowKnownErrorBodyHasShortDrainBudget(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Length", "100")
w.WriteHeader(503)
w.(http.Flusher).Flush()
<-r.Context().Done()
}))
defer server.Close()
c := testClient(t, Config{Timeout: 2 * time.Second})
start := time.Now()
err := c.JSON(context.Background(), "GET", server.URL, nil, nil)
var status *StatusError
if !errors.As(err, &status) || status.StatusCode != 503 || time.Since(start) > 500*time.Millisecond {
t.Fatal("slow error body delayed status", err, time.Since(start))
}
}
func TestJSONLimitAppliesAfterGzipExpansion(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Encoding", "gzip")
writer := gzip.NewWriter(w)
_, _ = io.WriteString(writer, strings.Repeat("x", 4096))
_ = writer.Close()
}))
defer server.Close()
c := testClient(t, Config{MaxResponseBytes: 64})
if err := c.JSON(context.Background(), "GET", server.URL, nil, nil); !errors.Is(err, ErrResponseTooLarge) {
t.Fatal("gzip bypassed response limit", err)
}
}

View File

@@ -76,7 +76,7 @@ func WithLogConfig(settings LogConfig) Option { return config.WithLogConfig(sett
func WithShutdownTimeout(timeout time.Duration) Option { return config.WithShutdownTimeout(timeout) }
func LoadConfig(prefix string) (Config, error) { return config.LoadFromEnv(prefix) }
func LoadConfig(path string) (Config, error) { return config.LoadFile(path) }
func SetupLogger(cfg LogConfig) (func() error, error) { return logx.Setup(cfg) }

Some files were not shown because too many files have changed in this diff Show More