Compare commits

...

16 Commits

Author SHA1 Message Date
d1a747c36f feat: release v0.26 portable i18n and presentation adapters 2026-09-29 13:31:42 +08:00
6a95cad184 test: record v0.25 remote transaction and MySQL acceptance 2026-09-26 20:32:23 +08:00
7a177744ab feat: expose transaction failure stages and conservative outcomes 2026-09-26 20:29:45 +08:00
8eddef9cc8 test: record v0.24 remote CORS and generated project acceptance 2026-09-26 19:55:41 +08:00
800974cfff feat: add configurable transport-independent CORS policy 2026-09-26 19:53:28 +08:00
9fade3bedb test: record v0.23 remote consumer and performance acceptance 2026-09-26 18:06:40 +08:00
3bbc6f60d7 feat: add precise diagnostics and optional client copy logging controls 2026-09-26 18:04:33 +08:00
69585dab61 test: record remote explicit configuration release acceptance 2026-09-26 16:16:37 +08:00
d598b2ebee feat: load explicit configuration references and remove implicit env overlays 2026-09-26 16:14:54 +08:00
4bf6e88cfe docs: record remote HTTP client release acceptance 2026-09-26 15:30:32 +08:00
96e7d9629b feat: simplify HTTP client proxy pools and authenticated JSON calls 2026-09-26 15:28:19 +08:00
068e85132f test: record remote configuration release acceptance 2026-09-26 14:49:28 +08:00
c60841f901 feat: unify generated configuration loading and extensible decoders 2026-09-26 14:47:33 +08:00
91a6beb2f3 test: verify remote copyx release and update usage reference 2026-09-25 17:31:48 +08:00
fd83235ce7 feat: replace mapping generation with checked copy utilities 2026-09-25 17:27:28 +08:00
ad9257d202 test: record remote nullable mapping release acceptance 2026-09-25 13:58:45 +08:00
269 changed files with 18686 additions and 3651 deletions

View File

@@ -11,7 +11,7 @@
| `openapi` | 路由与 OpenAPI 3.0.3 同时注册、DTO schema、响应信封 | 显式声明请求/响应;不自动推断任意 Handler 行为,不附带 Swagger UI | | `openapi` | 路由与 OpenAPI 3.0.3 同时注册、DTO schema、响应信封 | 显式声明请求/响应;不自动推断任意 Handler 行为,不附带 Swagger UI |
| `auth` | RS256/EdDSA JWT 验证与公钥轮换、可撤销 Session、Policy 接口 | 登录验证和业务授权规则由接入方提供;不自动获取 token 指定的密钥 | | `auth` | RS256/EdDSA JWT 验证与公钥轮换、可撤销 Session、Policy 接口 | 登录验证和业务授权规则由接入方提供;不自动获取 token 指定的密钥 |
| `telemetry` | HTTP 入站/出站、Redis、SQL spans,HTTP 指标和 traceId 日志关联 | 注入 OTel exporter/reader 后才外发;SQL Query span 不覆盖后续遍历;不接管全局 provider | | `telemetry` | HTTP 入站/出站、Redis、SQL spans,HTTP 指标和 traceId 日志关联 | 注入 OTel exporter/reader 后才外发;SQL Query span 不覆盖后续遍历;不接管全局 provider |
| `config` / `logx` | JSON/YAML 文件、环境变量覆盖、动态日志级别、轮转/脱敏/堆栈/慢请求 | 热更新只开放日志级别;连接池、监听、认证等静态变更要求重启 | | `config` / `logx` | JSON/YAML/XML 文件、显式环境引用、动态日志级别、轮转/脱敏/堆栈/慢请求 | 热更新只开放日志级别;连接池、监听、认证等静态变更要求重启 |
| `resilience` | 失败阈值、熔断冷却、单探针恢复、上游并发隔离 | 一个实例对应一个依赖;默认内存状态,不做跨进程全局熔断 | | `resilience` | 失败阈值、熔断冷却、单探针恢复、上游并发隔离 | 一个实例对应一个依赖;默认内存状态,不做跨进程全局熔断 |
| `middleware/redis` | 跨副本限流、幂等 claim/回放、容量限制 | 专用 Redis namespace;TTL 窗口内保护不等于永久业务去重 | | `middleware/redis` | 跨副本限流、幂等 claim/回放、容量限制 | 专用 Redis namespace;TTL 窗口内保护不等于永久业务去重 |
| `queue` / `queue/redis` | 持久消息接口、Streams 接管、有限重试、失败消息管理 | 至少一次交付,处理器必须幂等;持久性取决于 Redis 部署 | | `queue` / `queue/redis` | 持久消息接口、Streams 接管、有限重试、失败消息管理 | 至少一次交付,处理器必须幂等;持久性取决于 Redis 部署 |
@@ -102,7 +102,7 @@ err = consumer.Run(serviceContext, func(ctx context.Context, item queue.Delivery
## 配置与日志热更新 ## 配置与日志热更新
`config.LoadFile(path, prefix)` 按默认值→文件→非空环境变量加载并验证。两种文件格式都使用 JSON tag 的 lowerCamelCase,duration 可以写 `"5s"`;文件上限 1 MiB,拒绝未知字段、多文档和非法值。自定义 prefix 要包含期望的尾部下划线,例如 `BILLING_`。空环境变量沿用文件值,不清空已配置字段。 `config.LoadFile(path)` 使用默认值→文件字段显式引用→类型绑定→校验;没有自动环境覆盖。用 `${NAME}` 或 `${NAME:-fallback}` 指定来源,完整规则与迁移见 [CONFIG.md](CONFIG.md)。
```yaml ```yaml
address: '127.0.0.1:8080' address: '127.0.0.1:8080'
@@ -113,7 +113,7 @@ log:
format: json format: json
``` ```
先按配置创建 `logx.New(cfg.Log)`,再 `config.NewReloader(path, prefix, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`;后者阻塞,生命周期由应用负责。只有 `log.level` 允许变化,校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。 先按配置创建 `logx.New(cfg.Log)`,再 `config.NewReloader(path, logger)`。显式调用 `Reload()` 或运行 `Watch(ctx, time.Second, onError)`;后者阻塞,生命周期由应用负责。只有 `log.level` 允许变化,校验失败或静态字段变化保留原快照与日志级别。子 logger 共享动态级别。替换文件时建议使用临时文件 + rename。
## 熔断与观测 ## 熔断与观测

10
AGENTS.md Normal file
View File

@@ -0,0 +1,10 @@
# 开发约定
- 框架保持通用,不内置接入项目的业务类型、字段、错误码或业务规则。
- 2026-09-25 用户要求:每次功能开发完成,必须同步维护根 README.md。README 需要覆盖当前全部公开功能,解释用途、基本用法,并提供可运行示例或链接到 examples 内的完整示例。
- 新增、调整和删除功能都必须同步文档、示例、测试与发布说明。删除功能要明确说明已移除内容、影响范围和迁移步骤,不能继续把历史用法当作当前能力推荐。
- examples 使用通用场景,避免依赖业务项目、私有数据库或线上服务。必要的外部基础设施配置应明确说明。
- 2026-09-25 用户决定:移除 mappings/mappingFiles 及结构映射代码生成机制,改用可选 copyx 工具包。不得恢复另一套 JSON 或 Go 映射声明语言。
- 深拷贝的自定义转换结果不得与输入共享可变引用;发现共享应报错,不能降级成浅拷贝或静默接受。
- 2026-09-26 用户决定:配置文件显式声明固定值或 `${NAME}` / `${NAME:-fallback}`,旧自动环境覆盖、前缀/别名及 env/envDefault 标签模式移除,不得重新引入隐式二次覆盖。项目维护 Config/Default/Validate/文件,生成器维护 Load/main。

4
API.md
View File

@@ -48,7 +48,7 @@ go run ./cmd/go-omni new --api examples/contracts/user.api \
--log-output file --log-format json ../userService --log-output file --log-format json ../userService
``` ```
`api generate` 在新目录中也会生成完整项目。默认只监听 localhost;`GO_OMNI_HTTP_ADDR` 修改地址,`GO_OMNI_TRANSPORT=native` 切换网络层。配置顺序为默认值 → `etc/config.yaml` → 环境变量。文件日志支持现有轮转配置。远程试用使用与 CLI 一致的固定版本,见 [发布说明](RELEASE.md)。 `api generate` 在新目录中也会生成完整项目。默认只监听 localhost;`GO_OMNI_HTTP_ADDR` 修改地址,`GO_OMNI_TRANSPORT=native` 切换网络层。配置顺序为 Default() → `etc/config.yaml` 中的显式环境引用 → 类型绑定 → Validate();固定值不再被隐式覆盖。文件日志支持现有轮转配置。远程试用使用与 CLI 一致的固定版本,见 [发布说明](RELEASE.md)。
API 项目现在支持组合数据库、存储和日志: API 项目现在支持组合数据库、存储和日志:
@@ -63,7 +63,7 @@ go-omni new --api user.api --module example.com/userService \
初次选项记录在 `.goOmni/apiManifest.json`,以后直接 `api generate` 会继承。已有项目通过参数切换选项会报错,因为初始化代码属于业务方;可在新目录生成对照后手动迁移。`--force` 不允许覆盖 API 项目。new 与 api new 统一生成契约项目,不再使用旧式 CRUD 目录。 初次选项记录在 `.goOmni/apiManifest.json`,以后直接 `api generate` 会继承。已有项目通过参数切换选项会报错,因为初始化代码属于业务方;可在新目录生成对照后手动迁移。`--force` 不允许覆盖 API 项目。new 与 api new 统一生成契约项目,不再使用旧式 CRUD 目录。
MySQL 使用官方 `database/mysql` 结构化适配器,密码从 `GO_OMNI_DB_PASSWORD` 覆盖,非秘密池配置位于 YAML,环境变量优先。存储使用原有 `GO_OMNI_STORAGE_*`;API 项目不会额外生成公开的签名发放接口,鉴权和对象权限由业务实现。初始化失败会清理已打开资源,退出清理可重复调用。 MySQL 使用官方 `database/mysql` 结构化适配器,生成文件以 `password: "${GO_OMNI_DB_PASSWORD}"` 显式读取密码;其他字段可写固定值或引用,缺失项保留 Default()。存储配置也只读取文件明确引用的变量;API 项目不会额外生成公开的签名发放接口,鉴权和对象权限由业务实现。初始化失败会清理已打开资源,退出清理可重复调用。
HTTP 请求类型不能推导数据库表结构;迁移和 Repository 由业务自己维护。当前目录的完整示例见 [managedService](examples/managedService/README.md)。MySQL、事务审计和鉴权实现可参考 [旧目录用户管理示例](examples/userService/README.md),接入时按 [项目指南](PROJECT.md) 装配到 config/svc/logic。 HTTP 请求类型不能推导数据库表结构;迁移和 Repository 由业务自己维护。当前目录的完整示例见 [managedService](examples/managedService/README.md)。MySQL、事务审计和鉴权实现可参考 [旧目录用户管理示例](examples/userService/README.md),接入时按 [项目指南](PROJECT.md) 装配到 config/svc/logic。

View File

@@ -5,7 +5,7 @@
## 实施与验收清单 ## 实施与验收清单
- [x] 运行期:资源登记、逆序释放、幂等关闭、失败回滚、故障原因保留归框架;生成入口与项目依赖初始化边界明确。多 App 日志隔离、标准 log.Logger 适配、可信请求元数据与依赖中间件示例。 - [x] 运行期:资源登记、逆序释放、幂等关闭、失败回滚、故障原因保留归框架;生成入口与项目依赖初始化边界明确。多 App 日志隔离、标准 log.Logger 适配、可信请求元数据与依赖中间件示例。
- [x] 配置:项目完整结构体环境覆盖;默认值→文件→环境;嵌套、duration、slice、显式零值、空值策略、必填与脱敏;日志无副作用完整校验。 - [x] 配置:Default()→文件显式环境引用→类型绑定→Validate;嵌套、duration、slice、显式零值、必填/备用值、转义与安全诊断;日志无副作用完整校验。
- [x] 数据库:官方 MySQL 结构化配置/TLS/驱动日志适配,继续复用 OpenConnector 和唯一连接池;独立启动检查、失败清理、生成器接入;不内置版本/表结构/业务错误策略。 - [x] 数据库:官方 MySQL 结构化配置/TLS/驱动日志适配,继续复用 OpenConnector 和唯一连接池;独立启动检查、失败清理、生成器接入;不内置版本/表结构/业务错误策略。
- [x] 响应契约:数组、可空对象、成功 null、独立 HTTP 204;Logic/Handler/OpenAPI/漂移检查一致。 - [x] 响应契约:数组、可空对象、成功 null、独立 HTTP 204;Logic/Handler/OpenAPI/漂移检查一致。
- [x] 请求契约:未知 query、JSON UTF-8/charset、无 Body/必填空对象/可选 Body,App 默认与路由覆盖;scalar/any/数组/大小限制一致,普通 JSON 只绑定一次。 - [x] 请求契约:未知 query、JSON UTF-8/charset、无 Body/必填空对象/可选 Body,App 默认与路由覆盖;scalar/any/数组/大小限制一致,普通 JSON 只绑定一次。

View File

@@ -1,5 +1,74 @@
# Changelog # Changelog
## v0.26.0-preview(2026-09-29)
- 新增独立 i18n:不可变命名空间词典、源/默认/启用语言、BCP 47 匹配、显式回退、参数/条件/基数复数和有界资源加载;不依赖 HTTP、业务或数据库。
- Go/浏览器共用固定 CLDR 46.1 复数规则;JSON 资源版本、类型导出和按命名空间选择的语言包独立于业务错误格式。
- 新增可选 httpi18n/errori18n 适配;错误 translation 元数据、PublicMessage 展示入口,保留原业务码/状态/失败语义/原始原因和首次幂等响应。
- 新增 i18n check/export/types/extract;静态与动态键诊断、未知字段/重复定义/参数校验、显式覆盖;输入只读,不执行任意项目 Go。
- 新增无前端框架依赖的浏览器模块、按需加载/切换竞态/失败保留、CLI/HTTP/表单示例及通用验收。旧项目默认行为不变。
- 新 translation 生成代码要求 v0.26.0-preview 或具备能力的本地源码,未使用该元数据的模板最低版本保持 v0.22。
- 修复 JSON 对象/数组结束定界符读取失败时丢失原始错误的问题,保留请求体限额的 HTTP 413;共享 Go 目录测试工程改用声明支持的 Go 1.22。
## v0.25.0-preview(2026-09-26)
- database.InTx 新增 TxError、TxStage、TxOutcome,区分开启/回调/提交失败以及未执行回调、已确认回滚、结果不确定;原签名保留。
- 回调失败后只有显式回滚返回成功才标记 rolledBack;提交报错一律保守返回 unknown,不根据断网/超时或 ErrTxDone 推断回滚。不自动重试事务,不处理文件清理等业务决策。
- Err/RollbackErr 分开保留,errors.As/Is 可遍历两条原因链;业务错误目录和公开响应行为保持。nil context 接受,nil 回调在开启前报错,panic/Goexit 保持传播并尝试回滚。
- 新增阶段/取消竞争/确认丢失/并发/公开响应回归、真实 MySQL 驱动边界故障验收、性能比较和 examples/transactions;同步 README 与 database 指南。
- 错误现在有类型包装;直接错误相等、直接类型断言和错误文本依赖需改为 errors.Is/As。标准库异步回滚不公开的底层原因无法还原,结果保持 unknown;不改变 GORM 自己的 Transaction。
## v0.24.0-preview(2026-09-26)
- 新增可选 cors 包和 server.cors 配置;生成项目直接加载,手写项目使用 omni.WithCORS,独立 HTTP 服务可使用 Policy.Wrap。默认关闭,无新增第三方依赖。
- 精确来源白名单、方法/请求头、暴露头、凭据和预检缓存可配置;拒绝凭据与通配来源混用、非法来源/头及歧义的重复请求头,校验错误定位具体字段。
- 在路由和鉴权之前处理预检;实际请求仍执行原有中间件,统一响应、404/405、上传下载和流式能力保持。nethttp/native/HTTP2 共用策略。
- 补充真实生成项目、TLS 三种 transport、20 项真实浏览器检查、race/fuzz/性能基准;同步 README、配置指南、cors 指南及 examples/cors。
- 没有移除现有功能或修改默认跨域行为。代理已负责 CORS 的项目保持关闭,避免重复头;高级调用 Router().ServeHTTP 不经过应用外层 CORS。
## v0.23.0-preview(2026-09-26)
- MySQL/连接池、日志、HTTP 客户端、生命周期和请求策略返回字段级安全诊断,配置加载器保留文件/变量/位置与原始错误链。
- httpc.JSON 新增 WithSuccessStatuses,单次调用可限制允许的 2xx,默认兼容,拒绝响应继续关闭和有界回收。
- copyx.Check[S,T] / Copier.Check 预检嵌套类型与现有规则;修正空集合隐藏类型不匹配,动态接口和 converter 的具体行为继续运行时验证。
- logx.Config.TimeZone 绑定 Logger 实例,JSON/text/子 Logger/标准日志适配一致;不修改 time.Local 或业务时间属性。
- 无新映射文件、强制模块或第三方依赖;更新 README、各包指南和 developerTools 可运行示例。错误文本更精确,空集合中的确定类型错误现在会被拒绝。
## v0.22.0-preview(2026-09-26)
- 文件显式环境引用与备用值,字符串组合/转义,安全有界展开和类型绑定;变量值原样插入、不递归处理。
- 移除隐式环境覆盖、EnvOptions/前缀/别名及 env/envDefault;数据库统一使用文件加载,不再单独扫描环境。
- 生成 main/Load 新增 -check-config,精简配置文件,保留开发者扩展;错误字段建议和迁移检查。
- README、CONFIG、数据库/HTTP 客户端指南及示例同步新规则;配置中的响应模板需要 $$ 转义,.api 不变。
## v0.21.0-preview(2026-09-26)
- httpc 可配置环境/直连/固定代理、常用连接阶段超时和独立空闲池参数。
- JSON 新增逐请求 Header 快照与响应大小选项,保持有界读取、关闭和认证隔离。
- 成功请求少复制和退避计算,丢弃响应流式处理,错误体有界回收以改善连接复用。
- 增加真实本地代理、超时、连接限制/回收、并发、日志脱敏、gzip、fuzz 和修改前后基准。
- JSON 自定义方法接口签名需同步 variadic 参数;借用 Transport 的参数混配现在报错。详见 httpc/README.md。
## v0.20.0-preview(2026-09-26)
- 生成项目配置加载入口持续由生成器维护;保留业务字段、默认值、校验与 NewLoader 扩展文件。
- config.LoadProject 统一 JSON/YAML/XML、环境覆盖和校验;提供实例级 Decoder 注册。
- 文件/字段/环境/行列的安全诊断,Invalid/AtField 与原始错误链。
- 增加格式等价、精度、null/空数组、并发、资源限制、默认值隔离、真实生成项目与性能验收。
- 旧手写 Load 必须显式迁移;低层 API 保留。固定数组严格长度、YAML alias/merge/custom tag 拒绝。用法及限制见 CONFIG.md。
## v0.19.0-preview(2026-09-25)
- 新增通用可选 copyx:构造器、浅/深拷贝、精确类型转换、字段规则、nil/零值策略、并发复用。
- 深拷贝拒绝自定义结果共享源可变引用;原子提交目标、保留错误链、限制遍历及估算分配。
- 移除 JSON 结构映射配置与生成;提供明确迁移诊断,保留公共字段组合等其他生成能力。
- 同步 README 全部能力索引、COPY.md、可运行示例和开发文档约定。
- 验收与不兼容迁移见 [发布说明](RELEASE.md)、[copyx 验收](reports/copyxAcceptance.md)。v0.14–v0.18 的完整变更保留在发布说明历史段落。
## v0.13.0-preview(2026-09-24) ## v0.13.0-preview(2026-09-24)
- 新增通用 lifecycle.Controller,svc.RequestShutdown 支持依赖主动通知正常停止或致命故障,通知不阻塞,首个故障原因保留;生成入口排空 HTTP 后关闭依赖与日志,故障退出码为 1。 - 新增通用 lifecycle.Controller,svc.RequestShutdown 支持依赖主动通知正常停止或致命故障,通知不阻塞,首个故障原因保留;生成入口排空 HTTP 后关闭依赖与日志,故障退出码为 1。

199
CONFIG.md Normal file
View File

@@ -0,0 +1,199 @@
# 配置:文件明确声明值的来源
v0.22 起只使用这一条流程:
```text
项目 Default() → 解析选中的一个文件 → 解析字段值中的环境引用
→ 按 Go 类型绑定已提供的字段 → 项目 Validate() → 初始化日志和依赖
```
没有环境前缀、别名表或隐式二次覆盖。业务只维护 `Config`、`Default()`、`Validate()` 和配置文件。项目 Validate 中继续调用嵌套框架配置的 Validate;生成模板已包含这些调用。验证函数只检查值,不应打开资源。
## 直接使用
```go
type Config struct {
Server omniConfig.Config `json:"server"`
BatchSize int `json:"batchSize"`
Enabled bool `json:"enabled"`
Password string `json:"password"`
Timeout time.Duration `json:"timeout"`
}
func Default() Config {
return Config{Server: omniConfig.Default(), BatchSize: 100, Enabled: true,
Timeout: 3*time.Second}
}
func (c Config) Validate() error {
if err := c.Server.Validate(); err != nil {
return omniConfig.AtField("server", err)
}
if c.BatchSize < 0 { return omniConfig.Invalid("batchSize", "must not be negative") }
return nil
}
// 手写项目;生成项目的 Load 已自动调用此入口。
cfg, err := omniConfig.LoadProject("etc/config.yaml", Default())
```
```yaml
server:
address: "${HTTP_ADDR:-127.0.0.1:8080}"
password: "${DB_PASSWORD}"
timeout: "${REQUEST_TIMEOUT:-3s}"
enabled: false
```
文件未写 batchSize,保留 100;enabled 明确写 false,就使用 false。变量名字完全由文件决定,不需要对应 Go 字段名。配置文件可只列出部署需要变动的字段;不会自动导出/打印包含凭据的完整配置。
## 引用语法
| 文件里的值 | 含义 |
| --- | --- |
| `host: 127.0.0.1` | 固定值,不读取任何环境变量 |
| `password: "${DB_PASSWORD}"` | 未设置或空字符串报错,不回退到 Default() |
| `port: "${DB_PORT:-3306}"` | 未设置或为空时使用 3306 |
| `sessionToken: "${SESSION_TOKEN:-}"` | 未设置或为空时使用空字符串,是否合法由目标类型和 Validate 决定 |
| `url: "https://${HOST}:${PORT:-443}/v1"` | 普通字符串中可出现多个引用 |
| `literal: "$${DB_PASSWORD}"` | 得到字面量 `${DB_PASSWORD}`,不读取变量 |
| `literal: "$$cost"` | `$$` 转义一个 `$`,得到 `$cost` |
变量名遵循 `[A-Za-z_][A-Za-z0-9_]*`。只支持 `${NAME}`、`${NAME:-fallback}` 和 `$$` 转义,不支持 `${NAME-default}`、嵌套引用或 shell 表达式。普通单独的 `$` 保留。fallback 到第一个 `}` 结束,可包含冒号、`:-`、引号等文件格式允许的字符;需要包含 `}` 的值可使用固定值或环境变量。
解析只扫描文件的**已解码字段值**,不替换 YAML/JSON/XML 源文本,不扩展字段名,也不扫描 Default() 的字符串。插入的环境值和 fallback 不再展开。密码 `DDSFLKJlsf:saf:-123`、引号、换行、`${OTHER}` 均原样保留,不破坏文档结构。字符串不统一 TrimSpace;`"0"`、`"false"` 和全空格字符串均属于已提供。
同一变量每次加载最多读取一次;下一次加载会重新读取。没有全局变量值缓存。并发加载各自维护结果,不能把调用者正在修改的默认配置当作共享可变输入。
## 类型与覆盖
目标 Go 类型决定引用结果的转换:整数、浮点数、布尔值、`time.Duration`、字符串、指针及已有自定义 JSON/Text 反序列化。数值范围按目标位宽检查,整数不经过 float64。自定义反序列化仍由自定义类型解释文本,不猜测其业务含义。
- 文件缺失字段保留初始值;显式 0、false、空字符串正常覆盖,不用零值判断字段是否存在。
- `${MISSING}` 失败就报错;需要备用值必须写 `${MISSING:-value}`。
- 字符串编号 `"001"` 保持 `001`;绑定到 int 才转整数。环境里的 `null` 不自动变成空指针,`[1,2]` 不自动变成数组。
- 集合直接在文件里声明,可在每个标量元素中使用引用,例如 `ports: ["${PORT_A:-8080}", "${PORT_B:-8081}"]`。
- 数组整体替换;空 slice 保持非 nil;固定长度 Go 数组要求长度一致。map 合并键,已提供键的值整体替换。
- 固定文件 null 沿用 JSON 语义:清空指针/map/slice/interface,普通值字段保持初始值。需要区分 null 时使用指针。
- `any` 的引用结果保持字符串,不猜测动态类型。固定 JSON/YAML 数字使用 json.Number 保留精度。
- 引用成功后的值不会再覆盖一次。旧 `GO_OMNI_*` 名称只有出现在文件引用中才有作用;不隐式重置日志 file/rotation。
普通固定字符串仍按原有严格绑定规则处理;不会为了支持变量,把所有 `"123"` 都自动转换成数字。
## YAML、JSON、XML 一致
使用 Go 的 `json` 字段名,无需维护三份 tag。以下三份文件等价:
```yaml
port: "${PORT:-3306}"
password: "${PASSWORD}"
```
```json
{"port":"${PORT:-3306}","password":"${PASSWORD}"}
```
```xml
<config><port>${PORT:-3306}</port><password>${PASSWORD}</password></config>
```
XML 使用 `<config>` 根,标量占位符写在普通叶元素或 `type="string"` 元素中,由 Go 字段决定类型;显式 `type="number"` / `type="bool"` 表示文件里已经有合法数字/布尔常量。XML 数组用 `type="array"` 加 `<item>`;空对象 `type="object"`、null 用 `type="null"`。不从重复元素猜测数组。
`.api` 不经过配置加载器,响应模板 `${code}` 不变。**如果把响应模板写在 YAML/JSON/XML 配置文件里,须转义:**
```yaml
server:
response:
body:
code: "$${code}"
msg: "$${message}"
data: "$${data}"
```
加载后恢复为 `${code}` 等响应占位符,仍由响应编码器处理。
## 生成项目与检查
| 文件 | 维护方 |
| --- | --- |
| `main.go`、`internal/config/loadGen.go` | 生成器维护,纳入 `generate --check` |
| `internal/config/config.go` | 项目维护 Config、Default、Validate |
| `etc/config.yaml` 等选定文件 | 项目维护,重新生成不覆盖 |
| `internal/config/loader.go` | 初次生成即可使用,仅添加自定义格式时才修改 |
```sh
go run . -config etc/config.test.yaml -check-config
go run . -config etc/config.local.yaml
go build .
go-omni api generate --check
```
`-check-config` 使用与启动完全相同的 Load,通过时退出 0、失败时非零;不打印配置,不创建日志文件、数据库、后台组件或 HTTP 监听。它只校验配置,不检测真实数据库连接或证书握手。
`-config` 只选一个文件,不自动叠加 local/test/default。指定文件不存在时报错;`-config ''` 只使用 Default() 后校验,不读取任何环境变量,也不寻找其他文件。示例:[examples/configuration](examples/configuration/README.md)。
## 安全诊断与限制
```text
config "etc/config.yaml":7:9 env "DB_PORT" field "database.port": expected a valid int
```
`*config.Error` 包含 File/Field/Env/Line/Column/Reason。引用错误同时保留文件与环境变量名;组合引用失败可列出多个名字。行列定位原文件字段,不把环境值伪装成源文件文本。无法取得位置时仍提供可用的文件/字段信息。
用 `config.Invalid(field, publicReason)` 或 `config.AtField(prefix, err)` 报业务校验错误。未知字段给出唯一且接近的已声明字段建议,不自动修改。错误默认不打印原值;自定义 Reason 不应拼入秘密值。原始 cause 仍支持 errors.Is/As,不要把 Unwrap 后的错误或整个配置写到日志。
文件最多 1 MiB;单个环境值/展开字段最多 1 MiB,展开字符串合计/变量缓存分别最多 4 MiB;解析/展开限制 64 层、100000 个节点与引用计数。拒绝无效 UTF-8、多文档、重复字段、未知字段、非普通文件;YAML 不接受 alias/merge/custom tag,XML 不接受 DTD/外部实体。不执行 shell,不自动加载 `.env`。
## 框架模块的字段诊断
v0.23 起 MySQL(含连接池/TLS)、通用数据库池、日志(含轮转/时区)、HTTP 客户端、退出阶段和请求策略返回字段级安全原因。沿用项目已有的 `config.AtField("database", err)` / `config.AtField("server", err)`,自动组合嵌套字段,不需业务重复校验基础参数。
例如文件中 `database.port: "${DB_PORT}"` 的引用值为 70000:
```text
config "etc/config.yaml":3:9 env "DB_PORT" field "database.port": must be between 1 and 65535
```
日志级别非法指向 `server.log.level`;池参数指向 `database.pool.maxIdleConns` / `pingTimeout`。框架只公开已定义的安全原因,第三方/项目普通 error 仍显示 validation failed 并保留 errors.Is/As 原因链;业务使用 Invalid/AtField 提供公开原因。不要解析错误字符串;使用 `*config.Error` 字段。未提供字段的位置可能不可用,字段与原因仍保留。
生成项目现有 Load 和 `-check-config` 自动获得这些诊断。日志可在 `server.log.timeZone` 配置 IANA 名称,见 [实例时区](logx/README.md#实例时区)。
## 可选解析器与测试来源
内置格式无需项目手写加载器。需要额外格式时在已有 NewLoader 注册 Decoder,后续仍自动经过同一引用/绑定/校验流程:
```go
func NewLoader() (*omniConfig.Loader, error) {
return omniConfig.NewLoader(
omniConfig.WithDecoder("custom", omniConfig.JSONDecoder{}),
)
}
```
Decoder 返回 Document(map/string/bool/json.Number/nil/[]any,XML 类标量可用 Text),可提供 Positions。实例注册无全局 init;插件必须可并发调用、不修改共享输入,并且是可信 Go 代码。
测试或嵌入宿主可以用 `WithLookupEnv(func(string)(string,bool))` 替换引用查找函数。它仍然只读取文件明确引用的名称,不启用任何自动覆盖。普通业务不需要这个选项。
## v0.22 迁移:旧模式已移除
这是不兼容调整,不保留两套加载逻辑:
- 删除 `EnvOptions`、`LoadFromEnv`、`ApplyEnv`、`OverlayEnv`、`ApplyDefaults`、`ServerEnvAliases` 和 `env`/`envDefault` 标签处理;检出遗留标签会提示迁移。
- 删除 `database.ConfigFromEnv` / `database.ApplyEnv`;数据库配置作为项目 Config 的字段绑定。DSN 字段可在文件里使用 `dsn: "${DB_DSN}"`,序列化 database.Config 不输出 DSN 原值。
- `LoadProject(path, defaults[, loader])` 不再接收 EnvOptions;`LoadFile(path)`、`NewReloader(path, logger)` 不再接收前缀。
- `omni.LoadConfig(path)` 的参数现在是文件路径;空路径只用框架默认值。
迁移顺序:
1. 将 envDefault 默认值搬入 Default(),移除 env/envDefault 标签。需要文件绑定的旧 `json:"-"` 密钥字段改成明确 json 名称;不打印配置对象。
2. 按实际部署需要,在选定文件写环境引用。固定值保持固定,不按原前缀猜测。原先依赖环境切换日志输出的项目,应在文件明确配置 file/output/rotation。
3. 配置文件中的其他 `${...}` 模板改成 `$${...}` 字面量;`.api` 无需修改。
4. 手写项目改为 LoadProject。生成项目保留 Config/Default/Validate/NewLoader,删除旧手写 Load 后重新生成;main/loadGen 由生成器更新,业务配置文件不会被静默改写。
5. 升级匹配的 CLI 与框架,执行 generate、generate --check、go test、go build 和 `-check-config`,再使用真实部署配置启动。
低层 `DecodeFile(path, defaults[, loader])` 也解析显式引用,但不调用 Validate;默认值和原始 Document 都不会因失败而被部分修改。日志 Reloader 仍只允许 log.level 变化,其他静态字段变更要求重启。
## 国际化配置
通用国际化的 `i18n.Config` 可作为项目配置的字段显式加载并调用 Validate。正文词典使用 `i18n.LoadFS` / `DecodeJSON`,不经过环境引用展开。构建 Bundle 后通过 `WithLanguagePolicy` / `WithErrorLocalizer` 装配借用的只读实例;没有隐式环境变量或全局当前语言。见 [国际化指南](i18n/README.md) 和 [完整示例](examples/i18n/README.md)。
## 跨域配置
生成项目的 `server.cors` 由本加载流程直接绑定、校验并交给 App,不需要修改生成入口。支持固定来源数组或元素内显式环境引用;非法来源定位具体数组元素。默认关闭,完整配置与行为见 [CORS 指南](cors/README.md)。

166
COPY.md Normal file
View File

@@ -0,0 +1,166 @@
# 可选拷贝工具 copyx
v0.19 用普通 Go 调用取代 JSON 结构映射生成。`copyx` 只依赖标准库,不需要注册到 App,不执行 JSON 往返,也不会自动暴露数据库模型。普通 Logic 直接使用 `.api` 生成的 DTO;只有存在独立业务类型或需要隔离可变数据时才用它。
完整可运行代码:[examples/copying](examples/copying)。在框架根目录运行:
```sh
go run ./examples/copying
go test -race ./copyx ./examples/copying
```
## 最简单的调用
```go
import "git.yuntuo.hk/k2safe/go-omni/copyx"
type Request struct { Name string; Enabled bool }
type Input struct { Name string; Enabled bool }
req := Request{Name: "sample", Enabled: false}
var input Input
if err := copyx.Copy(&input, req); err != nil { return err }
```
同名、同类型字段直接赋值。默认不会跳过 `false`、`0`、空字符串。目标字段没有来源时报错;多余源字段默认忽略,使用 `WithStrictFields()` 可要求源字段也全部使用或显式忽略。
默认是浅拷贝:同类型指针、map、slice 共享引用,符合普通 Go 赋值。JSON tag 不参与字段匹配。不会自动调用方法、SQL Scanner/Valuer,或把 `int32`、`int64`、string、命名标量互相强转。
新字段:普通接口只改 `.api` 和使用该字段的 Logic。独立 DTO/业务类型都需要此字段时,两边各声明一次;同名同类型不增加拷贝配置。检查在运行时发生,实际类型对必须有测试,不能把运行时工具当作旧生成器的编译期字段覆盖检查。
## 类型预检
v0.23 新增可选预检,无需填写映射文件或构造测试值:
```go
// 参数方向是源 → 目标;与 Copy 使用同一组选项。
if err := copyx.Check[Request, BusinessInput](options...); err != nil {
return err
}
// 复用已有 Copier:传结果类型,不添加 Copy 所需的额外输出指针。
err := copier.Check(reflect.TypeFor[Request](), reflect.TypeFor[BusinessInput]())
```
适合 CI 测试或初始化检查;这是程序执行的类型检查,不是 Go 编译器自动证明。递归检查切片/数组元素、map 键值、指针、嵌套字段、改名、忽略和严格覆盖;用已访问类型对处理递归结构。不执行 converter,也不调用构造器,不读取实例值。错误路径如 `$.Rows[].Value: copyx: incompatible types: string -> int`。
Check 复用 Copy 的规则,包括浅拷贝对同类型容器的直接赋值:这种情况下内部 converter 不会自动逐项执行。需要处理内部字段时选择深拷贝或容器 converter。
- 未知具体值的源 interface 需要运行时验证;无法静态判定时返回 `ErrDynamicType`。项目可给该接口类型对提供精确 converter 并承担内容验证。
- 已注册 converter 的签名被接受,但函数不在 Check 中执行;其返回错误、panic、金额规则不能靠预检证明。
- nil 到非可空目标、map 键转换碰撞、深拷贝结果是否残留源引用继续在 Copy 时检查。Check 成功不会关闭这些保护。
- 实际 Copy 也修正了空集合的检查缺口:确定的嵌套类型不匹配不再因为切片/map 为空或数组长度为 0 而通过。需要这些转换时提供显式 converter;目标仍只在整体成功后更新。
- 不引入全局配置、映射语言或业务类型;Copier.Check 可并发调用,不共享每次检查的遍历状态。
示例:[developerTools](examples/developerTools/README.md)。普通同名同类型字段不需要额外配置。
## 深拷贝与构造器
```go
snapshot, err := copyx.Clone(source) // 保持源的静态类型,始终深拷贝
if err != nil { return err }
copier, err := copyx.New(copyx.WithDeepCopy(), copyx.WithStrictFields())
if err != nil { return err }
var result Output
if err := copier.Copy(&result, source); err != nil { return err }
```
`Copy` 的目标必须是非 nil 指针;`Clone[T]` 返回 `(T, error)`。重复执行时在初始化阶段创建 Copier 并复用;配置之后不可变,每次调用独立持有遍历状态,可以并发使用。源、目标的并发读写仍由调用方同步,自定义函数也须支持并发。
深拷贝会分配指针、map、slice,递归处理结构、数组、接口中的具体值。nil 集合与非 nil 空集合保持区别;slice 结果容量为长度,不复制长度之外的隐藏元素。普通循环和重复指针可处理。循环图应保留根指针,使用 `Clone(sourcePointer)` 或 `Copy(&targetPointer, sourcePointer)`;将循环指针根压成值结构会明确报错。
它保证结果不引用输入中的可变对象,不保证保留所有底层内存布局关系:不同重叠 slice 视图会分别复制,内部指针指向某个切片元素的关系也不重建。字符串和零尺寸值可安全共享;`time.Time` 按 Go 不可变值使用(Location 不做独立复制)。不要通过 unsafe 或 uintptr 隐藏可变引用后要求工具识别。
## 字段差异与自定义转换
```go
type Input struct { Name string; UpdatedAt time.Time; InternalID string }
type View struct { DisplayName string; UpdatedAt string; LocalNote string }
copier, err := copyx.New(
copyx.WithDeepCopy(),
copyx.WithStrictFields(),
copyx.WithFieldNames[Input, View](map[string]string{"Name": "DisplayName"}),
copyx.WithIgnoreSource[Input, View]("InternalID"),
copyx.WithIgnoreTarget[Input, View]("LocalNote"),
copyx.WithConverter(func(v time.Time) (string, error) {
return v.UTC().Format(time.RFC3339), nil
}),
)
if err != nil { return err }
out := View{LocalNote: "preserve"}
if err := copier.Copy(&out, input); err != nil { return err }
```
字段规则的类型参数必须是结构类型,字段名使用导出的 Go 字段名。构造时检查不存在的字段、重名及冲突。不支持字符串路径表达式;复杂嵌套差异为该嵌套类型注册 converter,或直接写 Go。
`WithIgnoreSource` 只是声明该源字段不参与使用;目标同名字段若仍需保留,还需 `WithIgnoreTarget`。忽略目标会保留原目标值,Clone 中则为零值。深拷贝时如果保留值仍与源共享可变引用,也会拒绝。
转换器的签名是 `func(S) (T, error)`,按**精确类型对**匹配,不能重复注册。金额、编号、日期格式由项目明确转换。需要逐字段不同规则时,给整个源/目标结构注册函数;函数返回独立的新目标。无需另一份映射配置或全局注册。
精确 converter 优先于 nil/零值策略,能收到 typed nil;例如 `func(*Money) (Money, error)` 可自行决定 nil 是否允许。绑定到 `any` 字段的 converter 能接收 nil interface,但顶层未带类型的 `nil` 没有精确类型可匹配。浅拷贝中已经直接共享的同类型容器不会继续逐项转换其内部;需要遍历内部时选择深拷贝,或为该容器类型注册 converter。
## 深拷贝不能原样返回可变引用
```go
copier, err := copyx.New(
copyx.WithDeepCopy(),
copyx.WithConverter(func(v []string) ([]string, error) {
return v, nil // 会返回 ErrSharedReference
}),
)
```
检查覆盖完整源图与转换结果,包括新父对象引用旧子对象、原 map、重叠子切片、`source[:0]` 的可写容量、指向源数组元素的指针,以及未导出字段中的引用。会检查每个 converter 的结果和最终目标,不允许静默降级为浅拷贝。
返回相同字符串、整数并不会报错,它们不是可变引用。正确切片 converter 可以写:
```go
copyx.WithConverter(func(v []string) ([]string, error) {
if v == nil { return nil, nil }
out := make([]string, len(v))
copy(out, v)
return out, nil
})
```
第三方 Decimal/自定义 Money 如果含未导出的指针,默认深拷贝拒绝;注册该类型的 converter,使用它自己的构造/Clone API 创建独立值,避免转成 float64 损失精度。框架不内置某一金额库的规则。无法独立复制的类型直接返回错误。
## nil、零值和错误
| 选项 | 行为 |
| --- | --- |
| 默认 `NilError` | nil → 可空目标保留 nil;nil → 非可空目标报 `ErrNil` |
| `WithNilPolicy(NilZero)` | 显式允许 nil → 零值;可空目标仍为 nil |
| `WithNilPolicy(NilSkip)` | nil 保留已有目标;新集合元素没有旧目标,保留元素零值 |
| `WithSkipZero()` | 显式跳过 Go 零值,包括 `false`、`0`、`""`;非 nil 空集合不是 Go 零值 |
所有 nil 策略都不把不兼容类型自动变成兼容类型。非 nil 指针与值之间可以按对应元素转换;nil 到非可空目标仍需选择策略。局部更新不能默认使用 NilZero/SkipZero,它们可能改变清空字段或布尔值的业务语义。
工具只能复制绑定后的 Go 值:如果输入类型已经将“缺失”和“JSON null”合并为 nil,无法恢复两者区别。需要三态时,应先设计包含 presence 信息的类型和绑定逻辑。
赋值使用临时结果,成功后才替换目标;转换失败不会交付部分结果。`*copyx.Error` 包含 Go 字段/索引路径,`errors.Is/As` 保留内部错误链,工具不主动在错误中加入源值。converter 自己返回的错误文本仍由项目负责。
converter 必须只读输入;其外部副作用、输入修改无法回滚,panic 原样传播。框架只检查与本次输入的共享引用,无法证明 converter 未返回外部全局缓存的引用。它是受信 Go 代码扩展,不是可运行不可信代码的沙箱。
## 支持范围与资源限制
- 支持结构体、指针、interface、slice、同长度 array、map。map key 转换后冲突或不可比较则报错。
- 导出匿名结构字段可按提升后的无歧义名称访问;源嵌入指针为 nil 且需要其字段时明确报错。跨类型目标包含私有字段时拒绝;同类型私有纯值可作为整体保留。
- 不克隆文件、数据库连接、网络连接、锁、原子状态。已知标准库资源/同步结构按值复制也拒绝;深拷贝中的非 nil func、chan、unsafe.Pointer 拒绝。其他封装资源类型应由项目排除,工具不能识别所有自定义句柄。
- 单次遍历深度上限 256、节点上限 200,000;内置分配的估算预算为 64 MiB,超限返回 `ErrLimit`。预算不是进程堆内存硬限制,reflect 元数据和 converter 自行分配不计入。大对象流式读写应使用 IO/存储接口。
- 只缓存不可变类型元数据,不缓存请求值、用户 converter 或实例策略。没有 `unsafe` 写入、JSON 往返、隐式数值截断或方法调用。
- 这是运行时反射工具,会比直接 Go 赋值多检查与分配。热点应复用 Copier,并用真实数据测量;性能记录见 [本版验收](reports/copyxAcceptance.md)。
## 从旧 mapping 迁移
v0.19 **彻底移除**映射配置加载、Go 类型加载、映射代码生成及 `pointerResult` 等附属选项。公共 `.api` 字段组合、scalar、中间件工厂、Repository 开关继续存在。
1. 先检查每个转换是否必要。普通 Logic 可直接使用生成 DTO;独立核心保留自己的类型。
2. 将 `mapping.Xxx(...)` 调用改为普通 Go 或小型 `copyx` helper。字段改名、nil、金额规则写在该 helper 同一处。指针返回可直接 `var out *Response; err := copier.Copy(&out, input)`。
3. 用真实类型对测试缺失/null/零值/空数组、字段覆盖、金额精度和错误链。需要源字段新增时提醒遗漏,启用 `WithStrictFields()` 并测试。
4. 从 `omni.gen.json` 删除 `mappings` / `mappingFiles`,删除无其他用途的映射 JSON 文件。
5. 确认已无调用,再删除 `internal/mapping/mappingsGen.go`。如之前手改过文件,先迁出手写内容。生成器遇到该旧文件会阻止生成并提示迁移,**不会静默删除**;其余业务文件照常保留。
6. 升级框架与 CLI 到同版,执行 generate、generate --check、go test ./...;多个服务再执行 workspace generate/check。新 manifest 自动去掉已移除文件的所有权记录。
历史发布记录保留旧能力说明,仅用于理解旧版本,不代表 v0.19 继续支持。

View File

@@ -1,8 +1,8 @@
# 通用业务开发指南(v0.18.0-preview) # 通用业务开发指南(v0.26.0-preview)
从 [README](README.md) 创建服务后,业务主要维护 `.api`、配置、`svc.Init`、Logic、Repository 和项目中间件。本页说明通用接入能力;字段、错误码、权限、表结构和事务边界都由项目定义。 从 [README](README.md) 创建服务后,业务主要维护 `.api`、配置、`svc.Init`、Logic、Repository 和项目中间件。本页说明通用接入能力;字段、错误码、权限、表结构和事务边界都由项目定义。
本版新增的公共字段组合、可选结构映射、自动中间件工厂和 Repository 开关,见 [生成能力指南](GENERATION.md)。 公共字段组合、自动中间件工厂和 Repository 开关,见 [生成能力指南](GENERATION.md)。必要的结构转换使用可选 [copyx](COPY.md),v0.19 已移除旧 JSON 映射生成。
## 返回数组、可空对象和 null ## 返回数组、可空对象和 null
@@ -100,42 +100,13 @@ defer reader.Close()
不声明 multipart 字段时,保留原来的 `body *httpx.Body` 流式入口,不创建临时文件。该方式适合特殊协议;项目应在不可撤销副作用之前完成校验并调用 `body.Complete()`。下载和原始回调继续使用 `@produces` / `@body(mode: raw)`。 不声明 multipart 字段时,保留原来的 `body *httpx.Body` 流式入口,不创建临时文件。该方式适合特殊协议;项目应在不可撤销副作用之前完成校验并调用 `body.Complete()`。下载和原始回调继续使用 `@produces` / `@body(mode: raw)`。
## 完整业务配置的环境覆盖 ## 文件声明配置来源
生成项目默认执行:代码与 `envDefault` 默认值 → JSON/YAML 文件 → 全结构体环境覆盖 → 既有框架环境别名 → `Validate`。增加业务字段不需要另外写 `os.Getenv`。 项目维护 Config、Default()、Validate() 和配置文件;生成器维护 Load/main。流程为默认值→选定文件→显式 `${NAME}` / `${NAME:-fallback}` 引用→类型绑定→校验。没有前缀、别名或 env/envDefault 隐式覆盖。
```go 固定值保持固定,缺失字段保留 Default(),0/false/空字符串正常覆盖;引用失败不回退初始化默认值。`$$` 转义 `$`,字符串凭据保留空格与特殊字符。
type ClientConfig struct {
Endpoint string `json:"endpoint" env:"CLIENT_ENDPOINT,required"`
Token string `json:"-" env:"CLIENT_TOKEN,required,secret,empty=reject"`
Timeout time.Duration `json:"timeout" envDefault:"3s"`
Retries []int `json:"retries"`
Enabled bool `json:"enabled"`
}
```
把字段加入项目 `Config` 后,默认环境名取 `GO_OMNI_` 加 JSON 路径的 UPPER_SNAKE_CASE,如 `GO_OMNI_CLIENT_TIMEOUT`。显式 env 名是完整名称,可以使用自己的业务前缀。`env:"-"` 排除整个子树;`json:"-"` 字段只有显式 env 标签才参与。 `go run . -config etc/config.test.yaml -check-config` 只校验,不打开日志、数据库或端口。生成/check/dry-run 保留业务文件;新的模板要求 v0.22。完整格式、错误定位、解析器与移除清单见 [CONFIG.md](CONFIG.md)。
duration 使用 `250ms`、`3s`;slice/map 使用 JSON,如 `[1,3,5]`、`{"retry":"2s"}`。支持嵌套、零/false、空数组、自定义 TextUnmarshaler。空环境值默认允许;可选择 `empty=ignore` 或 `empty=reject`。required 要求字符串非空、指针/集合非 nil;0 和 false 可以是合法业务值,范围由项目 Validate 决定。
手写项目可使用 `config.LoadProject(path, defaults, config.EnvOptions{Prefix:"MY_APP_"})`,或分别调用 `ApplyDefaults/DecodeFile/OverlayEnv`。读取失败不修改输入默认值;诊断不会打印输入值。不要主动打印含密钥的整个配置,或把底层解析错误展开到公开响应。Validate 不打开日志文件、数据库或存储连接;证书的可读性和握手信任在依赖初始化阶段验证。
### 一次覆盖完整配置与 HTTP/日志短名称
```go
cfg, err = config.OverlayEnv(cfg, config.EnvOptions{
Prefix: "MY_APP_",
Empty: "reject", // allow / ignore / reject,项目自行选择
Aliases: config.ServerEnvAliases("MY_APP_", "server"),
// Lookup 可注入测试或项目配置源;默认 os.LookupEnv。
})
```
不要再紧接着调用 `ApplyEnv(cfg.Server, ...)`。上面的调用同时处理完整配置和 HTTP_ADDR、SERVICE、LOG_LEVEL 等框架短名称。例如 `MY_APP_SERVER_ADDRESS` 与 `MY_APP_HTTP_ADDR` 指向同一字段,**完整名称优先**;未设置时才读取短名称。完整名称显式设空仍优先:allow 交给该字段解析(string 可清空;数字等仍要求合法值),ignore 保留文件/默认值,reject 报错。不会因 ignore 再退回别名。字段 `env` 标签的 empty 规则优先于全局规则。
自定义别名用 `EnvOptions.Aliases`,键是完整环境名、值是按优先级排列的别名数组。两种名称都使用同一 Lookup、空值策略和类型转换;布尔 false、数值 0 与未设置不同。集合始终使用 JSON,包括 `MY_APP_TRUSTED_PROXIES='["127.0.0.1"]'`。
新项目模板一次调用 OverlayEnv,默认 `Empty: "ignore"`。旧 `ApplyEnv` 保留原有忽略空值、修剪空白及 TRUSTED_PROXIES 逗号列表行为。已有项目的 `internal/config` 属于开发者文件,升级不覆盖它;按上例合并原来的两次覆盖调用后,保留项目自己的 Validate。日志输出的显式完整名称也优先于短名称。
## 依赖、日志与中间件 ## 依赖、日志与中间件
@@ -278,7 +249,7 @@ go-omni workspace check --file omni.workspace.json
## v0.13 项目升级 ## v0.13 项目升级
先使用与目标框架匹配的 CLI;CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.18.0-preview(或显式引用当前源码)。 先使用与目标框架匹配的 CLI;CLI 会提示与 go.mod 不同的版本。当前生成模板至少需要 v0.22.0-preview(或显式引用当前源码)。
```sh ```sh
go-omni api upgrade --dir . --dry-run go-omni api upgrade --dir . --dry-run

View File

@@ -1,18 +1,10 @@
# 公共字段、结构映射与生成选项 # 公共字段、中间件工厂与生成选项
本页对应 `v0.18.0-preview`。CLI 与框架依赖固定到同一版本: 本页对应 `v0.19.0-preview`。CLI 与框架依赖固定到同一版本,安装见 [README](README.md#准备环境与安装)。
```sh
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.18.0-preview
# 已有业务项目内执行:
go get git.yuntuo.hk/k2safe/go-omni@v0.18.0-preview
```
私有仓库访问设置见 [安装说明](README.md#准备环境与安装)。升级不会覆盖业务文件;已有中间件保持 handlers 模式,切换工厂模式按本页迁移步骤完成。
## 一份项目生成配置 ## 一份项目生成配置
新项目根目录生成开发者维护的 `omni.gen.json`: 新项目根目录的 `omni.gen.json` 只描述生成选项:
```json ```json
{ {
@@ -21,32 +13,11 @@ go get git.yuntuo.hk/k2safe/go-omni@v0.18.0-preview
} }
``` ```
该文件与 `.api` 一起提交。每次 `api generate`、`--check`、`--dry-run` 和 `workspace generate/check` 都会读取;`api validate/export/format` 只处理契约,不加载业务 Go 类型。 - `repository: none` 默认不创建空 Repository 文件;`group` 按 group 补齐可编辑骨架。已有文件始终保留;切回 none 不会删除业务文件。SQL、ORM 和仓储划分由项目决定。
- `middleware: factories` 是新项目默认值,自动调用带依赖的工厂。`handlers` 保留原有 `Defaults()` 及手动注册方式,供既有项目迁移。
- `repository: none` 是默认值,不创建空 Repository 文件。`group` 按 `.api` 的 group 补齐 `internal/repository/<group>/repository.go`;文件始终属于业务。切回 none 后不再补骨架,已有文件不删除、不覆盖。实际 SQL、ORM 和仓储划分由项目决定。 - 该文件与 `.api` 一起提交。generate/check/dry-run 和 workspace 采用相同规则;validate/export/format 只处理接口契约。
- `middleware: factories` 是新项目默认值,按下文自动初始化。`handlers` 保留原有 `Defaults()` 与 `func(next Handler) Handler` 接口,供既有项目逐步迁移。 - 旧 manifest 没有配置时补 `middleware: handlers`、`repository: none`。已有配置不能直接删除,避免意外切换中间件模式。
- `mappings` 可省略;只有明确列出的类型对才生成映射。未启用时不会运行 Go 类型加载命令,也不增加运行时依赖。 - v0.19 已删除 `mappings` 和 `mappingFiles`;遇到旧配置会明确报错,不读取子映射文件或执行映射生成。处理步骤见 [copyx 迁移](COPY.md#从旧-mapping-迁移)。
- 升级首次遇到没有配置的旧 manifest 时,补一份 `middleware: handlers` 的配置,保留原有手写注册;Repository 默认 none。生成配置建立后不能直接删除,否则生成失败,避免误关闭映射或改变中间件模式。
### 按模块拆分映射(v0.18 新增)
主配置可以保留少量内联 `mappings`,也可以只列出文件:
```json
{
"repository": "none",
"middleware": "factories",
"mappingFiles": ["mappings/requests.json", "mappings/responses.json"]
}
```
每个子文件只包含 `{"mappings": [...]}`,其中每条映射的写法与原来一致。先加载主配置内联定义,再按文件列表顺序加载;所有定义合并后统一校验,可以跨文件、向前引用 `mapping`。输出按映射名排序,文件拆分不改变函数名或语义。
- 路径相对项目根目录,必须是明确列出的 `.json` 文件。不展开通配符,不允许越界路径、符号链接、重复文件或多层导入;子文件不能覆盖 repository/middleware 等项目选项。
- 重名映射、指针辅助函数名冲突、未知映射与循环引用明确报错;文件之间不能用覆盖顺序隐藏冲突。重复定义会指出两个来源文件,字段生成错误也带来源文件。
- 文件均由项目维护,重新生成不改写它们。原来的单文件 `mappings` 继续有效。迁移时将完整条目移到子文件并删除原位置,不能复制后保留两份同名定义。
- 最多 64 个子文件、单文件 1 MiB、全部配置 4 MiB;合计仍最多 128 个映射。主配置和子文件一起提交 Git。
- `api generate`、`--check`、`--dry-run` 与 `workspace generate/check` 使用同一加载逻辑。缺少子文件或不完整配置立即失败,check/预览不写入;生成期间重新核对所有配置输入,发生变化时要求重试。
## 公共字段组合 ## 公共字段组合
@@ -73,151 +44,13 @@ type AdminRequest {
允许多层组合,上限 32 层、单类型 4096 字段、全契约展开 65536 字段。Go 匿名指针嵌入并非本功能;生成 DTO 中不会使用匿名嵌入。 允许多层组合,上限 32 层、单类型 4096 字段、全契约展开 65536 字段。Go 匿名指针嵌入并非本功能;生成 DTO 中不会使用匿名嵌入。
## 可选结构映射 ## DTO 与业务类型
`.api` 生成 DTO,业务独立维护输入/结果类型。项目明确选择需要自动赋值的类型对;不从数据库反向生成公开接口,不自动推断反向映射。 普通 Logic 直接接收生成的 Request、返回生成的 Response,不额外创建相同结构。只有独立核心或明确不同的输入/输出职责才保留业务类型。
假设 `types.AdminRequest` 有 `Name string`、`Salary amount.Money`、`CompanyID string`,业务的 `domain.EmployeeInput` 有 `EmployeeName string` 和同类型 `Salary`: 必要转换直接写 Go,或用可选 `copyx.Copy(&target, source)`。同名同类型字段自动赋值,字段改名和类型变化通过 Go 参数、转换函数明确设置,见 [完整示例](examples/copying/main.go)。无需 JSON 映射配置、包路径字符串或额外生成文件。
```json `copyx` 在运行时检查类型和字段,**不提供旧映射生成阶段的静态字段覆盖保证**;CI 应给实际类型对添加测试。普通业务字段新增通常只改 `.api` 和 Logic;独立 DTO/领域类型都需要此字段时,两边各加一次,同名同类型无需追加拷贝配置。
{
"repository": "none",
"middleware": "factories",
"mappings": [
{
"name": "ToEmployeeInput",
"source": {"package": "example.com/payroll/internal/types", "type": "AdminRequest"},
"target": {"package": "example.com/payroll/internal/domain", "type": "EmployeeInput"},
"fields": {
"EmployeeName": {"source": "Name"}
},
"ignoreSource": {
"CompanyID": "由 Logic 单独做企业范围授权,不属于请求体输入"
}
}
]
}
```
执行 `go-omni api generate` 后,产生独立的生成文件 `internal/mapping/mappingsGen.go`:
```go
func ToEmployeeInput(source types.AdminRequest) (domain.EmployeeInput, error) {
var result domain.EmployeeInput
result.EmployeeName = source.Name
result.Salary = source.Salary
return result, nil
}
```
包别名由生成器决定。业务调用 `mapping.ToEmployeeInput(*req)`;反向响应映射按另一条 source/target 规则声明。两个类型完全相同时通常无需映射。
### 类型与覆盖检查
自动赋值要求同名且 **Go 类型一致**(类型别名按 Go 类型系统处理),字段名使用 Go 名称,不按 JSON 标签猜对应关系。int32 → int64、`*Money` → Money、数值 ID → string 都必须显式转换。不会暗中补零、解引用或执行数字强转。
每个源字段必须被使用或出现在 `ignoreSource`;每个目标字段必须被赋值或出现在 `ignoreTarget`。忽略规则必须给出非空原因,目标忽略明确意味着保留零值;规则引用不存在的字段也会失败。新字段若无法自动匹配,会在生成阶段报错。两边同时添加的同名同类型字段会自动加入生成代码,`--check` 检出未重新生成的差异。
根类型是项目可导入的具名结构;支持值类型匿名嵌入,规则可用 `Employee.EmployeeName` 这样的实际 Go 字段路径。同名提升歧义、未导出字段和匿名指针嵌入拒绝处理。普通具名嵌套字段使用独立映射或转换函数,不猜测内部结构。泛型声明不能直接作为根;具体字段类型按 Go 类型系统检查。
配置变化、遗漏映射、类型错误会在写入前失败。映射生成与其他生成产物使用同一套文件所有权机制;手改映射生成文件会被拒绝覆盖。
### 显式转换函数
```json
"fields": {
"Salary": {
"source": "Salary",
"convert": {"package": "example.com/payroll/internal/convert", "name": "SalaryValue"}
}
}
```
支持这两种精确签名:
```go
func SalaryValue(source *amount.Money) amount.Money
func SalaryValue(source *amount.Money) (amount.Money, error)
```
项目选择一种实现。nil、默认值、金额精度、日期格式和错误码由函数负责;若两边已经是同一个 Money 类型,无需再次解析或序列化。转换函数应仅负责值转换,不执行 SQL 或改变授权范围。
错误通过 `%w` 加上映射名/目标字段路径,保留 `errors.Is/As` 和框架错误链;返回零目标值,不交付部分完成的结果。包装不主动添加源值或转换后的值。错误是否公开仍由项目错误目录和响应策略决定。
### 嵌套对象、数组与引用
`mapping` 指向同配置中的另一条映射,`elements` 明确要求逐项映射:
```json
"fields": {
"Summary": {"source": "Summary", "mapping": "ToSummary"},
"Items": {"source": "Items", "mapping": "ToItem", "elements": true}
}
```
`mapping` 与 `convert` 互斥。`elements` 可以配合其中任意一种,也可以对类型相同的元素直接赋值;支持 slice→slice、等长 array→array,禁止隐式互转。nil slice 保持 nil,非 nil 空 slice 保持非 nil,逐项失败的错误包含索引,循环映射声明被拒绝。
未选择 elements 的同类型指针、map、slice 使用普通 Go 赋值,**共享引用**,不做隐式深拷贝。elements 会分配新的目标 slice,但元素内的引用仍遵循其映射规则。需要深拷贝时显式提供转换函数。
映射不保存原 JSON 的字段存在信息。普通 `*T` 在绑定后可能已把缺失和 null 合并为 nil;无法在转换阶段恢复区分。有三态需要的项目必须先使用能携带该信息的输入类型及绑定扩展;没有声明三态时,保持当前输入类型的语义。完整表单更新和局部更新由业务决定,框架不把 nil 自动解释成“保留旧值”。
### 可空嵌套映射与显式 nil 策略(v0.18 新增)
假设已有 `ToChild` 定义 `types.Child → domain.Child`。父对象的字段规则可以直接复用它:
```json
"fields": {
"Child": {"source": "Child", "mapping": "ToChild"},
"Children": {"source": "Children", "mapping": "ToChild", "elements": true}
}
```
| 源字段 → 目标字段 | 明确选择 mapping 后的行为 |
| --- | --- |
| `S → T` | 调用原值映射 |
| `*S → *T` | 默认保留 nil;非 nil 调用值映射,并分配新的目标对象;也可配置 `nil: "error"` 拒绝 nil |
| `S → *T` | 调用值映射,成功后返回目标对象指针 |
| `*S → T` | 必须选择 `nil: "zero"` 或 `nil: "error"`,不猜默认值 |
`nil: "preserve"` 只适用于可空源和可空目标。`nil: "zero"` 只适用于可空源和非空目标:nil 直接产生目标 Go 零值,**不调用子映射或转换函数**。`nil: "error"` 在 nil 时返回带映射/字段路径的普通转换错误,HTTP 公开响应仍由项目的错误处理策略决定;需要领域错误码时使用项目 converter。
同一规则适用于显式 `elements: true` 下的 slice/等长数组元素,包括 `[]*S → []*T`。nil slice、非 nil 空 slice、nil 元素分别保留;元素失败保留索引与内部错误链,整个映射返回零目标值,不交付部分结果。只支持这里列出的单层指针,不自动处理多层指针或递归对象图。
普通同类型指针字段未指定 mapping 时仍直接赋值、共享引用;上述显式映射会创建目标子对象,但不承诺对其中所有引用深拷贝。业务不可变快照继续由项目设计。
简单的 `*T → T` 也可以单独指定 nil 策略,减少重复解引用函数:
```json
"fields": {
"Title": {"source": "Title", "nil": "zero"},
"Count": {"source": "Count", "nil": "error"}
}
```
这里必须是完全相同的 Go 元素类型,`*int32 → int64`、枚举 → string、数字编号 → string 仍需显式 converter。非 nil 的零值正常保留。`convert` 与 `nil` 不能同时设置,现有 converter 继续完整控制自己的空值、格式和错误语义。
### 可选指针返回辅助(v0.18 新增)
在映射条目上增加 `"pointerResult": true`,生成器保留 `ToResponse(source S) (T, error)`,另加:
```go
func ToResponsePtr(source S) (*T, error)
```
Logic 在业务调用成功后可直接写 `return mapping.ToResponsePtr(result)`。转换失败时返回 `nil, err`,错误链不变;成功才返回非 nil 指针。业务调用本身的错误仍由 Logic 处理,辅助函数不执行任何业务操作。
辅助函数名称固定为映射名加 `Ptr`,与其他映射/辅助函数同名会在生成前报错。输入仍是原来的值类型 S,不用 nil 输入表达业务错误;关闭该选项会移除辅助函数,使用它的业务代码需同时调整。
### 生成顺序与 CI
1. 先从 `.api` 生成项目,添加业务结构、scalar 和转换函数,执行 `go mod tidy`。
2. 在 `omni.gen.json` 列出映射,执行 `go-omni api generate`。
3. Logic 调用生成函数。后续改字段后重复生成,不修改 `*Gen.go`。
4. CI 同时运行 `go-omni api generate --check` 与 `go test ./...`。
启用映射后,CLI 通过 `go list -deps -export` 加载明确声明的包与依赖,按当前构建环境做类型检查,不执行业务代码、init 或转换函数。采用待生成 DTO 的临时 overlay,避免“新增字段还看旧 DTO”。go.mod/go.sum 的副本及 overlay 都在临时目录,check 不修改项目文件;Go 自己的构建/模块缓存可能更新。使用当前 GOOS/GOARCH/GOFLAGS、`GOWORK=off`;依赖用 go.mod 与 replace 表达,生成超时为 90 秒。
类型包/转换函数包不能依赖输出包 `internal/mapping`;把转换函数放在独立包中,避免导入环。若包本身无法编译,先运行 `go list -deps -export <相关包>` 排查;CLI 不回显可能含私有仓库凭据的 Go 下载输出。纯值字段转换只生成标准 Go 代码,不引入运行时反射搬运、JSON 往返、ORM 或队列依赖。
## 中间件工厂自动注册 ## 中间件工厂自动注册
@@ -271,9 +104,13 @@ factory 模式不再在 Init 写 `Middlewares["..."]`;残留非空手写注册
| 项目维护 | 生成器持续维护 | | 项目维护 | 生成器持续维护 |
| --- | --- | | --- | --- |
| `.api` 公共字段与接口 | 展开后的 DTO、Handler、OpenAPI | | `.api` 公共字段与接口 | 展开后的 DTO、Handler、OpenAPI |
| `omni.gen.json` 的映射与选项 | `internal/mapping/mappingsGen.go` | | `omni.gen.json` 的 Repository/中间件选项 | 对应骨架和注册代码;已有手写骨架不覆盖 |
| Go 业务类型、scalar、转换函数 | 编译期可检查的赋值与调用代码 | | 独立 Go 业务类型、scalar、可选 copyx 调用和测试 | 不生成结构转换文件 |
| middleware Dependencies、具名工厂、svc 访问器 | 中间件 registry、启动调用 | | middleware Dependencies、具名工厂、svc 访问器 | 中间件 registry、启动调用 |
| Logic、按需 Repository、配置、svc 初始化/清理登记 | main、runtime、路由接入 | | Logic、Repository、配置、svc 初始化/清理登记 | main、runtime、路由接入 |
回归样例位于 `cmd/go-omni/apigen/testdata/composition`,为独立三端服务,不导入任何 eTax 代码。 独立三端工资契约测试位于 `cmd/go-omni/apigen/testdata/composition`,不导入任何接入项目。它验证公共字段、Money scalar、三端 Handler、copyx、重新生成保留业务内容,以及新增字段不需要更新转换规则。
```sh
GO_OMNI_SCAFFOLD_INTEGRATION=1 go test ./cmd/go-omni/apigen -run TestCopyxThreeEndpointConsumer -count=1
```

View File

@@ -6,6 +6,24 @@
[examples/modules](examples/modules/README.md)。业务入口继续使用 `omni.New`,中间件依赖 [examples/modules](examples/modules/README.md)。业务入口继续使用 `omni.New`,中间件依赖
`httpx`,不依赖具体服务端网络实现。各模块按需创建、注入 Service,再注册到应用关闭流程。 `httpx`,不依赖具体服务端网络实现。各模块按需创建、注入 Service,再注册到应用关闭流程。
## 请求追踪与基础指标
`omni.New()` 的默认链包含 RequestID、访问日志(可配置开关)和 Recovery;无需重复挂载。业务通过 `c.Context()` 传递 requestId,`logx.FromContext(ctx)` 取得该 App 的 Logger。手动组装 httpx 链时,也可使用 `middleware.RequestID("X-Request-ID")`、`middleware.AccessLog()` 和 `middleware.Recovery()`。
轻量进程指标直接使用 `metrics`,无需 OTel collector:
```go
// import omni "git.yuntuo.hk/k2safe/go-omni"
// import "git.yuntuo.hk/k2safe/go-omni/metrics"
app := omni.New()
measurements := metrics.New()
app.Use(measurements.Middleware())
app.GET("/metrics", measurements.Handler())
// 也可以直接读取 measurements.Snapshot()。
```
记录请求数、失败数、处理中数量、累计耗时与 HTTP 状态分布;返回的是 JSON 快照,并非 Prometheus 文本格式。指标接口是否暴露以及访问权限由项目配置。需要 HTTP/SQL/Redis spans 和标准 OTel exporter 时,使用 [telemetry 示例](examples/platform/main.go)。需要跨域时配置 `server.cors`,由框架在路由与鉴权前处理预检;默认关闭。已有代理 CORS 可继续使用,避免两处重复设置。见 [CORS 配置与示例](cors/README.md)。
## 缓存:cache 与 cache/redis ## 缓存:cache 与 cache/redis
```go ```go
@@ -52,30 +70,26 @@ application.OnShutdown(client.Close)
```go ```go
client, err := httpc.New(httpc.Config{ client, err := httpc.New(httpc.Config{
Timeout: 3*time.Second, MaxRetries: 2, MaxConnsPerHost: 32, Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
MaxResponseBytes: 1 << 20, Timeout: 3*time.Second, DialTimeout: time.Second,
TLSHandshakeTimeout: time.Second, ResponseHeaderTimeout: time.Second,
MaxConnsPerHost: 32, MaxIdleConnsPerHost: 16, MaxIdleConns: 64,
IdleConnTimeout: 60*time.Second, MaxResponseBytes: 1<<20,
}) })
if err != nil { return err } if err != nil { return err }
application.OnShutdown(client.Close) application.OnShutdown(client.Close)
err = client.JSON(ctx, "GET", serviceURL, nil, &result) err = client.JSON(ctx, "GET", serviceURL, nil, &result,
httpc.WithHeader("Authorization", "Bearer "+token),
)
``` ```
客户端应在服务启动时创建并复用。默认使用独立的 `net/http.Transport` 连接池,每个目标主机 启动时创建并复用 Client。代理可选环境、直连、固定地址;不配置时保留标准库原代理策略。JSON 支持单次请求头和响应限额,不保存 Client 全局 Token。新阶段超时与空闲连接参数可选,0 沿用默认。
有连接上限;该上限不是整个进程的全局连接数上限。也可以注入自有 `http.RoundTripper`,
其连接限制和生命周期由调用方负责。这样出站实现同样可以替换。
总超时覆盖建连、重试、等待和读取响应体。`Do` 返回后调用方必须读取/关闭 `response.Body`; 成功响应读取有界;output=nil 流式丢弃,不整体缓存。小错误体只在大小/时间预算内尝试读完以复用连接;未知或大错误体及时关闭。Do 保留标准流式 Request/Response,由调用者限量读取并关闭 body。Close 拒绝新调用并关闭自有池空闲连接,项目先排空在途工作。
`JSON` 自动关闭,并限制解码前读取的字节数。`Do` 允许流式读取,本身不限制响应体大小。
`Close` 拒绝新请求、关闭自有池的空闲连接;在途请求应先排空。
`MaxRetries` 默认 0,最多 5。封装只对可重放的 GET/HEAD/OPTIONS 重试网络错误及 自定义 Transport 借用、不修改、不关闭;不能同时指定池/代理/阶段选项。总超时持续覆盖 HTTP 等待、重试和读取 body,但不抢占 JSON 编解码或不遵守 context 的自定义代码。MaxRetries 默认 0,只对可重放 GET/HEAD/OPTIONS 重试;3xx 不跟随。标准库和借用 Transport 自身仍可能重放,不能据此承诺业务只执行一次。
429/502/503/504;POST/PUT/PATCH/DELETE 不做应用层自动重试。标准库 Transport 和借用的
RoundTripper 自身的重放规则仍生效,不能据此承诺“只执行一次”。退避带抖动,尊重
`Retry-After`;服务器要求等待超过 `MaxBackoff` 时直接返回该响应,不提前重试。
重定向交给业务处理,3xx 不会自动跟随。
默认从 context 传递 `X-Request-ID`,日志只包含方法、主机、状态、尝试次数和收到响应头的耗时, 完整参数、NO_PROXY、TLS、安全日志、重试、签名兼容说明见 [httpc 指南](httpc/README.md),可运行 [HTTP Client 示例](examples/httpClient/README.md)。
不记录完整 URL、查询参数、凭证和响应体。非 2xx 的 JSON 调用返回只带状态码的 `StatusError`。
## 请求保护:middleware ## 请求保护:middleware

View File

@@ -1,6 +1,6 @@
# 生成项目与文件维护边界 # 生成项目与文件维护边界
v0.18.0-preview 提供受管理的生成项目。安装同版远程 CLI,生成的项目直接引用该版本,无需本地 replace。根目录入口支持 `go run .` 和 `go build .`;框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。 v0.26.0-preview 提供受管理的生成项目。安装同版远程 CLI,生成的项目直接引用该版本,无需本地 replace。根目录入口支持 `go run .` 和 `go build .`;框架本身仍然是可直接引用的库。安装和私有仓库访问见 [README](README.md#准备环境与安装)。
## 创建和日常开发 ## 创建和日常开发
@@ -38,7 +38,7 @@ ERROR [logicSignature] internal/logic/greet/greet.go:12 (GET /api/greet/:name):
这是轻量源码分析,不运行项目 init、不执行 Go 命令、不下载依赖。项目自定义类型别名、嵌入方法或 build tags 等无法可靠判断时,会给 REVIEW 并交给 Go 编译器验证;不把这些情况强行判为签名错误。它不检查方法体是否引用已删除字段,也不证明业务逻辑正确。CI 中应同时执行 `go-omni api generate --check` 和 `go test ./...`,按项目实际构建参数测试。当前版本首次接手缺少快照的 manifest v2 时,会明确提示建立基线,不猜测旧契约;Logic 签名检查仍会执行。 这是轻量源码分析,不运行项目 init、不执行 Go 命令、不下载依赖。项目自定义类型别名、嵌入方法或 build tags 等无法可靠判断时,会给 REVIEW 并交给 Go 编译器验证;不把这些情况强行判为签名错误。它不检查方法体是否引用已删除字段,也不证明业务逻辑正确。CI 中应同时执行 `go-omni api generate --check` 和 `go test ./...`,按项目实际构建参数测试。当前版本首次接手缺少快照的 manifest v2 时,会明确提示建立基线,不猜测旧契约;Logic 签名检查仍会执行。
本版新增生成选项、公共字段组合和映射,详见 [生成能力指南](GENERATION.md)。 生成选项、公共字段组合和可选 Go 拷贝工具,详见 [生成能力指南](GENERATION.md)。
## 哪些文件由用户维护 ## 哪些文件由用户维护
@@ -46,7 +46,8 @@ ERROR [logicSignature] internal/logic/greet/greet.go:12 (GET /api/greet/:name):
| --- | --- | | --- | --- |
| `api/*.api` | 路由、DTO、具名中间件、响应模板、请求/响应模式 | | `api/*.api` | 路由、DTO、具名中间件、响应模板、请求/响应模式 |
| `etc/config.yaml` | 运行参数;密钥通过环境变量提供 | | `etc/config.yaml` | 运行参数;密钥通过环境变量提供 |
| `internal/config/config.go` | 自定义配置字段、加载、默认值与校验 | | `internal/config/config.go` | 自定义配置字段、默认值与校验 |
| `internal/config/loader.go` | 需要额外文件格式时注册 Decoder |
| `internal/svc/serviceContext.go` | 依赖字段和初始化,持有加载后的 Config | | `internal/svc/serviceContext.go` | 依赖字段和初始化,持有加载后的 Config |
| `internal/svc/options.go` | AppOptions、GlobalMiddleware;注册 Go 错误目录和全局扩展 | | `internal/svc/options.go` | AppOptions、GlobalMiddleware;注册 Go 错误目录和全局扩展 |
| `internal/svc/databaseOptions.go` | 可选 MySQL 驱动与项目启动检查扩展 | | `internal/svc/databaseOptions.go` | 可选 MySQL 驱动与项目启动检查扩展 |
@@ -54,14 +55,15 @@ ERROR [logicSignature] internal/logic/greet/greet.go:12 (GET /api/greet/:name):
| `internal/logic/<group>/*.go` | 业务方法,使用标准 context.Context | | `internal/logic/<group>/*.go` | 业务方法,使用标准 context.Context |
| `internal/repository/<group>/*.go` | 数据访问;默认不补空骨架,可选 group 模式 | | `internal/repository/<group>/*.go` | 数据访问;默认不补空骨架,可选 group 模式 |
| `internal/middleware/*Factory.go`、`dependencies.go` | 中间件工厂与依赖小接口,svc 实现访问器 | | `internal/middleware/*Factory.go`、`dependencies.go` | 中间件工厂与依赖小接口,svc 实现访问器 |
| `omni.gen.json`、映射转换函数(支持 `mappingFiles` 引用子文件) | 生成选项、显式类型对与差异规则 | | `omni.gen.json` | Repository 和中间件生成选项;不再声明结构映射 |
| 可选的 Go 转换函数 / copyx 调用 | 只有独立类型确实需要转换时维护,见 [copyx](COPY.md) |
| 项目 Go 错误包 | 自行维护 Catalog 和常量,可跨服务共用 | | 项目 Go 错误包 | 自行维护 Catalog 和常量,可跨服务共用 |
| 自定义标量文件、`*Bind.go`、`*Manual.go` | 只有显式选择相应扩展时才生成/使用 | | 自定义标量文件、`*Bind.go`、`*Manual.go` | 只有显式选择相应扩展时才生成/使用 |
| 项目测试、迁移、部署脚本、go.mod/go.sum | 项目所有,生成器不重置其内容 | | 项目测试、迁移、部署脚本、go.mod/go.sum | 项目所有,生成器不重置其内容 |
以上骨架只补缺失文件,既有内容按字节保留。删除已废弃接口后,对应业务文件也不会被自动删除;由开发者整理编译器指出的失效类型或签名。 以上骨架只补缺失文件,既有内容按字节保留。删除已废弃接口后,对应业务文件也不会被自动删除;由开发者整理编译器指出的失效类型或签名。
生成器持续管理:`internal/svc/runtimeGen.go`、`main.go`、`internal/handler/*Gen.go`、分组的 `handlersGen.go`、`internal/types/*Gen.go`、可选映射 `internal/mapping/mappingsGen.go`、中间件 `registryGen.go`、`docs/openapi.json`、`.goOmni/apiManifest.json`。如果在 `.api` 里选择声明错误,还会管理 `internal/apierrors/errorsGen.go`。文件标注 DO NOT EDIT;手改会被拒绝覆盖,需先移走自定义内容。 生成器持续管理:`internal/config/loadGen.go`、`internal/svc/runtimeGen.go`、`main.go`、`internal/handler/*Gen.go`、分组的 `handlersGen.go`、`internal/types/*Gen.go`、中间件 `registryGen.go`、`docs/openapi.json`、`.goOmni/apiManifest.json`。如果在 `.api` 里选择声明错误,还会管理 `internal/apierrors/errorsGen.go`。文件标注 DO NOT EDIT;手改会被拒绝覆盖,需先移走自定义内容。
Logic 实例在注册时创建并由请求并发复用;请求数据用方法参数和局部变量传递,不存入共享 Logic/svc 字段。 Logic 实例在注册时创建并由请求并发复用;请求数据用方法参数和局部变量传递,不存入共享 Logic/svc 字段。
@@ -113,6 +115,8 @@ c.RequestShutdown(err)
`Lifecycle *lifecycle.Controller` 是 svc 持有的通用控制器,初始化时创建,生成入口使用它的 Context。该 Context 表示服务正在停止,不代表数据库已经关闭。不能把它当作在途请求的 Context,也不要在它取消时立刻关闭请求仍依赖的资源。发号器失去租约后拒绝发号等业务安全规则仍由项目实现。 `Lifecycle *lifecycle.Controller` 是 svc 持有的通用控制器,初始化时创建,生成入口使用它的 Context。该 Context 表示服务正在停止,不代表数据库已经关闭。不能把它当作在途请求的 Context,也不要在它取消时立刻关闭请求仍依赖的资源。发号器失去租约后拒绝发号等业务安全规则仍由项目实现。
配置加载入口 `internal/config/loadGen.go` 也由生成器维护。业务扩展 Config/Default/Validate;只有新格式才需要修改 NewLoader。JSON/YAML/XML、错误定位和旧 Load 迁移见 [配置指南](CONFIG.md)。
## 配置内置接口 ## 配置内置接口
生成项目在 `etc/config.yaml` 明确列出以下默认配置;可以独立关闭、修改路径,或让健康接口返回原始 JSON: 生成项目在 `etc/config.yaml` 明确列出以下默认配置;可以独立关闭、修改路径,或让健康接口返回原始 JSON:
@@ -139,7 +143,7 @@ server:
`responseMode: raw` 返回 `{"status":"ready"}` 等健康数据,有依赖时包含检查名称、结果与耗时;不包装公共体。存活检查不执行依赖探针;服务开始退出后,就绪检查返回 503,存活仍可返回 200,直到监听关闭。OpenAPI 始终输出规范文档本身,不使用业务响应包装;关闭 HTTP 文档端点不影响生成 `docs/openapi.json`。 `responseMode: raw` 返回 `{"status":"ready"}` 等健康数据,有依赖时包含检查名称、结果与耗时;不包装公共体。存活检查不执行依赖探针;服务开始退出后,就绪检查返回 503,存活仍可返回 200,直到监听关闭。OpenAPI 始终输出规范文档本身,不使用业务响应包装;关闭 HTTP 文档端点不影响生成 `docs/openapi.json`。
环境变量覆盖配置文件,例如 `GO_OMNI_ENDPOINTS_READINESS_PATH=/readyz`、`GO_OMNI_ENDPOINTS_READINESS_RESPONSE_MODE=raw`、`GO_OMNI_ENDPOINTS_OPENAPI_ENABLED=false`;liveness/readiness/openapi 都支持对应的 ENABLED 和 PATH。Config/AppOptions 的最终有效配置决定是否注册。 内置路由配置由项目文件明确设置,也可以使用 `${NAME:-fallback}` 引用;不再按字段名自动读取环境变量。Config/AppOptions 的最终有效配置决定是否注册。
关闭或改名后,旧路径不再由框架保留,项目可以在 `.api` 自己声明。启用的内置 GET 路由与业务路由相同或被业务参数/通配路径覆盖时,在创建 App、开始监听之前报错;不会覆盖业务路由。内置接口仍经过全局中间件;项目在鉴权中间件中明确决定这些路径的访问规则,框架不自动绕过鉴权。 关闭或改名后,旧路径不再由框架保留,项目可以在 `.api` 自己声明。启用的内置 GET 路由与业务路由相同或被业务参数/通配路径覆盖时,在创建 App、开始监听之前报错;不会覆盖业务路由。内置接口仍经过全局中间件;项目在鉴权中间件中明确决定这些路径的访问规则,框架不自动绕过鉴权。
@@ -211,7 +215,7 @@ get /download (DownloadReq)
## 已有服务接入步骤 ## 已有服务接入步骤
每个独立服务生成一个项目,module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.18.0-preview: 每个独立服务生成一个项目,module 名称、上层目录和共享业务包由项目选择。接入时固定 CLI 与框架版本为 v0.26.0-preview:
1. 先保留旧服务,整理实际路由及现有响应、鉴权和超时要求。在新目录执行 `go-omni new --api /path/to/service.api --module example.com/service ./service`;不对旧入口强制覆盖。 1. 先保留旧服务,整理实际路由及现有响应、鉴权和超时要求。在新目录执行 `go-omni new --api /path/to/service.api --module example.com/service ./service`;不对旧入口强制覆盖。
2. 在 `.api` 中选择 `@app(defaultResponse: ...)`,显式保留原项目的字段、成功码、错误码和 `dataMode`。需要独立回执的路由单独声明 raw/produces。普通路由、404/405 和提前返回的中间件一起验收。 2. 在 `.api` 中选择 `@app(defaultResponse: ...)`,显式保留原项目的字段、成功码、错误码和 `dataMode`。需要独立回执的路由单独声明 raw/produces。普通路由、404/405 和提前返回的中间件一起验收。

216
README.md
View File

@@ -2,24 +2,140 @@
GoOmni 是面向多个业务项目复用的 Go 后端框架:用 `.api` 定义接口、请求和返回类型,生成项目目录、HTTP 接入代码及 OpenAPI,业务代码负责 Logic 和 Repository。也可以在已有项目中直接使用 `omni.New()`,或单独引入日志、存储等模块。 GoOmni 是面向多个业务项目复用的 Go 后端框架:用 `.api` 定义接口、请求和返回类型,生成项目目录、HTTP 接入代码及 OpenAPI,业务代码负责 Logic 和 Repository。也可以在已有项目中直接使用 `omni.New()`,或单独引入日志、存储等模块。
**从这份 README 开始接入。** 当前版本 **`v0.18.0-preview`**,新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入;文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md),版本变化与验收见 [发布说明](RELEASE.md)。 **v0.26 改动**:通用 [i18n](i18n/README.md),提供独立词典、语言匹配、回退、命名参数/条件/复数、可选 HTTP 与错误适配、检查/导出 CLI 及前端运行时。文案由项目维护;[完整示例](examples/i18n/README.md)可直接运行。CLI 与框架使用同版 v0.26.0-preview。同时修复 Go 1.22 下请求体限额错误的识别。
**v0.17 提供**:公共字段组合、可选结构映射、带依赖的中间件工厂、按需 Repository 骨架,使用与迁移见 [生成能力指南](GENERATION.md)。 **从这份 README 开始接入。** 当前版本 **`v0.26.0-preview`**,新生成项目采用根目录入口和 `config / svc / logic / handler`。可以固定版本开展业务接入;文件维护边界和已有服务迁移步骤见 [项目指南](PROJECT.md),版本变化与验收见 [发布说明](RELEASE.md)。
**本版新增**:可空嵌套映射、映射配置按文件拆分、显式 nil 策略和可选指针返回辅助,见 [生成能力指南](GENERATION.md)。 **v0.25 改动**:`database.InTx` 返回可识别的事务阶段与结果,保留业务/驱动及回滚错误链;提交报错保守标记为结果不确定,不等同于已回滚。[使用规则](database/README.md#事务阶段与结果)、[可运行故障示例](examples/transactions/README.md)。
**v0.24 改动**:新增可选 CORS,生成项目通过 `server.cors` 自动接入,支持预检、错误响应和明确的来源白名单;默认关闭。[配置与规则](cors/README.md)、[可运行示例](examples/cors/README.md)。
**v0.23 改动**:配置校验定位具体字段和安全原因;`httpc.JSON` 可选限制成功状态;`copyx.Check` 无需样本预检类型;`logx` 支持实例独立时区。[四项完整示例](examples/developerTools/README.md)。
**v0.19 改动**:新增可选 `copyx` 工具,用普通 Go 调用完成浅拷贝、深拷贝和显式类型转换;深拷贝会拒绝自定义转换中残留的源对象可变引用。**移除了 `mappings` / `mappingFiles` 和映射代码生成**,已有使用者须按 [迁移步骤](COPY.md#从旧-mapping-迁移) 调整,不能直接删除仍在调用的旧文件。
普通接口的流程是:**修改 `.api` → 生成 Request/Response → 写 Logic → 测试**。有独立职责的业务类型才需要转换;可直接写 Go,也可以按需使用 [copyx](COPY.md)。公共字段组合、中间件工厂、Repository 开关继续支持,见 [生成能力指南](GENERATION.md)。
Go **1.22+**。框架不依赖 Gin,默认使用标准库网络层,提供可切换的 native HTTP/1.1 和实验性 HTTP/2 适配器。 Go **1.22+**。框架不依赖 Gin,默认使用标准库网络层,提供可切换的 native HTTP/1.1 和实验性 HTTP/2 适配器。
响应字段、业务码、登录方式、权限规则和数据库表结构由接入项目决定。文中的 `code/msg/data`、`200/9999` 都是示例配置,可以替换。 响应字段、业务码、登录方式、权限规则和数据库表结构由接入项目决定。文中的 `code/msg/data`、`200/9999` 都是示例配置,可以替换。
本版补齐 `returns([]string/[]int64/[]bool/[]any)`、OpenAPI 公共引用、workspace 一次导出共享 Go 错误目录,以及统一环境变量别名覆盖。详细配置与升级步骤见 [业务开发指南](DEVELOPMENT.md)。 契约支持 `returns([]string/[]int64/[]bool/[]any)`、OpenAPI 公共引用、workspace 一次导出共享 Go 错误目录,以及文件显式环境引用与配置检查。详细配置与升级步骤见 [业务开发指南](DEVELOPMENT.md)。
已有通用开发能力见 [业务开发指南](DEVELOPMENT.md):数组/null、严格请求策略、声明式上传、完整配置环境覆盖、资源生命周期、共享契约和升级工具。MySQL 直接使用 [官方适配器](database/mysql/README.md)。 已有通用开发能力见 [业务开发指南](DEVELOPMENT.md):数组/null、严格请求策略、声明式上传、文件显式配置引用、资源生命周期、共享契约和升级工具。MySQL 直接使用 [官方适配器](database/mysql/README.md)。
后台组件可在 svc 登记 [分阶段关闭](lifecycle/README.md#后台组件分阶段关闭),先停止接收工作并等待收尾,再关闭共享资源;无需绑定队列或 ORM。 后台组件可在 svc 登记 [分阶段关闭](lifecycle/README.md#后台组件分阶段关闭),先停止接收工作并等待收尾,再关闭共享资源;无需绑定队列或 ORM。
使用 GORM 的项目可以选择 [gormx 适配器](database/gormx/README.md),复用数据库初始化和连接池,并将 SQL 日志接入 logx。 使用 GORM 的项目可以选择 [gormx 适配器](database/gormx/README.md),复用数据库初始化和连接池,并将 SQL 日志接入 logx。
## 当前功能与使用入口
下表是当前公开能力的索引。每项给出调用入口和完整说明;模块按需使用,`omni.New()` 不会自动初始化数据库、Redis、ORM、队列或云存储。示例中的 import 统一以 `git.yuntuo.hk/k2safe/go-omni/` 开头。
| 功能与用途 | 怎么用 | 解释与示例 |
| --- | --- | --- |
| 项目、接口与文档生成 | `go-omni new`;改 `.api` 后 `api generate`;`--check` 检查漂移 | [下方快速开始](#新项目生成并运行)、[CLI](cmd/go-omni/README.md)、[完整生成项目](examples/managedService/README.md) |
| 公共字段与跨服务共享契约 | `.api` 组合对象字段;`omni.imports.json` 声明共享根;workspace 批量生成 | [公共字段示例](GENERATION.md#公共字段组合)、[共享契约与 workspace](DEVELOPMENT.md) |
| 中间件依赖工厂、可选 Repository | `omni.gen.json` 选择 factories / none;在 `New<Name>(deps)` 写一次实现 | [配置与完整工厂示例](GENERATION.md) |
| 路由、分组、请求上下文 | `omni.New()`、`app.GET` / `Group` / `Use`;业务使用 `context.Context` | [基本服务](examples/basic/main.go)、[已有项目接入](#已有项目按模块接入) |
| HTTP 网络与扩展适配 | 配置 transport 为 nethttp / native / http2;自定义 Router/Runner | [网络层配置与契约](transport/README.md)、[性能与限制](PERFORMANCE.md) |
| 浏览器跨域 CORS | `server.cors` / `omni.WithCORS`;独立 HTTP 用 `cors.New(...).Wrap(handler)` | [配置、默认值与协议边界](cors/README.md)、[鉴权与预检示例](examples/cors/README.md) |
| HTTP 老接口桥接 | 通过官方 Handler/Middleware 适配器接入标准 `net/http`,跟踪实际响应状态 | [接入说明](MIGRATION.md)、[桥接/上传/自定义 ID 示例](examples/compatibility/main.go) |
| 请求校验、动态 JSON、自定义标量 | `.api` 声明来源/校验/scalar;手写契约用 `contract.New[T]`;简单绑定可用 `validate.JSON/Query/URI` | [语法与例子](API.md)、[请求策略](DEVELOPMENT.md) |
| 数组、null、原始请求、上传下载 | `.api` 声明数组/可空返回、body/multipart 与大小限制;必要时使用 Bind/Manual 扩展 | [契约示例](examples/contracts)、[含上传/回调/下载的项目](examples/managedService/README.md) |
| 公共响应与特殊协议 | `.api` 定义 response 并 `@app(defaultResponse: ...)`;Go 用 `WithResponse` / Encoder | [响应配置](RESPONSE.md)、[响应模板契约](examples/contracts/responses.api) |
| 错误目录与动态提示 | `WithErrors(catalog)` 注册一次;`ByCode(...).WithParams(...).WithCause(err)` | [错误目录](errors/README.md)、[模板参数示例](examples/contracts/errorParams.api) |
| 通用国际化 | `i18n.New/Render`;可选 `WithLanguagePolicy/WithErrorLocalizer`;`go-omni i18n check/export/types/extract` | [资源格式与边界](i18n/README.md)、[CLI/HTTP/前端示例](examples/i18n/README.md)、[生成契约](examples/contracts/i18n.api) |
| OpenAPI 与错误公共引用 | `.api` 自动生成;手写接口可使用 `openapi` 注册器;Go 错误目录显式导出 | [契约工具](DEVELOPMENT.md)、[注册示例](examples/platform/main.go) |
| 结构化业务日志 | `logx.Info/Error`、`logx.FromContext(ctx)`、`logx.New`;支持独立时区、轮转、脱敏、慢请求和级别更新 | [下方日志示例](#日志)、[完整配置](logx/README.md) |
| 文件配置与环境引用 | Default()→JSON/YAML/XML 字段引用→类型绑定→校验;无隐式覆盖,支持 `-check-config` | [统一配置指南](CONFIG.md)、[完整可运行示例](examples/configuration/README.md)、[动态级别](ADVANCED.md#配置与日志热更新) |
| 生命周期与健康检查 | svc 登记 OnClose/OnStop/OnDrain,RequestShutdown 通知故障;健康路径/开关可配 | [生命周期用法](lifecycle/README.md)、[健康检查](health/README.md)、[生成项目扩展](PROJECT.md) |
| 数据库、连接池与事务 | `database.Open` / `OpenConnector`;`InTx` 的 `TxError` 提供阶段、已知结果与原因链 | [通用数据库](database/README.md)、[事务失败示例](examples/transactions/README.md)、[MySQL TLS/Logger](database/mysql/README.md) |
| 可选 GORM 与迁移 | gormx 复用已有连接池;`go-omni migrate` 预览,显式 apply | [GORM 示例](database/gormx/README.md)、[迁移用法](ADVANCED.md#迁移审计与分页) |
| 认证与权限接口 | 注入 Authenticator / Policy;JWT 或 Session 只是可选实现 | [认证说明](auth/README.md)、[鉴权完整例子](examples/platform/README.md) |
| 缓存 | `cache.NewMemory` 或 Redis 实现;业务面向 Store 的 Get/Set/Delete | [缓存配置/代码](MODULES.md)、[可运行模块服务](examples/modules/README.md) |
| 出站 HTTP、熔断 | `httpc.New` 配置代理、连接池和阶段超时;JSON 支持单次成功状态限制/请求头/限额;`resilience.New` 隔离故障 | [HTTP 完整用法](httpc/README.md)、[可运行示例](examples/httpClient/README.md)、[熔断/观测](ADVANCED.md#熔断与观测) |
| 请求追踪、访问日志、恢复、限流、幂等、超时 | `app.Use(...)` 按需组合 middleware;跨副本选择 Redis 实现 | [模块说明与限制](MODULES.md#请求追踪与基础指标)、[分布式实现](ADVANCED.md#分布式限流与幂等) |
| 本地后台任务与调度 | `task.New`、Submit、Every;有界队列、排空关闭 | [task 示例与生命周期](MODULES.md#后台执行task) |
| 持久队列 | 业务使用 Publisher / Consumer;Redis Streams 提供至少一次交付 | [队列初始化/消费示例](ADVANCED.md#可靠队列) |
| 文件存储、直传、云端分片 | Store 的 Put/Get/Delete;Local、S3/R2、OSS 按配置装配;扩展接口提供分片续传 | [存储接口与完整例子](storage/README.md)、[本地存储服务](examples/platform/README.md) |
| 日志关联、指标与链路 | `metrics.New().Middleware()`;telemetry 注入 OTel exporter/reader | [基础指标](MODULES.md#请求追踪与基础指标)、[OTel 用法](ADVANCED.md#熔断与观测)、[完整接入](examples/platform/main.go) |
| 审计与分页 | `audit.New` 注入 Sink/白名单;`pagination.Parse` / `NewResult` | [解释与调用例子](ADVANCED.md#迁移审计与分页) |
| 可选结构转换与深拷贝 | `copyx.Copy` / `Clone` / `New`;`Check[S,T]` 预检类型,Go converter 处理差异 | [下方示例](#可选结构转换与深拷贝)、[完整 copyx 指南](COPY.md)、[可运行代码](examples/copying) |
| 自定义适配器验收 | `transporttest.Run`、`storagetest.Run` / `RunUploads` 验证扩展契约 | [网络契约测试入口](transport/transporttest/router.go)、[存储契约测试入口](storage/storagetest/contract.go) |
## 浏览器跨域 CORS
同源部署保持默认关闭;需要跨域时,在生成项目的配置文件中加入:
```yaml
server:
cors:
enabled: true
allowedOrigins: [https://admin.example.com, https://merchant.example.com]
allowedMethods: [GET, HEAD, POST, PUT, PATCH, DELETE]
allowedHeaders: [Authorization, Content-Type]
exposedHeaders: [Content-Disposition, X-Request-ID]
allowCredentials: false
maxAge: 10m
```
不需要修改 main、Logic 或 `.api`。框架在鉴权前处理合法预检;实际请求仍鉴权,401/404/405/500 保持项目响应格式并附带允许来源的跨域头。没有 Origin 的服务端回调正常处理。配置只决定浏览器读取权限,不替代登录、验签或 CSRF 防护。
手写项目用 `omni.WithCORS(omni.CORSConfig{...})`,独立 HTTP 服务用 `cors.New(...).Wrap(handler)`;代理已经处理 CORS 时保持关闭。完整 [默认值与边界](cors/README.md)、[可运行示例](examples/cors/README.md)。
## 带认证的出站 HTTP
HTTP Client 在 svc 初始化一次,业务重复使用。普通内部请求可直接选择直连和连接参数,无需手写 Transport:
```go
client, err := httpc.New(httpc.Config{
Proxy: httpc.ProxyConfig{Mode: httpc.ProxyDirect},
Timeout: 3*time.Second, DialTimeout: time.Second,
TLSHandshakeTimeout: time.Second, ResponseHeaderTimeout: time.Second,
MaxConnsPerHost: 32, MaxIdleConnsPerHost: 16, MaxIdleConns: 64,
})
if err != nil { return err }
// 在 svc 生命周期登记 client.Close;关闭前先排空使用它的工作。
err = client.JSON(ctx, "GET", endpoint, nil, &result,
httpc.WithHeader("Authorization", "Bearer "+token),
httpc.WithMaxResponseBytes(64<<10),
)
```
请求头只作用于本次调用,不把用户 Token 留在共享 Client。默认代理/参数保持原行为;也可选择环境代理或固定代理。JSON 自动关闭并限制响应,Do 保留下载/流式读取扩展。完整配置、重试边界和已有自定义 Transport 的迁移见 [httpc](httpc/README.md)。
```sh
go run ./examples/httpClient
```
## 统一配置加载
业务只维护 Config、Default()、Validate() 和配置文件。生成器维护 Load/main;文件缺失字段保留默认值,显式 0/false/空字符串正常覆盖,固定值不会被同名环境变量再次覆盖。
```go
BatchSize int `json:"batchSize"` // 加入 Config;在 Default() 中设置 BatchSize: 100。
```
```yaml
server:
address: "${HTTP_ADDR:-127.0.0.1:8080}"
batchSize: "${BATCH_SIZE:-100}"
password: "${DB_PASSWORD}"
sessionToken: "${SESSION_TOKEN:-}"
```
`${NAME}` 未设置或为空就报错;`${NAME:-value}` 明确指定备用值。`$$` 转义 `$`;密码中的冒号、换行、引号和 `${...}` 原样插入,不二次解释。JSON/YAML/XML 共用解析后的字段处理和 Go 类型绑定,错误同时定位文件、字段、变量及可用行列,不打印原值。
```sh
go run ./examples/configuration -format xml
DEMO_CLIENT_TIMEOUT=500ms go run ./examples/configuration
# 在生成项目中,仅检查选中的文件,不初始化依赖:
go run . -config etc/config.test.yaml -check-config
```
完整 [语义与迁移](CONFIG.md)、[可运行示例](examples/configuration/README.md)。**v0.22 移除旧自动环境覆盖、前缀/别名及 env/envDefault 标签**,生成项目先迁移开发者配置再重新生成;文件中的响应模板 `${code}` 写成 `$${code}`,`.api` 保持原样。
## 选择接入方式 ## 选择接入方式
| 你的项目 | 从哪里开始 | | 你的项目 | 从哪里开始 |
@@ -40,7 +156,7 @@ module 地址是 `git.yuntuo.hk/k2safe/go-omni`。先确保本机 Go 和 Git 可
```sh ```sh
export GOPRIVATE="$(go env GOPRIVATE),git.yuntuo.hk" export GOPRIVATE="$(go env GOPRIVATE),git.yuntuo.hk"
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.18.0-preview go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.26.0-preview
export PATH="$(go env GOPATH)/bin:$PATH" export PATH="$(go env GOPATH)/bin:$PATH"
go-omni --help go-omni --help
``` ```
@@ -197,7 +313,7 @@ go-omni new --api catalog.api \
在业务项目的 **go.mod 所在目录**执行: 在业务项目的 **go.mod 所在目录**执行:
```sh ```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.18.0-preview go get git.yuntuo.hk/k2safe/go-omni@v0.26.0-preview
``` ```
已有项目可直接复用现有目录、Service、Repository 和数据库连接,先选择一个接口接入。也可以只用 `logx`、`storage` 等包,暂时保留原来的 HTTP 框架。希望入口和 HTTP 接入都交给生成器的项目,按 [已有服务接入步骤](PROJECT.md#已有服务接入步骤) 在新目录生成后迁入业务代码。 已有项目可直接复用现有目录、Service、Repository 和数据库连接,先选择一个接口接入。也可以只用 `logx`、`storage` 等包,暂时保留原来的 HTTP 框架。希望入口和 HTTP 接入都交给生成器的项目,按 [已有服务接入步骤](PROJECT.md#已有服务接入步骤) 在新目录生成后迁入业务代码。
@@ -391,6 +507,60 @@ return apierrors.TextTooLong("企业名称", 100)
更详细的配置、作用域和升级步骤见 [统一错误目录](errors/README.md)。 更详细的配置、作用域和升级步骤见 [统一错误目录](errors/README.md)。
### 可选结构转换与深拷贝
普通接口不需要任何拷贝配置,直接使用生成的 Request/Response。独立业务结构之间有转换需求时:
```go
// import "git.yuntuo.hk/k2safe/go-omni/copyx"
var input DomainInput
if err := copyx.Copy(&input, req); err != nil { return err }
// 源与目标之间的指针、map、slice 需要隔离时:
snapshot, err := copyx.Clone(source)
if err != nil { return err }
```
默认浅拷贝,同名同类型字段自动赋值,保留零值;默认不会把 nil 填成非可空目标的零值。不要为使用工具新增一套重复模型。
需要重复使用或自定义规则时创建 Copier:
```go
copier, err := copyx.New(
copyx.WithDeepCopy(),
copyx.WithFieldNames[Input, View](map[string]string{"Name": "DisplayName"}),
copyx.WithConverter(func(v time.Time) (string, error) {
return v.UTC().Format(time.RFC3339), nil
}),
)
if err != nil { return err }
if err := copier.Copy(&view, input); err != nil { return err }
```
自定义函数的源/目标类型由 Go 泛型确定。深拷贝时返回原指针、原 map、子切片或嵌套旧引用都会报错;字符串/整数原值可以返回。差异规则、nil 策略、字段覆盖、资源限制和原 mapping 移除步骤见 [copyx 指南](COPY.md),完整类型及可运行代码见 [examples/copying](examples/copying/main.go)。工具使用运行时检查,类型对需测试;深拷贝不适用于连接、锁等资源。
### 减少接入代码:四个常用入口
```yaml
server:
log:
timeZone: Asia/Shanghai
```
时区由 Logger 实例管理,不修改全局 `time.Local`。非法日志级别、数据库端口/连接池等配置会指出 `server.log.level`、`database.port` 等字段和原因,不打印输入值。详见 [配置诊断](CONFIG.md#框架模块的字段诊断)、[日志时区](logx/README.md#实例时区)。
```go
// 本次请求只接受 HTTP 200;默认仍接受所有 2xx。
err := client.JSON(ctx, http.MethodGet, url, nil, &result,
httpc.WithSuccessStatuses(http.StatusOK),
)
// CI/启动时检查类型,无需构造样本;仍使用 Copy 的同一组 Option。
err = copyx.Check[Request, BusinessInput](options...)
```
状态不符合要求返回 `*httpc.StatusError`,不会写入 result,响应体仍由框架关闭。Check 不执行 converter;动态 interface、具体 nil/转换失败和深拷贝引用检查有明确边界,见 [类型预检](COPY.md#类型预检)。完整可运行代码见 [developerTools](examples/developerTools/README.md)。
### 数据库、存储及其他模块 ### 数据库、存储及其他模块
生成项目在 `internal/svc` 中创建依赖,提供给 Logic/Repository,并用 `c.OnClose(db.Close)` 登记自有资源。生成的 main 在 HTTP 排空后关闭 svc,再关闭日志。直接使用框架的手写项目则可以用 `app.OnShutdown(db.Close)`;不要对同一资源重复登记。 生成项目在 `internal/svc` 中创建依赖,提供给 Logic/Repository,并用 `c.OnClose(db.Close)` 登记自有资源。生成的 main 在 HTTP 排空后关闭 svc,再关闭日志。直接使用框架的手写项目则可以用 `app.OnShutdown(db.Close)`;不要对同一资源重复登记。
@@ -409,20 +579,22 @@ API 项目的数据库迁移需要单独维护并显式执行;启动不会自
## 配置与运行 ## 配置与运行
生成项目从 `etc/config.yaml` 读取配置,优先级为 **默认值 → 配置文件 → 环境变量**。密钥通过环境变量提供。手写入口需显式调用配置加载函数并传给 `omni.New(omni.WithConfig(settings))`;单独调用 `omni.New()` 不会自动读取环境变量。 生成项目使用 **Default() → 选定文件的显式引用 → 类型绑定 → Validate()**。固定值不会被环境变量覆盖;配置规则、转义及迁移见 [配置指南](CONFIG.md)。手写入口显式加载配置,再传入 `omni.New(omni.WithConfig(settings))`。
| 环境变量 | 用途 | | 文件字段 | 用途 |
| --- | --- | | --- | --- |
| `GO_OMNI_HTTP_ADDR` | 监听地址;生成项目默认 `127.0.0.1:8080` | | `server.address` | 监听地址,生成项目默认 127.0.0.1:8080 |
| `GO_OMNI_TRANSPORT` | `nethttp`(默认)、`native`、`http2` | | `server.transport` | nethttp(默认)、native、http2 |
| `GO_OMNI_TLS_CERT_FILE` / `GO_OMNI_TLS_KEY_FILE` | 成对设置 TLS 证书和私钥路径 | | `server.tls.certFile/keyFile` | 成对配置 TLS 证书和私钥路径 |
| `GO_OMNI_HTTP_READ_HEADER_TIMEOUT` / `GO_OMNI_HTTP_READ_TIMEOUT` | 请求头 / 请求读取超时,如 `5s` / `30s` | | `server.http.readHeaderTimeout/readTimeout` | 请求头 / 请求读取超时 |
| `GO_OMNI_HTTP_WRITE_TIMEOUT` / `GO_OMNI_HTTP_IDLE_TIMEOUT` | 响应写入 / Keep-Alive 空闲超时 | | `server.http.writeTimeout/idleTimeout` | 响应写入 / Keep-Alive 空闲超时 |
| `GO_OMNI_HTTP_MAX_HEADER_BYTES` | 请求头大小限制,字节数 | | `server.http.maxHeaderBytes` | 请求头大小限制,字节数 |
| `GO_OMNI_SHUTDOWN_TIMEOUT` | 优雅退出等待时间,默认 `10s` | | `server.shutdownTimeout` | 优雅退出等待时间,默认 10s |
| `GO_OMNI_ENABLE_LOGGER` / `GO_OMNI_LOG_LEVEL` | 访问日志开关、日志级别 | | `server.enableLogger` / `server.log.level` | 访问日志开关、日志级别 |
| `GO_OMNI_TRUSTED_PROXIES` | 可解析转发地址头的可信代理 IP/CIDR,逗号分隔 | | `server.trustedProxies` | 可信代理 IP/CIDR,文件数组 |
| `GO_OMNI_DB_HOST/USER/PASSWORD/DATABASE` | 官方 MySQL 适配器的结构化连接配置 | | `database.host/user/password/database` | 官方 MySQL 结构化连接配置 |
初始生成文件已为地址、网络层和选中模块的必要配置声明环境引用;其他字段按需要添加固定值或引用。
例如在生成项目中切换 native: 例如在生成项目中切换 native:
@@ -467,7 +639,7 @@ go test ./...
`main.go`、`Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理,手动修改会报告冲突。Logic、Repository、中间件实现、配置和依赖装配属于项目,重复生成保留手写内容。契约变化会提示受影响的 Logic 文件和 DTO 字段,明确的签名错误会给出原签名与期望签名;生成后 `--check` 仍能检出未修复的问题。将 `.api`、`omni.gen.json`、生成文件和 `.goOmni/apiManifest.json` 一起提交,CI 同时运行 `generate --check` 与 `go test ./...`。详见 [契约变更提示与检查边界](PROJECT.md#契约变更提示)。 `main.go`、`Gen.go`、`docs/openapi.json` 和 manifest 由生成器管理,手动修改会报告冲突。Logic、Repository、中间件实现、配置和依赖装配属于项目,重复生成保留手写内容。契约变化会提示受影响的 Logic 文件和 DTO 字段,明确的签名错误会给出原签名与期望签名;生成后 `--check` 仍能检出未修复的问题。将 `.api`、`omni.gen.json`、生成文件和 `.goOmni/apiManifest.json` 一起提交,CI 同时运行 `generate --check` 与 `go test ./...`。详见 [契约变更提示与检查边界](PROJECT.md#契约变更提示)。
CLI 与业务 module 固定使用 `v0.18.0-preview`。v0.14.x 项目升级依赖与 CLI 后重新生成,框架维护的 main/runtimeGen 自动更新;v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理,不直接覆盖手写入口或业务文件。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。 CLI 与业务 module 固定使用 `v0.26.0-preview`。v0.21 及更早项目先按 [配置迁移](CONFIG.md#v022-迁移旧模式已移除) 整理旧标签、文件与 Load,再升级依赖与 CLI 并重新生成,框架维护的 main/runtimeGen 自动更新;v0.13 及更早项目按 [升级预览](DEVELOPMENT.md#v013-项目升级) 处理,不直接覆盖手写入口或业务文件。使用过旧 mapping 的项目必须先完成 [移除迁移](COPY.md#从旧-mapping-迁移),旧配置或旧生成文件会阻止生成。升级后执行生成检查与业务回归。版本变化、迁移边界和验收记录见 [发布说明](RELEASE.md)。
## 按需深入 ## 按需深入
@@ -483,3 +655,7 @@ CLI 与业务 module 固定使用 `v0.18.0-preview`。v0.14.x 项目升级依赖
| 网络性能、测试条件与适用范围 | [性能说明](PERFORMANCE.md)、[HTTP/2 验收](transport/native/HTTP2.md) | | 网络性能、测试条件与适用范围 | [性能说明](PERFORMANCE.md)、[HTTP/2 验收](transport/native/HTTP2.md) |
当前版本可用于固定版本接入与业务验证,仍标记为 preview。`.api` 借鉴 go-zero 的接口开发方式,支持范围以本仓库文档为准,不是完整 goctl 语法兼容层;HTTP/2 保持实验状态,OSS/R2 云账号实网验收不属于本次发布范围。 当前版本可用于固定版本接入与业务验证,仍标记为 preview。`.api` 借鉴 go-zero 的接口开发方式,支持范围以本仓库文档为准,不是完整 goctl 语法兼容层;HTTP/2 保持实验状态,OSS/R2 云账号实网验收不属于本次发布范围。
## 后续开发约定
每次功能开发完成,同步更新这份功能索引、用法说明、`examples/` 和发布记录;删除或改变功能必须注明影响与迁移。代码、示例和文档一起回归后再发布。约定已记录在 [AGENTS.md](AGENTS.md),后续开发继续遵循。

View File

@@ -1,4 +1,198 @@
# v0.18.0-preview # v0.26.0-preview
新增可独立使用的 i18n 底层模块及可选 HTTP/错误适配。项目维护文案和语言设置,框架提供不可变词典、语言匹配、回退、参数与分支、检查和导出;没有绑定业务身份、页面框架、数据库或响应字段名。
前后端共享版本化消息资源和固定 Unicode CLDR 46.1 基数复数规则。CLI 只读输入,支持源目录提取、缺译/参数/动态引用检查、按命名空间导出语言包和 TypeScript 类型。可选浏览器模块先加载并验证再切换,失败保留当前语言,旧请求不能覆盖新选择。
错误目录新增可选 `translation` 引用。内置响应使用展示译文,原目录文案继续参与契约校验;自定义 Encoder 通过 `PublicMessage()` 显式采用译文。业务码、HTTP 状态、成功/失败语义、cause、直接协议输出和幂等首次快照保持。旧配置与生成项目不启用时无需迁移。
完整 [使用指南](i18n/README.md)、[通用示例](examples/i18n/README.md)、[验收记录](reports/i18nAcceptance.md)。包含 translation 的新契约远程依赖最低版本为 v0.26.0-preview;未使用该元数据的模板最低版本仍为 v0.22。CLI 与框架使用同版。
同时修复 JSON 结束定界符读取失败时丢失原始错误的问题,使请求体超过外层限制时在 Go 1.22 中也保持 HTTP 413;共享目录测试工程使用 Go 1.22。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.26.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.26.0-preview
go-omni api generate --check
go test ./...
```
---
# v0.25.0-preview
事务失败可以按阶段和已知结果处理,避免把“提交错误”误判为“数据库没有保存”。
- `database.InTx` 签名不变,普通错误返回 `*database.TxError`;Stage 为 begin/work/commit,Outcome 为 notStarted/rolledBack/unknown。成功提交仍返回 nil。
- Commit 报错一律 unknown;回调失败且显式 Rollback 返回 nil 才标记 rolledBack。BeginTx 失败不执行回调。ErrTxDone 不作为回滚成功证据,框架不重试事务。
- `Err` 与 `RollbackErr` 保留两条原因链,可通过 errors.Is/As 识别;APIError、目录参数提示和公开响应继续工作。内部原因不自动进入 HTTP 提示。
- 提供无需外部数据库的 [故障演示](examples/transactions/README.md)。真实 MySQL 与性能结果见 [验收记录](reports/transactionAcceptance.md)。不引入新依赖,不包含文件删除、补偿任务或业务规则。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.25.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.25.0-preview
go-omni api generate --check
go test ./...
```
**兼容边界:** 既有 errors.Is/As 保持;直接 `err == sentinel`、直接类型断言或字符串匹配需调整。nil context 使用 Background,nil 回调在开启前返回 typed error。panic/Goexit 继续传播并尝试回滚,不转换为 error;标准库自动回滚不公开的原因无法还原,不能推断已回滚。结果仅覆盖按约定使用的当前事务,不覆盖非事务表、隐式提交 SQL、其他连接或外部副作用。GORM 原生 Transaction 没有被替换。
[database 使用规则](database/README.md#事务阶段与结果) 已同步。本轮 runtime 能力需要 v0.25,模板最低版本仍为 v0.22;没有新增必填配置,当前仍为 preview。
49 个测试包全量 race、Go 1.22、vet/build、取消竞争复测通过;成功事务没有新增分配。固定远程版本在空缓存下通过 77 项发布检查和 9 组生成消费者,耗时 71.93 秒。独立 MySQL TLS 容器的 53 条测试记录(含子用例)通过,包含真实提交/回滚后的确认丢失、可选 GORM、生成项目和后台关闭,耗时 28.49 秒。均无本地 replace,标签固定在 `7a177744ab2bb85f23efc0ec267fd905d349bd4b`。见 [远程发布记录](reports/transactionRemote.json)、[远程 MySQL 记录](reports/transactionMySQLRemote.json)。
---
# v0.24.0-preview(历史版本)
业务项目通过配置开启跨域,不需要手写 OPTIONS Handler 或逐条修改接口。
- 默认关闭,在 `server.cors` 显式配置来源、方法和请求头;统一文件加载、环境引用和字段级诊断自动生效。生成项目无需修改 main、svc 或 `.api`。
- 预检在鉴权前返回 204/403 空响应;实际请求继续鉴权、业务错误编码和路由处理。允许来源可读取 401/403/404/405/500,上传下载和流式能力保留。
- 只支持精确 HTTP(S) 来源,公开无凭据服务可配置单一 `*`;禁止 `*` 与凭据混用。配置初始化快照、请求并发隔离、Vary 缓存分流和有界请求头处理。
- 同一个 App 策略覆盖标准库、native 和 HTTP2;独立服务可使用 `cors.New(...).Wrap(handler)`。没有新增第三方依赖,没有移除现有功能。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.24.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.24.0-preview
go-omni api generate
go-omni api generate --check
go run . -config etc/config.yaml -check-config
```
现有项目不配置 CORS 时行为不变。代理已处理 CORS 的项目继续关闭;跨域不替代登录、权限或 CSRF 防护。通过 App/Engine/Run 接入,直接对外暴露底层 Router 会绕过应用外层策略。本轮新增 runtime 配置需 v0.24,未使用该配置的模板最低版本仍为 v0.22。
[配置与规则](cors/README.md)、[可运行示例](examples/cors/README.md)、[验收与性能](reports/corsAcceptance.md) 已同步。49 个测试包全量 race、Go 1.22、vet/build、实际生成项目、三种 transport 的 TLS 场景和 20 项真实浏览器检查通过;30 秒 fuzz 约 288 万次。当前仍为 preview。
空缓存安装远程版本后,9 组生成消费者和 70 项发布检查通过,耗时 66.78 秒,无本地 replace;包括下载模块的 cors/app race、三种 transport 的 TLS 检查和只改配置的生成项目。标签固定在 `800974cfff6228c2697373d9961d3c7c75611d08`,见 [远程记录](reports/corsRemote.json)。
---
# v0.23.0-preview(历史版本)
减少业务接入时重复维护的配置校验、HTTP 包装和类型检查。
- 配置诊断定位 database.port、database.pool.maxIdleConns、server.log.level/timeZone 等具体字段,保留文件、环境变量、行列和安全原因;凭据不进入公开提示,原始原因可 errors.Is/As。
- JSON 可传 `httpc.WithSuccessStatuses(200)`,默认仍接受全部 2xx;不匹配时不改目标,继续负责有界读取、关闭和连接回收,不添加新的重试规则。
- `copyx.Check[S,T](options...)` 或已有 Copier.Check 复用同一组 Go 选项,无需样本和映射文件;实际拷贝不再因集合为空而漏过确定的类型不匹配。动态 interface、nil 值、converter 失败及深拷贝共享引用仍有运行时检查。
- `server.log.timeZone: Asia/Shanghai` 配置实例时区;JSON/text/子日志一致,业务 time.Time 属性保持原样。极简镜像可用 `go build -tags timetzdata .` 内置时区数据,库不强制增加体积。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.23.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.23.0-preview
go-omni api generate
go-omni api generate --check
go run . -config etc/config.yaml -check-config
```
现有调用接口保留;配置错误文本现在更精确,业务应通过类型/字段识别错误。空集合里确定的类型错误会提前暴露,需要显式 converter 或调整实际类型,不应靠空值跳过校验。预检通过不代表 converter 对所有业务值必定成功。当前模板最低版本仍是 v0.22,本轮新增 runtime 能力需使用 v0.23。
[完整示例](examples/developerTools/README.md)、[配置](CONFIG.md)、[JSON 选项](httpc/README.md)、[类型预检](COPY.md#类型预检)、[日志时区](logx/README.md#实例时区)。验收与性能记录见 [本轮验收](reports/developerToolsAcceptance.md)。仍为 preview。
48 个测试包全量 race、Go 1.22、vet/build、实际生成消费者及约 392 万次 copyx fuzz 通过。空缓存安装远程版本后,66 项发布检查通过,包括下载模块的 config/httpc/copyx/logx race;耗时 57.06 秒,无本地 replace。标签固定在 `3bbc6f60d76cb9eae6f01fa6ca122eb1fbf481af`,见 [远程验收](reports/developerToolsRemote.json)。
---
# v0.22.0-preview(历史版本)
配置文件明确决定每个值的来源,项目只维护 Config、Default()、Validate() 和文件。
- `${NAME}` 必填,`${NAME:-fallback}` 在未设置/空字符串时使用备用值,`$$` 转义 `$`。支持字符串组合;环境值不二次展开,凭据空格和特殊字符原样保留。
- 在 JSON/YAML/XML 已解析的字段值上展开,再按目标 Go 类型绑定;缺失字段保留默认值,0/false/空字符串可显式覆盖。字段错误定位文件、环境名和可用行列,不回显值。
- 生成 Load/main 自动接入;新增 `-check-config`,选定一个文件检查,成功退出,不启动日志/依赖/监听。精简生成配置,重新生成保留业务 Config、Default、Validate 和文件。
- **不兼容变更:旧自动环境覆盖、前缀/别名、env/envDefault、database.ConfigFromEnv/ApplyEnv 已移除。** LoadFile/NewReloader 不再接收 prefix,omni.LoadConfig 参数改为文件路径。发现旧标签提示迁移。
- 配置文件中的响应模板用 `$${code}` 等转义;`.api` 中的 `${code}` 不变。日志 file/output/rotation 均由显式文件配置决定,不再隐式清除。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.22.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.22.0-preview
# 按 CONFIG.md 迁移开发者配置后:
go-omni api generate
go-omni api generate --check
go run . -config etc/config.yaml -check-config
```
[完整规则与迁移](CONFIG.md)、[可运行示例](examples/configuration/README.md)、[场景与性能验收](reports/explicitConfigurationAcceptance.md) 已同步。48 个测试包全量 race、实际生成消费者、Go 1.22、vet/build 和约 28 万次引用 fuzz 通过;小配置固定值/带引用加载约 25 µs,属于本机启动阶段基准。当前仍为 preview。
空缓存远程 CLI 安装、下载模块 config/httpc race、8 组实际生成消费者与 `-check-config` 验收通过,无本地 replace。标签固定在 `d598b2ebee8cde36859021e882a6bc1e0bb4fd4c`;见 [远程验收记录](reports/explicitConfigurationRemote.json)。
---
# v0.21.0-preview(历史版本)
让带认证的 HTTP 调用和内部直连可以直接使用 httpc,减少手写 Transport 与 JSON 响应处理。
- Proxy 支持环境、直连、固定代理;默认保留既有策略。
- 建连、TCP keepalive、TLS 握手、响应头、100-continue、空闲超时和独立空闲连接数量可选配置;不修改全局 Transport,不关闭借用实例。
- JSON 支持 WithHeader / WithHeaders / WithMaxResponseBytes;认证信息逐请求隔离,自动关闭和有界响应处理保留。
- 成功路径减少请求复制和无用退避;output=nil 按限额流式丢弃;小错误体在大小和时间预算内尝试复用连接。
- 不新增依赖、不增加业务规则;Do 保留标准流式扩展。CLI 最新版本与模板最低版本分开,本轮模板兼容门槛仍是 v0.20。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.21.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.21.0-preview
```
**兼容注意**:JSON 旧的直接调用仍有效,自定义 interface/函数值需要同步新增的 variadic 参数。自定义 Transport 与显式连接/代理/阶段参数混用现在报错,避免参数被忽略;迁移时在自己的 Transport 配置,或改用内置字段。没有删除原调用入口。
[参数与使用说明](httpc/README.md)、[可运行示例](examples/httpClient/README.md)、[场景与性能验收](reports/httpcAcceptance.md) 已同步到 README。48 个测试包全量 race、Go 1.22 兼容、vet/build、约 110 万次请求头 fuzz 通过。性能来自本机同条件基准,不能代替项目实际流量验证;仍为 preview。
空缓存远程安装、下载模块的 httpc race 与示例、8 组生成消费者验收通过,无本地 replace。标签固定在 `96e7d9629b5c8b78e9cfefbcb05da0ab702c242b`,见 [远程记录](reports/httpcRemote.json)。
---
# v0.20.0-preview(历史版本)
统一配置加载,减少生成项目重复维护的启动代码。
- 生成器维护 `internal/config/loadGen.go`,调用 `config.LoadProject`;项目只扩展 Config/Default/Validate,可选在 loader.go/NewLoader 注册新格式。入口、环境覆盖、日志输出规范化无需业务重复实现。
- JSON/YAML/XML 使用同一个加载与绑定流程;Decoder 通过实例注册,无全局 init 或额外强制依赖。内置格式、字段、文件大小与层数均有明确限制。
- `config.Error` 携带文件、字段、环境变量、可用行列及安全原因;`Invalid` / `AtField` 支持业务验证。保留内部错误链,不自动输出原值;大整数、空集合、null、默认值隔离和自定义反序列化有回归覆盖。
- 配置解析在日志、依赖和监听初始化之前;生成/check/dry-run 使用一致入口和文件所有权规则。
- **迁移要求:旧项目先整理手写 Load,再重新生成。** 不静默覆盖业务文件。低层配置 API 保留;新项目不再复制手写加载流程。固定 Go 数组严格检查长度,YAML alias/merge/custom tag 不再接受,避免模糊覆盖和扩展行为。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.20.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.20.0-preview
# 先按 CONFIG.md 迁移旧 Load,再执行:
go-omni api generate
go-omni api generate --check
go test ./...
go build .
```
完整用法、格式、脱敏边界和迁移见 [统一配置指南](CONFIG.md),可运行例子见 [examples/configuration](examples/configuration/README.md)。回归、模糊测试与启动加载性能见 [验收记录](reports/configurationAcceptance.md)。47 个测试包的全量 race、Go 1.22 兼容、vet/build、30 秒 fuzz 与 5 个基准场景三轮测量通过。空缓存远程安装和 8 组生成消费者回归通过,无本地 replace;标签固定在 `c60841f901a3bcc3e9582eece6cfb2cad008fb9e`,见 [远程记录](reports/configurationRemote.json)。仍为 preview;项目的实际配置和业务依赖应自行验收。
---
# v0.19.0-preview(历史版本)
将可选结构转换改为直接 Go 调用,减少业务项目需要维护的配置。普通接口继续使用 `.api` 生成 DTO,不要求增加独立模型或任何拷贝规则。
- 新增只依赖标准库的 `copyx`:Copy、Clone、可复用 Copier;浅/深拷贝、严格字段覆盖、nil/零值策略、字段改名、忽略字段、精确类型 converter。没有全局注册和隐式数字强转。
- 深拷贝检查整个输入和输出图。自定义 converter 原样返回指针、map、slice,或新对象仍嵌套旧引用时返回错误;保留普通循环图,拒绝资源对象及超出遍历/估算分配预算的输入。
- 转换失败不提交部分目标;保留错误路径和原始错误链。并发复用只共享不可变类型元数据;源/目标读写和 converter 副作用由调用方负责。
- **不兼容变更:删除 mappings / mappingFiles、Go 类型加载与映射代码生成。** 检出旧配置/旧生成文件时给出迁移提示;不会静默删除业务正在使用的代码。公共字段组合、scalar、中间件工厂、Repository 选项保留。
- README 补齐当前能力入口、解释和用法,新增 [完整 copyx 指南](COPY.md)、[可运行示例](examples/copying);[AGENTS.md](AGENTS.md) 记录每次修改同步文档/示例/移除说明的约定。
```sh
go get git.yuntuo.hk/k2safe/go-omni@v0.19.0-preview
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.19.0-preview
# 如用过旧 mapping,先按 COPY.md 完成迁移:
go-omni api generate
go-omni api generate --check
go test ./...
```
迁移步骤和语义边界见 [copyx](COPY.md#从旧-mapping-迁移)。新工具做运行时检查,不能声称具备旧生成器的编译期字段覆盖保证;实际类型对必须有测试。独立三端工资示例验证金额精度、缺失/null/零值、空集合、错误链和字段新增;不修改或代替接入项目的完整业务回归。
46 个测试包的全量 race(含实际生成项目)、vet/build、Go 1.22 兼容测试通过;30 秒 fuzz 无失败,14 种基准场景各测量三轮。验收条件、性能数据及本轮未覆盖的外部环境见 [v0.19 验收记录](reports/copyxAcceptance.md)。本版仍为 preview。 空缓存远程 CLI 与 7 组真实生成消费者验收通过,无本地 replace;标签固定在 `fd83235ce7fc15ce88a482d385d70e61f25d7b32`,详见 [远程记录](reports/copyxRemote.json)。
---
以下是历史版本记录,其中 mapping 功能已在 v0.19 移除,不应继续作为当前接入方式。
# v0.18.0-preview(历史版本)
补齐可空嵌套映射和按模块拆分配置,减少重复转换代码,业务空值规则仍由项目显式选择: 补齐可空嵌套映射和按模块拆分配置,减少重复转换代码,业务空值规则仍由项目显式选择:
@@ -19,6 +213,8 @@ v0.17 的单文件 mappings、现有 converter 和业务文件继续保留。配
使用与迁移见 [生成能力指南](GENERATION.md),回归和发布后验收见 [验收记录](reports/nullableMappingAcceptance.md)。本版仍为 preview。 使用与迁移见 [生成能力指南](GENERATION.md),回归和发布后验收见 [验收记录](reports/nullableMappingAcceptance.md)。本版仍为 preview。
全量 44 个测试包的 race、vet、build 和实际进程验收通过;空缓存远程 CLI 与 8 组实际生成消费者回归也通过,同版依赖且无本地 replace。标签固定在 `4563a6354fac19f22cd8d407fd76d98b8cc9c406`。
--- ---
# v0.17.0-preview(历史版本) # v0.17.0-preview(历史版本)
@@ -63,7 +259,7 @@ go-omni api generate --check
go test ./... go test ./...
``` ```
旧项目 config/Logic/svc 继续由开发者维护,不会被重新生成覆盖。已有环境配置需按 [一次覆盖完整配置](DEVELOPMENT.md#一次覆盖完整配置与-http日志短名称) 合并 OverlayEnv 与 ApplyEnv;共享错误目录接入见 [workspace 配置](DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。已有手写权限 Handler 切换自动生成前,先迁出实现并显式移除 Manual 文件,生成器不会静默删除。 旧项目 config/Logic/svc 继续由开发者维护,不会被重新生成覆盖。已有环境配置需按 本版当时的统一覆盖方式合并 OverlayEnv 与 ApplyEnv(该模式已在 v0.22 移除,当前升级见 [配置迁移](CONFIG.md#v022-迁移旧模式已移除));共享错误目录接入见 [workspace 配置](DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。已有手写权限 Handler 切换自动生成前,先迁出实现并显式移除 Manual 文件,生成器不会静默删除。
全量 race(含实际生成消费者)、vet、build、根入口/退出/再生成验证通过。eTax 只读契约验收:API 文档 20,751,149 → 847,171 字节,ID 文档 1,008,210 → 392,481 字节;70 个操作展开引用后与原文档完全一致。空缓存远程 CLI / 固定版本消费者验收也已通过,没有本地 replace;标签固定在 `64e3bd6cfced225476d58e95037249cd0a4fe695`。详见 [验收记录](reports/contractFollowupAcceptance.md)。这不代表已完成业务项目迁移。 全量 race(含实际生成消费者)、vet、build、根入口/退出/再生成验证通过。eTax 只读契约验收:API 文档 20,751,149 → 847,171 字节,ID 文档 1,008,210 → 392,481 字节;70 个操作展开引用后与原文档完全一致。空缓存远程 CLI / 固定版本消费者验收也已通过,没有本地 replace;标签固定在 `64e3bd6cfced225476d58e95037249cd0a4fe695`。详见 [验收记录](reports/contractFollowupAcceptance.md)。这不代表已完成业务项目迁移。

View File

@@ -1,5 +1,7 @@
# 公共响应体与独立接口协议 # 公共响应体与独立接口协议
v0.26 新增的 [国际化适配](i18n/README.md#接入-go-omni-错误与-http)保留现有响应契约。`Result.Message` 是校验使用的目录原文;`MessageOverride` 是可选展示译文,`PublicMessage()` 返回最终文案。内置 Formatter 自动使用译文;自定义 Encoder 显式选择,固定文本/XML 回执和直接输出不自动改写。
这些功能从 `v0.8.0-preview` 提供。CLI 与项目框架依赖请固定到同一版本;源码开发可显式使用 `--framework-dir`。`v0.7.0-preview.1` 不包含响应模板声明。 这些功能从 `v0.8.0-preview` 提供。CLI 与项目框架依赖请固定到同一版本;源码开发可显式使用 `--framework-dir`。`v0.7.0-preview.1` 不包含响应模板声明。
go-omni 的响应编码由应用或接口配置。没有配置时保留旧版默认值;业务可以定义自己的字段、嵌套结构、错误码类型、提示和空值行为。`response.Response` 仅保留为兼容 DTO,运行时与生成器不再通过它固定响应结构。 go-omni 的响应编码由应用或接口配置。没有配置时保留旧版默认值;业务可以定义自己的字段、嵌套结构、错误码类型、提示和空值行为。`response.Response` 仅保留为兼容 DTO,运行时与生成器不再通过它固定响应结构。

View File

@@ -1,5 +1,8 @@
# GoOmni 成熟度计划 # GoOmni 成熟度计划
当前发布基线为 v0.19.0-preview:普通接口通过 `.api` → DTO → Logic 开发;独立类型需要转换时可选 copyx。旧 JSON mapping 生成已移除,见 [迁移说明](COPY.md)。当前能力、使用入口和限制以 [README](README.md) 与 [发布记录](RELEASE.md) 为准。下文保留阶段性建设记录。
当前进展:`v0.14.0-preview` 已完成契约驱动的业务开发能力、通用 Runtime/配置和官方 MySQL 适配器,并通过全量 race、实际生成服务及空缓存远程验收。新项目/升级以 [README](README.md) 和 [开发指南](DEVELOPMENT.md) 为准;下方为历史阶段记录,完整本轮验收见 [报告](reports/businessDevelopmentAcceptance.md)。具体 eTax/ePay 业务迁移与业务回归仍由接入项目完成。 当前进展:`v0.14.0-preview` 已完成契约驱动的业务开发能力、通用 Runtime/配置和官方 MySQL 适配器,并通过全量 race、实际生成服务及空缓存远程验收。新项目/升级以 [README](README.md) 和 [开发指南](DEVELOPMENT.md) 为准;下方为历史阶段记录,完整本轮验收见 [报告](reports/businessDevelopmentAcceptance.md)。具体 eTax/ePay 业务迁移与业务回归仍由接入项目完成。
## 1. 新项目启动与发布流程(已完成,v0.1.1) ## 1. 新项目启动与发布流程(已完成,v0.1.1)

View File

@@ -12,6 +12,7 @@ import (
"time" "time"
"git.yuntuo.hk/k2safe/go-omni/config" "git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/cors"
apierrors "git.yuntuo.hk/k2safe/go-omni/errors" apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
"git.yuntuo.hk/k2safe/go-omni/httpx" "git.yuntuo.hk/k2safe/go-omni/httpx"
"git.yuntuo.hk/k2safe/go-omni/lifecycle" "git.yuntuo.hk/k2safe/go-omni/lifecycle"
@@ -24,6 +25,7 @@ import (
// httpx.Router, leaving the network implementation replaceable. // httpx.Router, leaving the network implementation replaceable.
type App struct { type App struct {
router httpx.Router router httpx.Router
handler http.Handler
config config.Config config config.Config
middlewares []httpx.Middleware middlewares []httpx.Middleware
closers []func() error closers []func() error
@@ -67,6 +69,11 @@ func configure(options ...config.Option) config.Config {
} }
func newApp(router httpx.Router, settings config.Config) *App { func newApp(router httpx.Router, settings config.Config) *App {
settings.CORS = settings.CORS.Clone()
corsPolicy, corsErr := cors.New(settings.CORS)
if corsErr != nil {
panic(config.AtField("cors", corsErr))
}
settings.Response = settings.Response.Clone() settings.Response = settings.Response.Clone()
settings.Errors = settings.Errors.Clone() settings.Errors = settings.Errors.Clone()
encoder, responseErr := response.New(settings.Response) encoder, responseErr := response.New(settings.Response)
@@ -93,6 +100,7 @@ func newApp(router httpx.Router, settings config.Config) *App {
} }
middlewares = append(middlewares, middleware.Recovery()) middlewares = append(middlewares, middleware.Recovery())
application := &App{router: router, config: settings, middlewares: middlewares, clientIPResolver: resolver, responseEncoder: responseEncoder} application := &App{router: router, config: settings, middlewares: middlewares, clientIPResolver: resolver, responseEncoder: responseEncoder}
application.handler = corsPolicy.Wrap(router)
if fallback, ok := router.(httpx.FallbackRouter); ok { if fallback, ok := router.(httpx.FallbackRouter); ok {
adaptError := func(status int, code, message string) http.Handler { adaptError := func(status int, code, message string) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
@@ -104,21 +112,23 @@ func newApp(router httpx.Router, settings config.Config) *App {
return application return application
} }
// Router returns the transport contract used by this application. // Router exposes the underlying routing contract for advanced registration.
// Serve requests through App or Engine so application CORS policy is applied.
func (a *App) Router() httpx.Router { return a.router } func (a *App) Router() httpx.Router { return a.router }
// Engine is kept as a compatibility name for the old API. It returns a // Engine is kept as a compatibility name for the old API. It returns a
// standard http.Handler, so callers do not depend on a router type. // standard http.Handler, so callers do not depend on a router type.
func (a *App) Engine() http.Handler { return a.router } func (a *App) Engine() http.Handler { return a.handler }
// ServeHTTP implements http.Handler for standard servers and HTTP tests. // ServeHTTP implements http.Handler for standard servers and HTTP tests.
func (a *App) ServeHTTP(writer http.ResponseWriter, request *http.Request) { func (a *App) ServeHTTP(writer http.ResponseWriter, request *http.Request) {
a.router.ServeHTTP(writer, request) a.handler.ServeHTTP(writer, request)
} }
// Config returns the effective application configuration. // Config returns the effective application configuration.
func (a *App) Config() config.Config { func (a *App) Config() config.Config {
settings := a.config settings := a.config
settings.CORS = settings.CORS.Clone()
settings.Response = settings.Response.Clone() settings.Response = settings.Response.Clone()
settings.Errors = settings.Errors.Clone() settings.Errors = settings.Errors.Clone()
settings.TrustedProxies = append([]string(nil), settings.TrustedProxies...) settings.TrustedProxies = append([]string(nil), settings.TrustedProxies...)
@@ -177,7 +187,7 @@ func (a *App) NewServer(address string) *http.Server {
} }
} }
cfg := a.config.HTTP cfg := a.config.HTTP
return &http.Server{Addr: address, Handler: a.router, return &http.Server{Addr: address, Handler: a.handler,
ReadHeaderTimeout: cfg.ReadHeaderTimeout, ReadTimeout: cfg.ReadTimeout, ReadHeaderTimeout: cfg.ReadHeaderTimeout, ReadTimeout: cfg.ReadTimeout,
WriteTimeout: cfg.WriteTimeout, IdleTimeout: cfg.IdleTimeout, WriteTimeout: cfg.WriteTimeout, IdleTimeout: cfg.IdleTimeout,
MaxHeaderBytes: cfg.MaxHeaderBytes, ErrorLog: cfg.ErrorLog} MaxHeaderBytes: cfg.MaxHeaderBytes, ErrorLog: cfg.ErrorLog}

187
app/cors_test.go Normal file
View File

@@ -0,0 +1,187 @@
package app_test
import (
"context"
"encoding/json"
"errors"
"io"
"net"
"net/http"
"net/http/httptest"
"strings"
"sync/atomic"
"testing"
"time"
omni "git.yuntuo.hk/k2safe/go-omni"
"git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/httpx"
"git.yuntuo.hk/k2safe/go-omni/internal/testutil"
"git.yuntuo.hk/k2safe/go-omni/response"
)
func corsApp() *omni.App {
return omni.New(omni.WithCORS(omni.CORSConfig{Enabled: true, AllowedOrigins: []string{"https://web.example"}, AllowedMethods: []string{"GET", "POST", "PUT"}, AllowedHeaders: []string{"Authorization", "Content-Type"}, ExposedHeaders: []string{"Content-Disposition"}}), omni.WithResponse(response.Config{CodeType: "int32", SuccessCode: 0, ErrorCode: 999, MessageField: "msg", DataMode: "always"}))
}
func TestAppCORSRoutingAndErrors(t *testing.T) {
a := corsApp()
defer a.Close()
var middlewareCalls int
a.Use(func(next omni.Handler) omni.Handler {
return func(c *omni.Context) error { middlewareCalls++; return next(c) }
})
a.GET("/secure", func(c *omni.Context) error { return c.OK("private") }, func(next omni.Handler) omni.Handler {
return func(c *omni.Context) error { return omni.Unauthorized("LOGIN_REQUIRED", "login required") }
})
a.GET("/failure", func(c *omni.Context) error { return omni.Internal(errors.New("private detail")) })
a.GET("/panic", func(c *omni.Context) error { panic("private panic") })
a.Handle("OPTIONS", "/options", func(c *omni.Context) error { return c.Text(200, "ordinary options") })
for name, entry := range map[string]http.Handler{"App": a, "Engine": a.Engine(), "NewServer": a.NewServer("").Handler} {
t.Run(name, func(t *testing.T) {
for _, tc := range []struct {
method, path string
status int
}{{"GET", "/secure", 401}, {"GET", "/failure", 500}, {"GET", "/panic", 500}, {"GET", "/absent", 404}, {"PUT", "/secure", 405}} {
r := httptest.NewRequest(tc.method, tc.path, nil)
r.Header.Set("Origin", "https://web.example")
w := httptest.NewRecorder()
entry.ServeHTTP(w, r)
var body map[string]any
err := json.Unmarshal(w.Body.Bytes(), &body)
if err != nil || w.Code != tc.status || w.Header().Get("Access-Control-Allow-Origin") != "https://web.example" || body["code"] != float64(999) || body["data"] != nil || body["msg"] == nil || strings.Contains(w.Body.String(), "private") {
t.Fatal(w.Code, w.Header(), w.Body.String(), err)
}
}
for _, path := range []string{"/secure", "/absent"} {
before := middlewareCalls
r := httptest.NewRequest("OPTIONS", path, nil)
r.Header.Set("Origin", "https://web.example")
r.Header.Set("Access-Control-Request-Method", "POST")
r.Header.Set("Access-Control-Request-Headers", "authorization,content-type")
w := httptest.NewRecorder()
entry.ServeHTTP(w, r)
if w.Code != 204 || w.Body.Len() != 0 || middlewareCalls != before {
t.Fatal(w.Code, w.Body.String(), middlewareCalls, before)
}
}
r := httptest.NewRequest("OPTIONS", "/options", nil)
w := httptest.NewRecorder()
entry.ServeHTTP(w, r)
if w.Code != 200 || w.Body.String() != "ordinary options" {
t.Fatal(w.Code, w.Body.String())
}
})
}
}
func TestAppCORSConfigIsolation(t *testing.T) {
cfg := config.Default()
cfg.CORS = omni.CORSConfig{Enabled: true, AllowedOrigins: []string{"https://one.example"}, AllowedHeaders: []string{"Authorization"}}
option := omni.WithConfig(cfg)
cfg.CORS.AllowedOrigins[0] = "https://evil.example"
a := omni.New(option)
defer a.Close()
b := omni.New(option)
defer b.Close()
a.Config().CORS.AllowedOrigins[0] = "https://evil.example"
for _, app := range []*omni.App{a, b} {
w := httptest.NewRecorder()
r := httptest.NewRequest("GET", "/absent", nil)
r.Header.Set("Origin", "https://one.example")
app.ServeHTTP(w, r)
if w.Header().Get("Access-Control-Allow-Origin") != "https://one.example" {
t.Fatal(w.Header())
}
}
defer func() {
if recover() == nil {
t.Error("invalid CORS passed direct New")
}
}()
omni.New(omni.WithCORS(omni.CORSConfig{Enabled: true}))
}
func TestCORSTransports(t *testing.T) {
for _, transport := range []string{omni.TransportNetHTTP, omni.TransportNative, omni.TransportHTTP2} {
t.Run(transport, func(t *testing.T) {
cert, key, tlsClient := testutil.CertificateFiles(t)
cfg := omni.CORSConfig{Enabled: true, AllowedOrigins: []string{"https://web.example"}, AllowedMethods: []string{"GET", "POST"}, AllowedHeaders: []string{"Content-Type"}, ExposedHeaders: []string{"Content-Disposition"}}
a := omni.New(omni.WithCORS(cfg), omni.WithTransport(transport), omni.WithTLS(cert, key))
var called atomic.Int32
a.Use(func(next omni.Handler) omni.Handler {
return func(c *omni.Context) error { called.Add(1); return next(c) }
})
a.POST("/upload", func(c *omni.Context) error {
b, err := io.ReadAll(c.Request().Body)
if err != nil {
return err
}
return c.Text(200, string(b))
})
a.GET("/download", func(c *omni.Context) error {
return c.WriteStream(200, "application/octet-stream", &httpx.Stream{Content: io.NopCloser(strings.NewReader("file bytes")), Name: "example.txt"})
})
listener, err := net.Listen("tcp", "127.0.0.1:0")
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- a.ServeContext(ctx, listener) }()
defer func() {
cancel()
select {
case err := <-done:
if err != nil {
t.Error(err)
}
case <-time.After(5 * time.Second):
t.Error("server did not close")
listener.Close()
}
}()
client := &http.Client{Timeout: 5 * time.Second, Transport: &http.Transport{TLSClientConfig: tlsClient, ForceAttemptHTTP2: true}}
defer client.CloseIdleConnections()
url := "https://" + listener.Addr().String()
for _, tc := range []struct {
method, path, body string
status int
}{{"OPTIONS", "/upload", "", 204}, {"POST", "/upload", "raw upload", 200}, {"GET", "/download", "file bytes", 200}, {"GET", "/missing", "", 404}} {
var input io.Reader
if tc.method == "POST" {
input = strings.NewReader(tc.body)
}
r, _ := http.NewRequest(tc.method, url+tc.path, input)
r.Header.Set("Origin", "https://web.example")
if tc.method == "OPTIONS" {
r.Header.Set("Access-Control-Request-Method", "POST")
r.Header.Set("Access-Control-Request-Headers", "content-type")
}
res, err := client.Do(r)
if err != nil {
t.Fatal(err)
}
body, err := io.ReadAll(res.Body)
res.Body.Close()
if err != nil {
t.Fatal(err)
}
if res.StatusCode != tc.status || res.Header.Get("Access-Control-Allow-Origin") != "https://web.example" {
t.Fatal(res.StatusCode, res.Header)
}
if tc.status == 200 && string(body) != tc.body {
t.Fatal(string(body))
}
if tc.status == 204 && (len(body) != 0 || called.Load() != 0) {
t.Fatal("preflight reached middleware")
}
if tc.path == "/download" && (!strings.Contains(res.Header.Get("Content-Disposition"), "example.txt") || res.Header.Get("Access-Control-Expose-Headers") != "Content-Disposition") {
t.Fatal(res.Header)
}
if transport == omni.TransportHTTP2 && res.ProtoMajor != 2 {
t.Fatal(res.Proto)
}
}
})
}
}

241
app/i18n_test.go Normal file
View File

@@ -0,0 +1,241 @@
package app_test
import (
"context"
"encoding/json"
"errors"
"net/http"
"net/http/httptest"
"strings"
"sync"
"testing"
omni "git.yuntuo.hk/k2safe/go-omni"
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
"git.yuntuo.hk/k2safe/go-omni/httpx"
"git.yuntuo.hk/k2safe/go-omni/i18n"
"git.yuntuo.hk/k2safe/go-omni/i18n/errori18n"
"git.yuntuo.hk/k2safe/go-omni/i18n/httpi18n"
"git.yuntuo.hk/k2safe/go-omni/middleware"
"git.yuntuo.hk/k2safe/go-omni/response"
)
func localizedCatalog(namespace string, status int) apierrors.Catalog {
return errori18n.Bind(apierrors.Catalog{DefaultStatus: status, Definitions: map[string]apierrors.Definition{
"SAME": {Code: int64(9223372036854775807), Message: namespace + ":{value}", Params: []apierrors.Parameter{{Name: "value", Type: "string"}}},
"INTERNAL_ERROR": {Status: 500, Code: int64(500), Message: "internal"},
"FIXED": {Status: 401, Code: int64(401), Message: "fixed {literal}"},
}}, namespace)
}
func localizedOptions(t *testing.T, catalogs ...apierrors.Catalog) (*httpi18n.Policy, *errori18n.Localizer) {
t.Helper()
docs := []i18n.Document{}
for _, catalog := range catalogs {
source, err := errori18n.Source(catalog, "en")
if err != nil {
t.Fatal(err)
}
docs = append(docs, source...)
for _, d := range source {
docs = append(docs, i18n.Document{Version: 1, Namespace: d.Namespace, Locale: "zh-Hans", Messages: map[string]i18n.Message{
"SAME": {Value: i18n.Pattern{Text: d.Namespace + "译文:{value}"}}, "FIXED": {Value: i18n.Pattern{Text: "需要认证"}}, "INTERNAL_ERROR": {Value: i18n.Pattern{Text: "系统错误"}},
}})
}
}
b, err := i18n.New(i18n.Config{SourceLocale: "en", DefaultLocale: "en", Locales: []string{"en", "zh-Hans"}}, docs...)
if err != nil {
t.Fatal(err)
}
l := errori18n.New(b, nil)
for _, c := range catalogs {
if err := l.Validate(c); err != nil {
t.Fatal(err)
}
}
p, err := httpi18n.New(b)
if err != nil {
t.Fatal(err)
}
return p, l
}
func TestI18nRouteScopesEarlyFailuresAndReplay(t *testing.T) {
global, local := localizedCatalog("app", 200), localizedCatalog("route", 409)
policy, localizer := localizedOptions(t, global, local)
format := response.Config{CodeType: "int64", SuccessCode: int64(0), ErrorCode: int64(999), Body: map[string]any{"error": map[string]any{"reason": "${message}", "id": "${code}"}, "ok": "${ok}", "data": "${data}"}, DataMode: "always", Messages: map[string]string{"FIXED": "fixed {literal}"}}
app := omni.New(omni.WithErrors(global), omni.WithResponse(format), omni.WithLanguagePolicy(policy), omni.WithErrorLocalizer(localizer))
defer app.Close()
private := errors.New("private-value")
app.Use(func(next httpx.Handler) httpx.Handler {
return func(c *httpx.Context) error {
switch c.Header("X-Early") {
case "auth":
return apierrors.ByCode("FIXED")
case "panic":
panic(private)
}
return next(c)
}
})
handler := func(c *httpx.Context) error {
if c.Header("X-Params") == "missing" {
return apierrors.ByCode("SAME")
}
return apierrors.ByCode("SAME").WithParams(map[string]any{"value": "{literal}"}).WithCause(private)
}
app.GET("/global", handler)
bridge := httpx.WrapMiddleware(func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { next.ServeHTTP(w, r) })
})
app.HandleWithResponse("GET", "/route", response.RawJSON(response.MustWithErrors(local, response.MustNew(format))), handler, bridge)
for _, path := range []string{"/global", "/route"} {
for _, locale := range []string{"en", "zh-CN"} {
for _, mode := range []string{"", "auth", "panic", "missing"} {
r := httptest.NewRequest("GET", path, nil)
r.Header.Set("Accept-Language", locale)
r.Header.Set("X-Early", mode)
r.Header.Set("X-Params", mode)
w := httptest.NewRecorder()
app.ServeHTTP(w, r)
var body struct {
Error struct {
Reason string `json:"reason"`
ID json.Number `json:"id"`
} `json:"error"`
OK bool `json:"ok"`
Data any `json:"data"`
}
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatal(err, w.Body.String())
}
ns, status := "app", 200
if path == "/route" {
ns, status = "route", 409
}
want := ns + ":{literal}"
if locale != "en" {
want = ns + "译文:{literal}"
}
if mode == "auth" {
status = 401
want = "fixed {literal}"
if locale != "en" {
want = "需要认证"
}
} else if mode == "panic" || mode == "missing" {
status = 500
want = "internal"
if locale != "en" {
want = "系统错误"
}
}
if w.Code != status || body.OK || body.Error.Reason != want || strings.Contains(w.Body.String(), "private-value") {
t.Fatal(path, locale, mode, w.Code, w.Body.String())
}
if mode == "" && body.Error.ID.String() != "9223372036854775807" {
t.Fatal("public code changed", body.Error)
}
}
}
}
for _, shared := range []bool{false, true} {
cfg := middleware.IdempotencyConfig{Scope: func(*httpx.Context) (string, error) { return "test", nil }}
if shared {
cfg.Store = &catalogStore{records: map[string]middleware.IdempotencyRecord{}}
}
idem, err := middleware.NewIdempotency(cfg)
if err != nil {
t.Fatal(err)
}
path := "/localReplay"
if shared {
path = "/sharedReplay"
}
calls := 0
app.POST(path, func(c *httpx.Context) error { calls++; return handler(c) }, idem.Middleware())
var first string
for _, locale := range []string{"en", "zh-CN"} {
r := httptest.NewRequest("POST", path, strings.NewReader(`{}`))
r.Header.Set("Idempotency-Key", "same")
r.Header.Set("Accept-Language", locale)
w := httptest.NewRecorder()
app.ServeHTTP(w, r)
if locale == "en" {
first = w.Body.String()
} else if w.Body.String() != first || w.Header().Get("Idempotency-Replayed") != "true" {
t.Fatal("replay changed", w.Body.String(), first)
}
}
if calls != 1 {
t.Fatal("language retriggered business", calls)
}
}
}
func TestI18nExplicitProtocolsAndConcurrentApps(t *testing.T) {
apps := []*omni.App{}
for _, ns := range []string{"one", "two"} {
c := localizedCatalog(ns, 200)
p, l := localizedOptions(t, c)
a := omni.New(omni.WithErrors(c), omni.WithResponse(response.Config{CodeType: "int64", SuccessCode: 0, ErrorCode: 999}), omni.WithLanguagePolicy(p), omni.WithErrorLocalizer(l))
defer a.Close()
a.GET("/", func(*httpx.Context) error { return apierrors.ByCode("FIXED") })
a.HandleWithResponse("GET", "/ack", response.EncoderFunc(func(r response.Result) (response.Payload, error) {
if !r.IsError() || r.Message != "fixed {literal}" || r.PublicMessage() != "需要认证" {
t.Error(r)
}
return response.Text(200, "ACK"), nil
}), func(*httpx.Context) error { return apierrors.ByCode("FIXED") })
a.GET("/direct", func(c *httpx.Context) error { return c.Text(200, "fixed-protocol") })
apps = append(apps, a)
}
var wg sync.WaitGroup
for n := 0; n < 60; n++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
locale, want := "en", "fixed {literal}"
if n%2 == 0 {
locale, want = "zh-CN", "需要认证"
}
r := httptest.NewRequest("GET", "/", nil)
r.Header.Set("Accept-Language", locale)
w := httptest.NewRecorder()
apps[n%len(apps)].ServeHTTP(w, r)
if w.Code != 401 || !strings.Contains(w.Body.String(), want) {
t.Error(w.Code, w.Body.String())
}
}(n)
}
wg.Wait()
for _, path := range []string{"/ack", "/direct"} {
r := httptest.NewRequest("GET", path, nil)
r.Header.Set("Accept-Language", "zh-CN")
w := httptest.NewRecorder()
apps[0].ServeHTTP(w, r)
want := "ACK"
if path == "/direct" {
want = "fixed-protocol"
}
if w.Body.String() != want {
t.Fatal(w.Body.String())
}
}
}
type panickingLocalizer struct{}
func (panickingLocalizer) Localize(context.Context, *apierrors.Registry, *apierrors.APIError) (string, bool) {
panic("presentation failed")
}
func TestI18nAdapterFailurePreservesBusiness(t *testing.T) {
c := localizedCatalog("test", 200)
a := omni.New(omni.WithErrors(c), omni.WithResponse(response.Config{CodeType: "int64", SuccessCode: 0, ErrorCode: 999}), omni.WithErrorLocalizer(panickingLocalizer{}))
defer a.Close()
a.GET("/", func(*httpx.Context) error { return apierrors.ByCode("FIXED") })
w := httptest.NewRecorder()
a.ServeHTTP(w, httptest.NewRequest("GET", "/", nil))
if w.Code != 401 || !strings.Contains(w.Body.String(), "fixed {literal}") {
t.Fatal(w.Code, w.Body.String())
}
}

View File

@@ -28,6 +28,9 @@ func (a *App) adaptResponse(encoder response.Encoder, handler httpx.Handler, mid
} }
} }
return http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) { return http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
if a.config.LanguagePolicy != nil {
request = a.config.LanguagePolicy.Apply(writer, request)
}
request = request.WithContext(logx.WithLogger(contract.WithRequestPolicy(request.Context(), a.config.Request), a.config.LogInstance)) request = request.WithContext(logx.WithLogger(contract.WithRequestPolicy(request.Context(), a.config.Request), a.config.LogInstance))
if len(a.config.TrustedProxies) > 0 { if len(a.config.TrustedProxies) > 0 {
request = request.WithContext(httpx.WithClientIPResolver(request.Context(), a.clientIPResolver)) request = request.WithContext(httpx.WithClientIPResolver(request.Context(), a.clientIPResolver))
@@ -40,6 +43,7 @@ func (a *App) adaptResponse(encoder response.Encoder, handler httpx.Handler, mid
requestContext.SetErrorRegistry(provider.ErrorRegistry()) requestContext.SetErrorRegistry(provider.ErrorRegistry())
} }
requestContext.SetResponseEncoder(encoder) requestContext.SetResponseEncoder(encoder)
requestContext.SetErrorLocalizer(a.config.ErrorLocalizer)
defer func() { httpx.CleanupMultipart(requestContext.Request()) }() defer func() { httpx.CleanupMultipart(requestContext.Request()) }()
if err := wrapped(requestContext); err != nil { if err := wrapped(requestContext); err != nil {
a.writeError(requestContext, err) a.writeError(requestContext, err)

View File

@@ -1,8 +1,10 @@
# go-omni CLI # go-omni CLI
v0.18 新增:可空嵌套映射、`mappingFiles` 配置拆分、显式 nil 策略和可选 `pointerResult`。原 v0.17 配置继续有效,见 [生成能力指南](../../GENERATION.md)。 v0.26 新增 `i18n check|export|types|extract`:检查独立语言资源、提取 Go 静态消息引用或已导出的错误目录,导出指定命名空间的前端包和 TypeScript 类型。输入只读,结果写 stdout;动态引用明确报告。完整命令见 [国际化工具链](../../i18n/README.md#cli-提取检查与导出)。`.api errorCatalog` 可声明 translation 引用,旧 runtime 在生成前拒绝,接入时使用同版 v0.26 CLI 与框架。
v0.17 增加公共字段组合、可选结构映射、自动中间件工厂和按需 Repository。配置和迁移见 [生成能力指南](../../GENERATION.md)。 v0.19 移除结构映射生成:`omni.gen.json` 只保留 Repository 和中间件选项。普通接口直接使用生成 DTO;需要类型转换时用普通 Go 或可选 [copyx](../../COPY.md)。旧 `mappings` / `mappingFiles` 必须按迁移步骤清理,CLI 不会静默删除仍在调用的旧生成文件。
公共字段组合、中间件工厂和按需 Repository 继续支持,见 [生成能力指南](../../GENERATION.md)。
v0.16 增加基础类型/动态 JSON 数组返回、OpenAPI 公共引用和 workspace Go Catalog 统一导出。见 [接入与筛选规则](../../DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。 v0.16 增加基础类型/动态 JSON 数组返回、OpenAPI 公共引用和 workspace Go Catalog 统一导出。见 [接入与筛选规则](../../DEVELOPMENT.md#同一份-go-错误目录供多个服务生成)。
@@ -14,7 +16,7 @@ v0.14 新增 `api format [--check]`、`api upgrade [--dry-run|--preview <空目
```sh ```sh
# 仓库读取权限与 PATH 配置见根目录 README # 仓库读取权限与 PATH 配置见根目录 README
go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.18.0-preview go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.26.0-preview
go-omni new --module example.com/hello ./hello go-omni new --module example.com/hello ./hello
# 从自己的契约生成;service.api 及其相对 import 文件需要在本地存在 # 从自己的契约生成;service.api 及其相对 import 文件需要在本地存在
@@ -49,14 +51,14 @@ main、Gen.go、OpenAPI 和 manifest 由生成器维护;业务文件按字节
## 版本与迁移 ## 版本与迁移
CLI 与框架依赖固定使用 v0.18.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod,不添加本地 replace;`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换: CLI 与框架依赖固定使用 v0.26.0-preview。远程安装的 CLI 默认把自身版本写入 go.mod,不添加本地 replace;`--framework-version` 可显式指定已发布的兼容版本。仅在开发框架源码时使用本地替换:
```sh ```sh
# 在 go-omni 源码仓库中执行 # 在 go-omni 源码仓库中执行
go run ./cmd/go-omni new --module example.com/hello --framework-dir . ../hello go run ./cmd/go-omni new --module example.com/hello --framework-dir . ../hello
``` ```
当前模板与依赖以 v0.18.0-preview 为基线,继续使用 manifest v2;GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级):原始模板可自动替换,含业务修改的基础文件按预览合并;通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务,不使用旧 `add` 命令的手动路由装配。 当前模板与依赖以 v0.26.0-preview 为基线,继续使用 manifest v2;GORM 可选适配器从 v0.14.1-preview 提供。已有项目使用 [定点升级与预览](../../DEVELOPMENT.md#v013-项目升级):原始模板可自动替换,含业务修改的基础文件按预览合并;通用清理由 Runtime 负责。独立工具命令不会被删除。受管理项目通过 `.api` 模块导入增加业务,不使用旧 `add` 命令的手动路由装配。
## 数据库迁移命令 ## 数据库迁移命令
@@ -69,3 +71,9 @@ go-omni migrate --dir migrations --apply
``` ```
SQL 命名示例 `0001_createUsers.up.sql`;预览只显示版本和 SHA-256,不连接数据库。失败保留 dirty 状态,不自动 down/force。细节见 [扩展模块](../../ADVANCED.md)。 SQL 命名示例 `0001_createUsers.up.sql`;预览只显示版本和 SHA-256,不连接数据库。失败保留 dirty 状态,不自动 down/force。细节见 [扩展模块](../../ADVANCED.md)。
### v0.22 文件显式配置
新生成 `internal/config/loadGen.go` 调用统一加载,纳入生成检查;项目 config.go 只维护字段、默认值和校验,loader.go 用于注册额外格式。JSON/YAML/XML 内置。旧手写 Load 会阻止生成,请先按 [迁移步骤](../../CONFIG.md#v022-迁移旧模式已移除) 整理,不会自动覆盖业务代码。
`go run . -config etc/config.test.yaml -check-config` 只做配置解析和校验,不初始化日志、数据库或监听。文件中的固定值不受环境自动覆盖;`${NAME}` 为必填引用,`${NAME:-fallback}` 提供备用值。

View File

@@ -0,0 +1,136 @@
package apigen
import (
"context"
"encoding/json"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
)
func TestGeneratedConfigurationConsumer(t *testing.T) {
if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") != "1" {
t.Skip("set GO_OMNI_SCAFFOLD_INTEGRATION=1")
}
dir := filepath.Join(t.TempDir(), "app")
api := filepath.Join(t.TempDir(), "service.api")
writeFile(t, api, "service configuration {\n@handler ping\nget /ping\n}\n")
o := Options{API: api, Dir: dir, Module: "example.com/configuration", FrameworkDir: frameworkRoot(t)}
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
main := readFile(t, filepath.Join(dir, "main.go"))
// The business only changes fields/defaults/validation, not the loading flow.
writeFile(t, filepath.Join(dir, "internal/config/config.go"), `package config
import("git.yuntuo.hk/k2safe/go-omni/config";"git.yuntuo.hk/k2safe/go-omni/logx")
type Config struct { Server config.Config `+"`json:\"server\"`"+`;Limit int `+"`json:\"limit\"`"+` }
func Default()Config{ c:=config.Default();c.Log.File="./var/log/app.log";c.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:10};return Config{Server:c,Limit:10} }
func(c Config)Validate()error{if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)};if c.Limit<0{return config.Invalid("limit","must not be negative")};return nil}
`)
writeFile(t, filepath.Join(dir, "internal/config/loader.go"), `package config
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
func NewLoader()(*omniConfig.Loader,error){return omniConfig.NewLoader(omniConfig.WithDecoder("custom",omniConfig.JSONDecoder{}))}
`)
svc := filepath.Join(dir, "internal/svc/serviceContext.go")
writeFile(t, svc, strings.Replace(readFile(t, svc), "return nil", "panic(\"dependencies must not initialize on invalid config\")", 1))
writeFile(t, filepath.Join(dir, "consumer_test.go"), readFile(t, "testdata/configuration/consumer_test.go"))
runConsumerGo(t, dir, "mod", "tidy")
runConsumerGo(t, dir, "test", "-race", "-count=1", "./...")
runConsumerGo(t, dir, "build", ".")
o.API = filepath.Join(dir, "api/service.api")
before := snapshot(t, dir)
for _, mode := range []string{"dry", "generate", "check"} {
o.DryRun = mode == "dry"
o.Check = mode == "check"
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("business files overwritten")
}
if main != readFile(t, filepath.Join(dir, "main.go")) {
t.Fatal("main changed")
}
load := filepath.Join(dir, "internal/config/loadGen.go")
original := readFile(t, load)
writeFile(t, load, original+"\n// drift\n")
if err := generateManagedTestProject(t, o); err == nil {
t.Fatal("loadGen drift accepted")
}
writeFile(t, load, original)
if os.Getenv("GO_OMNI_RELEASE_CLI") != "" {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "go", "list", "-m", "-json", frameworkModule)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GOWORK=off")
data, err := cmd.CombinedOutput()
if err != nil {
t.Fatal(err, string(data))
}
var dependency struct {
Version, Sum string
Replace any
}
if err = json.Unmarshal(data, &dependency); err != nil {
t.Fatal(err)
}
if dependency.Version != os.Getenv("GO_OMNI_RELEASE_VERSION") || dependency.Replace != nil || dependency.Sum == "" {
t.Fatal("not a remote consumer")
}
t.Logf("remote framework %s checksum %s; no local replace", dependency.Version, dependency.Sum)
}
}
func TestConfigLoadMigrationGuard(t *testing.T) {
dir := filepath.Join(t.TempDir(), "app")
o := Options{Dir: dir, Module: "example.com/upgrade", FrameworkDir: frameworkRoot(t)}
if _, err := New(o); err != nil {
t.Fatal(err)
}
o.API = filepath.Join(dir, "api/service.api")
matches, err := filepath.Glob(filepath.Join(dir, "api/*.api"))
if err != nil || len(matches) != 1 {
t.Fatal(matches, err)
}
o.API = matches[0]
writeFile(t, filepath.Join(dir, "internal/config/legacy.go"), "package config\nfunc Load(path string)(Config,error){return Default(),nil}\n")
before := snapshot(t, dir)
for _, mode := range []string{"dry", "generate", "check"} {
o.DryRun = mode == "dry"
o.Check = mode == "check"
_, err := Generate(o)
if err == nil || !strings.Contains(err.Error(), "CONFIG.md") {
t.Fatal("missing migration diagnostic", err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("migration guard changed files")
}
}
}
func TestRemovedEnvironmentTagsRejectBeforeGeneration(t *testing.T) {
dir := filepath.Join(t.TempDir(), "app")
o := Options{Dir: dir, Module: "example.com/migrate", FrameworkDir: frameworkRoot(t)}
if _, err := New(o); err != nil {
t.Fatal(err)
}
o.API = filepath.Join(dir, "api/service.api")
path := filepath.Join(dir, "internal/config/config.go")
source := readFile(t, path)
writeFile(t, path, strings.Replace(source, "json:\"server\"", "json:\"server\" env:\"OLD_SERVER\"", 1))
before := snapshot(t, dir)
for _, mode := range []string{"dry", "generate", "check"} {
o.DryRun = mode == "dry"
o.Check = mode == "check"
if _, err := Generate(o); err == nil || !strings.Contains(err.Error(), "env/envDefault") {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("migration rejection wrote files")
}
}
}

View File

@@ -55,6 +55,7 @@ func prepareMetadata(original *Spec) (*Spec, error) {
for i := range spec.Errors { for i := range spec.Errors {
spec.Errors[i].BusinessCode = normalized.Definitions[spec.Errors[i].Code].Code spec.Errors[i].BusinessCode = normalized.Definitions[spec.Errors[i].Code].Code
spec.Errors[i].Params = normalized.Definitions[spec.Errors[i].Code].Params spec.Errors[i].Params = normalized.Definitions[spec.Errors[i].Code].Params
spec.Errors[i].Translation = normalized.Definitions[spec.Errors[i].Code].Translation
} }
if err := validateErrorPolicies(&spec); err != nil { if err := validateErrorPolicies(&spec); err != nil {
return nil, err return nil, err

View File

@@ -0,0 +1,118 @@
package apigen
import (
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
)
func runConsumerGo(t *testing.T, dir string, args ...string) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "go", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GOWORK=off")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("go %v: %v\n%s", args, err, out)
}
}
func TestCopyxThreeEndpointConsumer(t *testing.T) {
if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") != "1" {
t.Skip("set GO_OMNI_SCAFFOLD_INTEGRATION=1")
}
dir := filepath.Join(t.TempDir(), "app")
o := Options{API: filepath.Join(frameworkRoot(t), "cmd/go-omni/apigen/testdata/composition/service.api"), Dir: dir, Module: "example.com/composition", FrameworkDir: frameworkRoot(t)}
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
for path, fixture := range map[string]string{
"internal/amount/money.go": "amount.go", "internal/domain/employee.go": "domain.go", "internal/convert/fields.go": "converters.go", "internal/convert/copy.go": "copy.go",
"internal/middleware/dependencies.go": "dependencies.go", "internal/middleware/accessFactory.go": "accessFactory.go", "internal/svc/client.go": "client.go", "consumer_test.go": "consumer_test.go",
} {
writeFile(t, filepath.Join(dir, path), readFile(t, "testdata/composition/"+fixture))
}
svcFile := filepath.Join(dir, "internal/svc/serviceContext.go")
source := strings.Replace(readFile(t, svcFile), "type ServiceContext struct {", "type ServiceContext struct {\n client *Client", 1)
source = strings.Replace(source, "return nil", "return c.initClient(ctx)", 1)
writeFile(t, svcFile, source)
configFile := filepath.Join(dir, "internal/config/config.go")
writeFile(t, configFile, strings.Replace(readFile(t, configFile), "type Config struct {", "type Config struct {\n FactoryMode string `json:\"factoryMode\"`", 1))
for _, group := range []string{"Enterprise", "Tenant", "Admin"} {
text := fmt.Sprintf(`package %s
import("context";"example.com/composition/internal/types";"example.com/composition/internal/convert";"example.com/composition/internal/domain")
func(s *Logic)Save(ctx context.Context,req *types.%sRequest)(*types.EmployeeResponse,error){
input,err:=convert.Employee(*req);if err!=nil{return nil,err}
result,err:=convert.Response(domain.Result{Employee:input,ID:9007199254740993});if err!=nil{return nil,err};return &result,nil
}
`, strings.ToLower(group), group)
writeFile(t, filepath.Join(dir, "internal/logic", strings.ToLower(group), "save.go"), text)
}
runConsumerGo(t, dir, "mod", "tidy")
runConsumerGo(t, dir, "test", "-race", "-count=1", "./...")
o.API = filepath.Join(dir, "api/service.api")
before := snapshot(t, dir)
o.DryRun = true
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
o.DryRun = false
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
o.Check = true
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("generation changed business files")
}
if _, err := os.Stat(filepath.Join(dir, "internal/mapping")); !os.IsNotExist(err) {
t.Fatal("mapping directory generated")
}
// Adding a common field requires one contract edit and one independent
// business field. Three endpoint converters need no new rules.
contract := readFile(t, o.API)
contract = strings.Replace(contract, "type SalaryFields {", "type SalaryFields {\n Note string `json:\"note,optional\"`", 1)
writeFile(t, o.API, contract)
domainPath := filepath.Join(dir, "internal/domain/employee.go")
writeFile(t, domainPath, strings.Replace(readFile(t, domainPath), "type Employee struct {", "type Employee struct {\n Note string", 1))
o.Check = false
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
o.Check = true
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
runConsumerGo(t, dir, "test", "-race", "-count=1", "./...")
runConsumerGo(t, dir, "build", ".")
if os.Getenv("GO_OMNI_RELEASE_CLI") != "" {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "go", "list", "-m", "-json", frameworkModule)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GOWORK=off")
data, err := cmd.CombinedOutput()
if err != nil {
t.Fatal(err, string(data))
}
var dependency struct {
Version, Sum string
Replace any
}
if err = json.Unmarshal(data, &dependency); err != nil {
t.Fatal(err)
}
if dependency.Version != os.Getenv("GO_OMNI_RELEASE_VERSION") || dependency.Replace != nil || dependency.Sum == "" {
t.Fatal("not a fixed remote consumer")
}
t.Logf("remote framework %s checksum %s; no local replace", dependency.Version, dependency.Sum)
}
}

View File

@@ -0,0 +1,59 @@
package apigen
import (
"encoding/json"
"os"
"os/exec"
"path/filepath"
"testing"
)
func TestGeneratedCORSConsumer(t *testing.T) {
if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") != "1" {
t.Skip("set GO_OMNI_SCAFFOLD_INTEGRATION=1")
}
dir := filepath.Join(t.TempDir(), "app")
api := filepath.Join(t.TempDir(), "service.api")
writeFile(t, api, "service corsdemo {\n@handler ping\nget /ping\n}\n")
o := Options{API: api, Dir: dir, Module: "example.com/corsdemo", FrameworkDir: frameworkRoot(t)}
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(dir, "consumer_test.go"), readFile(t, "testdata/cors/consumer_test.go"))
writeFile(t, filepath.Join(dir, "etc/config.yaml"), "server:\n log:\n output: discard\n cors:\n enabled: true\n allowedOrigins: [https://web.example]\n allowedMethods: [GET, POST, PUT]\n allowedHeaders: [Authorization, Content-Type]\n maxAge: 10m\n")
runConsumerGo(t, dir, "mod", "tidy")
runConsumerGo(t, dir, "test", "-race", "-count=1", "./...")
runConsumerGo(t, dir, "build", ".")
o.API = filepath.Join(dir, "api/service.api")
before := snapshot(t, dir)
for _, mode := range []string{"dry", "generate", "check"} {
o.DryRun = mode == "dry"
o.Check = mode == "check"
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("regeneration changed project files")
}
if os.Getenv("GO_OMNI_RELEASE_CLI") != "" {
cmd := exec.Command("go", "list", "-m", "-json", frameworkModule)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GOWORK=off")
out, err := cmd.CombinedOutput()
if err != nil {
t.Fatal(err, string(out))
}
var dependency struct {
Version, Sum string
Replace any
}
if err := json.Unmarshal(out, &dependency); err != nil {
t.Fatal(err)
}
if dependency.Version != os.Getenv("GO_OMNI_RELEASE_VERSION") || dependency.Replace != nil || dependency.Sum == "" {
t.Fatal("not a fixed remote consumer")
}
t.Logf("remote framework %s checksum %s; no local replace", dependency.Version, dependency.Sum)
}
}

View File

@@ -11,14 +11,15 @@ import (
// ErrorDefinition declares a public, stable error. Generated constructors return // ErrorDefinition declares a public, stable error. Generated constructors return
// a fresh error each time so WithCause cannot mutate a shared singleton. // a fresh error each time so WithCause cannot mutate a shared singleton.
type ErrorDefinition struct { type ErrorDefinition struct {
Name string `json:"name"` Name string `json:"name"`
BusinessCode any `json:"businessCode,omitempty"` BusinessCode any `json:"businessCode,omitempty"`
Params []apierrors.Parameter `json:"params,omitempty"` Params []apierrors.Parameter `json:"params,omitempty"`
Catalog string `json:"catalog,omitempty"` Translation *apierrors.Translation `json:"translation,omitempty"`
Status int `json:"status"` Catalog string `json:"catalog,omitempty"`
Code string `json:"code"` Status int `json:"status"`
Message string `json:"message"` Code string `json:"code"`
Position string `json:"-"` Message string `json:"message"`
Position string `json:"-"`
} }
type AuthDefinition struct { type AuthDefinition struct {
Name, Type, In, KeyName, Middleware, Description, BearerFormat, Position string Name, Type, In, KeyName, Middleware, Description, BearerFormat, Position string

View File

@@ -49,11 +49,12 @@ func (p *parser) parseErrorCatalog() error {
} }
} }
type entry struct { type entry struct {
Key string `json:"key"` Key string `json:"key"`
Code any `json:"code"` Code any `json:"code"`
Message string `json:"message"` Message string `json:"message"`
Status int `json:"status"` Status int `json:"status"`
Params []apierrors.Parameter `json:"params"` Params []apierrors.Parameter `json:"params"`
Translation *apierrors.Translation `json:"translation"`
} }
var entries map[string]entry var entries map[string]entry
if err := apierrors.DecodeCatalogJSON([]byte(props["errors"]), &entries); err != nil { if err := apierrors.DecodeCatalogJSON([]byte(props["errors"]), &entries); err != nil {
@@ -75,7 +76,7 @@ func (p *parser) parseErrorCatalog() error {
if e.Status == 0 { if e.Status == 0 {
e.Status = defaultStatus e.Status = defaultStatus
} }
p.spec.Errors = append(p.spec.Errors, ErrorDefinition{Name: fn, Status: e.Status, Code: e.Key, Message: e.Message, BusinessCode: e.Code, Params: e.Params, Catalog: name, Position: pos}) p.spec.Errors = append(p.spec.Errors, ErrorDefinition{Name: fn, Status: e.Status, Code: e.Key, Message: e.Message, BusinessCode: e.Code, Params: e.Params, Translation: e.Translation, Catalog: name, Position: pos})
} }
return nil return nil
} }
@@ -90,7 +91,7 @@ func errorCatalog(spec *Spec, names []string) apierrors.Catalog {
if names != nil && !selected[e.Name] { if names != nil && !selected[e.Name] {
continue continue
} }
c.Definitions[e.Code] = apierrors.Definition{Status: e.Status, Code: e.BusinessCode, Message: e.Message, Params: e.Params} c.Definitions[e.Code] = apierrors.Definition{Status: e.Status, Code: e.BusinessCode, Message: e.Message, Params: e.Params, Translation: e.Translation}
} }
return c return c
} }

View File

@@ -19,7 +19,10 @@ import (
) )
const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" const frameworkModule = "git.yuntuo.hk/k2safe/go-omni"
const minimumVersion = "v0.18.0-preview"
// Runtime-only releases need not raise the managed template compatibility floor.
const minimumVersion = "v0.22.0-preview"
const currentVersion = "v0.26.0-preview"
const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n" const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n"
type Options struct { type Options struct {
@@ -106,6 +109,9 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
if err = checkRuntimeExtension(directory); err != nil { if err = checkRuntimeExtension(directory); err != nil {
return Output{}, err return Output{}, err
} }
if err = checkConfigLoadExtension(directory); err != nil {
return Output{}, err
}
for _, r := range c.Spec.Routes { for _, r := range c.Spec.Routes {
if !r.Manual { if !r.Manual {
path := filepath.Join(directory, "internal/handler", r.Group, lower(r.Handler)+"Manual.go") path := filepath.Join(directory, "internal/handler", r.Group, lower(r.Handler)+"Manual.go")
@@ -152,7 +158,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
if !filepath.IsAbs(path) { if !filepath.IsAbs(path) {
path = filepath.Join(directory, path) path = filepath.Join(directory, path)
} }
if source, err := os.ReadFile(filepath.Join(path, "config", "env.go")); err == nil && strings.Contains(string(source), "func ServerEnvAliases(") { if source, err := os.ReadFile(filepath.Join(path, "config", "references.go")); err == nil && strings.Contains(string(source), "func WithLookupEnv(") {
localCurrent = true localCurrent = true
} }
} }
@@ -188,6 +194,13 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
} }
seeds["go.mod"] = data seeds["go.mod"] = data
} }
capabilityModule := moduleData
if seeded, ok := seeds["go.mod"]; ok {
capabilityModule = seeded
}
if err := checkI18nRuntime(c.Spec, capabilityModule, directory); err != nil {
return Output{}, err
}
if err = module.CheckImportPath(o.Module); err != nil { if err = module.CheckImportPath(o.Module); err != nil {
return Output{}, err return Output{}, err
} }
@@ -252,9 +265,6 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
} }
} }
} }
if err := renderMappings(o, generated, seeds); err != nil {
return Output{}, err
}
currentConfig, stillExists, err := readGeneratorConfig(directory) currentConfig, stillExists, err := readGeneratorConfig(directory)
if err != nil { if err != nil {
return Output{}, err return Output{}, err
@@ -265,7 +275,7 @@ func generate(o Options, c *Compiled, starter bool) (Output, error) {
return Output{}, errors.New("generation configuration changed during generation; retry") return Output{}, errors.New("generation configuration changed during generation; retry")
} }
out, err := apply(directory, o.Module, entry, generated, seeds, o.Check, o.DryRun, o.profile(), snapshotContract(c.Spec), o.lockHeld) out, err := apply(directory, o.Module, entry, generated, seeds, o.Check, o.DryRun, o.profile(), snapshotContract(c.Spec), o.lockHeld)
expectedVersion := minimumVersion expectedVersion := currentVersion
if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" && semver.IsValid(info.Main.Version) { if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" && semver.IsValid(info.Main.Version) {
expectedVersion = info.Main.Version expectedVersion = info.Main.Version
} }
@@ -304,7 +314,7 @@ func moduleFile(o Options) ([]byte, error) {
if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" { if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" {
version = info.Main.Version version = info.Main.Version
} else { } else {
version = minimumVersion version = currentVersion
} }
} }
if !semver.IsValid(version) || semver.Major(version) != "v0" && semver.Major(version) != "v1" { if !semver.IsValid(version) || semver.Major(version) != "v0" && semver.Major(version) != "v1" {

View File

@@ -15,14 +15,11 @@ import (
const generatorConfigFile = "omni.gen.json" const generatorConfigFile = "omni.gen.json"
// GeneratorConfig is developer-owned and checked on every generation. Mapping // GeneratorConfig contains developer-owned scaffolding options only.
// rules never change the public API or infer a persistence model.
type GeneratorConfig struct { type GeneratorConfig struct {
Repository string `json:"repository,omitempty"` Repository string `json:"repository,omitempty"`
Middleware string `json:"middleware,omitempty"` Middleware string `json:"middleware,omitempty"`
Mappings []Mapping `json:"mappings,omitempty"` inputDigest string
MappingFiles []string `json:"mappingFiles,omitempty"`
inputDigest string
} }
func readGeneratorConfig(root string) (GeneratorConfig, bool, error) { func readGeneratorConfig(root string) (GeneratorConfig, bool, error) {
@@ -40,7 +37,7 @@ func readGeneratorConfig(root string) (GeneratorConfig, bool, error) {
return cfg, false, err return cfg, false, err
} }
if previous.GeneratorConfig { if previous.GeneratorConfig {
return cfg, false, errors.New("omni.gen.json is missing; restore the project-owned generation options and mappings before regenerating") return cfg, false, errors.New("omni.gen.json is missing; restore the project-owned generation options before regenerating")
} }
cfg.Middleware = "handlers" cfg.Middleware = "handlers"
} else if !errors.Is(e, os.ErrNotExist) { } else if !errors.Is(e, os.ErrNotExist) {
@@ -54,72 +51,36 @@ func readGeneratorConfig(root string) (GeneratorConfig, bool, error) {
if !stat.Mode().IsRegular() || stat.Size() > 1<<20 { if !stat.Mode().IsRegular() || stat.Size() > 1<<20 {
return cfg, false, errors.New("omni.gen.json must be a regular file of at most 1 MiB") return cfg, false, errors.New("omni.gen.json must be a regular file of at most 1 MiB")
} }
raw, err := readMappingConfigFile(root, generatorConfigFile) raw, err := readGeneratorConfigFile(root, generatorConfigFile)
if err != nil { if err != nil {
return cfg, false, err return cfg, false, err
} }
var keys map[string]json.RawMessage
if err = apierrors.DecodeCatalogJSON(raw, &keys); err != nil {
return cfg, false, fmt.Errorf("omni.gen.json: %w", err)
}
for _, key := range []string{"mappings", "mappingFiles"} {
if _, exists := keys[key]; exists {
return cfg, false, errors.New("mapping generation was removed: migrate callers to copyx or ordinary Go, remove mappings/mappingFiles from omni.gen.json, then remove internal/mapping/mappingsGen.go; see COPY.md")
}
}
if err = apierrors.DecodeCatalogJSON(raw, &cfg); err != nil { if err = apierrors.DecodeCatalogJSON(raw, &cfg); err != nil {
return cfg, false, fmt.Errorf("omni.gen.json: %w", err) return cfg, false, fmt.Errorf("omni.gen.json: %w", err)
} }
// Child files only declare mappings. No recursive imports, globs, implicit digest := sha256.Sum256(raw)
// directory scans or access outside the project are supported. cfg.inputDigest = fmt.Sprintf("%x", digest)
if len(cfg.MappingFiles) > 64 {
return cfg, false, errors.New("omni.gen.json: at most 64 mappingFiles supported")
}
digest := sha256.New()
fmt.Fprintf(digest, "%s:%d:", generatorConfigFile, len(raw))
digest.Write(raw)
total := len(raw)
for i := range cfg.Mappings {
cfg.Mappings[i].origin = generatorConfigFile
}
seen := map[string]bool{generatorConfigFile: true}
for _, name := range cfg.MappingFiles {
key := strings.ToLower(name)
if seen[key] {
return cfg, false, fmt.Errorf("omni.gen.json: duplicate mapping file %q", name)
}
seen[key] = true
if !strings.HasSuffix(name, ".json") {
return cfg, false, fmt.Errorf("mapping file %q must have a .json extension", name)
}
data, err := readMappingConfigFile(root, name)
if err != nil {
return cfg, false, err
}
total += len(data)
if total > 4<<20 {
return cfg, false, errors.New("mapping configuration exceeds 4 MiB in total")
}
var file struct {
Mappings []Mapping `json:"mappings"`
}
if err = apierrors.DecodeCatalogJSON(data, &file); err != nil {
return cfg, false, fmt.Errorf("%s: %w (mapping files only support mappings; nested imports are not supported)", name, err)
}
for i := range file.Mappings {
file.Mappings[i].origin = name
}
cfg.Mappings = append(cfg.Mappings, file.Mappings...)
fmt.Fprintf(digest, "%s:%d:", name, len(data))
digest.Write(data)
}
cfg.inputDigest = fmt.Sprintf("%x", digest.Sum(nil))
if cfg.Repository != "none" && cfg.Repository != "group" { if cfg.Repository != "none" && cfg.Repository != "group" {
return cfg, false, errors.New("omni.gen.json: repository must be none or group") return cfg, false, errors.New("omni.gen.json: repository must be none or group")
} }
if cfg.Middleware != "factories" && cfg.Middleware != "handlers" { if cfg.Middleware != "factories" && cfg.Middleware != "handlers" {
return cfg, false, errors.New("omni.gen.json: middleware must be factories or handlers") return cfg, false, errors.New("omni.gen.json: middleware must be factories or handlers")
} }
if len(cfg.Mappings) > 128 {
return cfg, false, errors.New("omni.gen.json: at most 128 mappings supported")
}
return cfg, true, nil return cfg, true, nil
} }
// Read through a bounded reader as well as checking stat: a growing file cannot // Read through a bounded reader as well as checking stat: a growing file cannot
// make configuration loading unbounded. safePath also rejects symlink escapes. // make configuration loading unbounded. safePath also rejects symlink escapes.
func readMappingConfigFile(root, name string) ([]byte, error) { func readGeneratorConfigFile(root, name string) ([]byte, error) {
if err := safePath(root, name); err != nil { if err := safePath(root, name); err != nil {
return nil, err return nil, err
} }

View File

@@ -2,140 +2,95 @@ package apigen
import ( import (
"encoding/json" "encoding/json"
"fmt" "io"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
"testing" "testing"
) )
func TestMappingFileConfiguration(t *testing.T) { func TestWorkspaceRejectsRemovedMappingWithoutWrites(t *testing.T) {
dir := t.TempDir() root := t.TempDir()
defs := nullableMappings() for _, name := range []string{"one", "two"} {
writeGeneratorConfig(t, dir, GeneratorConfig{Repository: "none", Middleware: "factories", Mappings: defs[:1], MappingFiles: []string{"mappings/a.json", "mappings/b.json"}}) if _, err := New(Options{Dir: filepath.Join(root, name), Module: "example.com/" + name, FrameworkDir: frameworkRoot(t)}); err != nil {
writeMappingFile(t, dir, "mappings/a.json", defs[1:5]) t.Fatal(err)
writeMappingFile(t, dir, "mappings/b.json", defs[5:])
first, exists, err := readGeneratorConfig(dir)
if err != nil || !exists || len(first.Mappings) != len(defs) {
t.Fatal(first, err)
}
for i, m := range first.Mappings {
if m.Name != defs[i].Name {
t.Fatal("non-deterministic load order")
} }
} }
second, _, err := readGeneratorConfig(dir) firstAPI := filepath.Join(root, "one/api/service.api")
if err != nil || first.inputDigest != second.inputDigest { writeFile(t, firstAPI, strings.Replace(readFile(t, firstAPI), "Greet a visitor", "Changed description", 1))
t.Fatal("unstable config digest", err) writeFile(t, filepath.Join(root, "two", generatorConfigFile), `{"mappingFiles":["../../private.json"]}`)
} workspace := filepath.Join(root, "omni.workspace.json")
path := filepath.Join(dir, "mappings/a.json") writeFile(t, workspace, `{"services":[{"dir":"one"},{"dir":"two"}]}`)
writeFile(t, path, readFile(t, path)+"\n") before := snapshot(t, root)
changed, _, err := readGeneratorConfig(dir) for _, operation := range []string{"generate", "check"} {
if err != nil || changed.inputDigest == first.inputDigest { if err := Workspace(workspace, operation, io.Discard); err == nil || !strings.Contains(err.Error(), "mapping generation was removed") {
t.Fatal("child file change invisible to generation guard", err) t.Fatal(operation, err)
} }
// Duplicate definitions name both files instead of silently overriding. if !sameFiles(before, snapshot(t, root)) {
writeMappingFile(t, dir, "mappings/b.json", defs[1:2]) t.Fatal("workspace migration failure wrote files")
cfg, _, err := readGeneratorConfig(dir) }
if err != nil {
t.Fatal(err)
}
_, err = validateMappings(cfg.Mappings, "example.com/nullable")
if err == nil || !strings.Contains(err.Error(), "mappings/a.json") || !strings.Contains(err.Error(), "mappings/b.json") {
t.Fatal(err)
} }
} }
func TestMappingFileRejection(t *testing.T) { func TestRemovedMappingConfiguration(t *testing.T) {
for _, tc := range []struct{ name, root, child, want string }{ dir := filepath.Join(t.TempDir(), "project")
{"missing", `{"mappingFiles":["maps.json"]}`, "", "maps.json"}, if _, err := New(Options{Dir: dir, Module: "example.com/removed", FrameworkDir: frameworkRoot(t)}); err != nil {
{"escape", `{"mappingFiles":["../maps.json"]}`, "", "unsafe"}, t.Fatal(err)
{"absolute", `{"mappingFiles":["/maps.json"]}`, "", "unsafe"}, }
{"nonCanonical", `{"mappingFiles":["sub/../maps.json"]}`, "", "unsafe"}, for _, raw := range []string{`{"mappings":[]}`, `{"mappingFiles":[]}`, `{"mappings":null}`, `{"mappingFiles":["../../private.json"]}`} {
{"duplicate", `{"mappingFiles":["maps.json","maps.json"]}`, `{"mappings":[]}`, "duplicate mapping file"}, writeFile(t, filepath.Join(dir, generatorConfigFile), raw)
{"caseDuplicate", `{"mappingFiles":["maps.json","MAPS.json"]}`, `{"mappings":[]}`, "duplicate mapping file"}, before := snapshot(t, dir)
{"self", `{"mappingFiles":["omni.gen.json"]}`, "", "duplicate mapping file"}, for _, o := range []Options{{Dir: dir}, {Dir: dir, Check: true}, {Dir: dir, DryRun: true}} {
{"recursive", `{"mappingFiles":["maps.json"]}`, `{"mappingFiles":["omni.gen.json"]}`, "nested imports are not supported"}, o.API = filepath.Join(dir, "api/service.api")
{"childSettings", `{"mappingFiles":["maps.json"]}`, `{"repository":"none","mappings":[]}`, "maps.json"}, if _, err := Generate(o); err == nil || !strings.Contains(err.Error(), "mapping generation was removed") {
{"duplicateKeys", `{"mappingFiles":["maps.json"]}`, `{"mappings":[],"mappings":[]}`, "duplicate key"},
{"rootNull", `null`, "", "JSON object"},
{"childNull", `{"mappingFiles":["maps.json"]}`, `null`, "JSON object"},
{"unknownPolicy", `{"mappingFiles":["maps.json"]}`, `{"mappings":[{"unknown":true}]}`, "unknown field"},
{"trailingValue", `{"mappingFiles":["maps.json"]}`, `{"mappings":[]} {}`, "maps.json"},
{"tooLarge", `{"mappingFiles":["maps.json"]}`, strings.Repeat(" ", 1<<20) + `{}`, "1 MiB"},
} {
t.Run(tc.name, func(t *testing.T) {
dir := t.TempDir()
writeFile(t, filepath.Join(dir, generatorConfigFile), tc.root)
if tc.child != "" {
writeFile(t, filepath.Join(dir, "maps.json"), tc.child)
}
before := snapshot(t, dir)
_, _, err := readGeneratorConfig(dir)
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatal(err) t.Fatal(err)
} }
if !sameFiles(before, snapshot(t, dir)) { if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("read changed configuration") t.Fatal("rejected config wrote project")
} }
}) }
}
for _, directory := range []bool{false, true} {
t.Run(fmt.Sprintf("symlinkDirectory%t", directory), func(t *testing.T) {
dir := t.TempDir()
outside := t.TempDir()
writeMappingFile(t, outside, "maps.json", nil)
name := "maps.json"
target := filepath.Join(outside, name)
link := filepath.Join(dir, name)
if directory {
name = "linked/maps.json"
target = outside
link = filepath.Join(dir, "linked")
}
if err := os.Symlink(target, link); err != nil {
t.Fatal(err)
}
writeGeneratorConfig(t, dir, GeneratorConfig{Repository: "none", Middleware: "factories", MappingFiles: []string{name}})
if _, _, err := readGeneratorConfig(dir); err == nil || !strings.Contains(err.Error(), "symlink") {
t.Fatal(err)
}
})
} }
} }
func TestOldMappingFileRequiresExplicitMigration(t *testing.T) {
func TestMappingFileBounds(t *testing.T) { dir := filepath.Join(t.TempDir(), "project")
dir := t.TempDir() if _, err := New(Options{Dir: dir, Module: "example.com/removed", FrameworkDir: frameworkRoot(t)}); err != nil {
files := make([]string, 65)
for i := range files {
files[i] = fmt.Sprintf("maps%d.json", i)
}
writeGeneratorConfig(t, dir, GeneratorConfig{Repository: "none", Middleware: "factories", MappingFiles: files})
if _, _, err := readGeneratorConfig(dir); err == nil || !strings.Contains(err.Error(), "64") {
t.Fatal(err) t.Fatal(err)
} }
writeGeneratorConfig(t, dir, GeneratorConfig{Repository: "none", Middleware: "factories", MappingFiles: files[:4]}) name := "internal/mapping/mappingsGen.go"
for _, name := range files[:4] { body := generatedHeader + "package mapping\n"
writeFile(t, filepath.Join(dir, name), `{"mappings":[]}`+strings.Repeat(" ", (1<<20)-len(`{"mappings":[]}`))) writeFile(t, filepath.Join(dir, name), body)
} var m manifest
if _, _, err := readGeneratorConfig(dir); err == nil || !strings.Contains(err.Error(), "4 MiB") { if err := json.Unmarshal([]byte(readFile(t, filepath.Join(dir, manifestPath))), &m); err != nil {
t.Fatal(err) t.Fatal(err)
} }
// The definition limit applies after merging, not separately to each file. m.Generated[name] = digest([]byte(body))
defs := make([]Mapping, 129) raw, _ := json.Marshal(m)
for i := range defs { writeFile(t, filepath.Join(dir, manifestPath), string(raw))
defs[i] = nullableMappings()[0] before := snapshot(t, dir)
defs[i].Name = fmt.Sprintf("Map%d", i) o := Options{Dir: dir, API: filepath.Join(dir, "api/service.api")}
for _, mode := range []string{"generate", "check", "dry-run"} {
o.Check = mode == "check"
o.DryRun = mode == "dry-run"
if _, err := Generate(o); err == nil || !strings.Contains(err.Error(), "mapping generation was removed") {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("legacy file deleted or project changed")
}
} }
writeGeneratorConfig(t, dir, GeneratorConfig{Repository: "none", Middleware: "factories", MappingFiles: files[:2]}) if err := os.Remove(filepath.Join(dir, name)); err != nil {
writeMappingFile(t, dir, files[0], defs[:65])
writeMappingFile(t, dir, files[1], defs[65:])
if _, _, err := readGeneratorConfig(dir); err == nil || !strings.Contains(err.Error(), "128") {
t.Fatal(err) t.Fatal(err)
} }
// Origins/digests are loader metadata, never serialized into user configuration. o.Check = false
raw, err := json.Marshal(GeneratorConfig{inputDigest: "private"}) o.DryRun = false
if err != nil || strings.Contains(string(raw), "private") { if _, err := Generate(o); err != nil {
t.Fatal(string(raw), err) t.Fatal(err)
}
o.Check = true
if _, err := Generate(o); err != nil {
t.Fatal(err)
}
if strings.Contains(readFile(t, filepath.Join(dir, manifestPath)), name) {
t.Fatal("old ownership retained")
} }
} }

View File

@@ -0,0 +1,57 @@
package apigen
import (
"fmt"
"os"
"path/filepath"
"strings"
"golang.org/x/mod/modfile"
"golang.org/x/mod/semver"
)
// Translation metadata was introduced in this release. Consumers of older
// runtimes must upgrade or use an explicitly supported local replacement.
const i18nMinimumVersion = "v0.26.0-preview"
func checkI18nRuntime(spec *Spec, moduleData []byte, directory string) error {
needed := false
for _, e := range spec.Errors {
needed = needed || e.Translation != nil
}
if !needed {
return nil
}
mf, err := modfile.Parse("go.mod", moduleData, nil)
if err != nil {
return err
}
version := ""
for _, r := range mf.Require {
if r.Mod.Path == frameworkModule {
version = r.Mod.Version
}
}
for _, r := range mf.Replace {
if r.Old.Path != frameworkModule || r.Old.Version != "" && r.Old.Version != version {
continue
}
if r.New.Version != "" {
version = r.New.Version
continue
}
root := r.New.Path
if !filepath.IsAbs(root) {
root = filepath.Join(directory, root)
}
data, e := os.ReadFile(filepath.Join(root, "errors", "localization.go"))
if e == nil && strings.Contains(string(data), "type Translation struct") {
return nil
}
return fmt.Errorf("translation metadata requires a local framework with errors.Translation support")
}
if semver.Compare(version, i18nMinimumVersion) < 0 {
return fmt.Errorf("translation metadata requires %s or newer; use --framework-dir for unpublished source", i18nMinimumVersion)
}
return nil
}

View File

@@ -0,0 +1,42 @@
package apigen
import (
"path/filepath"
"strings"
"testing"
)
func TestI18nTranslationMetadataAndVersionBoundary(t *testing.T) {
spec, err := Parse(filepath.Join(frameworkRoot(t), "examples/contracts/i18n.api"))
if err != nil {
t.Fatal(err)
}
compiled, err := Compile(spec)
if err != nil {
t.Fatal(err)
}
ref := compiled.Spec.Errors[0].Translation
if ref == nil || ref.Namespace != "limits" || ref.Key != "exceeded" {
t.Fatal(ref)
}
if err := checkI18nRuntime(spec, []byte("module example.com/test\ngo 1.22\nrequire "+frameworkModule+" v0.25.0-preview\n"), t.TempDir()); err == nil {
t.Fatal("old runtime accepted")
}
generated, _, err := renderProject(Options{Module: "example.com/test"}, compiled, false)
if err != nil {
t.Fatal(err)
}
if !strings.Contains(string(generated["internal/apierrors/errorsGen.go"]), "translation") || !strings.Contains(string(generated["docs/openapi.json"]), "translation") {
t.Fatal("translation metadata lost")
}
bad := readFile(t, filepath.Join(frameworkRoot(t), "examples/contracts/i18n.api"))
bad = strings.Replace(bad, `"namespace":"limits"`, `"namespace":""`, 1)
parsed, err := parseText(t, bad)
if err == nil {
_, err = Compile(parsed)
}
if err == nil {
t.Fatal("invalid translation reference accepted")
}
}
func TestI18nGeneratedConsumer(t *testing.T) { testCatalogConsumer(t, "i18n.api", "i18n") }

View File

@@ -1,453 +0,0 @@
package apigen
import (
"fmt"
"go/ast"
"go/format"
goparser "go/parser"
gotoken "go/token"
"go/types"
"sort"
"strconv"
"strings"
"golang.org/x/mod/module"
)
// GoTypeRef names an existing, exported, non-generic Go struct.
type GoTypeRef struct {
Package string `json:"package"`
Type string `json:"type"`
}
type GoFunctionRef struct {
Package string `json:"package"`
Name string `json:"name"`
}
type MappingField struct {
Source string `json:"source"`
Convert *GoFunctionRef `json:"convert,omitempty"`
Mapping string `json:"mapping,omitempty"`
Elements bool `json:"elements,omitempty"`
Nil string `json:"nil,omitempty"`
}
type Mapping struct {
Name string `json:"name"`
PointerResult bool `json:"pointerResult,omitempty"`
origin string
Source GoTypeRef `json:"source"`
Target GoTypeRef `json:"target"`
Fields map[string]MappingField `json:"fields,omitempty"`
IgnoreSource map[string]string `json:"ignoreSource,omitempty"`
IgnoreTarget map[string]string `json:"ignoreTarget,omitempty"`
}
type mappingField struct {
path, name string
typ types.Type
}
type mappingTypes struct{ source, target types.Type }
type mappingRenderer struct {
importer types.Importer
imports map[string]string
definitions map[string]Mapping
types map[string]mappingTypes
}
func renderMappings(o Options, generated, seeds map[string][]byte) error {
if len(o.generation.Mappings) == 0 {
return nil
}
refs, err := validateMappings(o.generation.Mappings, o.Module)
if err != nil {
return err
}
importer, close, err := loadMappingTypes(o, refs, generated, seeds)
if err != nil {
return err
}
defer close()
r := mappingRenderer{importer: importer, imports: map[string]string{}, definitions: map[string]Mapping{}, types: map[string]mappingTypes{}}
for _, m := range o.generation.Mappings {
source, err := r.lookup(m.Source)
if err != nil {
return err
}
target, err := r.lookup(m.Target)
if err != nil {
return err
}
r.definitions[m.Name] = m
r.types[m.Name] = mappingTypes{source, target}
}
names := make([]string, 0, len(r.definitions))
for name := range r.definitions {
names = append(names, name)
}
sort.Strings(names)
var body strings.Builder
for _, name := range names {
source, err := r.render(r.definitions[name])
if err != nil {
return fmt.Errorf("%s: mapping %s: %w", mappingOrigin(r.definitions[name]), name, err)
}
body.WriteString(source)
}
imports := make([]string, 0, len(r.imports))
for path := range r.imports {
imports = append(imports, path)
}
sort.Strings(imports)
text := generatedHeader + "package mapping\nimport(\n"
for _, path := range imports {
text += r.imports[path] + " " + strconv.Quote(path) + "\n"
}
text += ")\n" + body.String()
code, err := format.Source([]byte(text))
if err != nil {
return fmt.Errorf("format mappings: %w", err)
}
// The generated package is type-checked before it is written. No application
// code (including init or converters) is executed during generation.
fset := gotoken.NewFileSet()
file, err := goparser.ParseFile(fset, "mappingsGen.go", code, 0)
if err != nil {
return err
}
check := types.Config{Importer: importer}
if _, err = check.Check(o.Module+"/internal/mapping", fset, []*ast.File{file}, nil); err != nil {
return fmt.Errorf("generated mapping type check: %w", err)
}
generated["internal/mapping/mappingsGen.go"] = code
return nil
}
func validateMappings(defs []Mapping, project string) ([]string, error) {
refs := map[string]bool{}
names := map[string]string{}
validPackage := func(path string) bool {
if module.CheckImportPath(path) != nil || path == project+"/internal/mapping" {
return false
}
// go/types does not enforce the compiler's internal import visibility.
if strings.HasPrefix(path, "internal/") {
return false
}
if i := strings.LastIndex(path, "/internal/"); i >= 0 {
return strings.HasPrefix(project+"/internal/mapping", path[:i]+"/")
}
return true
}
for _, m := range defs {
if !gotoken.IsIdentifier(m.Name) || !gotoken.IsExported(m.Name) {
return nil, fmt.Errorf("mapping name %q must be unique and exported", m.Name)
}
functions := []string{m.Name}
if m.PointerResult {
functions = append(functions, m.Name+"Ptr")
}
for _, name := range functions {
if prior, exists := names[name]; exists {
return nil, fmt.Errorf("mapping function %q must be unique: %s conflicts with %s", name, mappingOrigin(m), prior)
}
names[name] = mappingOrigin(m)
}
for _, ref := range []GoTypeRef{m.Source, m.Target} {
if !validPackage(ref.Package) || !gotoken.IsIdentifier(ref.Type) || !gotoken.IsExported(ref.Type) {
return nil, fmt.Errorf("mapping %s: invalid Go type reference", m.Name)
}
refs[ref.Package] = true
}
if len(m.Fields) > 4096 || len(m.IgnoreSource) > 4096 || len(m.IgnoreTarget) > 4096 {
return nil, fmt.Errorf("mapping %s: field rule limit exceeded", m.Name)
}
for target, f := range m.Fields {
if target == "" || f.Source == "" {
return nil, fmt.Errorf("mapping %s: explicit field rule requires target and source paths", m.Name)
}
if f.Nil != "" && f.Nil != "preserve" && f.Nil != "zero" && f.Nil != "error" {
return nil, fmt.Errorf("mapping %s.%s: nil must be preserve, zero or error", m.Name, target)
}
if f.Convert != nil && f.Nil != "" {
return nil, fmt.Errorf("mapping %s.%s: nil policy cannot be combined with a converter; the converter owns nil behavior", m.Name, target)
}
if f.Convert != nil {
if f.Mapping != "" || !validPackage(f.Convert.Package) || !gotoken.IsIdentifier(f.Convert.Name) || !gotoken.IsExported(f.Convert.Name) {
return nil, fmt.Errorf("mapping %s.%s: invalid or conflicting converter", m.Name, target)
}
refs[f.Convert.Package] = true
}
}
}
visiting, done := map[string]bool{}, map[string]bool{}
byName := map[string]Mapping{}
for _, m := range defs {
byName[m.Name] = m
}
var visit func(string) error
visit = func(name string) error {
if done[name] {
return nil
}
if visiting[name] {
return fmt.Errorf("mapping cycle at %s", name)
}
m, ok := byName[name]
if !ok {
return fmt.Errorf("unknown mapping %s", name)
}
visiting[name] = true
for _, f := range m.Fields {
if f.Mapping != "" {
if err := visit(f.Mapping); err != nil {
return err
}
}
}
delete(visiting, name)
done[name] = true
return nil
}
for _, m := range defs {
if err := visit(m.Name); err != nil {
return nil, err
}
}
out := make([]string, 0, len(refs))
for path := range refs {
out = append(out, path)
}
sort.Strings(out)
return out, nil
}
func (r *mappingRenderer) qualifier(p *types.Package) string {
if alias, ok := r.imports[p.Path()]; ok {
return alias
}
alias := fmt.Sprintf("pkg%d", len(r.imports))
r.imports[p.Path()] = alias
return alias
}
func (r *mappingRenderer) typeName(t types.Type) string { return types.TypeString(t, r.qualifier) }
func (r *mappingRenderer) lookup(ref GoTypeRef) (types.Type, error) {
pkg, err := r.importer.Import(ref.Package)
if err != nil {
return nil, fmt.Errorf("mapping type %s.%s: %w", ref.Package, ref.Type, err)
}
obj, ok := pkg.Scope().Lookup(ref.Type).(*types.TypeName)
if !ok {
return nil, fmt.Errorf("mapping type %s.%s not found", ref.Package, ref.Type)
}
typ := obj.Type()
if n, ok := typ.(*types.Named); ok && n.TypeParams() != nil && n.TypeParams().Len() > 0 {
return nil, fmt.Errorf("mapping root %s must be a non-generic struct", ref.Type)
}
if _, ok := typ.Underlying().(*types.Struct); !ok {
return nil, fmt.Errorf("mapping root %s must be a struct", ref.Type)
}
return typ, nil
}
func mappingFields(typ types.Type) (map[string]mappingField, error) {
out := map[string]mappingField{}
leaves := map[string]string{}
var walk func(types.Type, string, int) error
walk = func(typ types.Type, prefix string, depth int) error {
if depth > 32 {
return fmt.Errorf("embedded field nesting exceeds 32")
}
st, ok := typ.Underlying().(*types.Struct)
if !ok {
return fmt.Errorf("embedded field %s must be a value struct", prefix)
}
for i := 0; i < st.NumFields(); i++ {
f := st.Field(i)
path := prefix + f.Name()
if !f.Exported() {
return fmt.Errorf("unexported field %s cannot be mapped; use an explicit converter on the containing value", path)
}
if f.Embedded() {
if err := walk(f.Type(), path+".", depth+1); err != nil {
return err
}
continue
}
if previous, exists := leaves[f.Name()]; exists {
return fmt.Errorf("ambiguous embedded fields %s and %s", previous, path)
}
leaves[f.Name()] = path
out[path] = mappingField{path, f.Name(), f.Type()}
if len(out) > 4096 {
return fmt.Errorf("mapping field limit exceeded")
}
}
return nil
}
if err := walk(typ, "", 0); err != nil {
return nil, err
}
return out, nil
}
func (r *mappingRenderer) render(m Mapping) (string, error) {
mt := r.types[m.Name]
source, err := mappingFields(mt.source)
if err != nil {
return "", err
}
target, err := mappingFields(mt.target)
if err != nil {
return "", err
}
for key, reason := range m.IgnoreSource {
if _, ok := source[key]; !ok || strings.TrimSpace(reason) == "" {
return "", fmt.Errorf("ignoreSource %s requires an existing field and reason", key)
}
}
for key, reason := range m.IgnoreTarget {
if _, ok := target[key]; !ok || strings.TrimSpace(reason) == "" {
return "", fmt.Errorf("ignoreTarget %s requires an existing field and reason", key)
}
}
for key := range m.Fields {
if _, ok := target[key]; !ok {
return "", fmt.Errorf("unknown target field %s", key)
}
if _, ignored := m.IgnoreTarget[key]; ignored {
return "", fmt.Errorf("target %s is both mapped and ignored", key)
}
}
names := make([]string, 0, len(target))
for name := range target {
names = append(names, name)
}
sort.Strings(names)
used := map[string]bool{}
text := fmt.Sprintf("// %s maps one value. Pointer/map/slice assignments retain Go reference semantics.\nfunc %s(source %s)(%s,error){\nvar result %s\n", m.Name, m.Name, r.typeName(mt.source), r.typeName(mt.target), r.typeName(mt.target))
for _, path := range names {
dst := target[path]
if reason, ignored := m.IgnoreTarget[path]; ignored {
text += "// Target " + path + " is intentionally left at zero; see omni.gen.json.\n"
_ = reason
continue
}
rule, explicit := m.Fields[path]
if !explicit {
for key, f := range source {
if f.name == dst.name {
rule.Source = key
break
}
}
}
src, ok := source[rule.Source]
if !ok {
return "", fmt.Errorf("target %s has no source; add a field rule or ignoreTarget reason", path)
}
if _, ignored := m.IgnoreSource[src.path]; ignored {
return "", fmt.Errorf("source %s is both mapped and ignored", src.path)
}
used[src.path] = true
from, to := src.typ, dst.typ
srcExpr, dstExpr := "source."+src.path, "result."+path
if rule.Elements {
switch d := to.Underlying().(type) {
case *types.Slice:
s, ok := from.Underlying().(*types.Slice)
if !ok {
return "", fmt.Errorf("%s: elements requires slice to slice or equal-length array to array", path)
}
text += fmt.Sprintf("if %s!=nil {\n%s=make(%s,len(%s))\nfor index:=range %s {\n", srcExpr, dstExpr, r.typeName(to), srcExpr, srcExpr)
from, to = s.Elem(), d.Elem()
case *types.Array:
s, ok := from.Underlying().(*types.Array)
if !ok || s.Len() != d.Len() {
return "", fmt.Errorf("%s: array lengths must match", path)
}
text += fmt.Sprintf("for index:=range %s {\n", srcExpr)
from, to = s.Elem(), d.Elem()
default:
return "", fmt.Errorf("%s: elements requires a slice or array", path)
}
srcExpr += "[index]"
dstExpr += "[index]"
}
expr, fallible, err := r.expression(rule, from, to, srcExpr)
if err != nil {
return "", fmt.Errorf("%s -> %s: %w", src.path, path, err)
}
if fallible {
fmtPkg, err := r.importer.Import("fmt")
if err != nil {
return "", err
}
alias := r.qualifier(fmtPkg)
message := m.Name + "." + path + ": %w"
args := "err"
if rule.Elements {
message = m.Name + "." + path + "[%d]: %w"
args = "index,err"
}
text += fmt.Sprintf("{ value,err:=%s\nif err!=nil{var zero %s;return zero,%s.Errorf(%q,%s)}\n%s=value\n}\n", expr, r.typeName(mt.target), alias, message, args, dstExpr)
} else {
text += dstExpr + "=" + expr + "\n"
}
if rule.Elements {
text += "}\n"
if _, ok := dst.typ.Underlying().(*types.Slice); ok {
text += "}\n"
}
}
}
keys := make([]string, 0, len(source))
for key := range source {
keys = append(keys, key)
}
sort.Strings(keys)
for _, key := range keys {
if !used[key] {
if _, ignored := m.IgnoreSource[key]; !ignored {
return "", fmt.Errorf("unmapped source field %s; map it or provide an ignoreSource reason", key)
}
}
}
text += "return result,nil\n}\n"
if m.PointerResult {
text += fmt.Sprintf("// %sPtr returns nil on conversion failure and otherwise a pointer to the result.\nfunc %sPtr(source %s)(*%s,error){\nvalue,err:=%s(source)\nif err!=nil{return nil,err}\nreturn &value,nil\n}\n", m.Name, m.Name, r.typeName(mt.source), r.typeName(mt.target), m.Name)
}
return text, nil
}
func (r *mappingRenderer) expression(rule MappingField, from, to types.Type, source string) (string, bool, error) {
if rule.Mapping != "" {
return r.nestedExpression(rule, from, to, source)
}
if rule.Nil != "" {
return r.dereferenceExpression(rule, from, to, source)
}
if rule.Convert != nil {
pkg, err := r.importer.Import(rule.Convert.Package)
if err != nil {
return "", false, err
}
fn, ok := pkg.Scope().Lookup(rule.Convert.Name).(*types.Func)
if !ok {
return "", false, fmt.Errorf("converter function %s not found", rule.Convert.Name)
}
sig := fn.Type().(*types.Signature)
if sig.Variadic() || (sig.TypeParams() != nil && sig.TypeParams().Len() > 0) || sig.Params().Len() != 1 || sig.Results().Len() < 1 || sig.Results().Len() > 2 {
return "", false, fmt.Errorf("converter must have signature func(Source) Target or func(Source) (Target,error)")
}
if !types.Identical(sig.Params().At(0).Type(), from) || !types.Identical(sig.Results().At(0).Type(), to) {
return "", false, fmt.Errorf("converter signature does not match field types")
}
fallible := sig.Results().Len() == 2
if fallible && !types.Identical(sig.Results().At(1).Type(), types.Universe.Lookup("error").Type()) {
return "", false, fmt.Errorf("converter second result must be error")
}
return r.qualifier(pkg) + "." + rule.Convert.Name + "(" + source + ")", fallible, nil
}
if !types.Identical(from, to) {
return "", false, fmt.Errorf("types %s and %s differ; provide an explicit converter", types.TypeString(from, nil), types.TypeString(to, nil))
}
return source, false, nil
}

View File

@@ -1,175 +0,0 @@
package apigen
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"go/importer"
gotoken "go/token"
"go/types"
"io"
"os"
"os/exec"
"path/filepath"
"sort"
"strings"
"time"
"golang.org/x/mod/modfile"
)
// loadMappingTypes uses the Go compiler's export data, including the pending
// DTOs via an overlay. Module files and sum updates remain in a temporary dir.
// It compiles only explicitly selected packages and their dependencies; never
// runs project code. Build tags/GOOS follow the caller; GOWORK is disabled.
func loadMappingTypes(o Options, refs []string, generated, seeds map[string][]byte) (types.Importer, func(), error) {
// go list resolves its working directory (notably /var -> /private/var on
// macOS). Overlay keys must name that same physical path or stale DTOs win.
o.Dir = canonical(o.Dir)
temp, err := os.MkdirTemp("", "go-omni-mapping-*")
if err != nil {
return nil, nil, err
}
cleanup := func() { _ = os.RemoveAll(temp) }
fail := func(err error) (types.Importer, func(), error) { cleanup(); return nil, nil, err }
raw, err := os.ReadFile(filepath.Join(o.Dir, "go.mod"))
if err != nil {
return fail(errors.New("mapping generation requires an existing Go module; scaffold first and resolve dependencies"))
}
mf, err := modfile.Parse("go.mod", raw, nil)
if err != nil {
return fail(err)
}
replacements := append([]*modfile.Replace(nil), mf.Replace...)
for _, r := range replacements {
if r.New.Version == "" && !filepath.IsAbs(r.New.Path) {
if err = mf.AddReplace(r.Old.Path, r.Old.Version, filepath.Join(o.Dir, r.New.Path), ""); err != nil {
return fail(err)
}
}
}
raw, err = mf.Format()
if err != nil {
return fail(err)
}
modPath := filepath.Join(temp, "go.mod")
if err = os.WriteFile(modPath, raw, 0600); err != nil {
return fail(err)
}
if sums, e := os.ReadFile(filepath.Join(o.Dir, "go.sum")); e == nil {
if err = os.WriteFile(filepath.Join(temp, "go.sum"), sums, 0600); err != nil {
return fail(err)
}
} else if !errors.Is(e, os.ErrNotExist) {
return fail(e)
}
overlay := map[string]string{}
// Hide stale owned Go files, which may otherwise redeclare a renamed DTO.
var previous manifest
if data, e := os.ReadFile(filepath.Join(o.Dir, manifestPath)); e == nil {
if err = json.Unmarshal(data, &previous); err != nil {
return fail(err)
}
for name := range previous.Generated {
if strings.HasSuffix(name, ".go") {
if !ownedPath(name) {
return fail(fmt.Errorf("invalid owned path %s", name))
}
if err = safePath(o.Dir, name); err != nil {
return fail(err)
}
if _, ok := generated[name]; !ok {
overlay[filepath.Join(o.Dir, filepath.FromSlash(name))] = ""
}
}
}
} else if !errors.Is(e, os.ErrNotExist) {
return fail(e)
}
for i, files := range []map[string][]byte{generated, seeds} {
names := make([]string, 0, len(files))
for name := range files {
if strings.HasSuffix(name, ".go") {
names = append(names, name)
}
}
sort.Strings(names)
for _, name := range names {
if err = safePath(o.Dir, name); err != nil {
return fail(err)
}
path := filepath.Join(o.Dir, filepath.FromSlash(name))
if i == 1 {
if _, e := os.Stat(path); e == nil {
continue
} else if !errors.Is(e, os.ErrNotExist) {
return fail(e)
}
}
backing := filepath.Join(temp, fmt.Sprintf("overlay%d.go", len(overlay)))
if err = os.WriteFile(backing, files[name], 0600); err != nil {
return fail(err)
}
overlay[path] = backing
}
}
raw, err = json.Marshal(map[string]any{"Replace": overlay})
if err != nil {
return fail(err)
}
overlayPath := filepath.Join(temp, "overlay.json")
if err = os.WriteFile(overlayPath, raw, 0600); err != nil {
return fail(err)
}
args := []string{"list", "-deps", "-export", "-json", "-mod=readonly", "-modfile", modPath, "-overlay", overlayPath}
// fmt is used for fallible conversions even if neither struct imports it.
args = append(args, "fmt")
args = append(args, refs...)
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "go", args...)
cmd.Dir = o.Dir
cmd.Env = append(os.Environ(), "GOWORK=off")
cmd.WaitDelay = 2 * time.Second
stdout := &limitedCatalogBuffer{limit: 32 << 20}
stderr := &limitedCatalogBuffer{limit: 64 << 10}
cmd.Stdout = stdout
cmd.Stderr = stderr
if err = cmd.Run(); err != nil {
if ctx.Err() != nil {
return fail(fmt.Errorf("mapping type loading: %w", ctx.Err()))
}
// Go download errors may contain authenticated remote URLs. Keep arbitrary
// command output out of diagnostics; exact mapping errors follow type loading.
return fail(fmt.Errorf("mapping type loading failed (%v); run go list -deps -export for the configured source/target/converter packages; check dependencies, build tags and import cycles", err))
}
exports := map[string]string{}
dec := json.NewDecoder(bytes.NewReader(stdout.Bytes()))
for {
var p struct {
ImportPath, Export string
Error *struct{ Err string }
}
if err = dec.Decode(&p); err == io.EOF {
break
} else if err != nil {
return fail(err)
}
if p.Error != nil {
return fail(fmt.Errorf("mapping package %s could not be compiled", p.ImportPath))
}
if p.Export != "" {
exports[p.ImportPath] = p.Export
}
}
imp := importer.ForCompiler(gotoken.NewFileSet(), "gc", func(path string) (io.ReadCloser, error) {
file, ok := exports[path]
if !ok {
return nil, fmt.Errorf("no export data for mapping dependency %s", path)
}
return os.Open(file)
})
return imp, cleanup, nil
}

View File

@@ -1,250 +0,0 @@
package apigen
import (
"context"
"encoding/json"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
)
func nullableMappings() []Mapping {
pkg := "example.com/nullable/internal/"
ref := func(p, name string) GoTypeRef { return GoTypeRef{pkg + p, name} }
child := Mapping{Name: "ToChild", Source: ref("types", "Child"), Target: ref("domain", "Child"), Fields: map[string]MappingField{
"Name": {Source: "Name", Convert: &GoFunctionRef{pkg + "convert", "Name"}},
}}
parent := Mapping{Name: "ToParent", PointerResult: true, Source: ref("types", "Parent"), Target: ref("domain", "Parent"), Fields: map[string]MappingField{
"Child": {Source: "Child", Mapping: "ToChild"}, "Children": {Source: "Children", Mapping: "ToChild", Elements: true},
}}
defs := []Mapping{child, parent,
{Name: "ToWireChild", Source: ref("domain", "Child"), Target: ref("types", "Child")},
{Name: "ToWire", PointerResult: true, Source: ref("domain", "Parent"), Target: ref("types", "Parent"), Fields: map[string]MappingField{
"Child": {Source: "Child", Mapping: "ToWireChild"}, "Children": {Source: "Children", Mapping: "ToWireChild", Elements: true},
}},
{Name: "ToChildDTO", Source: ref("domain", "Child"), Target: ref("domain", "ChildDTO"), Fields: map[string]MappingField{"Name": {Source: "Name", Convert: &GoFunctionRef{pkg + "convert", "Name"}}}},
{Name: "Lift", Source: ref("domain", "ValueHolder"), Target: ref("domain", "PointerHolder"), Fields: map[string]MappingField{"Child": {Source: "Child", Mapping: "ToChildDTO"}}},
{Name: "Batch", Source: ref("domain", "InputBatch"), Target: ref("domain", "OutputBatch"), Fields: map[string]MappingField{"Fixed": {Source: "Fixed", Mapping: "ToChildDTO", Elements: true}}},
{Name: "ShallowCopy", Source: ref("domain", "Parent"), Target: ref("domain", "Parent")},
{Name: "RequirePointer", Source: ref("types", "Parent"), Target: ref("domain", "Parent"), Fields: map[string]MappingField{
"Child": {Source: "Child", Mapping: "ToChild", Nil: "error"}, "Children": {Source: "Children", Mapping: "ToChild", Elements: true},
}},
}
for _, p := range []struct{ name, policy string }{{"RequiredZero", "zero"}, {"RequiredError", "error"}} {
defs = append(defs, Mapping{Name: p.name, Source: ref("types", "Parent"), Target: ref("domain", "Required"), Fields: map[string]MappingField{
"Child": {Source: "Child", Mapping: "ToChild", Nil: p.policy},
}, IgnoreSource: map[string]string{"Children": "this explicit projection only includes the child"}})
}
for _, p := range []struct{ name, policy string }{{"ScalarZero", "zero"}, {"ScalarError", "error"}} {
m := Mapping{Name: p.name, Source: ref("domain", "ScalarInput"), Target: ref("domain", "ScalarOutput"), Fields: map[string]MappingField{}}
for _, name := range []string{"Text", "Count", "Enabled"} {
m.Fields[name] = MappingField{Source: name, Nil: p.policy}
}
defs = append(defs, m)
}
return defs
}
func writeMappingFile(t *testing.T, root, name string, defs []Mapping) {
t.Helper()
raw, err := json.MarshalIndent(struct {
Mappings []Mapping `json:"mappings"`
}{defs}, "", " ")
if err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(root, name), string(raw)+"\n")
}
func TestNullableMappingConsumer(t *testing.T) {
if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") != "1" {
t.Skip("set GO_OMNI_SCAFFOLD_INTEGRATION=1")
}
root := t.TempDir()
dir := filepath.Join(root, "app")
o := Options{API: filepath.Join(frameworkRoot(t), "cmd/go-omni/apigen/testdata/nullable/service.api"), Dir: dir, Module: "example.com/nullable", FrameworkDir: frameworkRoot(t)}
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
for path, fixture := range map[string]string{"internal/domain/model.go": "domain.go", "internal/convert/value.go": "convert.go"} {
writeFile(t, filepath.Join(dir, path), readFile(t, "testdata/nullable/"+fixture))
}
runCompositionGo(t, dir, "mod", "tidy")
defs := nullableMappings()
// References go across files, including forward references to ToChild.
writeMappingFile(t, dir, "mappings/parents.json", defs[1:])
writeMappingFile(t, dir, "mappings/children.json", defs[:1])
cfg := GeneratorConfig{Repository: "none", Middleware: "factories", MappingFiles: []string{"mappings/parents.json", "mappings/children.json"}}
writeGeneratorConfig(t, dir, cfg)
o.API = filepath.Join(dir, "api/service.api")
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(dir, "internal/logic/probe/roundTrip.go"), `package probe
import("context";"example.com/nullable/internal/types";"example.com/nullable/internal/mapping")
func(s *Logic)RoundTrip(ctx context.Context,req *types.Parent)(*types.Parent,error){
value,err:=mapping.ToParent(*req);if err!=nil{return nil,err}
return mapping.ToWirePtr(value)
}
`)
writeFile(t, filepath.Join(dir, "consumer_test.go"), readFile(t, "testdata/nullable/consumer_test.go"))
before := snapshot(t, dir)
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("regeneration changed business/config files")
}
o.Check = true
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("check wrote files")
}
runCompositionGo(t, dir, "test", "-race", "-count=1", "./...")
runCompositionGo(t, dir, "build", ".")
// Workspace preflight and check use the same split configuration loader.
workspace := filepath.Join(root, "omni.workspace.json")
writeFile(t, workspace, `{"services":[{"dir":"app"}]}`)
for _, command := range []string{"format", "generate", "check"} {
if err := nullableWorkspace(t, workspace, command); err != nil {
t.Fatal(command, err)
}
}
// All invalid configurations fail before either writing or previewing output.
for _, tc := range []struct {
name, want string
edit func([]Mapping) []Mapping
}{
{"missingNilPolicy", "requires explicit nil", func(d []Mapping) []Mapping {
d[9].Fields["Child"] = MappingField{Source: "Child", Mapping: "ToChild"}
return d
}},
{"preserveValue", "requires explicit nil", func(d []Mapping) []Mapping {
d[9].Fields["Child"] = MappingField{Source: "Child", Mapping: "ToChild", Nil: "preserve"}
return d
}},
{"zeroPointer", "pointer target", func(d []Mapping) []Mapping {
d[1].Fields["Child"] = MappingField{Source: "Child", Mapping: "ToChild", Nil: "zero"}
return d
}},
{"scalarMismatch", "identical *T to T", func(d []Mapping) []Mapping {
d[11].Fields["Text"] = MappingField{Source: "Count", Nil: "zero"}
return d
}},
{"valueNil", "requires a pointer source", func(d []Mapping) []Mapping {
d[5].Fields["Child"] = MappingField{Source: "Child", Mapping: "ToChildDTO", Nil: "zero"}
return d
}},
{"converterNil", "cannot be combined", func(d []Mapping) []Mapping {
f := d[0].Fields["Name"]
f.Nil = "error"
d[0].Fields["Name"] = f
return d
}},
{"pointerNameCollision", "must be unique", func(d []Mapping) []Mapping { m := d[0]; m.Name = "ToParentPtr"; return append(d, m) }},
{"mappingCycle", "cycle", func(d []Mapping) []Mapping {
d[0].Fields["Name"] = MappingField{Source: "Name", Mapping: "ToParent"}
return d
}},
{"unknownMapping", "unknown mapping", func(d []Mapping) []Mapping {
d[1].Fields["Child"] = MappingField{Source: "Child", Mapping: "Missing"}
return d
}},
{"unhandledField", "unmapped source", func(d []Mapping) []Mapping { d[9].IgnoreSource = nil; return d }},
} {
t.Run(tc.name, func(t *testing.T) {
changed := tc.edit(nullableMappings())
writeMappingFile(t, dir, "mappings/parents.json", changed[1:])
writeMappingFile(t, dir, "mappings/children.json", changed[:1])
defer writeMappingFile(t, dir, "mappings/parents.json", defs[1:])
defer writeMappingFile(t, dir, "mappings/children.json", defs[:1])
before := snapshot(t, dir)
for _, mode := range []string{"check", "preview", "generate"} {
current := o
current.Check = mode == "check"
current.DryRun = mode == "preview"
err := generateManagedTestProject(t, current)
if err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatal(mode, err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("invalid mapping wrote files")
}
}
})
}
// An edit inside a child file is picked up by --check; the config is never rewritten.
changed := nullableMappings()
changed[0].PointerResult = true
writeMappingFile(t, dir, "mappings/children.json", changed[:1])
before = snapshot(t, dir)
if err := generateManagedTestProject(t, o); err == nil {
t.Fatal("check missed child mapping drift")
}
if err := nullableWorkspace(t, workspace, "check"); err == nil {
t.Fatal("workspace missed child mapping drift")
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("drift check wrote files")
}
preview := o
preview.Check = false
preview.DryRun = true
if err := generateManagedTestProject(t, preview); err != nil {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("preview wrote files")
}
if err := nullableWorkspace(t, workspace, "generate"); err != nil {
t.Fatal(err)
}
if err := nullableWorkspace(t, workspace, "check"); err != nil {
t.Fatal(err)
}
if os.Getenv("GO_OMNI_RELEASE_CLI") != "" {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "go", "list", "-m", "-json", frameworkModule)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GOWORK=off")
data, err := cmd.CombinedOutput()
if err != nil {
t.Fatal(err, string(data))
}
var dependency struct {
Version, Sum string
Replace any
}
if err = json.Unmarshal(data, &dependency); err != nil {
t.Fatal(err)
}
if dependency.Version != os.Getenv("GO_OMNI_RELEASE_VERSION") || dependency.Replace != nil || dependency.Sum == "" {
t.Fatal("not a fixed remote consumer")
}
t.Logf("remote framework %s checksum %s; no local replace", dependency.Version, dependency.Sum)
}
}
func nullableWorkspace(t *testing.T, path, command string) error {
t.Helper()
cli := os.Getenv("GO_OMNI_RELEASE_CLI")
if cli == "" {
return Workspace(path, command, io.Discard)
}
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, cli, "workspace", command, "--file", path)
cmd.Dir = filepath.Dir(path)
if data, err := cmd.CombinedOutput(); err != nil {
return fmt.Errorf("released workspace CLI: %w\n%s", err, data)
}
return nil
}

View File

@@ -1,97 +0,0 @@
package apigen
import (
"fmt"
"go/types"
)
func mappingOrigin(m Mapping) string {
if m.origin != "" {
return m.origin
}
return generatorConfigFile
}
// Pointer adaptation is only selected by an explicit nested mapping. Ordinary
// same-type pointer assignments retain their existing reference semantics.
func (r *mappingRenderer) nestedExpression(rule MappingField, from, to types.Type, source string) (string, bool, error) {
m := r.types[rule.Mapping]
fromValue, toValue := from, to
src, sourcePointer := from.Underlying().(*types.Pointer)
dst, targetPointer := to.Underlying().(*types.Pointer)
if sourcePointer {
fromValue = src.Elem()
}
if targetPointer {
toValue = dst.Elem()
}
if !types.Identical(fromValue, m.source) || !types.Identical(toValue, m.target) {
return "", false, fmt.Errorf("nested mapping %s has incompatible types", rule.Mapping)
}
if !sourcePointer && rule.Nil != "" {
return "", false, fmt.Errorf("nil policy requires a pointer source")
}
if !sourcePointer && !targetPointer {
return rule.Mapping + "(" + source + ")", true, nil
}
policy := rule.Nil
if sourcePointer {
if targetPointer {
if policy == "" {
policy = "preserve"
}
if policy != "preserve" && policy != "error" {
return "", false, fmt.Errorf("pointer target requires nil preserve or error")
}
} else if policy != "zero" && policy != "error" {
return "", false, fmt.Errorf("pointer to value mapping requires explicit nil policy zero or error")
}
}
// Keep a typed local input so the source is evaluated once. The surrounding
// renderer adds the mapping/field/index error path and returns a zero parent.
body := "var zero " + r.typeName(to) + "\n"
input := "input"
if sourcePointer {
nilResult, err := r.nilResult(policy)
if err != nil {
return "", false, err
}
body += "if input==nil {" + nilResult + "}\n"
input = "*input"
}
body += "value,err:=" + rule.Mapping + "(" + input + ")\nif err!=nil{return zero,err}\n"
if targetPointer {
body += "return &value,nil\n"
} else {
body += "return value,nil\n"
}
return "func(input " + r.typeName(from) + ")(" + r.typeName(to) + ",error){\n" + body + "}(" + source + ")", true, nil
}
// A nil policy without a converter/mapping is deliberately limited to *T -> T.
// It never casts numbers, changes named types, or invents business defaults.
func (r *mappingRenderer) dereferenceExpression(rule MappingField, from, to types.Type, source string) (string, bool, error) {
pointer, ok := from.Underlying().(*types.Pointer)
if !ok || !types.Identical(pointer.Elem(), to) {
return "", false, fmt.Errorf("nil policy without a mapping requires identical *T to T types")
}
if rule.Nil != "zero" && rule.Nil != "error" {
return "", false, fmt.Errorf("pointer to value conversion requires nil zero or error")
}
nilResult, err := r.nilResult(rule.Nil)
if err != nil {
return "", false, err
}
return "func(input " + r.typeName(from) + ")(" + r.typeName(to) + ",error){\nvar zero " + r.typeName(to) + "\nif input==nil{" + nilResult + "}\nreturn *input,nil\n}(" + source + ")", true, nil
}
func (r *mappingRenderer) nilResult(policy string) (string, error) {
if policy != "error" {
return "return zero,nil", nil
}
pkg, err := r.importer.Import("fmt")
if err != nil {
return "", err
}
return "return zero," + r.qualifier(pkg) + ".Errorf(\"nil source is not allowed\")", nil
}

View File

@@ -1,266 +0,0 @@
package apigen
import (
"context"
"encoding/json"
"fmt"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
"time"
)
func compositionConfig() GeneratorConfig {
pkg := "example.com/composition/internal/"
cfg := GeneratorConfig{Repository: "none", Middleware: "factories"}
for _, name := range []string{"Enterprise", "Tenant", "Admin"} {
fields := map[string]MappingField{
"EmployeeName": {Source: "Name", Convert: &GoFunctionRef{pkg + "convert", "Name"}},
"Salary": {Source: "Salary", Convert: &GoFunctionRef{pkg + "convert", "Money"}},
"Enabled": {Source: "Enabled", Convert: &GoFunctionRef{pkg + "convert", "Bool"}},
}
m := Mapping{Name: "From" + name, Source: GoTypeRef{pkg + "types", name + "Request"}, Target: GoTypeRef{pkg + "domain", "Employee"}, Fields: fields}
if name != "Enterprise" {
m.IgnoreSource = map[string]string{"CompanyID": "Authorized separately by Logic; never copy path scope into body"}
}
cfg.Mappings = append(cfg.Mappings, m)
}
cfg.Mappings = append(cfg.Mappings, Mapping{Name: "ToResponse", Source: GoTypeRef{pkg + "domain", "Result"}, Target: GoTypeRef{pkg + "types", "EmployeeResponse"}, Fields: map[string]MappingField{"ID": {Source: "ID", Convert: &GoFunctionRef{pkg + "convert", "ID"}}}})
cfg.Mappings = append(cfg.Mappings, Mapping{Name: "ToRow", Source: GoTypeRef{pkg + "domain", "Row"}, Target: GoTypeRef{pkg + "domain", "RowDTO"}})
cfg.Mappings = append(cfg.Mappings, Mapping{Name: "ToBatch", Source: GoTypeRef{pkg + "domain", "Batch"}, Target: GoTypeRef{pkg + "domain", "BatchDTO"}, Fields: map[string]MappingField{
"Rows": {Source: "Rows", Mapping: "ToRow", Elements: true}, "Fixed": {Source: "Fixed", Mapping: "ToRow", Elements: true},
}})
return cfg
}
func writeGeneratorConfig(t *testing.T, dir string, cfg GeneratorConfig) {
t.Helper()
raw, err := json.MarshalIndent(cfg, "", " ")
if err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(dir, generatorConfigFile), string(raw)+"\n")
}
func runCompositionGo(t *testing.T, dir string, args ...string) {
t.Helper()
ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "go", args...)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GOWORK=off")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatalf("go %v: %v\n%s", args, err, out)
}
}
func TestCompositionMappingConsumer(t *testing.T) {
if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") != "1" {
t.Skip("set GO_OMNI_SCAFFOLD_INTEGRATION=1")
}
dir := filepath.Join(t.TempDir(), "app")
o := Options{API: filepath.Join(frameworkRoot(t), "cmd/go-omni/apigen/testdata/composition/service.api"), Dir: dir, Module: "example.com/composition", FrameworkDir: frameworkRoot(t)}
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
fixtures := map[string]string{
"internal/amount/money.go": "amount.go", "internal/domain/employee.go": "domain.go", "internal/convert/fields.go": "converters.go",
"internal/middleware/dependencies.go": "dependencies.go", "internal/middleware/accessFactory.go": "accessFactory.go", "internal/svc/client.go": "client.go", "consumer_test.go": "consumer_test.go",
}
for path, fixture := range fixtures {
writeFile(t, filepath.Join(dir, path), readFile(t, "testdata/composition/"+fixture))
}
svcFile := filepath.Join(dir, "internal/svc/serviceContext.go")
source := strings.Replace(readFile(t, svcFile), "type ServiceContext struct {", "type ServiceContext struct {\n client *Client", 1)
source = strings.Replace(source, "return nil", "return c.initClient(ctx)", 1)
writeFile(t, svcFile, source)
configFile := filepath.Join(dir, "internal/config/config.go")
writeFile(t, configFile, strings.Replace(readFile(t, configFile), "type Config struct {", "type Config struct {\n FactoryMode string `json:\"factoryMode\"`", 1))
// Resolve dependencies before opting into compile-time mapping. The first
// generation creates DTOs; the second sees current DTOs through an overlay.
// Logic does not yet import the generated mapping package.
testSource := readFile(t, filepath.Join(dir, "consumer_test.go"))
if err := os.Remove(filepath.Join(dir, "consumer_test.go")); err != nil {
t.Fatal(err)
}
runCompositionGo(t, dir, "mod", "tidy")
cfg := compositionConfig()
writeGeneratorConfig(t, dir, cfg)
o.API = filepath.Join(dir, "api/service.api")
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
writeFile(t, filepath.Join(dir, "consumer_test.go"), testSource)
for _, group := range []string{"Enterprise", "Tenant", "Admin"} {
text := fmt.Sprintf(`package %s
import("context";"example.com/composition/internal/types";"example.com/composition/internal/mapping";"example.com/composition/internal/domain")
func(s *Logic)Save(ctx context.Context,req *types.%sRequest)(*types.EmployeeResponse,error){
input,err:=mapping.From%s(*req);if err!=nil{return nil,err}
result,err:=mapping.ToResponse(domain.Result{Employee:input,ID:9007199254740993});if err!=nil{return nil,err};return &result,nil
}
`, strings.ToLower(group), group, group)
writeFile(t, filepath.Join(dir, "internal/logic", strings.ToLower(group), "save.go"), text)
}
before := snapshot(t, dir)
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("regeneration changed project")
}
o.Check = true
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
if !sameFiles(before, snapshot(t, dir)) {
t.Fatal("check changed project")
}
runCompositionGo(t, dir, "test", "-race", "-count=1", "./...")
runCompositionGo(t, dir, "build", ".")
generated := readFile(t, filepath.Join(dir, "internal/mapping/mappingsGen.go"))
if strings.Contains(generated, "encoding/json") || strings.Contains(generated, "reflect") || !strings.Contains(generated, "result.EmployeeName = value") {
t.Fatal("mapping does not emit typed assignments")
}
// Adding a source or target field is detected before any files are written.
for _, tc := range []struct{ name, path, before, after, want string }{
{"source", o.API, "Name *string", "NewField *string `json:\"newField,optional\"`\n Name *string", "unmapped source field NewField"},
{"target", filepath.Join(dir, "internal/domain/employee.go"), "EmployeeName string", "Unmapped string\n EmployeeName string", "target Unmapped has no source"},
} {
t.Run(tc.name, func(t *testing.T) {
original := readFile(t, tc.path)
writeFile(t, tc.path, strings.Replace(original, tc.before, tc.after, 1))
defer writeFile(t, tc.path, original)
snap := snapshot(t, dir)
if err := generateManagedTestProject(t, o); err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatal(err)
}
if !sameFiles(snap, snapshot(t, dir)) {
t.Fatal("failed check wrote files")
}
o.Check = false
defer func() { o.Check = true }()
if err := generateManagedTestProject(t, o); err == nil {
t.Fatal("generate accepted missing mapping")
}
if !sameFiles(snap, snapshot(t, dir)) {
t.Fatal("failed generate wrote files")
}
})
}
for _, tc := range []struct {
name string
edit func(*GeneratorConfig)
want string
}{
{"pointerPolicy", func(c *GeneratorConfig) { delete(c.Mappings[0].Fields, "Salary") }, "provide an explicit converter"},
{"unknownSource", func(c *GeneratorConfig) { c.Mappings[0].Fields["Salary"] = MappingField{Source: "Missing"} }, "has no source"},
{"wrongConverter", func(c *GeneratorConfig) {
f := c.Mappings[0].Fields["Salary"]
f.Convert.Name = "ID"
c.Mappings[0].Fields["Salary"] = f
}, "signature does not match"},
{"unhandledScope", func(c *GeneratorConfig) { c.Mappings[1].IgnoreSource = nil }, "unmapped source field CompanyID"},
{"badIgnore", func(c *GeneratorConfig) { c.Mappings[1].IgnoreSource["Missing"] = "stale rule" }, "ignoreSource Missing"},
{"arrayLength", func(c *GeneratorConfig) {
f := c.Mappings[5].Fields["Fixed"]
f.Source = "Rows"
c.Mappings[5].Fields["Fixed"] = f
}, "array lengths must match"},
} {
t.Run(tc.name, func(t *testing.T) {
changed := compositionConfig()
tc.edit(&changed)
writeGeneratorConfig(t, dir, changed)
defer writeGeneratorConfig(t, dir, cfg)
snap := snapshot(t, dir)
if err := generateManagedTestProject(t, o); err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatal(err)
}
if !sameFiles(snap, snapshot(t, dir)) {
t.Fatal("invalid rule wrote files")
}
})
}
// All diagnostics exercise current DTOs, not stale exported generated types.
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
// Exercise the repository switch with the same actual CLI as the mapper.
if _, err := os.Stat(filepath.Join(dir, "internal/repository")); !os.IsNotExist(err) {
t.Fatalf("default repository skeleton: %v", err)
}
o.Check = false
cfg.Repository = "group"
writeGeneratorConfig(t, dir, cfg)
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
repo := filepath.Join(dir, "internal/repository/enterprise/repository.go")
writeFile(t, repo, readFile(t, repo)+"\n// Project-owned persistence.\n")
cfg.Repository = "none"
writeGeneratorConfig(t, dir, cfg)
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
if !strings.Contains(readFile(t, repo), "Project-owned persistence.") {
t.Fatal("disabling skeleton overwrote business code")
}
o.Check = true
if err := generateManagedTestProject(t, o); err != nil {
t.Fatal(err)
}
if os.Getenv("GO_OMNI_RELEASE_CLI") != "" {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
cmd := exec.CommandContext(ctx, "go", "list", "-m", "-json", frameworkModule)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "GOWORK=off")
data, err := cmd.CombinedOutput()
if err != nil {
t.Fatal(err, string(data))
}
var dependency struct {
Version, Sum string
Replace any
}
if err = json.Unmarshal(data, &dependency); err != nil {
t.Fatal(err)
}
if dependency.Version != os.Getenv("GO_OMNI_RELEASE_VERSION") || dependency.Replace != nil || dependency.Sum == "" {
t.Fatal("not a fixed remote consumer")
}
t.Logf("remote framework %s checksum %s; no local replace", dependency.Version, dependency.Sum)
}
}
func TestMappingConfigurationRejection(t *testing.T) {
cases := []struct {
name string
edit func(*GeneratorConfig)
want string
}{
{"cycle", func(c *GeneratorConfig) {
c.Mappings[4].Fields = map[string]MappingField{"Value": {Source: "Value", Mapping: "ToRow"}}
}, "cycle"},
{"unknown", func(c *GeneratorConfig) {
c.Mappings[5].Fields["Rows"] = MappingField{Source: "Rows", Mapping: "Missing", Elements: true}
}, "unknown mapping"},
{"conflict", func(c *GeneratorConfig) {
f := c.Mappings[0].Fields["Salary"]
f.Mapping = "ToRow"
c.Mappings[0].Fields["Salary"] = f
}, "conflicting converter"},
{"internal", func(c *GeneratorConfig) { c.Mappings[0].Source.Package = "example.net/private/internal/types" }, "invalid Go type"},
{"outputCycle", func(c *GeneratorConfig) { c.Mappings[0].Source.Package = "example.com/composition/internal/mapping" }, "invalid Go type"},
{"duplicates", func(c *GeneratorConfig) { c.Mappings[1].Name = c.Mappings[0].Name }, "must be unique"},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
cfg := compositionConfig()
tc.edit(&cfg)
if _, err := validateMappings(cfg.Mappings, "example.com/composition"); err == nil || !strings.Contains(err.Error(), tc.want) {
t.Fatal(err)
}
})
}
}

View File

@@ -63,18 +63,23 @@ func TestAPIInfrastructureConsumers(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
test := `package config test := `package config
import("testing";"os") import("testing";"os";"strings")
func TestConfiguration(t *testing.T){ func TestConfiguration(t *testing.T){
t.Setenv("GO_OMNI_DB_DSN","test:test@tcp(127.0.0.1:1)/test") t.Setenv("GO_OMNI_DB_DATABASE","test");t.Setenv("GO_OMNI_DB_USER","test");t.Setenv("GO_OMNI_DB_PASSWORD","test")
t.Setenv("GO_OMNI_STORAGE_SIGNING_KEY","01234567890123456789012345678901") t.Setenv("GO_OMNI_STORAGE_SIGNING_KEY","01234567890123456789012345678901")
t.Setenv("GO_OMNI_STORAGE_ACCESS_KEY","test");t.Setenv("GO_OMNI_STORAGE_SECRET_KEY","test") t.Setenv("GO_OMNI_STORAGE_ACCESS_KEY","test");t.Setenv("GO_OMNI_STORAGE_SECRET_KEY","test")
t.Setenv("GO_OMNI_STORAGE_BUCKET","test");t.Setenv("GO_OMNI_STORAGE_ENDPOINT","ENDPOINT") t.Setenv("GO_OMNI_STORAGE_BUCKET","test");t.Setenv("GO_OMNI_STORAGE_ENDPOINT","ENDPOINT")
t.Setenv("GO_OMNI_STORAGE_ACCOUNT_ID","01234567890123456789012345678901") t.Setenv("GO_OMNI_STORAGE_ACCOUNT_ID","01234567890123456789012345678901")
cfg,err:=Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)} cfg,err:=Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
if cfg.Server.Log.File==""{t.Fatal("file logger absent")} if cfg.Server.Log.File==""{t.Fatal("file logger absent")}
t.Setenv("GO_OMNI_LOG_OUTPUT","stdout");cfg,err=Load("");if err!=nil||cfg.Server.Log.File!=""{t.Fatalf("override: %v",err)} t.Setenv("GO_OMNI_LOG_OUTPUT","stdout");t.Setenv("GO_OMNI_SERVER_LOG_FILE","./implicit.log")
t.Setenv("GO_OMNI_SERVER_LOG_FILE","./explicit.log");cfg,err=Load("");if err!=nil||cfg.Server.Log.File!="./explicit.log"{t.Fatalf("canonical log file: %v",err)} cfg,err=Load("../../etc/config.yaml");if err!=nil||cfg.Server.Log.File!=Default().Server.Log.File{t.Fatalf("implicit environment override: %v",err)}
t.Setenv("GO_OMNI_SERVER_LOG_OUTPUT","stderr");cfg,err=Load("");if err!=nil||cfg.Server.Log.Output!="stderr"{t.Fatalf("canonical output: %v",err)} raw,err:=os.ReadFile("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
// Add explicit fields while keeping storage and DB requirements.
raw=[]byte(strings.Replace(string(raw),"server:\n","server:\n log:\n output: '${OUTPUT:-stdout}'\n file: ''\n rotation:\n enabled: false\n",1))
path:="explicit.yaml";if err=os.WriteFile(path,raw,0600);err!=nil{t.Fatal(err)};defer os.Remove(path)
cfg,err=Load(path);if err!=nil||cfg.Server.Log.Output!="stdout"||cfg.Server.Log.File!=""||cfg.Server.Log.Rotation.Enabled{t.Fatalf("explicit file override: %v",err)}
_=os.Getenv("unused") _=os.Getenv("unused")
}` }`
endpoint := "" endpoint := ""

View File

@@ -19,8 +19,10 @@ import (
"MODULE/internal/svc" "MODULE/internal/svc"
) )
func main(){ func main(){
path:=flag.String("config","etc/config.yaml","configuration file; empty uses defaults and environment") path:=flag.String("config","etc/config.yaml","configuration file; empty uses only Default()")
check:=flag.Bool("check-config",false,"validate configuration and exit without starting dependencies")
flag.Parse() flag.Parse()
if *check {if _,err:=config.Load(*path);err!=nil{log.Fatal(err)};return}
ctx,stop:=signal.NotifyContext(context.Background(),os.Interrupt,syscall.SIGTERM) ctx,stop:=signal.NotifyContext(context.Background(),os.Interrupt,syscall.SIGTERM)
defer stop() defer stop()
if err:=run(ctx,*path);err!=nil{log.Fatal(err)} if err:=run(ctx,*path);err!=nil{log.Fatal(err)}
@@ -56,10 +58,18 @@ factories for new projects. Named middleware factories run after svc.Init;
implement Dependencies accessors on ServiceContext and New<Name>(deps) once. implement Dependencies accessors on ServiceContext and New<Name>(deps) once.
Unimplemented/nil factories fail startup and clean registered dependencies. Unimplemented/nil factories fail startup and clean registered dependencies.
Existing projects retain middleware: handlers until explicitly migrated. Existing projects retain middleware: handlers until explicitly migrated.
Optional mappings generate typed assignments in internal/mapping/mappingsGen.go; Ordinary Logic uses the generated Request/Response types directly. Where an
configure source/target types and explicit conversion rules after resolving module independent business type is needed, use ordinary Go or the optional copyx package.
dependencies. Missing fields fail generation and --check. See GENERATION.md in Configuration loading is generator-managed in internal/config/loadGen.go. Extend fields/defaults/Validate in config.go; additional decoders belong in loader.go/NewLoader. JSON/YAML/XML use explicit ${NAME} / ${NAME:-fallback} references; fixed values are never overridden by inferred environment names. Use -check-config to validate without opening dependencies. Framework validation
the framework repository for composition, null/reference semantics and migration. identifies fields such as server.log.level and database.port. server.log.timeZone
optionally selects a per-logger IANA zone. Optional server.cors is loaded and installed
automatically before routing/authentication; disabled by default. Configure allowedOrigins,
allowedMethods and allowedHeaders in the selected file; no main/handler edits.
See the framework CONFIG.md and cors/README.md.
There are no mappings/mappingFiles settings or generated mapping files in v0.19.
See COPY.md in the framework repository for copy policies and migration steps,
and GENERATION.md for field composition and middleware factories.
## Run ## Run
@@ -74,7 +84,7 @@ Configure server.endpoints to disable or move each built-in route. Health
responseMode is app by default (project format), or raw for plain health data. responseMode is app by default (project format), or raw for plain health data.
Probe HTTP statuses remain 200/503 even when business errors use HTTP 200. Probe HTTP statuses remain 200/503 even when business errors use HTTP 200.
OpenAPI remains an unwrapped document; disabled endpoints reserve no paths. OpenAPI remains an unwrapped document; disabled endpoints reserve no paths.
Set GO_OMNI_TRANSPORT=native to select the native transport without changing services. The default file explicitly references GO_OMNI_TRANSPORT; set it to native or edit server.transport.
The greeting starter works immediately; other new operation stubs return 501 until The greeting starter works immediately; other new operation stubs return 501 until
their business logic is implemented. Authentication hooks also fail closed with 501. their business logic is implemented. Authentication hooks also fail closed with 501.
@@ -136,7 +146,7 @@ Full reference: https://git.yuntuo.hk/k2safe/go-omni/src/branch/main/DEVELOPMENT
api/ # editable contract and imported types api/ # editable contract and imported types
main.go # GENERATED startup, lifecycle and route registration main.go # GENERATED startup, lifecycle and route registration
etc/config.yaml # lowerCamelCase configuration etc/config.yaml # lowerCamelCase configuration
internal/config/ # defaults -> file -> environment internal/config/ # defaults -> file references -> typed binding -> validation
internal/svc/ # dependency initialization and cleanup internal/svc/ # dependency initialization and cleanup
internal/types/*Gen.go # DTOs split by source API, one types package internal/types/*Gen.go # DTOs split by source API, one types package
internal/svc/runtimeGen.go # GENERATED lifecycle and infrastructure startup internal/svc/runtimeGen.go # GENERATED lifecycle and infrastructure startup
@@ -145,8 +155,7 @@ internal/handler/<group>/ # generated binding and Logic adapters
internal/logic/<group>/ # developer-owned business operations internal/logic/<group>/ # developer-owned business operations
internal/repository/<group>/ # optional; repository: group creates editable seeds internal/repository/<group>/ # optional; repository: group creates editable seeds
internal/middleware/ # typed dependency interface, factories and registry internal/middleware/ # typed dependency interface, factories and registry
omni.gen.json # developer-owned generator options and mappings omni.gen.json # developer-owned repository/middleware options
internal/mapping/*Gen.go # optional compile-checked struct mappings
docs/openapi.json # generated OpenAPI 3.0.3 docs/openapi.json # generated OpenAPI 3.0.3
.goOmni/apiManifest.json # ownership and content hashes .goOmni/apiManifest.json # ownership and content hashes
`+"```"+` `+"```"+`
@@ -238,13 +247,16 @@ func infrastructureREADME(o Options) string {
Selected database: %s; storage: %s; logging: %s/%s. Selected database: %s; storage: %s; logging: %s/%s.
MySQL uses database/sql through deps.DB, including InTx and a readiness probe. MySQL uses database/sql through deps.DB, including InTx and a readiness probe.
Use structured database/mysql configuration and GO_OMNI_DB_PASSWORD. Other connection/pool/TLS settings live in etc/config.yaml and GO_OMNI_DB_*. Use structured database/mysql configuration. The generated etc/config.yaml explicitly
references GO_OMNI_DB_PASSWORD and other required connection values; add fixed
values or explicit references for pool/TLS fields. No implicit environment overlay.
No tables are invented or migrated automatically from request/response DTOs. No tables are invented or migrated automatically from request/response DTOs.
The none/memory selection opens no database; implement repositories per business. The none/memory selection opens no database; implement repositories per business.
With storage enabled, services receive deps.Files, Uploads, Downloads and With storage enabled, services receive deps.Files, Uploads, Downloads and
UploadLinks capability interfaces. Local signed GET/HEAD/PUT uses /storage/object; UploadLinks capability interfaces. Local signed GET/HEAD/PUT uses /storage/object;
set GO_OMNI_STORAGE_SIGNING_KEY (32+ bytes) and GO_OMNI_STORAGE_PUBLIC_URL. the generated file references GO_OMNI_STORAGE_SIGNING_KEY (32+ bytes) and
GO_OMNI_STORAGE_PUBLIC_URL. File values may instead be fixed explicitly.
Cloud credentials use GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY; set bucket and endpoint Cloud credentials use GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY; set bucket and endpoint
or R2 accountId as appropriate. Signed URLs grant temporary access: authenticate or R2 accountId as appropriate. Signed URLs grant temporary access: authenticate
and authorize before issuing them. No public upload-link route is added. and authorize before issuing them. No public upload-link route is added.

View File

@@ -18,7 +18,6 @@ import (
"example.com/composition/internal/convert" "example.com/composition/internal/convert"
"example.com/composition/internal/domain" "example.com/composition/internal/domain"
"example.com/composition/internal/handler" "example.com/composition/internal/handler"
"example.com/composition/internal/mapping"
"example.com/composition/internal/middleware" "example.com/composition/internal/middleware"
"example.com/composition/internal/svc" "example.com/composition/internal/svc"
"example.com/composition/internal/types" "example.com/composition/internal/types"
@@ -86,16 +85,16 @@ func TestHTTPComposition(t *testing.T) {
} }
} }
} }
func TestFullFormAndMappingSemantics(t *testing.T) { func TestFullFormAndCopySemantics(t *testing.T) {
name := "employee" name := "employee"
money := amount.Money("2.00") money := amount.Money("2.00")
yes := true yes := true
req := types.EnterpriseRequest{Name: &name, Salary: &money, Enabled: &yes, Labels: []string{"one"}} req := types.EnterpriseRequest{Name: &name, Salary: &money, Enabled: &yes, Labels: []string{"one"}}
first, e := mapping.FromEnterprise(req) first, e := convert.Employee(req)
if e != nil { if e != nil {
t.Fatal(e) t.Fatal(e)
} }
second, e := mapping.FromEnterprise(types.EnterpriseRequest{}) second, e := convert.Employee(types.EnterpriseRequest{})
if e != nil { if e != nil {
t.Fatal(e) t.Fatal(e)
} }
@@ -108,26 +107,26 @@ func TestFullFormAndMappingSemantics(t *testing.T) {
t.Fatal("unexpected implicit deep copy") t.Fatal("unexpected implicit deep copy")
} }
req.Employer = &money req.Employer = &money
first, e = mapping.FromEnterprise(req) first, e = convert.Employee(req)
if e != nil || first.Employer != req.Employer { if e != nil || first.Employer != req.Employer {
t.Fatal("nullable pointer changed", e) t.Fatal("nullable pointer changed", e)
} }
bad := "reject" bad := "reject"
failed, e := mapping.FromEnterprise(types.EnterpriseRequest{Name: &bad}) failed, e := convert.Employee(types.EnterpriseRequest{Name: &bad})
if !errors.Is(e, convert.ErrName) || failed.EmployeeName != "" || !strings.Contains(e.Error(), "EmployeeName") { if !errors.Is(e, convert.ErrName) || failed.EmployeeName != "" || !strings.Contains(e.Error(), "EmployeeName") {
t.Fatal("converter cause/zero result lost", e) t.Fatal("converter cause/zero result lost", e)
} }
empty, e := mapping.ToBatch(domain.Batch{}) empty, e := convert.ToBatch(domain.Batch{})
if e != nil || empty.Rows != nil { if e != nil || empty.Rows != nil {
t.Fatal("nil slice changed", e) t.Fatal("nil slice changed", e)
} }
src := domain.Batch{Rows: []domain.Row{}, Fixed: [2]domain.Row{{Value: "a"}, {Value: "b"}}, Optional: &name} src := domain.Batch{Rows: []domain.Row{}, Fixed: [2]domain.Row{{Value: "a"}, {Value: "b"}}, Optional: &name}
out, e := mapping.ToBatch(src) out, e := convert.ToBatch(src)
if e != nil || out.Rows == nil || len(out.Rows) != 0 || out.Fixed[1].Value != "b" || out.Optional != src.Optional { if e != nil || out.Rows == nil || len(out.Rows) != 0 || out.Fixed[1].Value != "b" || out.Optional != src.Optional {
t.Fatal(out, e) t.Fatal(out, e)
} }
src.Rows = []domain.Row{{Value: "one"}} src.Rows = []domain.Row{{Value: "one"}}
out, e = mapping.ToBatch(src) out, e = convert.ToBatch(src)
if e != nil || out.Rows[0].Value != "one" { if e != nil || out.Rows[0].Value != "one" {
t.Fatal(out, e) t.Fatal(out, e)
} }
@@ -140,7 +139,7 @@ func TestFullFormAndMappingSemantics(t *testing.T) {
wg.Add(1) wg.Add(1)
go func() { go func() {
defer wg.Done() defer wg.Done()
result, err := mapping.FromEnterprise(types.EnterpriseRequest{Name: &name, Salary: &money}) result, err := convert.Employee(types.EnterpriseRequest{Name: &name, Salary: &money})
if err != nil || result.EmployeeName != name || result.Salary != money { if err != nil || result.EmployeeName != name || result.Salary != money {
t.Error(result, err) t.Error(result, err)
} }

View File

@@ -2,7 +2,6 @@ package convert
import ( import (
"errors" "errors"
"example.com/composition/internal/amount"
"strconv" "strconv"
) )
@@ -19,11 +18,4 @@ func Name(value *string) (string, error) {
} }
return *value, nil return *value, nil
} }
func Money(value *amount.Money) amount.Money {
if value == nil {
return ""
}
return *value
}
func Bool(value *bool) bool { return value != nil && *value }
func ID(value int64) string { return strconv.FormatInt(value, 10) } func ID(value int64) string { return strconv.FormatInt(value, 10) }

View File

@@ -0,0 +1,29 @@
package convert
import (
"example.com/composition/internal/domain"
"example.com/composition/internal/types"
"git.yuntuo.hk/k2safe/go-omni/copyx"
)
// The three endpoints reuse one ordinary Go helper and their existing scalar.
// There is no field list or package/type name JSON to keep in sync.
func Employee[S any](source S) (domain.Employee, error) {
var target domain.Employee
err := copyx.Copy(&target, source,
copyx.WithFieldNames[S, domain.Employee](map[string]string{"Name": "EmployeeName"}),
copyx.WithNilPolicy(copyx.NilZero),
copyx.WithConverter(Name),
)
return target, err
}
func Response(source domain.Result) (types.EmployeeResponse, error) {
var target types.EmployeeResponse
err := copyx.Copy(&target, source, copyx.WithConverter(func(v int64) (string, error) { return ID(v), nil }))
return target, err
}
func ToBatch(source domain.Batch) (domain.BatchDTO, error) {
var target domain.BatchDTO
err := copyx.Copy(&target, source)
return target, err
}

View File

@@ -14,7 +14,7 @@ type Result struct {
ID int64 ID int64
} }
// Distinct container item types exercise explicitly selected element mappings. // Distinct container item types exercise explicit Go collection copying.
type Row struct{ Value string } type Row struct{ Value string }
type RowDTO struct{ Value string } type RowDTO struct{ Value string }
type Batch struct { type Batch struct {

View File

@@ -0,0 +1,113 @@
package main
import (
"context"
"errors"
"example.com/configuration/internal/config"
omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
"os"
"os/exec"
"path/filepath"
"strings"
"testing"
)
func TestGeneratedConfiguration(t *testing.T) {
for ext, source := range map[string]string{"json": `{"limit":"${LIMIT:-7}"}`, "yaml": "limit: '${LIMIT:-7}'\n", "xml": "<config><limit>${LIMIT:-7}</limit></config>", "custom": `{"limit":"${LIMIT:-7}"}`} {
t.Run(ext, func(t *testing.T) {
t.Setenv("LIMIT", "")
t.Setenv("GO_OMNI_LIMIT", "999")
path := filepath.Join(t.TempDir(), "app."+ext)
if err := os.WriteFile(path, []byte(source), 0600); err != nil {
t.Fatal(err)
}
cfg, err := config.Load(path)
if err != nil || cfg.Limit != 7 {
t.Fatal(cfg.Limit, err)
}
t.Setenv("LIMIT", "0")
cfg, err = config.Load(path)
if err != nil || cfg.Limit != 0 {
t.Fatal(cfg.Limit, err)
}
})
}
t.Setenv("GO_OMNI_LOG_OUTPUT", "stderr")
cfg, err := config.Load("")
if err != nil || cfg.Limit != 10 || cfg.Server.Log.File != "./var/log/app.log" || !cfg.Server.Log.Rotation.Enabled {
t.Fatal("implicit overlay", err)
}
path := filepath.Join(t.TempDir(), "bad.yaml")
os.WriteFile(path, []byte("limit: '${LIMIT}'\n"), 0600)
t.Setenv("LIMIT", "-1")
// svc.Init panics if reached. Both invalid startup and successful check must
// stop before opening the configured file logger or project dependencies.
err = run(context.Background(), path)
var d *omniConfig.Error
if !errors.As(err, &d) || d.Field != "limit" || d.File != path || d.Line != 1 || d.Env != "LIMIT" {
t.Fatal(err)
}
t.Setenv("LIMIT", "private-invalid")
_, err = config.Load(path)
if !errors.As(err, &d) || d.Env != "LIMIT" || strings.Contains(err.Error(), "private-invalid") {
t.Fatal(err)
}
}
func TestConfigurationCheckProcess(t *testing.T) {
exe := filepath.Join(t.TempDir(), "server")
cmd := exec.Command("go", "build", "-o", exe, ".")
if out, err := cmd.CombinedOutput(); err != nil {
t.Fatal(err, string(out))
}
dir := t.TempDir()
path := filepath.Join(dir, "selected.yaml")
if err := os.WriteFile(path, []byte("limit: '${LIMIT:-0}'\nserver:\n address: 127.0.0.1:1\n"), 0600); err != nil {
t.Fatal(err)
}
// A broken neighboring default/local/test file must never be layered in.
for _, name := range []string{"config.yaml", "config.local.yaml", "config.test.yaml"} {
os.WriteFile(filepath.Join(dir, name), []byte("broken: ["), 0600)
}
cmd = exec.Command(exe, "-config", path, "-check-config")
cmd.Dir = dir
if out, err := cmd.CombinedOutput(); err != nil || len(out) != 0 {
t.Fatal(err, string(out))
}
if _, err := os.Stat(filepath.Join(dir, "var")); !os.IsNotExist(err) {
t.Fatal("check opened logger/dependencies")
}
os.WriteFile(path, []byte("limit: '${MISSING_REQUIRED}'"), 0600)
cmd = exec.Command(exe, "-config", path, "-check-config")
cmd.Dir = dir
cmd.Env = append(os.Environ(), "MISSING_REQUIRED=")
out, err := cmd.CombinedOutput()
if err == nil || !strings.Contains(string(out), "MISSING_REQUIRED") || strings.Contains(string(out), "panic") {
t.Fatal(err, string(out))
}
cmd = exec.Command(exe, "-config", path)
cmd.Dir = dir
cmd.Env = append(os.Environ(), "MISSING_REQUIRED=")
if out, err := cmd.CombinedOutput(); err == nil || strings.Contains(string(out), "panic") {
t.Fatal(err, string(out))
}
}
func TestGeneratedModuleFieldDiagnostic(t *testing.T) {
path := filepath.Join(t.TempDir(), "config.yaml")
t.Setenv("EXAMPLE_LOG_LEVEL", "invalid-private-level")
if err := os.WriteFile(path, []byte("server:\n log:\n level: '${EXAMPLE_LOG_LEVEL}'\n"), 0600); err != nil {
t.Fatal(err)
}
err := run(context.Background(), path)
var d *omniConfig.Error
if !errors.As(err, &d) || d.Field != "server.log.level" || d.Line != 3 || d.Env != "EXAMPLE_LOG_LEVEL" || !strings.Contains(d.Reason, "debug") || strings.Contains(err.Error(), "invalid-private-level") {
t.Fatal(err)
}
if err := os.WriteFile(path, []byte("server:\n log:\n timeZone: Asia/Shanghai\n"), 0600); err != nil {
t.Fatal(err)
}
cfg, err := config.Load(path)
if err != nil || cfg.Server.Log.TimeZone != "Asia/Shanghai" {
t.Fatal(err)
}
}

View File

@@ -0,0 +1,69 @@
package main
import (
"context"
"errors"
"net/http/httptest"
"os"
"path/filepath"
"strings"
"testing"
"example.com/corsdemo/internal/config"
"example.com/corsdemo/internal/handler"
"example.com/corsdemo/internal/svc"
omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/logx"
)
func TestGeneratedCORSConfiguration(t *testing.T) {
cfg, err := config.Load("etc/config.yaml")
if err != nil {
t.Fatal(err)
}
logger, err := logx.New(cfg.Server.Log)
if err != nil {
t.Fatal(err)
}
defer logger.Close()
deps, err := svc.New(context.Background(), logger, cfg)
if err != nil {
t.Fatal(err)
}
defer deps.Close()
a, err := handler.NewApp(deps)
if err != nil {
t.Fatal(err)
}
defer a.Close()
for _, tc := range []struct {
method, path string
status int
}{{"GET", "/ping", 501}, {"GET", "/missing", 404}, {"PUT", "/ping", 405}, {"GET", "/health/live", 200}, {"GET", "/health/ready", 200}, {"GET", "/openapi.json", 200}, {"OPTIONS", "/ping", 204}} {
r := httptest.NewRequest(tc.method, tc.path, nil)
r.Header.Set("Origin", "https://web.example")
if tc.method == "OPTIONS" {
r.Header.Set("Access-Control-Request-Method", "GET")
r.Header.Set("Access-Control-Request-Headers", "authorization")
}
w := httptest.NewRecorder()
a.ServeHTTP(w, r)
if w.Code != tc.status || w.Header().Get("Access-Control-Allow-Origin") != "https://web.example" {
t.Fatal(tc, w.Code, w.Header(), w.Body.String())
}
if tc.status == 204 && (w.Body.Len() != 0 || w.Header().Get("Access-Control-Max-Age") != "600") {
t.Fatal(w)
}
}
// Startup config validation uses the normal generated run/Load, before log
// files or dependencies. No changes to generated main/HTTP composition.
path := filepath.Join(t.TempDir(), "invalid.yaml")
if err := os.WriteFile(path, []byte("server:\n cors:\n enabled: true\n allowedOrigins: ['*']\n allowCredentials: true\n"), 0600); err != nil {
t.Fatal(err)
}
err = run(context.Background(), path)
var d *omniConfig.Error
if !errors.As(err, &d) || d.Field != "server.cors.allowedOrigins[0]" || !strings.Contains(d.Reason, "allowCredentials") {
t.Fatal(err)
}
}

View File

@@ -0,0 +1,70 @@
package main_test
import (
"context"
"encoding/json"
"io"
"net/http/httptest"
"testing"
"example.com/errorCatalog/internal/apierrors"
"example.com/errorCatalog/internal/handler"
"example.com/errorCatalog/internal/svc"
omni "git.yuntuo.hk/k2safe/go-omni"
"git.yuntuo.hk/k2safe/go-omni/httpx"
"git.yuntuo.hk/k2safe/go-omni/i18n"
"git.yuntuo.hk/k2safe/go-omni/i18n/errori18n"
"git.yuntuo.hk/k2safe/go-omni/i18n/httpi18n"
"git.yuntuo.hk/k2safe/go-omni/logx"
)
func TestGeneratedTranslationAndParameterCompatibility(t *testing.T) {
catalog := apierrors.Catalog()
source, err := errori18n.Source(catalog, "en")
if err != nil {
t.Fatal(err)
}
source = append(source, i18n.Document{Version: 1, Locale: "zh-Hans", Namespace: "limits", Messages: map[string]i18n.Message{"exceeded": {Value: i18n.Pattern{Text: "上限:{max}"}}}})
b, err := i18n.New(i18n.Config{SourceLocale: "en", DefaultLocale: "en", Locales: []string{"en", "zh-Hans"}}, source...)
if err != nil {
t.Fatal(err)
}
p, err := httpi18n.New(b)
if err != nil {
t.Fatal(err)
}
lc := logx.Default()
lc.Writer = io.Discard
logger, err := logx.New(lc)
if err != nil {
t.Fatal(err)
}
defer logger.Close()
deps, err := svc.New(context.Background(), logger)
if err != nil {
t.Fatal(err)
}
defer deps.Close()
a := omni.New(omni.WithLanguagePolicy(p), omni.WithErrorLocalizer(errori18n.New(b, nil)))
defer a.Close()
a.Use(func(next httpx.Handler) httpx.Handler {
return func(c *httpx.Context) error { return apierrors.LimitExceeded(9223372036854775807) }
})
if err := handler.RegisterRoutes(a, deps); err != nil {
t.Fatal(err)
}
r := httptest.NewRequest("GET", "/value", nil)
r.Header.Set("Accept-Language", "zh-CN")
w := httptest.NewRecorder()
a.ServeHTTP(w, r)
var body map[string]any
if err := json.Unmarshal(w.Body.Bytes(), &body); err != nil {
t.Fatal(err)
}
if w.Code != 200 || body["detail"] != "上限:9223372036854775807" || body["code"] != float64(400001) || body["ok"] != false {
t.Fatal(w.Code, body)
}
if apierrors.LimitExceeded(5).Message != "Limit: 5" {
t.Fatal("non-HTTP constructor changed")
}
}

View File

@@ -123,7 +123,7 @@ func TestGeneratedTransfers(t *testing.T) {
func configFile(t *testing.T, address string, fail bool) string { func configFile(t *testing.T, address string, fail bool) string {
t.Helper() t.Helper()
path := filepath.Join(t.TempDir(), "config.json") path := filepath.Join(t.TempDir(), "config.json")
data, _ := json.Marshal(map[string]any{"server": map[string]any{"address": address}, "failInit": fail}) data, _ := json.Marshal(map[string]any{"server": map[string]any{"address": address, "transport": "${GO_OMNI_TRANSPORT:-nethttp}"}, "failInit": fail})
if err := os.WriteFile(path, data, 0600); err != nil { if err := os.WriteFile(path, data, 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -24,7 +24,7 @@ func phasedConfig(t *testing.T, address, mode string, fail bool) string {
if mode == "stuck" { if mode == "stuck" {
drainTimeout = "50ms" drainTimeout = "50ms"
} }
data, _ := json.Marshal(map[string]any{"backgroundMode": mode, "failInit": fail, "server": map[string]any{"address": address, "lifecycle": map[string]string{"stopTimeout": "1s", "drainTimeout": drainTimeout, "cancelTimeout": "50ms"}}}) data, _ := json.Marshal(map[string]any{"backgroundMode": mode, "failInit": fail, "server": map[string]any{"address": address, "transport": "${GO_OMNI_TRANSPORT:-nethttp}", "lifecycle": map[string]string{"stopTimeout": "1s", "drainTimeout": drainTimeout, "cancelTimeout": "50ms"}}})
if err := os.WriteFile(path, data, 0600); err != nil { if err := os.WriteFile(path, data, 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -1,170 +0,0 @@
package main
import (
"context"
"encoding/json"
"errors"
"io"
"net/http/httptest"
"strings"
"sync"
"testing"
"example.com/nullable/internal/convert"
"example.com/nullable/internal/domain"
"example.com/nullable/internal/handler"
"example.com/nullable/internal/mapping"
"example.com/nullable/internal/svc"
"example.com/nullable/internal/types"
"git.yuntuo.hk/k2safe/go-omni/logx"
)
func TestNullableMappings(t *testing.T) {
for _, src := range []types.Parent{{}, {Children: []*types.Child{}}, {Child: &types.Child{}, Children: []*types.Child{nil, {Name: "two"}}}} {
out, err := mapping.ToParent(src)
if err != nil || (src.Child == nil) != (out.Child == nil) || (src.Children == nil) != (out.Children == nil) || len(src.Children) != len(out.Children) {
t.Fatal(src, out, err)
}
if src.Child != nil && out.Child.Name != src.Child.Name {
t.Fatal("zero child changed")
}
if len(out.Children) > 0 && (out.Children[0] != nil || out.Children[1].Name != "two") {
t.Fatal(out)
}
}
src := types.Parent{Child: &types.Child{Name: "one"}, Children: []*types.Child{{Name: "two"}}}
out, err := mapping.ToParentPtr(src)
if err != nil || out == nil || out.Child.Name != "one" {
t.Fatal(out, err)
}
out.Child.Name = "changed"
out.Children[0].Name = "changed"
if src.Child.Name != "one" || src.Children[0].Name != "two" {
t.Fatal("nested mapped values alias input")
}
plain := domain.Parent{Child: &domain.Child{Name: "shared"}}
copy, err := mapping.ShallowCopy(plain)
if err != nil || copy.Child != plain.Child {
t.Fatal("ordinary assignment semantics changed", err)
}
zero, err := mapping.RequiredZero(types.Parent{})
if err != nil || zero.Child.Name != "" {
t.Fatal(zero, err)
}
for _, fn := range []func(types.Parent) (domain.Required, error){mapping.RequiredZero, mapping.RequiredError} {
out, err := fn(types.Parent{Child: &types.Child{Name: "required"}})
if err != nil || out.Child.Name != "required" {
t.Fatal(out, err)
}
}
_, err = mapping.RequiredError(types.Parent{})
if err == nil || !strings.Contains(err.Error(), "RequiredError.Child") {
t.Fatal(err)
}
_, err = mapping.RequirePointer(types.Parent{})
if err == nil || !strings.Contains(err.Error(), "RequirePointer.Child") {
t.Fatal(err)
}
lift, err := mapping.Lift(domain.ValueHolder{Child: domain.Child{Name: "lift"}})
if err != nil || lift.Child == nil || lift.Child.Name != "lift" {
t.Fatal(lift, err)
}
batch, err := mapping.Batch(domain.InputBatch{Fixed: [2]*domain.Child{nil, {Name: "array"}}})
if err != nil || batch.Fixed[0] != nil || batch.Fixed[1].Name != "array" {
t.Fatal(batch, err)
}
_, err = mapping.Batch(domain.InputBatch{Fixed: [2]*domain.Child{nil, {Name: "bad"}}})
if !errors.Is(err, convert.ErrValue) || !strings.Contains(err.Error(), "Batch.Fixed[1]") {
t.Fatal(err)
}
for _, input := range []types.Parent{{Child: &types.Child{Name: "bad"}}, {Children: []*types.Child{nil, {Name: "bad"}}}} {
failed, err := mapping.ToParentPtr(input)
if failed != nil || !errors.Is(err, convert.ErrValue) || !strings.Contains(err.Error(), "ToChild.Name") {
t.Fatal(failed, err)
}
if len(input.Children) > 0 && !strings.Contains(err.Error(), "ToParent.Children[1]") {
t.Fatal(err)
}
value, err := mapping.ToParent(input)
if value.Child != nil || value.Children != nil || !errors.Is(err, convert.ErrValue) {
t.Fatal("partial result escaped", err)
}
}
}
func TestExplicitNilAndConcurrentCalls(t *testing.T) {
zero, err := mapping.ScalarZero(domain.ScalarInput{})
if err != nil || zero != (domain.ScalarOutput{}) {
t.Fatal(zero, err)
}
_, err = mapping.ScalarError(domain.ScalarInput{})
if err == nil {
t.Fatal("nil accepted")
}
text, count, enabled := "", int64(0), false
out, err := mapping.ScalarError(domain.ScalarInput{Text: &text, Count: &count, Enabled: &enabled})
if err != nil || out != (domain.ScalarOutput{}) {
t.Fatal("non-nil zero rejected", out, err)
}
var wg sync.WaitGroup
for i := 0; i < 32; i++ {
wg.Add(1)
go func() {
defer wg.Done()
out, err := mapping.ToParent(types.Parent{Child: &types.Child{Name: "parallel"}})
if err != nil || out.Child == nil || out.Child.Name != "parallel" {
t.Error(out, err)
}
}()
}
wg.Wait()
}
func TestHTTPNullableMapping(t *testing.T) {
cfg := logx.Default()
cfg.Writer = io.Discard
logger, err := logx.New(cfg)
if err != nil {
t.Fatal(err)
}
defer logger.Close()
deps, err := svc.New(context.Background(), logger)
if err != nil {
t.Fatal(err)
}
defer deps.Close()
app, err := handler.NewApp(deps)
if err != nil {
t.Fatal(err)
}
defer app.Close()
for _, body := range []string{`{}`, `{"child":null}`, `{"child":{"name":""}}`, `{"child":{"name":"one"},"children":[null,{"name":"two"}]}`, `{"children":[]}`} {
w := httptest.NewRecorder()
req := httptest.NewRequest("POST", "/", strings.NewReader(body))
req.Header.Set("Content-Type", "application/json")
app.Engine().ServeHTTP(w, req)
if w.Code != 200 {
t.Fatal(w.Code, w.Body.String())
}
var input, output types.Parent
if err := json.Unmarshal([]byte(body), &input); err != nil {
t.Fatal(err)
}
if err := json.Unmarshal(w.Body.Bytes(), &output); err != nil {
t.Fatal(err)
}
if (input.Child == nil) != (output.Child == nil) || (input.Children == nil) != (output.Children == nil) || len(input.Children) != len(output.Children) {
t.Fatal(body, w.Body.String())
}
if input.Child != nil && input.Child.Name != output.Child.Name {
t.Fatal(w.Body.String())
}
}
w := httptest.NewRecorder()
req := httptest.NewRequest("POST", "/", strings.NewReader(`{"child":{"name":"bad"}}`))
req.Header.Set("Content-Type", "application/json")
app.Engine().ServeHTTP(w, req)
if w.Code != 500 || strings.Contains(w.Body.String(), convert.ErrValue.Error()) || strings.Contains(w.Body.String(), "ToChild") {
t.Fatal("private cause leaked", w.Code, w.Body.String())
}
}

View File

@@ -1,15 +0,0 @@
package convert
import (
"errors"
"fmt"
)
var ErrValue = errors.New("private conversion cause")
func Name(value string) (string, error) {
if value == "bad" {
return "", fmt.Errorf("conversion refused: %w", ErrValue)
}
return value, nil
}

View File

@@ -1,23 +0,0 @@
package domain
type Child struct{ Name string }
type ChildDTO struct{ Name string }
type Parent struct {
Child *Child
Children []*Child
}
type Required struct{ Child Child }
type ValueHolder struct{ Child Child }
type PointerHolder struct{ Child *ChildDTO }
type InputBatch struct{ Fixed [2]*Child }
type OutputBatch struct{ Fixed [2]*ChildDTO }
type ScalarInput struct {
Text *string
Count *int64
Enabled *bool
}
type ScalarOutput struct {
Text string
Count int64
Enabled bool
}

View File

@@ -1,15 +0,0 @@
type Child {
Name string `json:"name"`
}
type Parent {
Child *Child `json:"child,optional"`
Children []*Child `json:"children,optional"`
}
@server (
group: probe
response: raw
)
service probe {
@handler roundTrip
post / (Parent) returns (Parent)
}

View File

@@ -14,7 +14,9 @@ import (
"golang.org/x/mod/semver" "golang.org/x/mod/semver"
"os" "os"
"path/filepath" "path/filepath"
"reflect"
"sort" "sort"
"strconv"
"strings" "strings"
) )
@@ -30,6 +32,55 @@ type UpgradePlan struct {
Module, Entry string Module, Entry string
} }
// Handwritten old loaders must be migrated explicitly, never silently replaced
// or left competing with the generated entry point.
func checkConfigLoadExtension(root string) error {
dir := filepath.Join(root, "internal/config")
entries, err := os.ReadDir(dir)
if errors.Is(err, os.ErrNotExist) {
return nil
}
if err != nil {
return err
}
for _, entry := range entries {
name := entry.Name()
if entry.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") || name == "loadGen.go" {
continue
}
rel := "internal/config/" + name
if err := safePath(root, rel); err != nil {
return err
}
f, err := goparser.ParseFile(gotoken.NewFileSet(), filepath.Join(root, rel), nil, 0)
if err != nil {
return fmt.Errorf("parse %s: %w", rel, err)
}
var removed bool
ast.Inspect(f, func(n ast.Node) bool {
field, ok := n.(*ast.Field)
if ok && field.Tag != nil {
if raw, e := strconv.Unquote(field.Tag.Value); e == nil {
tag := reflect.StructTag(raw)
_, a := tag.Lookup("env")
_, b := tag.Lookup("envDefault")
removed = removed || a || b
}
}
return true
})
if removed {
return fmt.Errorf("%s: env/envDefault tags were removed; migrate defaults and explicit file references before generating; see CONFIG.md", rel)
}
for _, decl := range f.Decls {
if fn, ok := decl.(*ast.FuncDecl); ok && fn.Recv == nil && fn.Name.Name == "Load" {
return fmt.Errorf("%s: migrate the project-owned Load function before generating internal/config/loadGen.go; retain Config/Default/Validate and move decoder options to NewLoader; see CONFIG.md", rel)
}
}
}
return nil
}
// PlanUpgrade compares infrastructure against the exact previous release seeds. // PlanUpgrade compares infrastructure against the exact previous release seeds.
// A custom file is never classified as safe to replace. Logic, middleware, // A custom file is never classified as safe to replace. Logic, middleware,
// options and repositories outside the changed templates are left untouched. // options and repositories outside the changed templates are left untouched.

View File

@@ -39,7 +39,7 @@ func oldInfrastructureProject(t *testing.T) string {
raw, _ := json.Marshal(m) raw, _ := json.Marshal(m)
writeFile(t, filepath.Join(dir, manifestPath), string(raw)) writeFile(t, filepath.Join(dir, manifestPath), string(raw))
path := filepath.Join(dir, "go.mod") path := filepath.Join(dir, "go.mod")
writeFile(t, path, strings.ReplaceAll(readFile(t, path), minimumVersion, "v0.13.0-preview")) writeFile(t, path, strings.ReplaceAll(readFile(t, path), currentVersion, "v0.13.0-preview"))
return dir return dir
} }
@@ -74,7 +74,7 @@ func TestUpgradeV014PreservesExtensions(t *testing.T) {
raw, _ := json.Marshal(m) raw, _ := json.Marshal(m)
writeFile(t, filepath.Join(dir, manifestPath), string(raw)) writeFile(t, filepath.Join(dir, manifestPath), string(raw))
mod := filepath.Join(dir, "go.mod") mod := filepath.Join(dir, "go.mod")
writeFile(t, mod, strings.ReplaceAll(readFile(t, mod), minimumVersion, "v0.14.1-preview")) writeFile(t, mod, strings.ReplaceAll(readFile(t, mod), currentVersion, "v0.14.1-preview"))
owned := map[string]string{} owned := map[string]string{}
for _, name := range []string{"internal/svc/serviceContext.go", "internal/config/config.go", "etc/config.yaml"} { for _, name := range []string{"internal/svc/serviceContext.go", "internal/config/config.go", "etc/config.yaml"} {
path := filepath.Join(dir, name) path := filepath.Join(dir, name)

View File

@@ -13,7 +13,7 @@ import (
func TestWorkspaceSharedGoCatalog(t *testing.T) { func TestWorkspaceSharedGoCatalog(t *testing.T) {
root := t.TempDir() root := t.TempDir()
writeFile(t, filepath.Join(root, "go.mod"), fmt.Sprintf("module example.com/catalogs\n\ngo 1.24.0\nrequire %s v0.15.0-preview\nreplace %s => %s\n", frameworkModule, frameworkModule, strconvQuote(frameworkRoot(t)))) writeFile(t, filepath.Join(root, "go.mod"), fmt.Sprintf("module example.com/catalogs\n\ngo 1.22.0\nrequire %s v0.15.0-preview\nreplace %s => %s\n", frameworkModule, frameworkModule, strconvQuote(frameworkRoot(t))))
writeFile(t, filepath.Join(root, "internal/apierrors/catalog.go"), `package apierrors writeFile(t, filepath.Join(root, "internal/apierrors/catalog.go"), `package apierrors
import errors "git.yuntuo.hk/k2safe/go-omni/errors" import errors "git.yuntuo.hk/k2safe/go-omni/errors"
func Catalog() errors.Catalog {return errors.Catalog{DefaultStatus:400,Definitions:map[string]errors.Definition{ func Catalog() errors.Catalog {return errors.Catalog{DefaultStatus:400,Definitions:map[string]errors.Definition{

View File

@@ -117,6 +117,17 @@ func apply(root, module, entry string, generated, seeds map[string][]byte, check
var changes []change var changes []change
// Check ALL previous outputs, even stale ones, before writing any new file. // Check ALL previous outputs, even stale ones, before writing any new file.
for name, hash := range old.Generated { for name, hash := range old.Generated {
// Migration guard only; old mappings are never regenerated or silently
// deleted. Once the project removes them, forget the stale ownership.
if name == "internal/mapping/mappingsGen.go" {
if err = safePath(root, name); err != nil {
return result, err
}
if _, e := os.Lstat(filepath.Join(root, name)); !errors.Is(e, os.ErrNotExist) {
return result, errors.New("mapping generation was removed: migrate callers to copyx or ordinary Go and remove internal/mapping/mappingsGen.go before generating; see COPY.md")
}
continue
}
if !ownedPath(name) { if !ownedPath(name) {
return result, fmt.Errorf("api: invalid owned path %q", name) return result, fmt.Errorf("api: invalid owned path %q", name)
} }
@@ -273,7 +284,7 @@ func apply(root, module, entry string, generated, seeds map[string][]byte, check
} }
func digest(data []byte) string { sum := sha256.Sum256(data); return hex.EncodeToString(sum[:]) } func digest(data []byte) string { sum := sha256.Sum256(data); return hex.EncodeToString(sum[:]) }
func ownedPath(name string) bool { func ownedPath(name string) bool {
if name == "internal/mapping/mappingsGen.go" { if name == "internal/config/loadGen.go" {
return true return true
} }
if strings.HasPrefix(name, "internal/types/dto") && strings.HasSuffix(name, "Gen.go") && goName.MatchString(strings.TrimSuffix(filepath.Base(name), ".go")) { if strings.HasPrefix(name, "internal/types/dto") && strings.HasSuffix(name, "Gen.go") && goName.MatchString(strings.TrimSuffix(filepath.Base(name), ".go")) {

195
cmd/go-omni/i18n.go Normal file
View File

@@ -0,0 +1,195 @@
package main
import (
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"os"
"strings"
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
"git.yuntuo.hk/k2safe/go-omni/i18n"
"git.yuntuo.hk/k2safe/go-omni/i18n/errori18n"
"git.yuntuo.hk/k2safe/go-omni/i18n/i18ntool"
)
type i18nFlags []string
func (v *i18nFlags) String() string { return strings.Join(*v, ",") }
func (v *i18nFlags) Set(s string) error {
if s == "" {
return errors.New("empty value")
}
*v = append(*v, s)
return nil
}
func runI18n(args []string) error { return runI18nTo(args, os.Stdout) }
func runI18nTo(args []string, w io.Writer) error {
if len(args) == 0 {
return errors.New("expected check, export, types or extract")
}
fs := flag.NewFlagSet("i18n "+args[0], flag.ContinueOnError)
fs.SetOutput(io.Discard)
var resources, namespaces, goFiles i18nFlags
configPath := fs.String("config", "", "i18n Config JSON file")
locale := fs.String("locale", "", "selected/source locale")
catalog := fs.String("catalog", "", "exported errors.Catalog JSON (extract only)")
fs.Var(&resources, "resource", "resource JSON file (repeatable)")
fs.Var(&namespaces, "namespace", "export namespace (repeatable)")
fs.Var(&goFiles, "go", "Go source file (repeatable)")
if err := fs.Parse(args[1:]); err != nil {
return err
}
if fs.NArg() != 0 {
return errors.New("unexpected positional arguments")
}
encode := func(v any) error { e := json.NewEncoder(w); e.SetIndent("", " "); return e.Encode(v) }
if args[0] == "extract" && *catalog != "" {
if len(goFiles) > 0 || len(resources) > 0 || *configPath != "" || len(namespaces) != 1 || *locale == "" {
return errors.New("catalog extraction requires --catalog, --locale and exactly one --namespace")
}
raw, err := readI18nFile(*catalog)
if err != nil {
return err
}
var c apierrors.Catalog
if err := apierrors.DecodeCatalogJSON(raw, &c); err != nil {
return err
}
docs, err := errori18n.Source(errori18n.Bind(c, namespaces[0]), *locale)
if err != nil {
return err
}
for _, doc := range docs {
if doc.Namespace == namespaces[0] {
return encode(doc)
}
}
return errors.New("catalog has no messages in the selected namespace")
}
if *catalog != "" {
return errors.New("--catalog is only supported by extract")
}
if args[0] == "extract" {
if len(goFiles) == 0 || len(resources) > 0 || len(namespaces) > 0 || *configPath != "" || *locale != "" {
return errors.New("Go extraction requires one or more --go files")
}
out := i18ntool.Extraction{References: []i18ntool.Reference{}, Findings: []i18ntool.Finding{}}
for _, path := range goFiles {
raw, err := readI18nFile(path)
if err != nil {
return err
}
result, err := i18ntool.ExtractGo(path, raw)
if err != nil {
return err
}
out.References = append(out.References, result.References...)
out.Findings = append(out.Findings, result.Findings...)
}
return encode(out)
}
if args[0] != "check" && args[0] != "export" && args[0] != "types" {
return errors.New("expected check, export, types or extract")
}
if *configPath == "" || len(resources) == 0 {
return errors.New("--config and at least one --resource are required")
}
data, err := readI18nFile(*configPath)
if err != nil {
return err
}
var cfg i18n.Config
if err := i18n.DecodeJSON(data, &cfg); err != nil {
return err
}
docs := []i18n.Document{}
total := 0
for _, path := range resources {
raw, err := readI18nFile(path)
if err != nil {
return err
}
total += len(raw)
if total > 4<<20 {
return errors.New("resources exceed 4 MiB")
}
var doc i18n.Document
if err := i18n.DecodeJSON(raw, &doc); err != nil {
return fmt.Errorf("%s: %w", path, err)
}
docs = append(docs, doc)
}
bundle, err := i18n.New(cfg, docs...)
if err != nil {
return err
}
switch args[0] {
case "export":
if len(goFiles) > 0 {
return errors.New("--go is only supported by check/extract")
}
return bundle.Export(w, *locale, namespaces...)
case "types":
if *locale != "" || len(goFiles) > 0 {
return errors.New("types does not accept --locale or --go")
}
return bundle.TypeScript(w, namespaces...)
case "check":
if *locale != "" || len(namespaces) > 0 {
return errors.New("check validates all configured locales and namespaces")
}
findings := bundle.Check()
for _, path := range goFiles {
raw, e := readI18nFile(path)
if e != nil {
return e
}
ex, e := i18ntool.ExtractGo(path, raw)
if e != nil {
return e
}
for _, r := range ex.References {
if _, ok := bundle.Schema(i18n.Ref{Namespace: r.Namespace, Key: r.Key}); !ok {
findings = append(findings, i18n.Diagnostic{Kind: "missing", Namespace: r.Namespace, Key: r.Key, Problem: fmt.Sprintf("referenced by %s:%d", r.File, r.Line)})
}
}
for _, f := range ex.Findings {
findings = append(findings, i18n.Diagnostic{Kind: "dynamic", Problem: fmt.Sprintf("%s:%d: %s", f.File, f.Line, f.Problem)})
}
}
if err := encode(findings); err != nil {
return err
}
if len(findings) > 0 {
return errors.New("internationalization checks failed")
}
return nil
}
return nil
}
func readI18nFile(path string) ([]byte, error) {
f, err := os.Open(path)
if err != nil {
return nil, err
}
defer f.Close()
s, err := f.Stat()
if err != nil {
return nil, err
}
if !s.Mode().IsRegular() {
return nil, errors.New("expected a regular file")
}
b, err := io.ReadAll(io.LimitReader(f, (1<<20)+1))
if err != nil {
return nil, err
}
if len(b) > 1<<20 {
return nil, errors.New("file exceeds 1 MiB")
}
return b, nil
}

80
cmd/go-omni/i18n_test.go Normal file
View File

@@ -0,0 +1,80 @@
package main
import (
"bytes"
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"git.yuntuo.hk/k2safe/go-omni/i18n"
)
func TestI18nCommandsAreReadOnlyAndCheckMissing(t *testing.T) {
dir := t.TempDir()
files := map[string]string{
"config.json": `{"sourceLocale":"en","defaultLocale":"en","locales":["en","fr"]}`,
"en.json": `{"version":1,"namespace":"ui","locale":"en","messages":{"hello":{"params":{"type":"string"},"value":{"text":"Hello {type}"}}}}`,
"fr.json": `{"version":1,"namespace":"ui","locale":"fr","messages":{"hello":{"value":{"text":"Bonjour {type}"}}}}`,
"use.go": "package demo\nimport words \"git.yuntuo.hk/k2safe/go-omni/i18n\"\nvar key=words.Ref{Namespace:\"ui\",Key:\"hello\"}\n",
}
for path, data := range files {
if err := os.WriteFile(filepath.Join(dir, path), []byte(data), 0600); err != nil {
t.Fatal(err)
}
}
base := []string{"--config", filepath.Join(dir, "config.json"), "--resource", filepath.Join(dir, "en.json"), "--resource", filepath.Join(dir, "fr.json")}
var out bytes.Buffer
args := append([]string{"check"}, base...)
args = append(args, "--go", filepath.Join(dir, "use.go"))
if err := runI18nTo(args, &out); err != nil || strings.TrimSpace(out.String()) != "[]" {
t.Fatal(err, out.String())
}
out.Reset()
args = append([]string{"export"}, base...)
args = append(args, "--locale", "fr", "--namespace", "ui")
if err := runI18nTo(args, &out); err != nil {
t.Fatal(err)
}
var pack i18n.Pack
if err := json.Unmarshal(out.Bytes(), &pack); err != nil || pack.Locale != "fr" || len(pack.Documents) != 2 {
t.Fatal(err, pack)
}
out.Reset()
args = append([]string{"types"}, base...)
args = append(args, "--namespace", "ui")
if err := runI18nTo(args, &out); err != nil || !strings.Contains(out.String(), `"type": string`) {
t.Fatal(err, out.String())
}
out.Reset()
if err := runI18nTo([]string{"extract", "--go", filepath.Join(dir, "use.go")}, &out); err != nil || !strings.Contains(out.String(), `"key": "hello"`) {
t.Fatal(err, out.String())
}
for path, want := range files {
got, _ := os.ReadFile(filepath.Join(dir, path))
if string(got) != want {
t.Fatal("command mutated", path)
}
}
out.Reset()
args = append([]string{"check"}, base[:4]...)
if err := runI18nTo(args, &out); err == nil || !strings.Contains(out.String(), "direct translation missing") {
t.Fatal(err, out.String())
}
}
func TestI18nCatalogExtractionPreservesLiteralBraces(t *testing.T) {
path := filepath.Join(t.TempDir(), "errors.json")
if err := os.WriteFile(path, []byte(`{"definitions":{"FIXED":{"code":1,"message":"${NAME}"}}}`), 0600); err != nil {
t.Fatal(err)
}
var out bytes.Buffer
if err := runI18nTo([]string{"extract", "--catalog", path, "--namespace", "errors", "--locale", "en"}, &out); err != nil {
t.Fatal(err)
}
var d i18n.Document
if err := i18n.DecodeJSON(out.Bytes(), &d); err != nil || !d.Messages["FIXED"].Value.Literal || d.Messages["FIXED"].Value.Text != "${NAME}" {
t.Fatal(err, d)
}
}

View File

@@ -18,6 +18,11 @@ func main() {
} }
switch os.Args[1] { switch os.Args[1] {
case "i18n":
if err := runI18n(os.Args[2:]); err != nil {
fmt.Fprintln(os.Stderr, "go-omni i18n:", err)
os.Exit(1)
}
case "workspace": case "workspace":
if err := runWorkspace(os.Args[2:]); err != nil { if err := runWorkspace(os.Args[2:]); err != nil {
fmt.Fprintln(os.Stderr, "go-omni workspace:", err) fmt.Fprintln(os.Stderr, "go-omni workspace:", err)
@@ -170,6 +175,7 @@ func usage() {
fmt.Fprintln(os.Stderr, "GoOmni project tool") fmt.Fprintln(os.Stderr, "GoOmni project tool")
fmt.Fprintln(os.Stderr, "") fmt.Fprintln(os.Stderr, "")
fmt.Fprintln(os.Stderr, "Usage:") fmt.Fprintln(os.Stderr, "Usage:")
fmt.Fprintln(os.Stderr, " go-omni i18n <check|export|types|extract> Validate and export independent language resources")
fmt.Fprintln(os.Stderr, " go-omni migrate [--dir migrations] [--apply] [--dsn-env GO_OMNI_DB_DSN]") fmt.Fprintln(os.Stderr, " go-omni migrate [--dir migrations] [--apply] [--dsn-env GO_OMNI_DB_DSN]")
fmt.Fprintln(os.Stderr, " go-omni new [--module <module>] <directory>") fmt.Fprintln(os.Stderr, " go-omni new [--module <module>] <directory>")
fmt.Fprintln(os.Stderr, " go-omni new --api <file.api> [--module <module>] <directory>") fmt.Fprintln(os.Stderr, " go-omni new --api <file.api> [--module <module>] <directory>")

View File

@@ -8,7 +8,7 @@ import (
) )
func TestNewWithRemoteFrameworkVersion(t *testing.T) { func TestNewWithRemoteFrameworkVersion(t *testing.T) {
for _, flag := range [][]string{{"--framework-version", "v0.18.0-preview"}, {"--framework-version=v0.18.0-preview"}} { for _, flag := range [][]string{{"--framework-version", "v0.22.0-preview"}, {"--framework-version=v0.22.0-preview"}} {
project := filepath.Join(t.TempDir(), "billing") project := filepath.Join(t.TempDir(), "billing")
args := append([]string{"--module", "example.com/billing"}, flag...) args := append([]string{"--module", "example.com/billing"}, flag...)
if err := runNew(append(args, project)); err != nil { if err := runNew(append(args, project)); err != nil {
@@ -18,7 +18,7 @@ func TestNewWithRemoteFrameworkVersion(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if !strings.Contains(string(content), "go-omni v0.18.0-preview") || strings.Contains(string(content), "replace ") { if !strings.Contains(string(content), "go-omni v0.22.0-preview") || strings.Contains(string(content), "replace ") {
t.Fatalf("CLI did not pin remote dependency: %s", content) t.Fatalf("CLI did not pin remote dependency: %s", content)
} }
} }

View File

@@ -21,6 +21,8 @@ func apiInfrastructure(module, database, storage, logOutput, logFormat string, l
"internal/svc/runtimeGen.go": "templates/api-runtime.go.tmpl", "internal/svc/runtimeGen.go": "templates/api-runtime.go.tmpl",
"internal/svc/options.go": "templates/api-options.go.tmpl", "internal/svc/options.go": "templates/api-options.go.tmpl",
"internal/config/config.go": "templates/api-config.go.tmpl", "internal/config/config.go": "templates/api-config.go.tmpl",
"internal/config/loadGen.go": "templates/api-load.go.tmpl",
"internal/config/loader.go": "templates/api-loader.go.tmpl",
"etc/config.yaml": "templates/modules-config.yaml.tmpl", "etc/config.yaml": "templates/modules-config.yaml.tmpl",
"internal/svc/serviceContext.go": "templates/api-context.go.tmpl", "internal/svc/serviceContext.go": "templates/api-context.go.tmpl",
} }
@@ -31,6 +33,8 @@ func apiInfrastructure(module, database, storage, logOutput, logFormat string, l
files["internal/svc/storage.go"] = "templates/modules-storage.go.tmpl" files["internal/svc/storage.go"] = "templates/modules-storage.go.tmpl"
} }
if legacy { if legacy {
delete(files, "internal/config/loadGen.go")
delete(files, "internal/config/loader.go")
delete(files, "internal/svc/runtimeGen.go") delete(files, "internal/svc/runtimeGen.go")
delete(files, "internal/svc/databaseOptions.go") delete(files, "internal/svc/databaseOptions.go")
files["internal/svc/close.go"] = "templates/api-close.go.tmpl" files["internal/svc/close.go"] = "templates/api-close.go.tmpl"

View File

@@ -66,9 +66,13 @@ func TestGenerateModuleProfiles(t *testing.T) {
t.Fatal("replacement does not resolve", err) t.Fatal("replacement does not resolve", err)
} }
cfg, _ := os.ReadFile(filepath.Join(project, "etc/config.yaml")) cfg, _ := os.ReadFile(filepath.Join(project, "etc/config.yaml"))
if strings.Contains(string(cfg), "secretKey:") || strings.Contains(string(cfg), "filesToken:") { for _, line := range strings.Split(string(cfg), "\n") {
t.Fatal("secret defaults emitted") key, value, ok := strings.Cut(strings.TrimSpace(line), ":")
if ok && (key == "secretKey" || key == "filesToken" || key == "signingKey") && !strings.Contains(value, "${GO_OMNI_") {
t.Fatal("secret must be an explicit reference")
}
} }
}) })
} }
} }
@@ -76,6 +80,7 @@ func TestModuleOptionsRejectBeforeWriting(t *testing.T) {
for _, options := range []Options{ for _, options := range []Options{
{LogOutput: "invalid"}, {LogFormat: "xml"}, {Storage: "ftp"}, {LogOutput: "invalid"}, {LogFormat: "xml"}, {Storage: "ftp"},
{Storage: "local", FrameworkVersion: "v0.6.0-preview"}, {Storage: "local", FrameworkVersion: "v0.6.0-preview"},
{FrameworkVersion: "v0.21.0-preview"},
{Module: "example.com/x\nrequire malicious v1.0.0"}, {Module: "example.com/x\nrequire malicious v1.0.0"},
} { } {
options.Dir = filepath.Join(t.TempDir(), "project") options.Dir = filepath.Join(t.TempDir(), "project")

View File

@@ -34,9 +34,13 @@ type Options struct {
} }
var files = map[string]string{ var files = map[string]string{
"internal/settings/config.go": "templates/modules-settings.go.tmpl",
"internal/settings/config_test.go": "templates/modules-settings_test.go.tmpl",
"etc/config.yaml": "templates/modules-config.yaml.tmpl",
"go.mod": "templates/go.mod.tmpl", "go.mod": "templates/go.mod.tmpl",
"README.md": "templates/README.md.tmpl", "README.md": "templates/README.md.tmpl",
"cmd/server/main.go": "templates/cmd-server-main.go.tmpl", "cmd/server/main.go": "templates/modules-main.go.tmpl",
"internal/model/user.go": "templates/internal-model-user.go.tmpl", "internal/model/user.go": "templates/internal-model-user.go.tmpl",
"internal/model/errors.go": "templates/internal-model-errors.go.tmpl", "internal/model/errors.go": "templates/internal-model-errors.go.tmpl",
"internal/repository/users.go": "templates/internal-repository-users.go.tmpl", "internal/repository/users.go": "templates/internal-repository-users.go.tmpl",
@@ -109,7 +113,7 @@ func Generate(options Options) error {
if options.FrameworkDir != "" || options.FrameworkVersion == "" { if options.FrameworkDir != "" || options.FrameworkVersion == "" {
replace = frameworkReplace(directory, options.FrameworkDir) replace = frameworkReplace(directory, options.FrameworkDir)
} }
if modules && replace == "" && semver.Compare(version, modulesFrameworkVersion) < 0 && version != installedFrameworkVersion() { if replace == "" && semver.Compare(version, modulesFrameworkVersion) < 0 && version != installedFrameworkVersion() {
return fmt.Errorf("module templates require %s or newer; use --framework-dir for unpublished source", modulesFrameworkVersion) return fmt.Errorf("module templates require %s or newer; use --framework-dir for unpublished source", modulesFrameworkVersion)
} }
data := templateData{ data := templateData{

View File

@@ -39,7 +39,7 @@ func TestGenerateCreatesServiceLayout(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("read generated server: %v", err) t.Fatalf("read generated server: %v", err)
} }
for _, expected := range []string{"config.LoadFromEnv(\"\")", "logx.SetDefault(logger)", "omni.WithConfig(settings)", "servicecontext.New(ctx, logger)", "services.Health.ReadinessHandler()", "application.RunContext(ctx, settings.Address)"} { for _, expected := range []string{"settings.Load(path)", "logx.SetDefault(logger)", "omni.WithConfig(cfg.Server)", "servicecontext.New(ctx, logger, cfg)", "services.Health.ReadinessHandler()", "application.RunContext(ctx, cfg.Server.Address)"} {
if !strings.Contains(string(mainContent), expected) { if !strings.Contains(string(mainContent), expected) {
t.Fatalf("generated server missing %q: %s", expected, mainContent) t.Fatalf("generated server missing %q: %s", expected, mainContent)
} }
@@ -110,7 +110,7 @@ func TestFrameworkReplaceUsesCanonicalPaths(t *testing.T) {
} }
func TestGenerateRemoteProjectPinsVersion(t *testing.T) { func TestGenerateRemoteProjectPinsVersion(t *testing.T) {
for _, version := range []string{"v0.1.0", "v1.2.3-rc.1", "v0.0.0-20260920140624-8b19aa79eea0"} { for _, version := range []string{"v0.22.0-preview", "v1.2.3-rc.1", "v0.22.1-0.20260926000000-8b19aa79eea0"} {
t.Run(version, func(t *testing.T) { t.Run(version, func(t *testing.T) {
project := filepath.Join(t.TempDir(), "billing") project := filepath.Join(t.TempDir(), "billing")
if err := Generate(Options{Dir: project, Module: "example.com/billing", FrameworkVersion: version}); err != nil { if err := Generate(Options{Dir: project, Module: "example.com/billing", FrameworkVersion: version}); err != nil {

View File

@@ -32,46 +32,44 @@ go mod tidy
go run ./cmd/server go run ./cmd/server
``` ```
运行时配置从 `GO_OMNI_*` 环境变量加载,例如: 配置入口是 `internal/settings.Config`、`Default()`、`Validate()` 和 `etc/config.yaml`。
只有文件里的 `${NAME}` / `${NAME:-fallback}` 会读取环境变量,固定值不受进程环境覆盖。生成的文件已声明地址、网络层,以及所选数据库/存储的必要引用;其他参数直接写进文件。
```sh ```yaml
GO_OMNI_ENABLE_LOGGER=true GO_OMNI_LOG_LEVEL=debug go run ./cmd/server server:
address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
transport: "${GO_OMNI_TRANSPORT:-nethttp}"
log:
level: debug
shutdownTimeout: 10s
``` ```
`GO_OMNI_HTTP_ADDR` 可以修改监听地址。生成项目默认提供 `/health/live` 和 用 `-config` 选择一个 YAML/JSON/XML 文件;`-config ''` 只使用 Default();`-check-config` 仅加载和校验,不创建依赖或监听:
`/health/ready`,日志、请求 ID 和优雅退出等基础能力由 GoOmni 统一初始化;业务依赖集中放在项目自己的 `internal/servicecontext`。
服务使用 `signal.NotifyContext` 处理 `SIGINT` / `SIGTERM`。部署时构建并运行服务二进制,
使信号直接到达服务进程:
```sh ```sh
go run ./cmd/server -config etc/config.yaml -check-config
go build -o server ./cmd/server go build -o server ./cmd/server
./server ./server -config etc/config.yaml
``` ```
`GO_OMNI_SHUTDOWN_TIMEOUT` 控制等待请求完成的时间,默认 `10s`。 服务通过 `signal.NotifyContext` 处理 `SIGINT` / `SIGTERM`,先排空 HTTP 请求,再关闭依赖和日志。生成项目默认提供 `/health/live` 和 `/health/ready`,业务依赖集中在 `internal/servicecontext`。
使用支持统一网络配置的框架版本时,设置 `GO_OMNI_TRANSPORT=native` 即可切换为自研 HTTP/1.1; 通过文件中的引用设置 `GO_OMNI_TRANSPORT=native` 可切换自研 HTTP/1.1,默认 nethttp。HTTPS 在文件的 `server.tls.certFile` / `server.tls.keyFile` 成对配置;实验 HTTP/2 选择 `http2` 且必须配置 TLS。不会自动读取未在文件声明的 TLS 环境变量。
默认 `nethttp`。`GO_OMNI_TLS_CERT_FILE` 和 `GO_OMNI_TLS_KEY_FILE` 配对启用 HTTPS。
实验版 HTTP/2 使用 `GO_OMNI_TRANSPORT=http2`,必须同时配置 TLS 证书和私钥。
业务 Handler / Service / Repository 不需要修改;三种适配器共用上述退出等待时间。
{{if eq .Database "mysql"}} {{if eq .Database "mysql"}}
`/health/ready` 已注册数据库探针:连接失败返回 `503`,数据库恢复后重新返回 `200`。 `/health/ready` 已注册数据库探针,连接失败返回 503,恢复后返回 200;`/health/live` 只检查进程存活。连接池参数直接写入 `database`,缺失字段保留 Default():
`/health/live` 始终只检查服务存活。`GO_OMNI_DB_PING_TIMEOUT` 限制初始化检查和数据库探针的等待时间。
退出时先排空 HTTP 请求,再关闭数据库连接池,最后关闭日志。
数据库连接池参数: ```yaml
database:
```text driver: mysql
GO_OMNI_DB_MAX_OPEN_CONNS=10 dsn: "${GO_OMNI_DB_DSN}"
GO_OMNI_DB_MAX_IDLE_CONNS=5 maxOpenConns: 10
GO_OMNI_DB_CONN_MAX_LIFETIME=30m maxIdleConns: 5
GO_OMNI_DB_CONN_MAX_IDLE_TIME=5m connMaxLifetime: 30m
GO_OMNI_DB_PING_TIMEOUT=5s connMaxIdleTime: 5m
pingTimeout: 5s
``` ```
`MAX_IDLE_CONNS=0` 禁用空闲连接,`MAX_OPEN_CONNS=0` 表示不限制连接数。 `maxIdleConns: 0` 禁用空闲连接;此项目要求 maxOpenConns 为正数。项目启用 MySQL parseTime,并按 UTC 读取 DATETIME;JSON 字段保持 lowerCamelCase。
项目会启用 MySQL `parseTime` 并按 UTC 读取 DATETIME 字段,JSON 字段保持 lowerCamelCase。
事务内复用 Repository(在业务代码中导入 `database/sql` 和项目的 `internal/repository`): 事务内复用 Repository(在业务代码中导入 `database/sql` 和项目的 `internal/repository`):

View File

@@ -1,10 +1,9 @@
package settings package settings
import ( import (
"os"
{{if and (not .API) (ne .Storage "none")}}"strings"{{end}} {{if and (not .API) (ne .Storage "none")}}"strings"{{end}}
"time" "time"
"git.yuntuo.hk/k2safe/go-omni/config" "git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/logx" {{if eq .LogOutput "file"}}"git.yuntuo.hk/k2safe/go-omni/logx"{{end}}
{{if eq .Database "mysql"}}mysqladapter "git.yuntuo.hk/k2safe/go-omni/database/mysql"{{end}} {{if eq .Database "mysql"}}mysqladapter "git.yuntuo.hk/k2safe/go-omni/database/mysql"{{end}}
{{if ne .Storage "none"}}"errors" {{if ne .Storage "none"}}"errors"
{{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}} {{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}}
@@ -12,8 +11,8 @@ import (
) )
type Config struct { type Config struct {
Server config.Config `json:"server"` Server config.Config `json:"server"`
{{if eq .Database "mysql"}}Database mysqladapter.Config `json:"database" env:"GO_OMNI_DB"`{{end}} {{if eq .Database "mysql"}}Database mysqladapter.Config `json:"database"`{{end}}
{{if ne .Storage "none"}}Storage StorageConfig `json:"storage" env:"GO_OMNI_STORAGE"`{{end}} {{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}}
} }
func Default() Config { func Default() Config {
server:=config.Default() server:=config.Default()
@@ -35,22 +34,10 @@ func Default() Config {
}{{end}}} }{{end}}}
} }
// Load applies defaults -> file -> typed environment -> validation.
// New fields automatically use GO_OMNI_<JSON_PATH>, or an explicit env tag.
func Load(path string)(Config,error){
cfg:=Default();var err error
cfg,err=config.ApplyDefaults(cfg);if err!=nil{return Config{},err}
if path!="" {cfg,err=config.DecodeFile(path,cfg);if err!=nil{return Config{},err}}
cfg,err=config.OverlayEnv(cfg,config.EnvOptions{Prefix:"GO_OMNI_",Empty:"ignore",Aliases:config.ServerEnvAliases("GO_OMNI_","server")});if err!=nil{return Config{},err}
output,outputSet:=os.LookupEnv("GO_OMNI_SERVER_LOG_OUTPUT");if !outputSet {output=os.Getenv("GO_OMNI_LOG_OUTPUT")}
file,fileSet:=os.LookupEnv("GO_OMNI_SERVER_LOG_FILE");if !fileSet {file=os.Getenv("GO_OMNI_LOG_FILE")}
if output!="" && file=="" {cfg.Server.Log.File="";cfg.Server.Log.Rotation=logx.RotationConfig{}}
return cfg,cfg.Validate()
}
func(c Config)Validate()error{ func(c Config)Validate()error{
if err:=c.Server.Validate();err!=nil{return err} if err:=c.Server.Validate();err!=nil{return config.AtField("server",err)}
{{if eq .Database "mysql"}}if err:=c.Database.Validate();err!=nil{return err}{{end}} {{if eq .Database "mysql"}}if err:=c.Database.Validate();err!=nil{return config.AtField("database",err)}{{end}}
{{if ne .Storage "none"}}if err:=c.Storage.Validate();err!=nil{return err}{{end}} {{if ne .Storage "none"}}if err:=c.Storage.Validate();err!=nil{return config.AtField("storage",err)}{{end}}
return nil return nil
} }
{{if ne .Storage "none"}} {{if ne .Storage "none"}}
@@ -67,12 +54,12 @@ type StorageConfig struct {
ContentTypes []string `json:"contentTypes"` ContentTypes []string `json:"contentTypes"`
LinkTTL time.Duration `json:"linkTtl"` LinkTTL time.Duration `json:"linkTtl"`
RequestTimeout time.Duration `json:"requestTimeout"` RequestTimeout time.Duration `json:"requestTimeout"`
// Secrets are environment-only and cannot be populated by config files. // File values may explicitly reference environment variables. Never log this struct.
FilesToken string `json:"-" env:"GO_OMNI_FILES_TOKEN,secret"` FilesToken string `json:"filesToken"`
SigningKey string `json:"-" env:"GO_OMNI_STORAGE_SIGNING_KEY,secret"` SigningKey string `json:"signingKey"`
AccessKey string `json:"-" env:"GO_OMNI_STORAGE_ACCESS_KEY,secret"` AccessKey string `json:"accessKey"`
SecretKey string `json:"-" env:"GO_OMNI_STORAGE_SECRET_KEY,secret"` SecretKey string `json:"secretKey"`
SessionToken string `json:"-" env:"GO_OMNI_STORAGE_SESSION_TOKEN,secret"` SessionToken string `json:"sessionToken"`
} }
func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}} func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}}
func (s StorageConfig) Validate()error { func (s StorageConfig) Validate()error {
@@ -80,14 +67,14 @@ func (s StorageConfig) Validate()error {
if err:=s.Policy().Validate();err!=nil{return err} if err:=s.Policy().Validate();err!=nil{return err}
if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")} if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")}
if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")} if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")}
{{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("set GO_OMNI_FILES_TOKEN to a token of at least 32 bytes without whitespace")}{{end}} {{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("storage.filesToken requires at least 32 bytes without whitespace")}{{end}}
{{if eq .Storage "local"}} {{if eq .Storage "local"}}
if s.Root=="" {return errors.New("storage.root is required")} if s.Root=="" {return errors.New("storage.root is required")}
u,err:=url.Parse(s.PublicURL) u,err:=url.Parse(s.PublicURL)
if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")} if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")}
if len(s.SigningKey)<32 {return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")} if len(s.SigningKey)<32 {return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")}
{{else}} {{else}}
if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region and GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY are required")} if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region, accessKey and secretKey are required")}
{{if eq .Storage "r2"}} {{if eq .Storage "r2"}}
if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")} if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")}
if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")} if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")}

View File

@@ -0,0 +1,8 @@
package config
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
// Load resolves explicit file references, binds onto Default and validates.
// Ordinary projects only edit Config, Default, Validate and the selected file.
func Load(path string)(Config,error){
loader,err:=NewLoader();if err!=nil{return Config{},err}
return omniConfig.LoadProject(path,Default(),loader)
}

View File

@@ -0,0 +1,7 @@
package config
import omniConfig "git.yuntuo.hk/k2safe/go-omni/config"
// NewLoader is project-owned. JSON/YAML/XML work without changes. Register
// additional format decoders here only when the project needs them.
func NewLoader() (*omniConfig.Loader,error) { return omniConfig.NewLoader() }

View File

@@ -1,57 +0,0 @@
package main
import (
"context"
"log"
"os"
"os/signal"
"strings"
"syscall"
omni "git.yuntuo.hk/k2safe/go-omni"
"git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/logx"
"{{.Module}}/internal/handler"
"{{.Module}}/internal/service"
"{{.Module}}/internal/servicecontext"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
settings, err := config.LoadFromEnv("")
if err != nil {
return err
}
if strings.TrimSpace(settings.Log.Service) == "" {
settings.Log.Service = "{{.Module}}"
}
logger, err := logx.New(settings.Log)
if err != nil {
return err
}
defer logger.Close()
// Access logs and business logs share the same configured sink.
logx.SetDefault(logger)
services, err := servicecontext.New(ctx, logger)
if err != nil { return err }
defer services.Close()
userService := service.NewUsers(services)
userHandler := handler.NewUsers(userService)
application := omni.New(omni.WithConfig(settings))
application.OnShutdown(logger.Close)
application.OnShutdown(services.Close)
application.GET("/health/live", services.Health.LivenessHandler())
application.GET("/health/ready", services.Health.ReadinessHandler())
userHandler.RegisterRoutes(application)
return application.RunContext(ctx, settings.Address)
}

View File

@@ -13,6 +13,7 @@ import (
"git.yuntuo.hk/k2safe/go-omni/database" "git.yuntuo.hk/k2safe/go-omni/database"
"git.yuntuo.hk/k2safe/go-omni/logx" "git.yuntuo.hk/k2safe/go-omni/logx"
"{{.Module}}/internal/model" "{{.Module}}/internal/model"
"{{.Module}}/internal/settings"
"{{.Module}}/internal/repository" "{{.Module}}/internal/repository"
"{{.Module}}/internal/servicecontext" "{{.Module}}/internal/servicecontext"
) )
@@ -23,15 +24,12 @@ func databaseContext(t *testing.T) *servicecontext.Context {
t.Helper() t.Helper()
dsn := os.Getenv("GO_OMNI_TEST_MYSQL_DSN") dsn := os.Getenv("GO_OMNI_TEST_MYSQL_DSN")
if dsn == "" { t.Skip("set GO_OMNI_TEST_MYSQL_DSN to a dedicated test database") } if dsn == "" { t.Skip("set GO_OMNI_TEST_MYSQL_DSN to a dedicated test database") }
for _, key := range []string{"MAX_OPEN_CONNS", "MAX_IDLE_CONNS", "CONN_MAX_LIFETIME", "CONN_MAX_IDLE_TIME", "PING_TIMEOUT"} {
t.Setenv("GO_OMNI_DB_"+key, "")
}
t.Setenv("GO_OMNI_DB_DSN", dsn) t.Setenv("GO_OMNI_DB_DSN", dsn)
t.Setenv("GO_OMNI_DB_DRIVER", "mysql")
logger, err := logx.New(logx.Config{Writer: io.Discard}) logger, err := logx.New(logx.Config{Writer: io.Discard})
if err != nil { t.Fatal(err) } if err != nil { t.Fatal(err) }
t.Cleanup(func() { _ = logger.Close() }) t.Cleanup(func() { _ = logger.Close() })
services, err := servicecontext.New(context.Background(), logger) cfg,err:=settings.Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
services, err := servicecontext.New(context.Background(), logger,cfg)
if err != nil { t.Fatal(err) } if err != nil { t.Fatal(err) }
t.Cleanup(func() { _ = services.Close() }) t.Cleanup(func() { _ = services.Close() })
return services return services

View File

@@ -2,6 +2,7 @@ package servicecontext
import ( import (
"context" "context"
"{{.Module}}/internal/settings"
{{if and .Modules (ne .Storage "none")}}"errors" {{if and .Modules (ne .Storage "none")}}"errors"
"net/http" "net/http"
"sync" "sync"
@@ -38,16 +39,16 @@ type Context struct {
// New initializes the project's dependencies before the HTTP listener opens. // New initializes the project's dependencies before the HTTP listener opens.
// The caller owns Logger; Close releases only resources created here. // The caller owns Logger; Close releases only resources created here.
func New(ctx context.Context, logger *logx.Logger) (*Context, error) { func New(ctx context.Context, logger *logx.Logger, project settings.Config) (*Context, error) {
if ctx == nil { ctx = context.Background() } if ctx == nil { ctx = context.Background() }
if err := ctx.Err(); err != nil { return nil, err } if err := ctx.Err(); err != nil { return nil, err }
checker, err := health.New() checker, err := health.New()
if err != nil { return nil, err } if err != nil { return nil, err }
services := &Context{Logger: logger, Health: checker} services := &Context{Logger: logger, Health: checker}
{{if eq .Database "mysql"}} {{if eq .Database "mysql"}}
cfg, err := database.ConfigFromEnv("") cfg:=project.Database
if err != nil { return nil, err } if err:=cfg.Validate();err!=nil{return nil,err}
if cfg.Driver != "mysql" { return nil, fmt.Errorf("this project requires GO_OMNI_DB_DRIVER=mysql") } if cfg.Driver != "mysql" { return nil, fmt.Errorf("this project requires database.driver=mysql") }
dsn, err := mysql.ParseDSN(cfg.DSN) dsn, err := mysql.ParseDSN(cfg.DSN)
if err != nil { return nil, fmt.Errorf("parse mysql connection settings: %w", err) } if err != nil { return nil, fmt.Errorf("parse mysql connection settings: %w", err) }
if dsn.DBName == "" { return nil, fmt.Errorf("mysql connection settings require a database name") } if dsn.DBName == "" { return nil, fmt.Errorf("mysql connection settings require a database name") }

View File

@@ -2,7 +2,7 @@
本项目选用日志输出 `{{.LogOutput}}`、格式 `{{.LogFormat}}`、对象存储 `{{.Storage}}`。数据库选择 `{{.Database}}` 独立于对象存储。 本项目选用日志输出 `{{.LogOutput}}`、格式 `{{.LogFormat}}`、对象存储 `{{.Storage}}`。数据库选择 `{{.Database}}` 独立于对象存储。
配置入口为 `etc/config.yaml`,通过 `go run ./cmd/server -config /path/to/config.yaml` 指定其他 JSON/YAML 文件;`-config ''` 只用代码默认值和环境变量。优先级:默认值 → 文件 → 非空环境变量。未知字段、无效参数会在监听前拒绝。 配置入口为 `etc/config.yaml`,通过 `go run ./cmd/server -config /path/to/config.yaml` 指定其他 JSON/YAML/XML 文件;`-config ''` 只用代码默认值。文件用 `${NAME}` 或 `${NAME:-fallback}` 显式引用变量,固定值不受隐式环境覆盖;`-check-config` 只检查、不初始化依赖。未知字段、无效参数会在监听前拒绝。
```sh ```sh
go mod tidy go mod tidy
@@ -24,7 +24,7 @@ go run ./cmd/server
{{if eq .Database "mysql"}}还需按 README 配置独立 MySQL 数据库并应用 schema.sql;应用不自动迁移。{{end}} {{if eq .Database "mysql"}}还需按 README 配置独立 MySQL 数据库并应用 schema.sql;应用不自动迁移。{{end}}
默认仅监听 `127.0.0.1:8080`。`GO_OMNI_HTTP_ADDR` 覆盖地址;本地签名 URL 的外部地址通过 `GO_OMNI_STORAGE_PUBLIC_URL` 单独配置,始终以 `/storage/object` 结尾,不使用请求 Host 自动拼接可信链接。 默认仅监听 `127.0.0.1:8080`。文件中的 `${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}` 设置地址;本地签名 URL 的外部地址通过 `GO_OMNI_STORAGE_PUBLIC_URL` 单独配置,始终以 `/storage/object` 结尾,不使用请求 Host 自动拼接可信链接。
## 日志 ## 日志
@@ -35,7 +35,7 @@ curl -H 'Content-Type: application/json' -H 'X-Request-ID: example-request' \
-d '{"name":"Example","email":"example@example.com"}' http://127.0.0.1:8080/users -d '{"name":"Example","email":"example@example.com"}' http://127.0.0.1:8080/users
``` ```
`GO_OMNI_LOG_LEVEL`、`GO_OMNI_LOG_FORMAT`、`GO_OMNI_LOG_FILE` 覆盖对应项;显式设置 `GO_OMNI_LOG_OUTPUT=stdout` 且未设置 LOG_FILE 时,切换到控制台并关闭文件轮转。慢请求阈值为 500ms,可通过 `GO_OMNI_SLOW_REQUEST_THRESHOLD` 调整。字段脱敏不扫描 msg/error 中的自由文本;不要把凭据或签名 URL 拼到日志中。 日志参数直接写在 `server.log`,例如 `level: debug`;也可显式写 `level: "${LOG_LEVEL:-info}"`。切换文件日志为控制台时,明确设置 `file: ""`、`output: stdout` 和 `rotation.enabled: false`,不会隐式重置其他字段。慢请求阈值使用 `server.slowRequestThreshold`,默认 500ms。字段脱敏不扫描 msg/error 自由文本;不要把凭据或签名 URL 拼到日志中。
{{if ne .Storage "none"}}## 文件示例 {{if ne .Storage "none"}}## 文件示例
@@ -66,18 +66,18 @@ curl -H "Authorization: Bearer $GO_OMNI_FILES_TOKEN" -H 'Content-Type: applicati
响应包括随机 `data.key` 和 `data.request`(method/url/headers/expiresAt)。按返回的 method 和 headers 用原始文件字节上传到 URL;不能额外包 FormData。浏览器自动填写 Content-Length,其余签名 Header 原样发送;云桶需配置允许业务域名的 CORS。云端直传只绑定声明类型,实际内容需上传后验证;本地和 API 中转上传会检查允许类型及前缀嗅探结果。未完成内容验证前不要把暂存文件当作可信文件公开。 响应包括随机 `data.key` 和 `data.request`(method/url/headers/expiresAt)。按返回的 method 和 headers 用原始文件字节上传到 URL;不能额外包 FormData。浏览器自动填写 Content-Length,其余签名 Header 原样发送;云桶需配置允许业务域名的 CORS。云端直传只绑定声明类型,实际内容需上传后验证;本地和 API 中转上传会检查允许类型及前缀嗅探结果。未完成内容验证前不要把暂存文件当作可信文件公开。
| 参数 | 默认值 / 环境变量 | | 文件字段 | 默认值 / 含义 |
| --- | --- | | --- | --- |
| 大小 | 32 MiB / GO_OMNI_STORAGE_MAX_BYTES | | `storage.maxBytes` | 32 MiB |
| 类型 | text/plain、image/png、image/jpeg、application/pdf / GO_OMNI_STORAGE_CONTENT_TYPES(逗号分隔) | | `storage.contentTypes` | text/plain、image/png、image/jpeg、application/pdf;文件数组 |
| 链接有效期 | 5m / GO_OMNI_STORAGE_LINK_TTL(1s..1h) | | `storage.linkTtl` | 5m,允许 1s..1h |
| 请求超时 | 30s / GO_OMNI_STORAGE_REQUEST_TIMEOUT(>0 且 <=5m) | | `storage.requestTimeout` | 30s,大于 0 且不超过 5m |
| 本地目录 | var/files / GO_OMNI_STORAGE_ROOT | | `storage.root` | var/files |
| 本地外部 URL | http://127.0.0.1:8080/storage/object / GO_OMNI_STORAGE_PUBLIC_URL | | `storage.publicUrl` | http://127.0.0.1:8080/storage/object |
| 云配置 | GO_OMNI_STORAGE_ENDPOINT、REGION、BUCKET、ACCOUNT_ID、SECURE | | `storage.endpoint/region/bucket/accountId/secure` | 对应平台连接配置 |
| 环境密钥 | GO_OMNI_FILES_TOKEN、GO_OMNI_STORAGE_SIGNING_KEY、GO_OMNI_STORAGE_ACCESS_KEY、SECRET_KEY、SESSION_TOKEN | | `storage.filesToken/signingKey/accessKey/secretKey/sessionToken` | 文件显式引用环境变量提供凭据 |
全部 JSON/YAML 字段使用 lowerCamelCase。密钥字段仅允许环境变量,配置文件中写入会报未知字段;不要把运行环境文件提交到仓库。S3 的 endpoint 为 host[:port],OSS 的可选 endpoint 为 HTTPS origin;R2 从 accountId 派生官方 endpoint,region 为 auto。仅 S3/MinIO 的本地测试可设置 SECURE=false。 全部 JSON/YAML/XML 字段使用 lowerCamelCase。文件可使用固定值或显式环境引用,不按字段名自动推导环境变量;凭据建议使用 `${NAME}`,不把含秘密的配置提交到仓库。S3 endpoint 为 host[:port],OSS 可选 endpoint 为 HTTPS origin;R2 从 accountId 派生官方 endpoint,region 为 auto。仅 S3/MinIO 本地测试可设置 `storage.secure: false`。
错误响应:未授权 401、无 Content-Length 411、超限 413、类型不允许 415、长度不符/非法 key 400、对象不存在 404、存储超时 504,其余内部错误统一隐藏细节。底层签名入口遵循存储协议返回 201/403 等状态,流式传输且访问日志保留实际状态。 错误响应:未授权 401、无 Content-Length 411、超限 413、类型不允许 415、长度不符/非法 key 400、对象不存在 404、存储超时 504,其余内部错误统一隐藏细节。底层签名入口遵循存储协议返回 201/403 等状态,流式传输且访问日志保留实际状态。

View File

@@ -1,68 +1,30 @@
# Non-secret settings only. Environment variables override these values. # Only fields listed here override Default(). Environment names are explicit.
server: server:
address: '127.0.0.1:8080' # Optional cross-origin access: server.cors (disabled by default). See framework cors/README.md.
enableLogger: true address: "${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}"
slowRequestThreshold: 500ms transport: "${GO_OMNI_TRANSPORT:-nethttp}"
shutdownTimeout: 10s
{{if .API}} lifecycle:
stopTimeout: 5s
drainTimeout: 10s
cancelTimeout: 5s
endpoints:
liveness:
enabled: true
path: /health/live
responseMode: app
readiness:
enabled: true
path: /health/ready
responseMode: app
openapi:
enabled: true
path: /openapi.json
{{end}}
log:
service: {{printf "%q" .Module}}
level: info
format: {{.LogFormat}}
{{if eq .LogOutput "file"}} file: './var/log/server.log'
rotation:
enabled: true
maxSizeMb: 100
maxBackups: 10
maxAgeDays: 7
compress: true
{{else}} output: {{.LogOutput}}
{{end}} redactKeys: []
{{if ne .Storage "none"}}storage: {{if ne .Storage "none"}}storage:
provider: {{.Storage}} {{if not .API}} filesToken: "${GO_OMNI_FILES_TOKEN}"
maxBytes: 33554432 {{end}}{{if eq .Storage "local"}} root: "${GO_OMNI_STORAGE_ROOT:-var/files}"
contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf'] publicUrl: "${GO_OMNI_STORAGE_PUBLIC_URL:-http://127.0.0.1:8080/storage/object}"
linkTtl: 5m signingKey: "${GO_OMNI_STORAGE_SIGNING_KEY}"
requestTimeout: 30s {{else}} bucket: "${GO_OMNI_STORAGE_BUCKET}"
{{if eq .Storage "local"}} root: './var/files' accessKey: "${GO_OMNI_STORAGE_ACCESS_KEY}"
publicUrl: 'http://127.0.0.1:8080/storage/object' secretKey: "${GO_OMNI_STORAGE_SECRET_KEY}"
{{else}} bucket: '' # set GO_OMNI_STORAGE_BUCKET sessionToken: "${GO_OMNI_STORAGE_SESSION_TOKEN:-}"
secure: true {{if eq .Storage "r2"}} accountId: "${GO_OMNI_STORAGE_ACCOUNT_ID}"
{{if eq .Storage "r2"}} accountId: '' # set GO_OMNI_STORAGE_ACCOUNT_ID {{else if eq .Storage "s3"}} endpoint: "${GO_OMNI_STORAGE_ENDPOINT}"
region: auto region: "${GO_OMNI_STORAGE_REGION:-us-east-1}"
{{else if eq .Storage "s3"}} endpoint: '' # set GO_OMNI_STORAGE_ENDPOINT (host[:port]) {{else}} region: "${GO_OMNI_STORAGE_REGION:-cn-hangzhou}"
region: us-east-1
{{else}} region: cn-hangzhou
endpoint: '' # empty uses the official regional endpoint
{{end}}{{end}}{{end}} {{end}}{{end}}{{end}}
{{if and .API (eq .Database "mysql")}}database: {{if and .API (eq .Database "mysql")}}database:
host: 127.0.0.1 host: "${GO_OMNI_DB_HOST:-127.0.0.1}"
port: 3306 port: "${GO_OMNI_DB_PORT:-3306}"
database: '' # GO_OMNI_DB_DATABASE database: "${GO_OMNI_DB_DATABASE}"
user: '' # GO_OMNI_DB_USER; password via GO_OMNI_DB_PASSWORD user: "${GO_OMNI_DB_USER}"
tls: password: "${GO_OMNI_DB_PASSWORD}"
mode: required {{end}}
pool: {{if and (not .API) (eq .Database "mysql")}}database:
maxOpenConns: 10 driver: mysql
maxIdleConns: 5 dsn: "${GO_OMNI_DB_DSN}"
connMaxLifetime: 30m
connMaxIdleTime: 5m
pingTimeout: 5s
{{end}} {{end}}

View File

@@ -18,8 +18,10 @@ import (
) )
func main(){ func main(){
path:=flag.String("config","etc/config.yaml","JSON/YAML settings; empty uses defaults and environment") path:=flag.String("config","etc/config.yaml","configuration file; empty uses only Default()")
check:=flag.Bool("check-config",false,"validate without starting dependencies")
flag.Parse() flag.Parse()
if *check {if _,err:=settings.Load(*path);err!=nil{log.Fatal(err)};return}
if err:=run(*path);err!=nil{log.Fatal(err)} if err:=run(*path);err!=nil{log.Fatal(err)}
} }
func run(path string)(runErr error){ func run(path string)(runErr error){
@@ -31,7 +33,7 @@ func run(path string)(runErr error){
if err!=nil{return err} if err!=nil{return err}
defer func(){runErr=errors.Join(runErr,logger.Close())}() defer func(){runErr=errors.Join(runErr,logger.Close())}()
logx.SetDefault(logger) logx.SetDefault(logger)
services,err:=servicecontext.New(ctx,logger) services,err:=servicecontext.New(ctx,logger,cfg)
if err!=nil{return err} if err!=nil{return err}
defer func(){runErr=errors.Join(runErr,services.Close())}() defer func(){runErr=errors.Join(runErr,services.Close())}()
{{if ne .Storage "none"}}if err=services.InitStorage(ctx,cfg.Storage);err!=nil{return err}{{end}} {{if ne .Storage "none"}}if err=services.InitStorage(ctx,cfg.Storage);err!=nil{return err}{{end}}

View File

@@ -3,21 +3,19 @@ package settings
import ( import (
"errors" "errors"
"os"
"strings" "strings"
"time" "time"
{{if ne .Storage "none"}}"fmt" {{if ne .Storage "none"}}
{{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}} {{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}}
"strconv"
"git.yuntuo.hk/k2safe/go-omni/storage"{{end}} "git.yuntuo.hk/k2safe/go-omni/storage"{{end}}
{{if and .API (eq .Database "mysql")}}"git.yuntuo.hk/k2safe/go-omni/database"{{end}} {{if eq .Database "mysql"}}"git.yuntuo.hk/k2safe/go-omni/database"{{end}}
"git.yuntuo.hk/k2safe/go-omni/config" "git.yuntuo.hk/k2safe/go-omni/config"
"git.yuntuo.hk/k2safe/go-omni/logx" {{if eq .LogOutput "file"}}"git.yuntuo.hk/k2safe/go-omni/logx"{{end}}
) )
type Config struct { type Config struct {
Server config.Config `json:"server"` Server config.Config `json:"server"`
{{if and .API (eq .Database "mysql")}}Database database.Config `json:"database"`{{end}} {{if eq .Database "mysql"}}Database database.Config `json:"database"`{{end}}
{{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}} {{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}}
} }
@@ -31,7 +29,8 @@ func Default() Config {
{{if eq .LogOutput "file"}}server.Log.File="./var/log/server.log" {{if eq .LogOutput "file"}}server.Log.File="./var/log/server.log"
server.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:100,MaxBackups:10,MaxAgeDays:7,Compress:true} server.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:100,MaxBackups:10,MaxAgeDays:7,Compress:true}
{{else}}server.Log.Output={{printf "%q" .LogOutput}}{{end}} {{else}}server.Log.Output={{printf "%q" .LogOutput}}{{end}}
return Config{Server:server{{if and .API (eq .Database "mysql")}},Database:database.DefaultConfig(){{end}}{{if ne .Storage "none"}},Storage:StorageConfig{ {{if eq .Database "mysql"}}db:=database.DefaultConfig();db.Driver="mysql"{{end}}
return Config{Server:server{{if eq .Database "mysql"}},Database:db{{end}}{{if ne .Storage "none"}},Storage:StorageConfig{
Provider:{{printf "%q" .Storage}}, Root:"./var/files", PublicURL:"http://127.0.0.1:8080/storage/object", Provider:{{printf "%q" .Storage}}, Root:"./var/files", PublicURL:"http://127.0.0.1:8080/storage/object",
Secure:true, MaxBytes:32<<20, ContentTypes:[]string{"text/plain","image/png","image/jpeg","application/pdf"}, Secure:true, MaxBytes:32<<20, ContentTypes:[]string{"text/plain","image/png","image/jpeg","application/pdf"},
LinkTTL:5*time.Minute, RequestTimeout:30*time.Second, LinkTTL:5*time.Minute, RequestTimeout:30*time.Second,
@@ -41,27 +40,10 @@ func Default() Config {
}{{end}}} }{{end}}}
} }
// Load uses defaults -> strict YAML/JSON -> nonempty environment variables. // Load uses Default(), the selected file's explicit references, and Validate.
// Pass an empty path to use only defaults and environment. func Load(path string)(Config,error){return config.LoadProject(path,Default())}
func Load(path string)(Config,error) {
cfg:=Default()
var err error
if path!="" { cfg,err=config.DecodeFile(path,cfg);if err!=nil{return Config{},err} }
cfg.Server,err=config.ApplyEnv(cfg.Server,"")
if err!=nil{return Config{},err}
// A stream explicitly selected by environment overrides a file profile.
if output:=strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT"));output!="" && strings.TrimSpace(os.Getenv("GO_OMNI_LOG_FILE"))=="" {
cfg.Server.Log.File="";cfg.Server.Log.Rotation=logx.RotationConfig{}
}
{{if ne .Storage "none"}}
if err=cfg.Storage.applyEnv();err!=nil{return Config{},err}
{{end}}
{{if and .API (eq .Database "mysql")}}cfg.Database,err=database.ApplyEnv(cfg.Database,"");if err!=nil{return Config{},err}{{end}}
if err=cfg.Validate();err!=nil{return Config{},err}
return cfg,nil
}
func (c Config) Validate()error { func (c Config) Validate()error {
{{if and .API (eq .Database "mysql")}} {{if eq .Database "mysql"}}
if err:=c.Database.Validate();err!=nil{return err} if err:=c.Database.Validate();err!=nil{return err}
if c.Database.Driver!="mysql" || c.Database.MaxOpenConns<1 {return errors.New("database requires driver mysql and a positive maxOpenConns")} if c.Database.Driver!="mysql" || c.Database.MaxOpenConns<1 {return errors.New("database requires driver mysql and a positive maxOpenConns")}
{{end}} {{end}}
@@ -85,45 +67,27 @@ type StorageConfig struct {
ContentTypes []string `json:"contentTypes"` ContentTypes []string `json:"contentTypes"`
LinkTTL time.Duration `json:"linkTtl"` LinkTTL time.Duration `json:"linkTtl"`
RequestTimeout time.Duration `json:"requestTimeout"` RequestTimeout time.Duration `json:"requestTimeout"`
// Secrets are environment-only and cannot be populated by config files. // Use explicit environment references in the configuration file for secrets.
FilesToken string `json:"-"` FilesToken string `json:"filesToken"`
SigningKey string `json:"-"` SigningKey string `json:"signingKey"`
AccessKey string `json:"-"` AccessKey string `json:"accessKey"`
SecretKey string `json:"-"` SecretKey string `json:"secretKey"`
SessionToken string `json:"-"` SessionToken string `json:"sessionToken"`
} }
func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}} func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}}
func (s *StorageConfig) applyEnv()error {
values:=map[string]*string{
"GO_OMNI_STORAGE_ROOT":&s.Root,"GO_OMNI_STORAGE_PUBLIC_URL":&s.PublicURL,
"GO_OMNI_STORAGE_ENDPOINT":&s.Endpoint,"GO_OMNI_STORAGE_REGION":&s.Region,
"GO_OMNI_STORAGE_BUCKET":&s.Bucket,"GO_OMNI_STORAGE_ACCOUNT_ID":&s.AccountID,
"GO_OMNI_STORAGE_SIGNING_KEY":&s.SigningKey,"GO_OMNI_FILES_TOKEN":&s.FilesToken,
"GO_OMNI_STORAGE_ACCESS_KEY":&s.AccessKey,"GO_OMNI_STORAGE_SECRET_KEY":&s.SecretKey,
"GO_OMNI_STORAGE_SESSION_TOKEN":&s.SessionToken,
}
for name,target:=range values {if value:=strings.TrimSpace(os.Getenv(name));value!=""{*target=value}}
for name,target:=range map[string]*time.Duration{"GO_OMNI_STORAGE_LINK_TTL":&s.LinkTTL,"GO_OMNI_STORAGE_REQUEST_TIMEOUT":&s.RequestTimeout} {
if value:=strings.TrimSpace(os.Getenv(name));value!="" {v,err:=time.ParseDuration(value);if err!=nil{return fmt.Errorf("%s must be a duration",name)};*target=v}
}
if value:=os.Getenv("GO_OMNI_STORAGE_MAX_BYTES");value!=""{v,err:=strconv.ParseInt(value,10,64);if err!=nil{return errors.New("GO_OMNI_STORAGE_MAX_BYTES must be an integer")};s.MaxBytes=v}
if value:=os.Getenv("GO_OMNI_STORAGE_SECURE");value!=""{v,err:=strconv.ParseBool(value);if err!=nil{return errors.New("GO_OMNI_STORAGE_SECURE must be a boolean")};s.Secure=v}
if value:=strings.TrimSpace(os.Getenv("GO_OMNI_STORAGE_CONTENT_TYPES"));value!="" {s.ContentTypes=strings.Split(value,",");for i:=range s.ContentTypes{s.ContentTypes[i]=strings.TrimSpace(s.ContentTypes[i])}}
return nil
}
func (s StorageConfig) Validate()error { func (s StorageConfig) Validate()error {
if s.Provider!={{printf "%q" .Storage}} {return errors.New("storage.provider must match the adapter selected at generation")} if s.Provider!={{printf "%q" .Storage}} {return errors.New("storage.provider must match the adapter selected at generation")}
if err:=s.Policy().Validate();err!=nil{return err} if err:=s.Policy().Validate();err!=nil{return err}
if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")} if s.LinkTTL<time.Second || s.LinkTTL>time.Hour{return errors.New("storage.linkTtl must be 1s..1h")}
if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")} if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")}
{{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("set GO_OMNI_FILES_TOKEN to a token of at least 32 bytes without whitespace")}{{end}} {{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("storage.filesToken requires at least 32 bytes without whitespace")}{{end}}
{{if eq .Storage "local"}} {{if eq .Storage "local"}}
if s.Root=="" {return errors.New("storage.root is required")} if s.Root=="" {return errors.New("storage.root is required")}
u,err:=url.Parse(s.PublicURL) u,err:=url.Parse(s.PublicURL)
if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")} if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")}
if len(s.SigningKey)<32 {return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")} if len(s.SigningKey)<32 {return errors.New("storage.signingKey requires a stable secret of at least 32 bytes")}
{{else}} {{else}}
if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region and GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY are required")} if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region, accessKey and secretKey are required")}
{{if eq .Storage "r2"}} {{if eq .Storage "r2"}}
if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")} if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")}
if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")} if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")}

View File

@@ -21,18 +21,18 @@ func environment(t *testing.T){
} }
func TestSettingsPrecedenceAndStrictness(t *testing.T){ func TestSettingsPrecedenceAndStrictness(t *testing.T){
environment(t) environment(t)
{{if eq .Database "mysql"}}t.Setenv("GO_OMNI_DB_DSN","test:secret@tcp(localhost)/demo"){{end}}
source,err:=os.ReadFile("../../etc/config.yaml");if err!=nil{t.Fatal(err)}
path:=filepath.Join(t.TempDir(),"config.yaml") path:=filepath.Join(t.TempDir(),"config.yaml")
if err:=os.WriteFile(path,[]byte("server:\n address: '127.0.0.1:8123'\n log:\n level: debug\n"),0600);err!=nil{t.Fatal(err)} source=[]byte(strings.Replace(string(source),"${GO_OMNI_HTTP_ADDR:-127.0.0.1:8080}","127.0.0.1:8123",1))
if err=os.WriteFile(path,source,0600);err!=nil{t.Fatal(err)}
t.Setenv("GO_OMNI_HTTP_ADDR","127.0.0.1:9123") t.Setenv("GO_OMNI_HTTP_ADDR","127.0.0.1:9123")
t.Setenv("GO_OMNI_LOG_OUTPUT","stdout") t.Setenv("GO_OMNI_LOG_OUTPUT","stdout")
cfg,err:=Load(path);if err!=nil{t.Fatal(err)} cfg,err:=Load(path);if err!=nil{t.Fatal(err)}
if cfg.Server.Address!="127.0.0.1:9123" || cfg.Server.Log.Level!="debug" || cfg.Server.Log.Output!="stdout" || cfg.Server.Log.File!="" || cfg.Server.Log.Rotation.Enabled {t.Fatal("configuration precedence failed")} initial:=Default()
for _,body:=range []string{"unexpected: true","server:\n slowRequestThreshold: -1s","server:\n log:\n format: invalid"{{if ne .Storage "none"}},"storage:\n filesToken: forbidden","storage:\n maxBytes: -1","storage:\n provider: unsupported"{{end}}}{ if cfg.Server.Address!="127.0.0.1:8123"||cfg.Server.Log.Output!=initial.Server.Log.Output||cfg.Server.Log.File!=initial.Server.Log.File {t.Fatal("implicit overwrite")}
if err:=os.WriteFile(path,[]byte(body),0600);err!=nil{t.Fatal(err)} for _,body:=range []string{"unexpected: true","server:\n slowRequestThreshold: -1s","server:\n log:\n format: invalid"}{
if _,err:=Load(path);err==nil{t.Fatal("invalid configuration accepted")} if err=os.WriteFile(path,[]byte(body),0600);err!=nil{t.Fatal(err)}
if _,err=Load(path);err==nil{t.Fatal("invalid configuration accepted")}
} }
{{if ne .Storage "none"}}
t.Setenv("GO_OMNI_FILES_TOKEN","")
if _,err:=Load("");err==nil{t.Fatal("missing file token accepted")}
{{end}}
} }

View File

@@ -10,11 +10,11 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni"
// Source builds can carry an unpublished VCS version (including +dirty). // Source builds can carry an unpublished VCS version (including +dirty).
// Keep this fallback in sync with the next release; installed CLIs use their // Keep this fallback in sync with the next release; installed CLIs use their
// downloaded module version, which also has a module checksum. // downloaded module version, which also has a module checksum.
const defaultFrameworkVersion = "v0.18.0-preview" const defaultFrameworkVersion = "v0.26.0-preview"
// This target is not a claim that the release already exists. Source users // This target is not a claim that the release already exists. Source users
// select a local framework checkout until the module templates are published. // select a local framework checkout until the module templates are published.
const modulesFrameworkVersion = "v0.7.0-preview" const modulesFrameworkVersion = "v0.22.0-preview"
func installedFrameworkVersion() string { func installedFrameworkVersion() string {
info, _ := debug.ReadBuildInfo() info, _ := debug.ReadBuildInfo()

View File

@@ -1,66 +0,0 @@
package config
import (
"strings"
"testing"
"time"
)
func TestUnifiedEnvAliases(t *testing.T) {
type project struct {
Server Config `json:"server"`
Label string `json:"label"`
}
base := project{Server: Default(), Label: "base"}
for _, policy := range []string{"allow", "ignore", "reject"} {
t.Run(policy, func(t *testing.T) {
env := map[string]string{"TEST_HTTP_ADDR": "127.0.0.1:8099", "TEST_LOG_SERVICE": "", "TEST_SERVICE": "", "TEST_LABEL": ""}
opts := EnvOptions{Prefix: "TEST_", Empty: policy, Aliases: ServerEnvAliases("TEST_", "server"), Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}
got, err := OverlayEnv(base, opts)
if policy == "reject" {
if err == nil || !strings.Contains(err.Error(), "empty") {
t.Fatal(err)
}
return
}
if err != nil {
t.Fatal(err)
}
if got.Server.Address != "127.0.0.1:8099" {
t.Fatal("alias ignored")
}
if policy == "allow" && got.Label != "" || policy == "ignore" && got.Label != "base" {
t.Fatal("empty policy differs")
}
env["TEST_SERVER_ADDRESS"] = "127.0.0.1:9001"
env["TEST_HTTP_READ_TIMEOUT"] = "3s"
env["TEST_ENABLE_LOGGER"] = "false"
env["TEST_TRUSTED_PROXIES"] = `["127.0.0.1"]`
got, err = OverlayEnv(base, opts)
if err != nil || got.Server.Address != "127.0.0.1:9001" || got.Server.HTTP.ReadTimeout != 3*time.Second || got.Server.EnableLogger || len(got.Server.TrustedProxies) != 1 {
t.Fatal(got, err)
}
env["TEST_SERVER_ADDRESS"] = ""
got, err = OverlayEnv(base, opts)
if err != nil {
t.Fatal(err)
}
expected := ""
if policy == "ignore" {
expected = base.Server.Address
}
if got.Server.Address != expected {
t.Fatal("empty canonical unexpectedly fell through to alias")
}
})
}
if base.Server.Address != ":8080" || base.Label != "base" {
t.Fatal("defaults mutated")
}
}
func TestAliasTypeErrorsDoNotExposeValue(t *testing.T) {
opts := EnvOptions{Prefix: "X_", Empty: "reject", Aliases: ServerEnvAliases("X_", ""), Lookup: func(k string) (string, bool) { return "private-invalid", k == "X_HTTP_READ_TIMEOUT" }}
if _, err := OverlayEnv(Default(), opts); err == nil || strings.Contains(err.Error(), "private-invalid") {
t.Fatal(err)
}
}

336
config/bind.go Normal file
View File

@@ -0,0 +1,336 @@
package config
import (
"encoding/json"
"fmt"
"reflect"
"sort"
"strings"
)
var jsonUnmarshalerType = reflect.TypeOf((*json.Unmarshaler)(nil)).Elem()
type documentBinder struct {
doc Document
nodes int
}
func (b *documentBinder) step(path string, depth int) error {
b.nodes++
if depth > maxConfigDepth || b.nodes > maxConfigNodes {
return b.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil)
}
return nil
}
func bindDocument[T any](doc Document, defaults T) (T, error) {
var zero T
base := reflect.ValueOf(&defaults).Elem()
if base.Kind() != reflect.Struct && (base.Kind() != reflect.Pointer || base.Type().Elem().Kind() != reflect.Struct) {
return zero, &Error{Reason: "project configuration must be a struct"}
}
result, err := cloneConfiguration(base, 0)
if err != nil {
return zero, &Error{Reason: "cannot clone configuration defaults", cause: err}
}
b := documentBinder{doc: doc}
if err = b.bind(doc.Value, result, "", 0, false); err != nil {
return zero, err
}
return result.Interface().(T), nil
}
func (b *documentBinder) decode(v any, dst reflect.Value, path string) (err error) {
defer func() {
if recover() != nil {
err = b.doc.diagnostic(path, "custom value decoder panicked", nil)
}
}()
raw, err := json.Marshal(v)
if err != nil {
return b.doc.diagnostic(path, "unsupported configuration value", err)
}
if err = json.Unmarshal(raw, dst.Addr().Interface()); err != nil {
return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err)
}
return nil
}
func (b *documentBinder) bind(v any, dst reflect.Value, path string, depth int, quoted bool) error {
if err := b.step(path, depth); err != nil {
return err
}
if text, ok := v.(resolvedText); ok {
if dst.Kind() != reflect.Pointer && dst.Type() != durationType && dst.CanAddr() && !dst.Addr().Type().Implements(jsonUnmarshalerType) && !dst.Addr().Type().Implements(textType) {
switch dst.Kind() {
case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64:
if err := parseTextValue(dst, string(text)); err != nil {
return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err)
}
return nil
}
}
if dst.Kind() != reflect.Pointer {
v = string(text)
}
}
if v == nil {
return b.decode(nil, dst, path)
} // Preserve encoding/json null semantics.
if dst.Kind() == reflect.Pointer {
if dst.IsNil() {
dst.Set(reflect.New(dst.Type().Elem()))
}
return b.bind(v, dst.Elem(), path, depth+1, quoted)
}
if dst.Type() == durationType {
var raw string
switch x := v.(type) {
case string:
raw = x
case Text:
raw = string(x)
default:
return b.decode(v, dst, path)
}
if err := parseTextValue(dst, strings.TrimSpace(raw)); err != nil {
return b.doc.diagnostic(path, "expected a duration such as 5s", err)
}
return nil
}
if dst.CanAddr() && dst.Addr().Type().Implements(jsonUnmarshalerType) {
normalized, err := b.dynamic(v, path, depth+1)
if err != nil {
return err
}
return b.decode(normalized, dst, path)
}
if dst.CanAddr() && dst.Addr().Type().Implements(textType) {
if text, ok := v.(Text); ok {
v = string(text)
}
return b.decode(v, dst, path)
}
if text, ok := v.(Text); ok {
if dst.Kind() == reflect.String || dst.Kind() == reflect.Interface || (dst.Kind() == reflect.Slice && dst.Type().Elem().Kind() == reflect.Uint8) {
return b.decode(string(text), dst, path)
}
switch dst.Kind() {
case reflect.Bool, reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Float32, reflect.Float64:
if err := parseTextValue(dst, strings.TrimSpace(string(text))); err != nil {
return b.doc.diagnostic(path, "expected a valid "+dst.Type().String(), err)
}
return nil
default:
return b.doc.diagnostic(path, "expected an object or array, not scalar text", nil)
}
}
if quoted {
s, ok := v.(string)
if !ok {
return b.doc.diagnostic(path, "expected a JSON-encoded string", nil)
}
if err := json.Unmarshal([]byte(s), dst.Addr().Interface()); err != nil {
return b.doc.diagnostic(path, "invalid string-encoded value", err)
}
return nil
}
switch dst.Kind() {
case reflect.Struct:
m, ok := v.(map[string]any)
if !ok {
return b.doc.diagnostic(path, "expected an object", nil)
}
fields := configurationFields(dst.Type())
used := map[string]bool{}
for _, name := range sortedKeys(m) {
child := joinField(path, name)
f, ok := fields[name]
canonical := name
if !ok {
for n, field := range fields {
if strings.EqualFold(n, name) {
if ok {
return b.doc.diagnostic(child, "ambiguous field name", nil)
}
f, ok, canonical = field, true, n
}
}
}
if !ok {
return b.doc.diagnostic(child, unknownFieldReason(name, fields), nil)
}
if used[canonical] {
return b.doc.diagnostic(child, "duplicate field binding", nil)
}
used[canonical] = true
target := dst
for _, index := range f.Index {
if target.Kind() == reflect.Pointer {
if target.IsNil() {
if !target.CanSet() {
return b.doc.diagnostic(child, "field cannot be assigned", nil)
}
target.Set(reflect.New(target.Type().Elem()))
}
target = target.Elem()
}
target = target.Field(index)
}
if !target.CanSet() {
return b.doc.diagnostic(child, "field cannot be assigned", nil)
}
if err := b.bind(m[name], target, child, depth+1, strings.Contains(","+f.Tag.Get("json")+",", ",string,")); err != nil {
return err
}
}
return nil
case reflect.Map:
m, ok := v.(map[string]any)
if !ok || dst.Type().Key().Kind() != reflect.String {
return b.doc.diagnostic(path, "expected an object with string keys", nil)
}
if dst.IsNil() {
dst.Set(reflect.MakeMap(dst.Type()))
}
for _, key := range sortedKeys(m) {
value := reflect.New(dst.Type().Elem()).Elem()
if err := b.bind(m[key], value, joinField(path, key), depth+1, false); err != nil {
return err
}
k := reflect.New(dst.Type().Key()).Elem()
k.SetString(key)
dst.SetMapIndex(k, value)
}
return nil
case reflect.Slice, reflect.Array:
if _, ok := v.(string); ok && dst.Kind() == reflect.Slice && dst.Type().Elem().Kind() == reflect.Uint8 {
return b.decode(v, dst, path)
}
items, ok := v.([]any)
if !ok {
return b.doc.diagnostic(path, "expected an array", nil)
}
if dst.Kind() == reflect.Array && len(items) != dst.Len() {
return b.doc.diagnostic(path, "array length does not match its declared type", nil)
}
if dst.Kind() == reflect.Slice {
dst.Set(reflect.MakeSlice(dst.Type(), len(items), len(items)))
}
for i, value := range items {
if err := b.bind(value, dst.Index(i), fmt.Sprintf("%s[%d]", path, i), depth+1, false); err != nil {
return err
}
}
return nil
case reflect.Interface:
v, err := b.dynamic(v, path, depth+1)
if err != nil {
return err
}
if dst.Type().NumMethod() == 0 {
dst.Set(reflect.ValueOf(v))
return nil
}
return b.decode(v, dst, path)
default:
return b.decode(v, dst, path)
}
}
func sortedKeys(m map[string]any) []string {
keys := make([]string, 0, len(m))
for k := range m {
keys = append(keys, k)
}
sort.Strings(keys)
return keys
}
func (b *documentBinder) dynamic(v any, path string, depth int) (any, error) {
if err := b.step(path, depth); err != nil {
return nil, err
}
switch x := v.(type) {
case Text:
return string(x), nil
case resolvedText:
return string(x), nil
case map[string]any:
out := make(map[string]any, len(x))
for _, k := range sortedKeys(x) {
c, err := b.dynamic(x[k], joinField(path, k), depth+1)
if err != nil {
return nil, err
}
out[k] = c
}
return out, nil
case []any:
out := make([]any, len(x))
for i, v := range x {
c, err := b.dynamic(v, fmt.Sprintf("%s[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
out[i] = c
}
return out, nil
default:
return v, nil
}
}
// Match encoding/json's field promotion: the shallowest declaration wins;
// explicit tags win ties, and ambiguous fields remain unavailable.
func configurationFields(t reflect.Type) map[string]reflect.StructField {
type candidate struct {
field reflect.StructField
tagged bool
}
candidates := map[string][]candidate{}
var visit func(reflect.Type, []int, map[reflect.Type]bool)
visit = func(current reflect.Type, prefix []int, visiting map[reflect.Type]bool) {
if visiting[current] {
return
}
visiting[current] = true
defer delete(visiting, current)
for i := 0; i < current.NumField(); i++ {
f := current.Field(i)
name := strings.Split(f.Tag.Get("json"), ",")[0]
if name == "-" {
continue
}
child := f.Type
if child.Kind() == reflect.Pointer {
child = child.Elem()
}
if f.PkgPath != "" && (!f.Anonymous || child.Kind() != reflect.Struct) {
continue
}
f.Index = append(append([]int(nil), prefix...), i)
if name == "" && f.Anonymous && child.Kind() == reflect.Struct {
visit(child, f.Index, visiting)
continue
}
tagged := name != ""
if name == "" {
name = f.Name
}
candidates[name] = append(candidates[name], candidate{f, tagged})
}
}
visit(t, nil, map[reflect.Type]bool{})
fields := map[string]reflect.StructField{}
for name, list := range candidates {
sort.Slice(list, func(i, j int) bool {
if len(list[i].field.Index) != len(list[j].field.Index) {
return len(list[i].field.Index) < len(list[j].field.Index)
}
return list[i].tagged && !list[j].tagged
})
if len(list) > 1 && len(list[0].field.Index) == len(list[1].field.Index) && list[0].tagged == list[1].tagged {
continue
}
fields[name] = list[0].field
}
return fields
}

View File

@@ -5,8 +5,10 @@ import (
"time" "time"
"git.yuntuo.hk/k2safe/go-omni/contract" "git.yuntuo.hk/k2safe/go-omni/contract"
"git.yuntuo.hk/k2safe/go-omni/cors"
apierrors "git.yuntuo.hk/k2safe/go-omni/errors" apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
"git.yuntuo.hk/k2safe/go-omni/httpx" "git.yuntuo.hk/k2safe/go-omni/httpx"
"git.yuntuo.hk/k2safe/go-omni/i18n/httpi18n"
"git.yuntuo.hk/k2safe/go-omni/lifecycle" "git.yuntuo.hk/k2safe/go-omni/lifecycle"
"git.yuntuo.hk/k2safe/go-omni/logx" "git.yuntuo.hk/k2safe/go-omni/logx"
"git.yuntuo.hk/k2safe/go-omni/response" "git.yuntuo.hk/k2safe/go-omni/response"
@@ -28,6 +30,7 @@ type TLSConfig struct {
// Config controls the default runtime behavior of an App. // Config controls the default runtime behavior of an App.
type Config struct { type Config struct {
CORS cors.Config `json:"cors"`
Request contract.RequestPolicy `json:"request"` Request contract.RequestPolicy `json:"request"`
Address string `json:"address"` Address string `json:"address"`
Transport string `json:"transport"` Transport string `json:"transport"`
@@ -45,6 +48,8 @@ type Config struct {
Response response.Config `json:"response"` Response response.Config `json:"response"`
ResponseEncoder response.Encoder `json:"-"` ResponseEncoder response.Encoder `json:"-"`
LogInstance *logx.Logger `json:"-"` LogInstance *logx.Logger `json:"-"`
LanguagePolicy *httpi18n.Policy `json:"-"`
ErrorLocalizer apierrors.Localizer `json:"-"`
} }
func Default() Config { func Default() Config {
@@ -61,20 +66,23 @@ func Default() Config {
// Validate checks configuration values before an application starts. // Validate checks configuration values before an application starts.
func (c Config) Validate() error { func (c Config) Validate() error {
if err := c.CORS.Validate(); err != nil {
return AtField("cors", err)
}
if err := c.Lifecycle.Validate(); err != nil { if err := c.Lifecycle.Validate(); err != nil {
return err return AtField("lifecycle", err)
} }
if err := c.Request.Validate(); err != nil { if err := c.Request.Validate(); err != nil {
return err return AtField("request", err)
} }
if err := c.Log.Validate(); err != nil { if err := c.Log.Validate(); err != nil {
return err return AtField("log", err)
} }
if err := c.Endpoints.Validate(); err != nil { if err := c.Endpoints.Validate(); err != nil {
return err return err
} }
if err := c.Errors.Validate(); err != nil { if err := c.Errors.Validate(); err != nil {
return err return AtField("errors", err)
} }
encoder := c.ResponseEncoder encoder := c.ResponseEncoder
if encoder == nil { if encoder == nil {
@@ -90,7 +98,7 @@ func (c Config) Validate() error {
return err return err
} }
if err := c.Response.Validate(); err != nil { if err := c.Response.Validate(); err != nil {
return err return AtField("response", err)
} }
if c.SlowRequestThreshold < 0 { if c.SlowRequestThreshold < 0 {
return &ConfigError{Field: "slowRequestThreshold", Message: "must be nonnegative"} return &ConfigError{Field: "slowRequestThreshold", Message: "must be nonnegative"}
@@ -99,13 +107,13 @@ func (c Config) Validate() error {
return &ConfigError{Field: "address", Message: "is required"} return &ConfigError{Field: "address", Message: "is required"}
} }
if strings.TrimSpace(c.RequestIDHeader) == "" { if strings.TrimSpace(c.RequestIDHeader) == "" {
return &ConfigError{Field: "request_id_header", Message: "is required"} return &ConfigError{Field: "requestIdHeader", Message: "is required"}
} }
if c.ShutdownTimeout <= 0 { if c.ShutdownTimeout <= 0 {
return &ConfigError{Field: "shutdown_timeout", Message: "must be positive"} return &ConfigError{Field: "shutdownTimeout", Message: "must be positive"}
} }
if _, err := httpx.NewClientIPResolver(c.TrustedProxies...); err != nil { if _, err := httpx.NewClientIPResolver(c.TrustedProxies...); err != nil {
return &ConfigError{Field: "trusted_proxies", Message: err.Error()} return &Error{Field: "trustedProxies", Reason: "expected IP addresses or CIDRs", cause: err}
} }
return c.ValidateTransport() return c.ValidateTransport()
} }
@@ -188,12 +196,14 @@ func WithTrustedProxies(proxies ...string) Option {
// WithConfig applies a complete configuration loaded from a file or // WithConfig applies a complete configuration loaded from a file or
// environment. Later options can still override individual fields. // environment. Later options can still override individual fields.
func WithConfig(settings Config) Option { func WithConfig(settings Config) Option {
settings.CORS = settings.CORS.Clone()
settings.Response = settings.Response.Clone() settings.Response = settings.Response.Clone()
settings.Errors = settings.Errors.Clone() settings.Errors = settings.Errors.Clone()
settings.TrustedProxies = append([]string(nil), settings.TrustedProxies...) settings.TrustedProxies = append([]string(nil), settings.TrustedProxies...)
settings.Log.RedactKeys = append([]string(nil), settings.Log.RedactKeys...) settings.Log.RedactKeys = append([]string(nil), settings.Log.RedactKeys...)
return func(cfg *Config) { return func(cfg *Config) {
*cfg = settings *cfg = settings
cfg.CORS = settings.CORS.Clone()
cfg.Response = settings.Response.Clone() cfg.Response = settings.Response.Clone()
cfg.Errors = settings.Errors.Clone() cfg.Errors = settings.Errors.Clone()
cfg.TrustedProxies = append([]string(nil), settings.TrustedProxies...) cfg.TrustedProxies = append([]string(nil), settings.TrustedProxies...)
@@ -201,6 +211,12 @@ func WithConfig(settings Config) Option {
} }
} }
// WithCORS installs an app-local CORS policy, before routing and authentication.
func WithCORS(settings cors.Config) Option {
settings = settings.Clone()
return func(cfg *Config) { cfg.CORS = settings.Clone() }
}
// WithResponse configures this application's default JSON envelope. // WithResponse configures this application's default JSON envelope.
func WithResponse(settings response.Config) Option { func WithResponse(settings response.Config) Option {
settings = settings.Clone() settings = settings.Clone()
@@ -240,6 +256,15 @@ func WithErrors(catalog apierrors.Catalog) Option {
// WithLogInstance borrows a logger for this App without changing process logging. // WithLogInstance borrows a logger for this App without changing process logging.
func WithLogInstance(logger *logx.Logger) Option { return func(c *Config) { c.LogInstance = logger } } func WithLogInstance(logger *logx.Logger) Option { return func(c *Config) { c.LogInstance = logger } }
// WithLanguagePolicy borrows an immutable HTTP language policy. Resource loading
// remains explicit in the project; configuration strings are not dictionaries.
func WithLanguagePolicy(policy *httpi18n.Policy) Option {
return func(c *Config) { c.LanguagePolicy = policy }
}
func WithErrorLocalizer(localizer apierrors.Localizer) Option {
return func(c *Config) { c.ErrorLocalizer = localizer }
}
func WithRequestPolicy(policy contract.RequestPolicy) Option { func WithRequestPolicy(policy contract.RequestPolicy) Option {
return func(c *Config) { c.Request = policy } return func(c *Config) { c.Request = policy }
} }

View File

@@ -1,132 +1,6 @@
package config package config
import ( import "testing"
"strings"
"testing"
"time"
)
func TestLoadFromEnvUsesDefaults(t *testing.T) {
for _, key := range []string{
"GO_OMNI_HTTP_ADDR", "GO_OMNI_REQUEST_ID_HEADER", "GO_OMNI_ENABLE_LOGGER",
"GO_OMNI_TRUSTED_PROXIES",
"GO_OMNI_TRANSPORT", "GO_OMNI_TLS_CERT_FILE", "GO_OMNI_TLS_KEY_FILE",
"GO_OMNI_LOG_LEVEL", "GO_OMNI_LOG_FORMAT", "GO_OMNI_LOG_OUTPUT",
"GO_OMNI_LOG_FILE", "GO_OMNI_SERVICE", "GO_OMNI_ENVIRONMENT",
"GO_OMNI_LOG_ADD_SOURCE", "GO_OMNI_SHUTDOWN_TIMEOUT",
"GO_OMNI_SLOW_REQUEST_THRESHOLD",
} {
t.Setenv(key, "")
}
cfg, err := LoadFromEnv("")
if err != nil {
t.Fatal(err)
}
if cfg.Address != ":8080" || cfg.Transport != TransportNetHTTP || cfg.TLS != (TLSConfig{}) || cfg.RequestIDHeader != "X-Request-ID" || cfg.ShutdownTimeout != 10*time.Second {
t.Fatalf("unexpected defaults: %+v", cfg)
}
if cfg.EnableLogger || cfg.Log.Level != "info" || cfg.Log.Format != "json" || cfg.Log.Output != "stderr" {
t.Fatalf("unexpected logger defaults: %+v", cfg.Log)
}
}
func TestTransportEnvironment(t *testing.T) {
for _, test := range []struct {
name, transport, cert, key, errorField string
}{
{"default", "", "", "", ""},
{"native", "native", "", "", ""},
{"nativeTLS", "native", "server.crt", "server.key", ""},
{"http2", "http2", "server.crt", "server.key", ""},
{"unknown", "typo", "", "", "transport"},
{"certOnly", "nethttp", "server.crt", "", "tls"},
{"keyOnly", "native", "", "server.key", "tls"},
{"http2WithoutTLS", "http2", "", "", "tls"},
} {
t.Run(test.name, func(t *testing.T) {
t.Setenv("TRANSPORT_TEST_TRANSPORT", test.transport)
t.Setenv("TRANSPORT_TEST_TLS_CERT_FILE", test.cert)
t.Setenv("TRANSPORT_TEST_TLS_KEY_FILE", test.key)
cfg, err := LoadFromEnv("TRANSPORT_TEST_")
if test.errorField != "" {
if err == nil || !strings.Contains(err.Error(), test.errorField) {
t.Fatalf("unexpected error: %v", err)
}
return
}
if err != nil {
t.Fatal(err)
}
want := test.transport
if want == "" {
want = TransportNetHTTP
}
if cfg.Transport != want || cfg.TLS.CertFile != test.cert || cfg.TLS.KeyFile != test.key {
t.Fatalf("settings: %+v", cfg)
}
})
}
}
func TestLoadFromEnvOverridesSettings(t *testing.T) {
t.Setenv("GO_OMNI_HTTP_ADDR", "127.0.0.1:9090")
t.Setenv("GO_OMNI_REQUEST_ID_HEADER", "X-Correlation-ID")
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "127.0.0.1, 10.0.0.0/8")
t.Setenv("GO_OMNI_ENABLE_LOGGER", "true")
t.Setenv("GO_OMNI_LOG_LEVEL", "debug")
t.Setenv("GO_OMNI_LOG_FORMAT", "text")
t.Setenv("GO_OMNI_LOG_OUTPUT", "stdout")
t.Setenv("GO_OMNI_SERVICE", "billing")
t.Setenv("GO_OMNI_ENVIRONMENT", "test")
t.Setenv("GO_OMNI_LOG_ADD_SOURCE", "true")
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "3s")
t.Setenv("GO_OMNI_SLOW_REQUEST_THRESHOLD", "500ms")
cfg, err := LoadFromEnv("")
if err != nil {
t.Fatal(err)
}
if cfg.Address != "127.0.0.1:9090" || cfg.RequestIDHeader != "X-Correlation-ID" || !cfg.EnableLogger || len(cfg.TrustedProxies) != 2 || cfg.TrustedProxies[1] != "10.0.0.0/8" {
t.Fatalf("unexpected application settings: %+v", cfg)
}
if cfg.ShutdownTimeout != 3*time.Second || cfg.Log.Level != "debug" || cfg.Log.Format != "text" || cfg.Log.Output != "stdout" || cfg.Log.Service != "billing" || cfg.Log.Environment != "test" || !cfg.Log.AddSource {
t.Fatalf("unexpected logger settings: %+v", cfg.Log)
}
if cfg.SlowRequestThreshold != 500*time.Millisecond {
t.Fatal(cfg.SlowRequestThreshold)
}
}
func TestLoadFromEnvRejectsInvalidValues(t *testing.T) {
t.Setenv("GO_OMNI_ENABLE_LOGGER", "sometimes")
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "ENABLE_LOGGER") {
t.Fatalf("expected invalid logger flag error, got %v", err)
}
t.Setenv("GO_OMNI_ENABLE_LOGGER", "")
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "later")
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "SHUTDOWN_TIMEOUT") {
t.Fatalf("expected invalid timeout error, got %v", err)
}
t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "1s")
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "proxy.example.com")
if _, err := LoadFromEnv(""); err == nil || !strings.Contains(err.Error(), "trusted_proxies") {
t.Fatalf("expected invalid trusted proxy error, got %v", err)
}
t.Setenv("GO_OMNI_TRUSTED_PROXIES", "")
cfg := Default()
cfg.Address = ""
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "address") {
t.Fatalf("expected invalid address error, got %v", err)
}
cfg = Default()
cfg.RequestIDHeader = " "
if err := cfg.Validate(); err == nil || !strings.Contains(err.Error(), "request_id_header") {
t.Fatalf("expected invalid request header error, got %v", err)
}
}
func TestWithConfigAndWithLogConfig(t *testing.T) { func TestWithConfigAndWithLogConfig(t *testing.T) {
settings := Default() settings := Default()
@@ -147,23 +21,3 @@ func TestWithConfigAndWithLogConfig(t *testing.T) {
t.Fatalf("log config was not replaced: %+v", cfg.Log) t.Fatalf("log config was not replaced: %+v", cfg.Log)
} }
} }
func TestApplyEnvPreservesProjectDefaults(t *testing.T) {
cfg := Default()
cfg.Address = "127.0.0.1:8123"
cfg.Log.RedactKeys = []string{"bankAccount"}
cfg.TrustedProxies = []string{"127.0.0.1"}
t.Setenv("OVERLAY_TEST_LOG_LEVEL", "debug")
got, err := ApplyEnv(cfg, "OVERLAY_TEST_")
if err != nil {
t.Fatal(err)
}
if got.Address != cfg.Address || got.Log.Level != "debug" {
t.Fatal(got)
}
got.Log.RedactKeys[0] = "changed"
got.TrustedProxies[0] = "192.0.2.1"
if cfg.Log.RedactKeys[0] != "bankAccount" || cfg.TrustedProxies[0] != "127.0.0.1" {
t.Fatal("caller-owned defaults mutated")
}
}

51
config/cors_test.go Normal file
View File

@@ -0,0 +1,51 @@
package config_test
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"git.yuntuo.hk/k2safe/go-omni/config"
)
type corsProject struct {
Server config.Config `json:"server"`
}
func (c corsProject) Validate() error { return config.AtField("server", c.Server.Validate()) }
func TestCORSConfiguration(t *testing.T) {
t.Setenv("WEB_ORIGIN", "https://web.example")
for ext, source := range map[string]string{
"yaml": "server:\n cors:\n enabled: true\n allowedOrigins:\n - '${WEB_ORIGIN}'\n allowedMethods: [GET, POST]\n allowedHeaders: [Authorization, Content-Type]\n maxAge: 10m\n",
"json": `{"server":{"cors":{"enabled":true,"allowedOrigins":["${WEB_ORIGIN}"],"allowedMethods":["GET","POST"],"allowedHeaders":["Authorization","Content-Type"],"maxAge":"10m"}}}`,
"xml": `<config><server><cors><enabled type="bool">true</enabled><allowedOrigins type="array"><item>${WEB_ORIGIN}</item></allowedOrigins><allowedMethods type="array"><item>GET</item><item>POST</item></allowedMethods><allowedHeaders type="array"><item>Authorization</item><item>Content-Type</item></allowedHeaders><maxAge>10m</maxAge></cors></server></config>`,
} {
t.Run(ext, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "config."+ext)
if err := os.WriteFile(path, []byte(source), 0600); err != nil {
t.Fatal(err)
}
cfg, err := config.LoadProject(path, corsProject{Server: config.Default()})
if err != nil {
t.Fatal(err)
}
if !cfg.Server.CORS.Enabled || cfg.Server.CORS.AllowedOrigins[0] != "https://web.example" || cfg.Server.CORS.MaxAge != 10*time.Minute {
t.Fatal(cfg.Server.CORS)
}
})
}
path := filepath.Join(t.TempDir(), "bad.yaml")
if err := os.WriteFile(path, []byte("server:\n cors:\n enabled: true\n allowedOrigins:\n - '${WEB_ORIGIN}'\n"), 0600); err != nil {
t.Fatal(err)
}
t.Setenv("WEB_ORIGIN", "https://user:secret@web.example")
_, err := config.LoadProject(path, corsProject{Server: config.Default()})
var d *config.Error
if !errors.As(err, &d) || d.Field != "server.cors.allowedOrigins[0]" || d.Env != "WEB_ORIGIN" || d.Line != 5 || strings.Contains(err.Error(), "secret") {
t.Fatal(err)
}
}

398
config/decoders.go Normal file
View File

@@ -0,0 +1,398 @@
package config
import (
"bytes"
"encoding/json"
"encoding/xml"
"errors"
"fmt"
"io"
"regexp"
"sort"
"strconv"
"strings"
"unicode/utf8"
"gopkg.in/yaml.v3"
)
type JSONDecoder struct{}
type YAMLDecoder struct{}
type XMLDecoder struct{}
type parseState struct {
doc Document
nodes int
}
func newParseState() *parseState { return &parseState{doc: Document{Positions: map[string]Position{}}} }
func (p *parseState) enter(path string, pos Position, depth int) error {
p.nodes++
p.doc.Positions[path] = pos
if depth > maxConfigDepth || p.nodes > maxConfigNodes {
return p.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil)
}
return nil
}
type sourcePositions struct{ starts []int }
func newSourcePositions(data []byte) sourcePositions {
p := sourcePositions{starts: []int{0}}
for i, c := range data {
if c == '\n' {
p.starts = append(p.starts, i+1)
}
}
return p
}
func (p sourcePositions) at(offset int64) Position {
if offset < 0 {
offset = 0
}
i := sort.Search(len(p.starts), func(i int) bool { return int64(p.starts[i]) > offset }) - 1
return Position{Line: i + 1, Column: int(offset) - p.starts[i] + 1}
}
var yamlLine = regexp.MustCompile(`line ([0-9]+)`)
func yamlSyntax(err error) *Error {
d := &Error{Reason: "invalid YAML syntax", cause: err}
if m := yamlLine.FindStringSubmatch(err.Error()); len(m) == 2 {
d.Line, _ = strconv.Atoi(m[1])
d.Column = 1
}
return d
}
func (JSONDecoder) Decode(data []byte) (Document, error) {
p := newParseState()
if len(data) > maxConfigBytes {
return p.doc, &Error{Reason: "file exceeds 1 MiB"}
}
if !utf8.Valid(data) {
return p.doc, &Error{Reason: "configuration must be UTF-8"}
}
positions := newSourcePositions(data)
d := json.NewDecoder(bytes.NewReader(data))
d.UseNumber()
var parse func(string, int) (any, error)
parse = func(path string, depth int) (any, error) {
offset := d.InputOffset()
for offset < int64(len(data)) && strings.ContainsRune(" \t\r\n:,", rune(data[offset])) {
offset++
}
if err := p.enter(path, positions.at(offset), depth); err != nil {
return nil, err
}
token, err := d.Token()
if err != nil {
pos := positions.at(d.InputOffset())
var syntax *json.SyntaxError
if errors.As(err, &syntax) {
pos = positions.at(syntax.Offset - 1)
}
return nil, &Error{Field: path, Line: pos.Line, Column: pos.Column, Reason: "invalid JSON syntax", cause: err}
}
if delim, ok := token.(json.Delim); ok {
switch delim {
case '{':
out := map[string]any{}
for d.More() {
k, e := d.Token()
if e != nil {
return nil, p.doc.diagnostic(path, "invalid object key", e)
}
key, ok := k.(string)
if !ok {
return nil, p.doc.diagnostic(path, "object key must be a string", nil)
}
child := joinField(path, key)
if _, exists := out[key]; exists {
return nil, p.doc.diagnostic(child, "duplicate field", nil)
}
out[key], e = parse(child, depth+1)
if e != nil {
return nil, e
}
}
if _, err = d.Token(); err != nil {
return nil, p.doc.diagnostic(path, "unterminated object", err)
}
return out, nil
case '[':
out := []any{}
for d.More() {
v, e := parse(fmt.Sprintf("%s[%d]", path, len(out)), depth+1)
if e != nil {
return nil, e
}
out = append(out, v)
}
if _, err = d.Token(); err != nil {
return nil, p.doc.diagnostic(path, "unterminated array", err)
}
return out, nil
default:
return nil, p.doc.diagnostic(path, "unexpected JSON delimiter", nil)
}
}
return token, nil
}
var err error
p.doc.Value, err = parse("", 0)
if err == nil {
if _, e := d.Token(); e != io.EOF {
err = p.doc.diagnostic("", "exactly one JSON document required", e)
}
}
return p.doc, err
}
func (YAMLDecoder) Decode(data []byte) (Document, error) {
p := newParseState()
if len(data) > maxConfigBytes {
return p.doc, &Error{Reason: "file exceeds 1 MiB"}
}
if !utf8.Valid(data) {
return p.doc, &Error{Reason: "configuration must be UTF-8"}
}
d := yaml.NewDecoder(bytes.NewReader(data))
var root yaml.Node
if err := d.Decode(&root); err != nil {
return p.doc, yamlSyntax(err)
}
var extra yaml.Node
if err := d.Decode(&extra); err != io.EOF {
return p.doc, &Error{Reason: "exactly one YAML document required", cause: err}
}
var walk func(*yaml.Node, string, int) (any, error)
walk = func(n *yaml.Node, path string, depth int) (any, error) {
if err := p.enter(path, Position{n.Line, n.Column}, depth); err != nil {
return nil, err
}
switch n.Kind {
case yaml.MappingNode:
out := map[string]any{}
for i := 0; i < len(n.Content); i += 2 {
k, v := n.Content[i], n.Content[i+1]
if k.Tag != "!!str" {
return nil, p.doc.diagnostic(path, "mapping keys must be strings; merge keys are not supported", nil)
}
child := joinField(path, k.Value)
if _, ok := out[k.Value]; ok {
return nil, &Error{Field: child, Line: k.Line, Column: k.Column, Reason: "duplicate field"}
}
x, err := walk(v, child, depth+1)
if err != nil {
return nil, err
}
out[k.Value] = x
}
return out, nil
case yaml.SequenceNode:
out := make([]any, len(n.Content))
for i, c := range n.Content {
var err error
out[i], err = walk(c, fmt.Sprintf("%s[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
}
return out, nil
case yaml.AliasNode:
return nil, p.doc.diagnostic(path, "YAML aliases are not supported", nil)
case yaml.ScalarNode:
switch n.Tag {
case "!!str", "!!timestamp":
return n.Value, nil
case "!!null":
return nil, nil
case "!!bool", "!!int", "!!float":
var v any
if err := n.Decode(&v); err != nil {
return nil, p.doc.diagnostic(path, "invalid YAML scalar", err)
}
raw, err := json.Marshal(v)
if err != nil {
return nil, p.doc.diagnostic(path, "number must be finite", err)
}
var out any
dec := json.NewDecoder(bytes.NewReader(raw))
dec.UseNumber()
if err = dec.Decode(&out); err != nil {
return nil, p.doc.diagnostic(path, "invalid YAML scalar", err)
}
return out, nil
default:
return nil, p.doc.diagnostic(path, "unsupported YAML tag", nil)
}
}
return nil, p.doc.diagnostic(path, "unsupported YAML node", nil)
}
if len(root.Content) != 1 {
return p.doc, &Error{Reason: "one nonempty YAML document required"}
}
var err error
p.doc.Value, err = walk(root.Content[0], "", 0)
return p.doc, err
}
// XML uses a config root, element field names, type="array" with item children,
// type="object" for empty objects, and type="null" for explicit null. Scalar
// text is bound using the destination type; optional number/bool/string types
// let dynamic any values preserve their intended JSON type.
func (XMLDecoder) Decode(data []byte) (Document, error) {
p := newParseState()
if len(data) > maxConfigBytes {
return p.doc, &Error{Reason: "file exceeds 1 MiB"}
}
if !utf8.Valid(data) {
return p.doc, &Error{Reason: "configuration must be UTF-8"}
}
d := xml.NewDecoder(bytes.NewReader(data))
var parse func(xml.StartElement, string, int, Position) (any, error)
parse = func(start xml.StartElement, path string, depth int, pos Position) (any, error) {
if err := p.enter(path, pos, depth); err != nil {
return nil, err
}
if start.Name.Space != "" {
return nil, p.doc.diagnostic(path, "XML namespaces are not supported", nil)
}
kind := ""
attrSeen := false
for _, a := range start.Attr {
if a.Name.Space != "" || a.Name.Local != "type" || attrSeen {
return nil, p.doc.diagnostic(path, "only one XML type attribute is supported", nil)
}
kind = a.Value
attrSeen = true
}
if depth == 0 && kind == "" {
kind = "object"
}
switch kind {
case "", "object", "array", "null", "string", "number", "bool":
default:
return nil, p.doc.diagnostic(path, "unsupported XML type", nil)
}
children := map[string]any{}
array := []any{}
var text strings.Builder
for {
line, column := d.InputPos()
token, err := d.Token()
if err != nil {
return nil, &Error{Field: path, Line: line, Column: column, Reason: "invalid XML syntax", cause: err}
}
switch x := token.(type) {
case xml.StartElement:
if kind != "" && kind != "object" && kind != "array" {
return nil, p.doc.diagnostic(path, "scalar XML element cannot contain children", nil)
}
child := joinField(path, x.Name.Local)
if kind == "array" {
if x.Name.Local != "item" {
return nil, p.doc.diagnostic(path, "XML array children must be item elements", nil)
}
child = fmt.Sprintf("%s[%d]", path, len(array))
} else if _, ok := children[x.Name.Local]; ok {
return nil, p.doc.diagnostic(child, "duplicate field; use type=\"array\" for lists", nil)
}
v, e := parse(x, child, depth+1, Position{line, column})
if e != nil {
return nil, e
}
if kind == "array" {
array = append(array, v)
} else {
children[x.Name.Local] = v
}
case xml.CharData:
text.Write([]byte(x))
case xml.Comment:
case xml.EndElement:
raw := text.String()
if len(children) > 0 || kind == "array" || kind == "object" {
if strings.TrimSpace(raw) != "" {
return nil, p.doc.diagnostic(path, "mixed XML text and elements are not supported", nil)
}
if kind == "array" {
return array, nil
}
return children, nil
}
switch kind {
case "null":
if strings.TrimSpace(raw) != "" {
return nil, p.doc.diagnostic(path, "null element must be empty", nil)
}
return nil, nil
case "string":
return raw, nil
case "number":
var n any
dec := json.NewDecoder(strings.NewReader(strings.TrimSpace(raw)))
dec.UseNumber()
if err := dec.Decode(&n); err != nil {
return nil, p.doc.diagnostic(path, "invalid number", err)
}
if _, ok := n.(json.Number); !ok {
return nil, p.doc.diagnostic(path, "expected number", nil)
}
if _, e := dec.Token(); e != io.EOF {
return nil, p.doc.diagnostic(path, "invalid number", e)
}
return n, nil
case "bool":
b, e := strconv.ParseBool(strings.TrimSpace(raw))
if e != nil {
return nil, p.doc.diagnostic(path, "expected boolean", e)
}
return b, nil
}
return Text(raw), nil
default:
return nil, p.doc.diagnostic(path, "XML directives and processing instructions are not supported", nil)
}
}
}
found := false
prolog := false
for {
line, column := d.InputPos()
token, err := d.Token()
if err == io.EOF {
break
}
if err != nil {
return p.doc, &Error{Line: line, Column: column, Reason: "invalid XML syntax", cause: err}
}
switch x := token.(type) {
case xml.StartElement:
if found || x.Name.Local != "config" {
return p.doc, &Error{Line: line, Column: column, Reason: "exactly one config root required"}
}
found = true
p.doc.Value, err = parse(x, "", 0, Position{line, column})
if err != nil {
return p.doc, err
}
case xml.CharData:
if strings.TrimSpace(string(x)) != "" {
return p.doc, &Error{Reason: "unexpected text outside XML root"}
}
case xml.Comment:
case xml.ProcInst:
if found || prolog || x.Target != "xml" {
return p.doc, &Error{Reason: "XML processing instructions are not supported"}
}
prolog = true
default:
return p.doc, &Error{Reason: "XML directives are not supported"}
}
}
if !found {
return p.doc, &Error{Reason: "config root required"}
}
return p.doc, nil
}

101
config/diagnostic.go Normal file
View File

@@ -0,0 +1,101 @@
package config
import (
"errors"
"fmt"
"strings"
"git.yuntuo.hk/k2safe/go-omni/internal/configdiag"
)
// Error is a value-free diagnostic. Cause may contain private parser details;
// errors.Is/As remain available, but Error never formats Cause automatically.
type Error struct {
File, Field, Env, Reason string
Line, Column int
cause error
}
func (e *Error) Error() string {
where := "config"
if e.File != "" {
where = fmt.Sprintf("config %q", e.File)
}
if e.Line > 0 {
where += fmt.Sprintf(":%d:%d", e.Line, e.Column)
}
if e.Env != "" {
where += fmt.Sprintf(" env %q", e.Env)
}
if e.Field != "" {
where += fmt.Sprintf(" field %q", e.Field)
}
return where + ": " + e.Reason
}
func (e *Error) Unwrap() error { return e.cause }
// DecodeError is retained for callers of errors.As from earlier releases.
type DecodeError = Error
// Invalid reports a public, value-free validation reason for a configuration field.
func Invalid(field, reason string) error { return &ConfigError{Field: field, Message: reason} }
// AtField prefixes a nested validation error without losing its original cause.
// Unstructured validation errors are deliberately not echoed; use Invalid when
// a custom validator needs a public reason rather than a private error message.
func AtField(field string, err error) error {
if err == nil {
return nil
}
d := validationError(err)
d.Field = joinField(field, d.Field)
return d
}
func validationError(err error) *Error {
var d *Error
if errors.As(err, &d) {
out := *d
out.cause = err
return &out
}
var nested *configdiag.Error
if errors.As(err, &nested) {
return &Error{Field: nested.Field, Reason: nested.Reason, cause: err}
}
var c *ConfigError
if errors.As(err, &c) {
return &Error{Field: c.Field, Reason: c.Message, cause: err}
}
return &Error{Reason: "validation failed", cause: err}
}
func joinField(parent, child string) string {
if parent == "" {
return child
}
if child == "" {
return parent
}
if strings.HasPrefix(child, "[") {
return parent + child
}
return parent + "." + child
}
// Position uses one-based source line/column numbers. Zero means unavailable.
type Position struct{ Line, Column int }
// Document is the common decoder result. Values use JSON-compatible data:
// string, bool, json.Number, nil, []any, map[string]any. Text is available for
// formats such as XML whose scalar tokens have no inherent JSON type.
type Document struct {
Value any
Positions map[string]Position
environments map[string][]string
}
type Text string
func (d Document) diagnostic(field, reason string, cause error) *Error {
p := d.Positions[field]
return &Error{Field: field, Reason: reason, Line: p.Line, Column: p.Column, cause: cause}
}

View File

@@ -18,14 +18,15 @@ func TestEndpointConfiguration(t *testing.T) {
if cfg.Endpoints.Liveness.Enabled || cfg.Endpoints.OpenAPI.Enabled || !cfg.Endpoints.Readiness.Enabled || cfg.Endpoints.Readiness.ResponseMode != "raw" { if cfg.Endpoints.Liveness.Enabled || cfg.Endpoints.OpenAPI.Enabled || !cfg.Endpoints.Readiness.Enabled || cfg.Endpoints.Readiness.ResponseMode != "raw" {
t.Fatal(cfg.Endpoints) t.Fatal(cfg.Endpoints)
} }
refs := configFile(t, "yaml", "endpoints:\n readiness:\n path: ${TEST_ENDPOINTS_READINESS_PATH}\n openapi:\n enabled: ${TEST_ENDPOINTS_OPENAPI_ENABLED}\n")
t.Setenv("TEST_ENDPOINTS_READINESS_PATH", "/internal/ready") t.Setenv("TEST_ENDPOINTS_READINESS_PATH", "/internal/ready")
t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "true") t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "true")
cfg, err = ApplyEnv(cfg, "TEST_") cfg, err = LoadProject(refs, cfg)
if err != nil || cfg.Endpoints.Readiness.Path != "/internal/ready" || !cfg.Endpoints.OpenAPI.Enabled { if err != nil || cfg.Endpoints.Readiness.Path != "/internal/ready" || !cfg.Endpoints.OpenAPI.Enabled {
t.Fatal(cfg.Endpoints, err) t.Fatal(cfg.Endpoints, err)
} }
t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "invalid") t.Setenv("TEST_ENDPOINTS_OPENAPI_ENABLED", "invalid")
if _, err = ApplyEnv(cfg, "TEST_"); err == nil { if _, err = LoadProject(refs, cfg); err == nil {
t.Fatal("invalid bool accepted") t.Fatal("invalid bool accepted")
} }
} }

View File

@@ -1,83 +0,0 @@
package config
import (
"encoding/json"
"os"
"strings"
)
const defaultEnvPrefix = "GO_OMNI_"
// ServerEnvAliases returns the documented short names for Config fields.
// field is the JSON path of Config in a project (usually "server"), or ""
// for Config itself. Canonical names win over aliases. Both use the same
// EnvOptions.Empty/Lookup and typed parser; collections use JSON syntax.
func ServerEnvAliases(prefix, field string) map[string][]string {
root := prefix
if field != "" {
for _, part := range strings.Split(field, ".") {
root += envName(part) + "_"
}
}
names := map[string]string{
"ADDRESS": "HTTP_ADDR", "TRANSPORT": "TRANSPORT", "TLS_CERT_FILE": "TLS_CERT_FILE", "TLS_KEY_FILE": "TLS_KEY_FILE",
"REQUEST_ID_HEADER": "REQUEST_ID_HEADER", "ENABLE_LOGGER": "ENABLE_LOGGER", "SLOW_REQUEST_THRESHOLD": "SLOW_REQUEST_THRESHOLD",
"TRUSTED_PROXIES": "TRUSTED_PROXIES", "SHUTDOWN_TIMEOUT": "SHUTDOWN_TIMEOUT",
"LOG_LEVEL": "LOG_LEVEL", "LOG_FORMAT": "LOG_FORMAT", "LOG_OUTPUT": "LOG_OUTPUT", "LOG_FILE": "LOG_FILE", "LOG_SERVICE": "SERVICE", "LOG_ENVIRONMENT": "ENVIRONMENT", "LOG_ADD_SOURCE": "LOG_ADD_SOURCE",
"HTTP_READ_HEADER_TIMEOUT": "HTTP_READ_HEADER_TIMEOUT", "HTTP_READ_TIMEOUT": "HTTP_READ_TIMEOUT", "HTTP_WRITE_TIMEOUT": "HTTP_WRITE_TIMEOUT", "HTTP_IDLE_TIMEOUT": "HTTP_IDLE_TIMEOUT", "HTTP_MAX_HEADER_BYTES": "HTTP_MAX_HEADER_BYTES",
"LIFECYCLE_STOP_TIMEOUT": "LIFECYCLE_STOP_TIMEOUT", "LIFECYCLE_DRAIN_TIMEOUT": "LIFECYCLE_DRAIN_TIMEOUT", "LIFECYCLE_CANCEL_TIMEOUT": "LIFECYCLE_CANCEL_TIMEOUT",
}
for _, endpoint := range []string{"LIVENESS", "READINESS", "OPENAPI"} {
for _, part := range []string{"ENABLED", "PATH", "RESPONSE_MODE"} {
if endpoint == "OPENAPI" && part == "RESPONSE_MODE" {
continue
}
name := "ENDPOINTS_" + endpoint + "_" + part
names[name] = name
}
}
aliases := map[string][]string{}
for key, alias := range names {
if root+key != prefix+alias {
aliases[root+key] = []string{prefix + alias}
}
}
return aliases
}
// LoadFromEnv loads defaults using the legacy short-name conventions.
func LoadFromEnv(prefix string) (Config, error) { return ApplyEnv(Default(), prefix) }
// ApplyEnv retains legacy whitespace trimming, ignored empty variables and
// comma-separated TRUSTED_PROXIES. New projects should use OverlayEnv with
// ServerEnvAliases for one explicit empty-value policy across the whole config.
func ApplyEnv(defaults Config, prefix string) (Config, error) {
prefix = strings.TrimSpace(prefix)
if prefix == "" {
prefix = defaultEnvPrefix
}
lookup := func(key string) (string, bool) {
raw, ok := os.LookupEnv(key)
raw = strings.TrimSpace(raw)
if ok && raw != "" && key == prefix+"TRUSTED_PROXIES" && !strings.HasPrefix(raw, "[") {
values := strings.Split(raw, ",")
for i := range values {
values[i] = strings.TrimSpace(values[i])
}
data, _ := json.Marshal(values)
raw = string(data)
}
return raw, ok
}
defaults.Errors = defaults.Errors.Clone()
defaults.TrustedProxies = append([]string(nil), defaults.TrustedProxies...)
defaults.Log.RedactKeys = append([]string(nil), defaults.Log.RedactKeys...)
cfg, err := OverlayEnv(defaults, EnvOptions{Prefix: prefix, Empty: "ignore", Lookup: lookup, Aliases: ServerEnvAliases(prefix, "")})
if err != nil {
return Config{}, err
}
if err = cfg.Validate(); err != nil {
return Config{}, err
}
return cfg, nil
}

View File

@@ -21,7 +21,7 @@ func TestCatalogConfigFiles(t *testing.T) {
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
cfg, err := config.LoadFile(path, "CATALOG_FIXTURE_") cfg, err := config.LoadFile(path)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -40,7 +40,7 @@ func TestCatalogConfigFiles(t *testing.T) {
if err := os.WriteFile(path, []byte(body), 0600); err != nil { if err := os.WriteFile(path, []byte(body), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if _, err := config.LoadFile(path, "CATALOG_FIXTURE_"); err == nil { if _, err := config.LoadFile(path); err == nil {
t.Fatal("accepted invalid catalog", body) t.Fatal("accepted invalid catalog", body)
} }
} }
@@ -57,7 +57,7 @@ func TestCatalogReloadSnapshotIsolation(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
defer logger.Close() defer logger.Close()
reloader, err := config.NewReloader(path, "CATALOG_RELOAD_", logger) reloader, err := config.NewReloader(path, logger)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -20,7 +20,7 @@ func TestTemplateConfigFiles(t *testing.T) {
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
cfg, err := config.LoadFile(path, "TEMPLATE_TEST_") cfg, err := config.LoadFile(path)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -1,190 +1,41 @@
package config package config
import ( import (
"bytes"
"context" "context"
"encoding/json"
"errors" "errors"
"io"
"os"
"path/filepath"
"reflect" "reflect"
"strings"
"sync" "sync"
"time" "time"
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
"git.yuntuo.hk/k2safe/go-omni/logx" "git.yuntuo.hk/k2safe/go-omni/logx"
"gopkg.in/yaml.v3"
) )
// DecodeFile merges a bounded JSON/YAML document into defaults. It uses JSON // LoadFile loads and validates framework configuration using explicit file references.
// field names for both formats, rejects unknown fields and accepts durations func LoadFile(path string) (Config, error) { return LoadProject(path, Default()) }
// such as "3s". Application-specific validation remains with the caller.
func DecodeFile[T any](path string, defaults T) (T, error) {
result, err := decodeFile(path, defaults)
if err != nil {
var zero T
return zero, &DecodeError{cause: err}
}
return result, nil
}
// DecodeError keeps input values out of public diagnostics while preserving
// errors.Is/As for explicitly handled underlying I/O and syntax errors.
type DecodeError struct{ cause error }
func (e *DecodeError) Error() string {
return "config: cannot decode file; check its path, format, field names and value types"
}
func (e *DecodeError) Unwrap() error { return e.cause }
func decodeFile[T any](path string, defaults T) (T, error) {
var zero T
f, err := os.Open(path)
if err != nil {
return zero, err
}
defer f.Close()
data, err := io.ReadAll(io.LimitReader(f, (1<<20)+1))
if err != nil {
return zero, err
}
if len(data) > 1<<20 {
return zero, errors.New("config: file exceeds 1 MiB")
}
var value any
switch strings.ToLower(filepath.Ext(path)) {
case ".yaml", ".yml":
dec := yaml.NewDecoder(bytes.NewReader(data))
if err := dec.Decode(&value); err != nil {
return zero, err
}
var extra any
if err := dec.Decode(&extra); err != io.EOF {
return zero, errors.New("config: one YAML document required")
}
case ".json":
if err := apierrors.DecodeCatalogJSON(data, &value); err != nil {
return zero, err
}
default:
return zero, errors.New("config: expected .json, .yaml or .yml")
}
if _, ok := value.(map[string]any); !ok {
return zero, errors.New("config: root must be an object")
}
value, err = durations(value, reflect.TypeOf(defaults))
if err != nil {
return zero, err
}
encoded, err := json.Marshal(value)
if err != nil {
return zero, err
}
// Clone configuration without serializing borrowed json:"-" capabilities.
var result T
original, err := cloneConfiguration(reflect.ValueOf(&defaults).Elem(), 0)
if err != nil {
return zero, err
}
reflect.ValueOf(&result).Elem().Set(original)
dec := json.NewDecoder(bytes.NewReader(encoded))
dec.DisallowUnknownFields()
if err = dec.Decode(&result); err != nil {
return zero, err
}
return result, nil
}
func durations(v any, t reflect.Type) (any, error) {
if t == nil {
return v, nil
}
for t.Kind() == reflect.Pointer {
t = t.Elem()
}
if t == reflect.TypeOf(time.Duration(0)) {
if s, ok := v.(string); ok {
d, err := time.ParseDuration(s)
return int64(d), err
}
return v, nil
}
if t.Kind() == reflect.Struct {
if m, ok := v.(map[string]any); ok {
for n := 0; n < t.NumField(); n++ {
field := t.Field(n)
name := strings.Split(field.Tag.Get("json"), ",")[0]
if name == "-" {
continue
}
if name == "" {
name = field.Name
}
if item, exists := m[name]; exists {
converted, err := durations(item, field.Type)
if err != nil {
return nil, err
}
m[name] = converted
}
}
}
}
if t.Kind() == reflect.Map && t.Key().Kind() == reflect.String {
if values, ok := v.(map[string]any); ok {
for key, value := range values {
converted, err := durations(value, t.Elem())
if err != nil {
return nil, err
}
values[key] = converted
}
}
}
if t.Kind() == reflect.Slice {
if items, ok := v.([]any); ok {
for n, item := range items {
converted, err := durations(item, t.Elem())
if err != nil {
return nil, err
}
items[n] = converted
}
}
}
return v, nil
}
func LoadFile(path, prefix string) (Config, error) {
cfg, err := DecodeFile(path, Default())
if err != nil {
return Config{}, err
}
return ApplyEnv(cfg, prefix)
}
// Reloader only accepts log.level changes. Static changes and invalid files // Reloader only accepts log.level changes. Static changes and invalid files
// leave the current snapshot and logger untouched. Watch does not own a goroutine. // leave the current snapshot and logger untouched. Watch does not own a goroutine.
type Reloader struct { type Reloader struct {
mu sync.Mutex mu sync.Mutex
path, prefix string path string
current Config current Config
logger *logx.Logger logger *logx.Logger
} }
func NewReloader(path, prefix string, logger *logx.Logger) (*Reloader, error) { func NewReloader(path string, logger *logx.Logger) (*Reloader, error) {
cfg, err := LoadFile(path, prefix) cfg, err := LoadFile(path)
if err != nil { if err != nil {
return nil, err return nil, err
} }
if err = logger.SetLevel(cfg.Log.Level); err != nil { if err = logger.SetLevel(cfg.Log.Level); err != nil {
return nil, err return nil, err
} }
return &Reloader{path: path, prefix: prefix, current: cfg, logger: logger}, nil return &Reloader{path: path, current: cfg, logger: logger}, nil
} }
func (r *Reloader) Reload() error { func (r *Reloader) Reload() error {
r.mu.Lock() r.mu.Lock()
defer r.mu.Unlock() defer r.mu.Unlock()
next, err := LoadFile(r.path, r.prefix) next, err := LoadFile(r.path)
if err != nil { if err != nil {
return err return err
} }

View File

@@ -20,8 +20,8 @@ func TestFileMergeAndSafeReload(t *testing.T) {
} }
write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: info\n") write("address: ':8123'\nshutdownTimeout: 7s\nlog:\n level: info\n")
t.Setenv("TEST_MODULE_HTTP_ADDR", ":9123") t.Setenv("TEST_MODULE_HTTP_ADDR", ":9123")
cfg, err := LoadFile(path, "TEST_MODULE_") cfg, err := LoadFile(path)
if err != nil || cfg.Address != ":9123" || cfg.ShutdownTimeout != 7*time.Second { if err != nil || cfg.Address != ":8123" || cfg.ShutdownTimeout != 7*time.Second {
t.Fatal(cfg, err) t.Fatal(cfg, err)
} }
var output bytes.Buffer var output bytes.Buffer
@@ -33,7 +33,7 @@ func TestFileMergeAndSafeReload(t *testing.T) {
} }
defer l.Close() defer l.Close()
child := l.With("scope", "test") child := l.With("scope", "test")
r, err := NewReloader(path, "TEST_MODULE_", l) r, err := NewReloader(path, l)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }

View File

@@ -22,7 +22,7 @@ func TestHTTPFileAndEnvironment(t *testing.T) {
if err := os.WriteFile(path, []byte(file.body), 0600); err != nil { if err := os.WriteFile(path, []byte(file.body), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
cfg, err := config.LoadFile(path, "HTTP_TEST_") cfg, err := config.LoadFile(path)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -35,7 +35,10 @@ func TestHTTPFileAndEnvironment(t *testing.T) {
t.Setenv("HTTP_TEST_HTTP_WRITE_TIMEOUT", "8s") t.Setenv("HTTP_TEST_HTTP_WRITE_TIMEOUT", "8s")
t.Setenv("HTTP_TEST_HTTP_IDLE_TIMEOUT", "9s") t.Setenv("HTTP_TEST_HTTP_IDLE_TIMEOUT", "9s")
t.Setenv("HTTP_TEST_HTTP_MAX_HEADER_BYTES", "16384") t.Setenv("HTTP_TEST_HTTP_MAX_HEADER_BYTES", "16384")
got, err := config.ApplyEnv(cfg, "HTTP_TEST_") if err := os.WriteFile(path, []byte(`{"http":{"readHeaderTimeout":"${HTTP_TEST_HTTP_READ_HEADER_TIMEOUT}","readTimeout":"${HTTP_TEST_HTTP_READ_TIMEOUT}","writeTimeout":"${HTTP_TEST_HTTP_WRITE_TIMEOUT}","idleTimeout":"${HTTP_TEST_HTTP_IDLE_TIMEOUT}","maxHeaderBytes":"${HTTP_TEST_HTTP_MAX_HEADER_BYTES}"}}`), 0600); err != nil {
t.Fatal(err)
}
got, err := config.LoadProject(path, cfg)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -55,7 +58,12 @@ func TestHTTPInvalidAndUnsupportedSettings(t *testing.T) {
value = "-1s" value = "-1s"
} }
t.Setenv("HTTP_INVALID_HTTP_"+key, value) t.Setenv("HTTP_INVALID_HTTP_"+key, value)
if _, err := config.LoadFromEnv("HTTP_INVALID_"); err == nil { names := map[string]string{"READ_HEADER_TIMEOUT": "readHeaderTimeout", "READ_TIMEOUT": "readTimeout", "WRITE_TIMEOUT": "writeTimeout", "IDLE_TIMEOUT": "idleTimeout", "MAX_HEADER_BYTES": "maxHeaderBytes"}
path := filepath.Join(t.TempDir(), "config.json")
if err := os.WriteFile(path, []byte(`{"http":{"`+names[key]+`":"${HTTP_INVALID_HTTP_`+key+`}"}}`), 0600); err != nil {
t.Fatal(err)
}
if _, err := config.LoadFile(path); err == nil {
t.Fatal("invalid HTTP value accepted") t.Fatal("invalid HTTP value accepted")
} }
}) })

View File

@@ -14,7 +14,7 @@ func TestLifecycleConfiguration(t *testing.T) {
if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: 2s\n drainTimeout: 3s\n cancelTimeout: 4s\n"), 0600); err != nil { if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: 2s\n drainTimeout: 3s\n cancelTimeout: 4s\n"), 0600); err != nil {
t.Fatal(err) t.Fatal(err)
} }
cfg, err := config.LoadFile(path, "PHASE_TEST_") cfg, err := config.LoadFile(path)
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -24,13 +24,16 @@ func TestLifecycleConfiguration(t *testing.T) {
for _, key := range []string{"STOP_TIMEOUT", "DRAIN_TIMEOUT", "CANCEL_TIMEOUT"} { for _, key := range []string{"STOP_TIMEOUT", "DRAIN_TIMEOUT", "CANCEL_TIMEOUT"} {
t.Setenv("PHASE_TEST_LIFECYCLE_"+key, "6s") t.Setenv("PHASE_TEST_LIFECYCLE_"+key, "6s")
} }
cfg, err = config.ApplyEnv(cfg, "PHASE_TEST_") if err := os.WriteFile(path, []byte("lifecycle:\n stopTimeout: ${PHASE_TEST_LIFECYCLE_STOP_TIMEOUT}\n drainTimeout: ${PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT}\n cancelTimeout: ${PHASE_TEST_LIFECYCLE_CANCEL_TIMEOUT}\n"), 0600); err != nil {
t.Fatal(err)
}
cfg, err = config.LoadProject(path, cfg)
if err != nil || cfg.Lifecycle.StopTimeout != 6*time.Second || cfg.Lifecycle.DrainTimeout != 6*time.Second || cfg.Lifecycle.CancelTimeout != 6*time.Second { if err != nil || cfg.Lifecycle.StopTimeout != 6*time.Second || cfg.Lifecycle.DrainTimeout != 6*time.Second || cfg.Lifecycle.CancelTimeout != 6*time.Second {
t.Fatal(cfg.Lifecycle, err) t.Fatal(cfg.Lifecycle, err)
} }
for _, v := range []string{"bad", "-1s"} { for _, v := range []string{"bad", "-1s"} {
t.Setenv("PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT", v) t.Setenv("PHASE_TEST_LIFECYCLE_DRAIN_TIMEOUT", v)
if _, err := config.ApplyEnv(cfg, "PHASE_TEST_"); err == nil { if _, err := config.LoadProject(path, cfg); err == nil {
t.Fatal("invalid duration accepted") t.Fatal("invalid duration accepted")
} }
} }

177
config/loader.go Normal file
View File

@@ -0,0 +1,177 @@
package config
import (
"errors"
"io"
"os"
"path/filepath"
"reflect"
"strings"
)
const maxConfigBytes = 1 << 20
const maxConfigDepth = 64
const maxConfigNodes = 100000
// Decoder must not retain or mutate input and must support concurrent calls
// when its Loader is shared. It is trusted application code, not a sandbox.
type Decoder interface {
Decode([]byte) (Document, error)
}
type DecoderFunc func([]byte) (Document, error)
func (f DecoderFunc) Decode(data []byte) (Document, error) { return f(data) }
// Loader owns an immutable per-instance decoder registry. No global registration.
type Loader struct {
decoders map[string]Decoder
sealed bool
lookup func(string) (string, bool)
}
type LoaderOption func(*Loader) error
// WithDecoder adds a new extension; builtin/duplicate extensions cannot be replaced.
func WithDecoder(extension string, decoder Decoder) LoaderOption {
return func(l *Loader) error {
if l == nil || l.decoders == nil || l.sealed {
return errors.New("config: decoder registry cannot be changed after construction")
}
ext := strings.ToLower(strings.TrimPrefix(extension, "."))
if ext == "" || strings.ContainsAny(ext, "./\\ \t\r\n") {
return errors.New("config: invalid decoder extension")
}
if decoder == nil {
return errors.New("config: nil decoder")
}
v := reflect.ValueOf(decoder)
if (v.Kind() == reflect.Pointer || v.Kind() == reflect.Func || v.Kind() == reflect.Map || v.Kind() == reflect.Slice || v.Kind() == reflect.Interface) && v.IsNil() {
return errors.New("config: nil decoder")
}
if _, exists := l.decoders[ext]; exists {
return errors.New("config: decoder extension already registered")
}
l.decoders[ext] = decoder
return nil
}
}
func NewLoader(options ...LoaderOption) (*Loader, error) {
l := &Loader{lookup: os.LookupEnv, decoders: map[string]Decoder{"json": JSONDecoder{}, "yaml": YAMLDecoder{}, "yml": YAMLDecoder{}, "xml": XMLDecoder{}}}
for _, o := range options {
if o == nil {
return nil, errors.New("config: nil loader option")
}
if err := o(l); err != nil {
return nil, err
}
}
l.sealed = true
return l, nil
}
var defaultLoader, _ = NewLoader()
func selectLoader(loaders []*Loader) (*Loader, error) {
if len(loaders) == 0 {
return defaultLoader, nil
}
if len(loaders) != 1 || loaders[0] == nil {
return nil, errors.New("config: expected one non-nil loader")
}
return loaders[0], nil
}
func (l *Loader) read(path string) (doc Document, err error) {
defer func() {
if err != nil {
var d *Error
if errors.As(err, &d) {
c := *d
c.File = path
err = &c
} else {
reason := "cannot read configuration file"
if errors.Is(err, os.ErrNotExist) {
reason = "configuration file does not exist"
} else if errors.Is(err, os.ErrPermission) {
reason = "permission denied reading configuration file"
}
err = &Error{File: path, Reason: reason, cause: err}
}
}
}()
decoder := l.decoders[strings.TrimPrefix(strings.ToLower(filepath.Ext(path)), ".")]
if decoder == nil {
return doc, &Error{Reason: "unsupported configuration format"}
}
info, e := os.Stat(path)
if e != nil {
return doc, e
}
if !info.Mode().IsRegular() {
return doc, &Error{Reason: "configuration must be a regular file"}
}
f, e := os.Open(path)
if e != nil {
return doc, e
}
defer f.Close()
info, e = f.Stat()
if e != nil {
return doc, e
}
if !info.Mode().IsRegular() {
return doc, &Error{Reason: "configuration must be a regular file"}
}
data, e := io.ReadAll(io.LimitReader(f, maxConfigBytes+1))
if e != nil {
return doc, e
}
if len(data) > maxConfigBytes {
return doc, &Error{Reason: "file exceeds 1 MiB"}
}
doc, err = decodeSafely(decoder, data)
if err != nil {
return doc, err
}
if _, ok := doc.Value.(map[string]any); !ok {
return doc, doc.diagnostic("", "root must be an object", nil)
}
return resolveDocument(doc, l.lookup)
}
func decodeSafely(decoder Decoder, data []byte) (doc Document, err error) {
defer func() {
if recover() != nil {
err = &Error{Reason: "decoder panicked"}
}
}()
doc, err = decoder.Decode(data)
if err != nil {
var d *Error
if !errors.As(err, &d) {
err = &Error{Reason: "invalid configuration syntax", cause: err}
}
}
return
}
// DecodeFile overlays a bounded document onto independent defaults. With no
// loader argument it accepts JSON/YAML/XML. It never opens dependencies.
func DecodeFile[T any](path string, defaults T, loaders ...*Loader) (T, error) {
var zero T
l, err := selectLoader(loaders)
if err != nil {
return zero, err
}
doc, err := l.read(path)
if err != nil {
return zero, err
}
if err := rejectEnvironmentTags(reflect.TypeOf(defaults), map[reflect.Type]bool{}); err != nil {
return zero, err
}
result, err := bindDocument(doc, defaults)
if err != nil {
return zero, locateError(err, path, doc)
}
return result, nil
}

377
config/loader_test.go Normal file
View File

@@ -0,0 +1,377 @@
package config
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"reflect"
"strings"
"sync"
"testing"
"time"
)
type loaderExample struct {
Server Config `json:"server"`
Upload struct {
MaxBytes int64 `json:"maxBytes"`
} `json:"upload"`
Values []int64 `json:"values"`
Optional *string `json:"optional"`
Enabled bool `json:"enabled"`
Dynamic any `json:"dynamic"`
}
func (c loaderExample) Validate() error {
if err := c.Server.Validate(); err != nil {
return AtField("server", err)
}
if c.Upload.MaxBytes < 0 {
return Invalid("upload.maxBytes", "must be nonnegative")
}
return nil
}
func noEnvironment(string) (string, bool) { return "", false }
func configFile(t *testing.T, ext, raw string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "service."+ext)
if err := os.WriteFile(path, []byte(raw), 0600); err != nil {
t.Fatal(err)
}
return path
}
func TestUnifiedFormatAndPrecedence(t *testing.T) {
formats := map[string]string{
"json": `{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":"${APP_UPLOAD_MAX_BYTES}"},"values":[9007199254740993],"optional":null,"enabled":"${APP_ENABLED}","dynamic":{"id":9007199254740993}}`,
"yaml": "server:\n http:\n readTimeout: 2s\nupload:\n maxBytes: \"${APP_UPLOAD_MAX_BYTES}\"\nvalues: [9007199254740993]\noptional: null\nenabled: \"${APP_ENABLED}\"\ndynamic:\n id: 9007199254740993\n",
"xml": `<config><server><http><readTimeout>2s</readTimeout></http></server><upload><maxBytes>${APP_UPLOAD_MAX_BYTES}</maxBytes></upload><values type="array"><item>9007199254740993</item></values><optional type="null"/><enabled>${APP_ENABLED}</enabled><dynamic><id type="number">9007199254740993</id></dynamic></config>`,
}
var reference loaderExample
for _, ext := range []string{"json", "yaml", "xml"} {
t.Run(ext, func(t *testing.T) {
old := "old"
base := loaderExample{Server: Default(), Optional: &old}
cfg, err := LoadProject(configFile(t, ext, formats[ext]), base, testLoader(func(key string) (string, bool) {
v, ok := map[string]string{"APP_UPLOAD_MAX_BYTES": "0", "APP_ENABLED": "false"}[key]
return v, ok
}))
if err != nil {
t.Fatal(err)
}
if cfg.Upload.MaxBytes != 0 || cfg.Enabled || cfg.Optional != nil || cfg.Values[0] != 9007199254740993 || cfg.Server.HTTP.ReadTimeout != 2*time.Second {
t.Fatal("values changed", cfg)
}
if cfg.Dynamic.(map[string]any)["id"] != json.Number("9007199254740993") {
t.Fatal("dynamic integer precision lost")
}
if ext == "json" {
reference = cfg
} else if !reflect.DeepEqual(reference, cfg) {
t.Fatal("format-dependent semantics")
}
if base.Optional == nil || *base.Optional != "old" {
t.Fatal("defaults mutated")
}
})
}
cfg, err := LoadProject("", loaderExample{Server: Default()}, testLoader(noEnvironment))
if err != nil || cfg.Upload.MaxBytes != 0 {
t.Fatal(cfg, err)
}
}
func TestConfigDiagnosticsAndCauses(t *testing.T) {
for _, tc := range []struct {
ext, body, field string
line bool
}{
{"json", "{\n\"upload\": {\"maxBytes\":\"private-value\"}}", "upload.maxBytes", true},
{"yaml", "server:\n http:\n readTimeout: private-value", "server.http.readTimeout", true},
{"xml", "<config>\n<upload><maxBytes>private-value</maxBytes></upload></config>", "upload.maxBytes", true},
{"json", `{"upload":{"maxBytes":-1}}`, "upload.maxBytes", true},
{"yaml", "server:\n http:\n readTimeout: -1s", "server.http.readTimeout", true},
{"json", `{"upload":{"typo":3}}`, "upload.typo", true},
} {
t.Run(tc.ext+tc.field, func(t *testing.T) {
path := configFile(t, tc.ext, tc.body)
_, err := LoadProject(path, loaderExample{Server: Default()}, testLoader(noEnvironment))
var d *Error
if !errors.As(err, &d) || d.File != path || d.Field != tc.field || d.Reason == "" || tc.line && d.Line < 1 {
t.Fatalf("missing location: %#v %v", d, err)
}
if strings.Contains(err.Error(), "private-value") {
t.Fatal("secret leaked", err)
}
})
}
path := filepath.Join(t.TempDir(), "missing.json")
_, err := LoadProject(path, loaderExample{})
if !errors.Is(err, os.ErrNotExist) {
t.Fatal("I/O cause discarded", err)
}
_, err = DecodeFile(configFile(t, "json", `{"count":"private-value"}`), struct {
Count int `json:"count"`
}{})
var typed *json.UnmarshalTypeError
if !errors.As(err, &typed) {
t.Fatal("type cause discarded", err)
}
_, err = LoadProject(configFile(t, "yaml", "upload:\n maxBytes: \"${APP_UPLOAD_MAX_BYTES}\""), loaderExample{Server: Default()}, testLoader(func(k string) (string, bool) { return "-1", k == "APP_UPLOAD_MAX_BYTES" }))
var d *Error
if !errors.As(err, &d) || d.Env != "APP_UPLOAD_MAX_BYTES" || d.Field != "upload.maxBytes" || d.File == "" || d.Line != 2 {
t.Fatal("missing file and variable provenance", err)
}
}
func TestMalformedFormatsAndLimits(t *testing.T) {
for ext, bodies := range map[string][]string{
"json": {`{"enabled":true,"enabled":false}`, `{} {}`, `{"upload":`, `{"enabled":true,"Enabled":false}`, strings.Repeat(`{"a":`, 70) + `0` + strings.Repeat(`}`, 70)},
"yaml": {"enabled: true\nenabled: false", "---\n{}\n---\n{}", "a: &x [1]\nb: *x", "a:\n <<: {}", "upload: [\n"},
"xml": {`<!DOCTYPE config><config/>`, `<config>&private-value;</config>`, `<config/><config/>`, `<config xmlns="x"/>`, `<config type="" type=""/>`, `<config><enabled>true</enabled><enabled>false</enabled></config>`, `<config>text<enabled>true</enabled></config>`, `<config><values type="array"><bad>1</bad></values></config>`},
} {
for i, raw := range bodies {
t.Run(fmt.Sprintf("%s%d", ext, i), func(t *testing.T) {
_, err := LoadProject(configFile(t, ext, raw), loaderExample{Server: Default()}, testLoader(noEnvironment))
if err == nil || strings.Contains(err.Error(), "private-value") {
t.Fatal(err)
}
})
}
}
if _, err := DecodeFile(configFile(t, "json", strings.Repeat(" ", maxConfigBytes+1)), loaderExample{}); err == nil {
t.Fatal("unbounded file")
}
if _, err := DecodeFile(configFile(t, "json", string([]byte{'{', '"', 0xff, '"', ':', '0', '}'})), loaderExample{}); err == nil {
t.Fatal("invalid UTF-8")
}
}
func TestDecoderRegistryAndConcurrentUse(t *testing.T) {
decoder := DecoderFunc(func([]byte) (Document, error) {
return Document{Value: map[string]any{"upload": map[string]any{"maxBytes": json.Number("42")}}}, nil
})
l, err := NewLoader(WithDecoder("custom", decoder))
if err != nil {
t.Fatal(err)
}
path := configFile(t, "custom", "ignored")
var wg sync.WaitGroup
for i := 0; i < 24; i++ {
wg.Add(1)
go func() {
defer wg.Done()
cfg, e := LoadProject(path, loaderExample{Server: Default()}, l)
if e != nil || cfg.Upload.MaxBytes != 42 {
t.Error(e)
}
}()
}
wg.Wait()
if _, err := DecodeFile(path, loaderExample{}); err == nil {
t.Fatal("decoder registration leaked globally")
}
for _, options := range [][]LoaderOption{{WithDecoder("json", decoder)}, {WithDecoder("x", decoder), WithDecoder("x", decoder)}, {WithDecoder("x", DecoderFunc(nil))}} {
if _, err = NewLoader(options...); err == nil {
t.Fatal("invalid registration")
}
}
l, _ = NewLoader(WithDecoder("panic", DecoderFunc(func([]byte) (Document, error) { panic("private-value") })))
_, err = DecodeFile(configFile(t, "panic", ""), loaderExample{}, l)
if err == nil || strings.Contains(err.Error(), "private-value") {
t.Fatal(err)
}
cycle := map[string]any{}
cycle["self"] = cycle
l, _ = NewLoader(WithDecoder("cycle", DecoderFunc(func([]byte) (Document, error) { return Document{Value: map[string]any{"dynamic": cycle}}, nil })))
if _, err = DecodeFile(configFile(t, "cycle", ""), loaderExample{}, l); err == nil {
t.Fatal("decoder bypassed traversal limit")
}
}
func FuzzConfigurationDecoders(f *testing.F) {
f.Add(uint8(0), []byte(`{"count":1}`))
f.Add(uint8(1), []byte("count: 1"))
f.Add(uint8(2), []byte("<config><count>1</count></config>"))
f.Fuzz(func(t *testing.T, format uint8, data []byte) {
if len(data) > 8192 {
data = data[:8192]
}
ds := []Decoder{JSONDecoder{}, YAMLDecoder{}, XMLDecoder{}}
doc, err := ds[int(format)%len(ds)].Decode(data)
if err == nil {
_, _ = bindDocument(doc, struct {
Count int `json:"count"`
}{})
}
})
}
func BenchmarkConfiguration(b *testing.B) {
for name, tc := range map[string]struct {
d Decoder
raw string
}{"json": {JSONDecoder{}, `{"count":9007199254740993,"timeout":"5s","values":[1,2,3]}`}, "yaml": {YAMLDecoder{}, "count: 9007199254740993\ntimeout: 5s\nvalues: [1,2,3]"}, "xml": {XMLDecoder{}, `<config><count>9007199254740993</count><timeout>5s</timeout><values type="array"><item>1</item><item>2</item><item>3</item></values></config>`}} {
b.Run(name, func(b *testing.B) {
b.ReportAllocs()
raw := []byte(tc.raw)
for i := 0; i < b.N; i++ {
doc, err := tc.d.Decode(raw)
if err != nil {
b.Fatal(err)
}
_, err = bindDocument(doc, struct {
Count int64 `json:"count"`
Timeout time.Duration `json:"timeout"`
Values []int `json:"values"`
}{})
if err != nil {
b.Fatal(err)
}
}
})
}
}
// Text-only scalars and embedded config fields use the same behavior in each format.
type testText struct{ Value string }
func (v *testText) UnmarshalText(data []byte) error { v.Value = string(data); return nil }
func TestConfigScalarAndEmbedding(t *testing.T) {
type Fields struct {
Name string `json:"name"`
}
type C struct {
Fields
Nested Fields `json:"nested"`
Scalar testText `json:"scalar"`
Bytes []byte `json:"bytes"`
}
cases := map[string]string{
"json": `{"name":"outer","nested":{"name":"inner"},"scalar":"text","bytes":"aGk="}`,
"yaml": "name: outer\nnested:\n name: inner\nscalar: text\nbytes: aGk=\n",
"xml": `<config><name>outer</name><nested><name>inner</name></nested><scalar>text</scalar><bytes>aGk=</bytes></config>`,
}
for ext, data := range cases {
t.Run(ext, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "config."+ext)
os.WriteFile(path, []byte(data), 0600)
got, err := DecodeFile(path, &C{})
if err != nil {
t.Fatal(err)
}
if got.Name != "outer" || got.Nested.Name != "inner" || got.Scalar.Value != "text" || string(got.Bytes) != "hi" {
t.Fatal(got)
}
})
}
type Tagged struct {
Fields `json:"nested"`
}
doc, _ := (JSONDecoder{}).Decode([]byte(`{"name":"wrong"}`))
if _, err := bindDocument(doc, Tagged{}); err == nil {
t.Fatal("tagged embed incorrectly flattened")
}
}
func TestConfigDefaultsIsolationAndEmptyValues(t *testing.T) {
type Child struct {
Name string `json:"name"`
}
type C struct {
Child *Child `json:"child"`
Items []int `json:"items"`
Labels map[string]string `json:"labels"`
Enabled bool `json:"enabled"`
}
base := C{Child: &Child{Name: "old"}, Items: []int{1}, Labels: map[string]string{"old": "yes"}, Enabled: true}
doc, _ := (JSONDecoder{}).Decode([]byte(`{"child":{"name":"new"},"labels":{"new":"yes"},"items":[],"enabled":false}`))
got, err := bindDocument(doc, base)
if err != nil {
t.Fatal(err)
}
if got.Child.Name != "new" || got.Items == nil || len(got.Items) != 0 || got.Enabled {
t.Fatal(got)
}
if base.Child.Name != "old" || len(base.Items) != 1 || len(base.Labels) != 1 {
t.Fatal("mutated defaults")
}
bad, _ := (JSONDecoder{}).Decode([]byte(`{"child":{"name":"changed"},"items":"bad"}`))
if _, err := bindDocument(bad, base); err == nil {
t.Fatal("invalid input accepted")
}
if base.Child.Name != "old" {
t.Fatal("failed load changed default pointer")
}
null, _ := (JSONDecoder{}).Decode([]byte(`{"child":null,"items":null,"labels":null,"enabled":null}`))
got, err = bindDocument(null, base)
if err != nil || got.Child != nil || got.Items != nil || got.Labels != nil || !got.Enabled {
t.Fatal(got, err)
}
}
func TestDefaultStageDoesNotReadEnvironment(t *testing.T) {
reads := 0
type C struct {
Value int `json:"value"`
}
got, err := LoadProject("", C{Value: 3}, testLoader(func(string) (string, bool) { reads++; return "9", true }))
if err != nil || got.Value != 3 || reads != 0 {
t.Fatal(got, err, reads)
}
}
func testLoader(lookup func(string) (string, bool)) *Loader {
l, err := NewLoader(WithLookupEnv(lookup))
if err != nil {
panic(err)
}
return l
}
func TestDecoderRegistryFrozen(t *testing.T) {
l, err := NewLoader()
if err != nil {
t.Fatal(err)
}
if err = WithDecoder("extra", JSONDecoder{})(l); err == nil {
t.Fatal("mutable registry")
}
}
func BenchmarkConfigurationStartup(b *testing.B) {
path := filepath.Join(b.TempDir(), "app.json")
if err := os.WriteFile(path, []byte(`{"server":{"http":{"readTimeout":"2s"}},"upload":{"maxBytes":4096}}`), 0600); err != nil {
b.Fatal(err)
}
defaults := loaderExample{Server: Default()}
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := LoadProject(path, defaults); err != nil {
b.Fatal(err)
}
}
}
func BenchmarkConfigurationLarge(b *testing.B) {
type Entry struct {
Timeout time.Duration `json:"timeout"`
Count int `json:"count"`
}
type Large struct {
Entries []Entry `json:"entries"`
}
raw := []byte(`{"entries":[` + strings.TrimSuffix(strings.Repeat(`{"timeout":"3s","count":10},`, 1000), ",") + `]}`)
b.SetBytes(int64(len(raw)))
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
doc, err := (JSONDecoder{}).Decode(raw)
if err != nil {
b.Fatal(err)
}
if _, err = bindDocument(doc, Large{}); err != nil {
b.Fatal(err)
}
}
}

View File

@@ -0,0 +1,85 @@
package config_test
import (
"errors"
"os"
"path/filepath"
"strings"
"testing"
"git.yuntuo.hk/k2safe/go-omni/config"
mysql "git.yuntuo.hk/k2safe/go-omni/database/mysql"
"git.yuntuo.hk/k2safe/go-omni/httpc"
)
type moduleConfig struct {
Database mysql.Config `json:"database"`
Server config.Config `json:"server"`
Client httpc.Config `json:"client"`
}
func (c moduleConfig) Validate() error {
if err := c.Database.Validate(); err != nil {
return config.AtField("database", err)
}
if err := c.Server.Validate(); err != nil {
return config.AtField("server", err)
}
return config.AtField("client", c.Client.Validate())
}
func TestModuleFieldDiagnostics(t *testing.T) {
for _, tc := range []struct{ name, source, field, reason string }{
{"port", "database:\n port: 70000\n", "database.port", "1 and 65535"},
{"host", "database:\n host: 'invalid host'\n", "database.host", "hostname"},
{"pool", "database:\n pool:\n maxIdleConns: 999\n", "database.pool.maxIdleConns", "maxOpenConns"},
{"ping", "database:\n pool:\n pingTimeout: 0s\n", "database.pool.pingTimeout", "positive"},
{"timeout", "database:\n readTimeout: -1s\n", "database.readTimeout", "nonnegative"},
{"mysqlZone", "database:\n location: Secret/Not_A_Zone\n", "database.location", "IANA"},
{"tlsPair", "database:\n tls:\n certFile: secret-file\n keyFile: ''\n", "database.tls.keyFile", "certFile"},
{"level", "server:\n log:\n level: secret-level\n", "server.log.level", "debug"},
{"format", "server:\n log:\n format: secret-format\n", "server.log.format", "json or text"},
{"rotation", "server:\n log:\n rotation:\n maxBackups: -1\n", "server.log.rotation.maxBackups", "10000"},
{"zone", "server:\n log:\n timeZone: Secret/Not_A_Zone\n", "server.log.timeZone", "IANA"},
{"lifecycle", "server:\n lifecycle:\n stopTimeout: -1s\n", "server.lifecycle.stopTimeout", "nonnegative"},
{"policy", "server:\n request:\n body: secret-mode\n", "server.request.body", "required"},
{"clientPool", "client:\n maxIdleConns: -1\n", "client.maxIdleConns", "131072"},
{"proxy", "client:\n proxy:\n mode: url\n url: 'https://private:secret@localhost/path'\n", "client.proxy.url", "origin"},
} {
t.Run(tc.name, func(t *testing.T) {
path := filepath.Join(t.TempDir(), "config.yaml")
if err := os.WriteFile(path, []byte(tc.source), 0600); err != nil {
t.Fatal(err)
}
_, err := config.LoadProject(path, moduleConfig{Database: mysql.DefaultConfig(), Server: config.Default()})
var d *config.Error
if !errors.As(err, &d) || d.Field != tc.field || d.File != path || d.Line <= 1 || d.Column <= 0 || !strings.Contains(d.Reason, tc.reason) {
t.Fatalf("diagnostic: %v", err)
}
if strings.Contains(err.Error(), "secret") || strings.Contains(err.Error(), "Secret/") || strings.Contains(err.Error(), "private") {
t.Fatalf("value leaked: %v", err)
}
})
}
}
func TestModuleDiagnosticsKeepReferenceAndCause(t *testing.T) {
t.Setenv("CHECK_PORT", "70000")
path := filepath.Join(t.TempDir(), "config.yaml")
os.WriteFile(path, []byte("database:\n port: '${CHECK_PORT}'\n"), 0600)
_, err := config.LoadProject(path, moduleConfig{Database: mysql.DefaultConfig(), Server: config.Default()})
var d *config.Error
if !errors.As(err, &d) || d.Field != "database.port" || d.Env != "CHECK_PORT" || d.Line != 2 {
t.Fatal(err)
}
original := mysql.DefaultConfig()
original.Port = 70000
cause := original.Validate()
wrapped := config.AtField("database", cause)
if !errors.Is(wrapped, cause) {
t.Fatal("lost original validation error")
}
// Unknown project errors continue to be redacted rather than echoed.
wrapped = config.AtField("custom", errors.New("private-password"))
if strings.Contains(wrapped.Error(), "private-password") {
t.Fatal(wrapped)
}
}

View File

@@ -1,221 +1,105 @@
package config package config
import ( import (
"bytes"
"encoding" "encoding"
"encoding/json"
"errors" "errors"
"fmt"
apierrors "git.yuntuo.hk/k2safe/go-omni/errors"
"math" "math"
"os"
"reflect" "reflect"
"strconv" "strconv"
"strings" "strings"
"time" "time"
"unicode"
) )
// EnvOptions controls typed project configuration overlays. Explicit env tags // LoadProject clones defaults, resolves only explicit file references, binds the
// are absolute names; otherwise JSON names become PREFIX_NESTED_FIELD names. // provided fields and validates. Empty path uses only defaults. No env tags,
// Empty is allow (default), ignore or reject. No input values appear in errors. // prefixes, aliases or implicit environment overlays are applied.
type EnvOptions struct { func LoadProject[T any](path string, defaults T, loaders ...*Loader) (T, error) {
Prefix string
Empty string
Lookup func(string) (string, bool)
// Aliases maps canonical absolute environment names to ordered fallback names.
// A present canonical value wins, including an explicit empty value.
Aliases map[string][]string
}
// ApplyDefaults applies envDefault tags to zero fields before file overlays.
func ApplyDefaults[T any](base T) (T, error) { return projectOverlay(base, EnvOptions{}, true) }
// LoadProject applies code/tag defaults, a strict optional file, environment,
// then the project's Validate method, before any logger or dependency is opened.
// envDefault is only applied to zero-valued defaults before reading the file.
func LoadProject[T any](path string, defaults T, options EnvOptions) (T, error) {
var zero T var zero T
cfg, err := projectOverlay(defaults, options, true) loader, err := selectLoader(loaders)
if err != nil { if err != nil {
return zero, err return zero, err
} }
if err = rejectEnvironmentTags(reflect.TypeOf(defaults), map[reflect.Type]bool{}); err != nil {
return zero, err
}
doc := Document{Value: map[string]any{}}
if path != "" { if path != "" {
cfg, err = DecodeFile(path, cfg) doc, err = loader.read(path)
if err != nil { if err != nil {
return zero, errors.New("config: cannot decode file; check its path, format, field names and value types") return zero, err
} }
} }
cfg, err = OverlayEnv(cfg, options) cfg, err := bindDocument(doc, defaults)
if err != nil { if err != nil {
return zero, err return zero, locateError(err, path, doc)
} }
if valid, ok := any(&cfg).(interface{ Validate() error }); ok { if valid, ok := any(&cfg).(interface{ Validate() error }); ok {
if err = valid.Validate(); err != nil { if err = valid.Validate(); err != nil {
return zero, err return zero, locateError(err, path, doc)
}
} else if valid, ok := any(cfg).(interface{ Validate() error }); ok {
if err = valid.Validate(); err != nil {
return zero, locateError(err, path, doc)
} }
} }
return cfg, nil return cfg, nil
} }
func locateError(err error, path string, doc Document) *Error {
d := validationError(err)
d.File = path
if d.Line == 0 {
p := doc.Positions[d.Field]
d.Line = p.Line
d.Column = p.Column
}
for field := d.Field; field != ""; {
if names := doc.environments[field]; len(names) > 0 {
d.Env = strings.Join(names, ", ")
break
}
i := strings.LastIndexAny(field, ".[")
if i < 0 {
break
}
field = field[:i]
}
return d
}
// OverlayEnv overlays the entire project struct without mutating the input. // Reject removed declarations rather than silently ignoring deployment rules.
// Tags: env:"CUSTOM_NAME,required,secret,empty=reject"; env:"-" excludes a func rejectEnvironmentTags(t reflect.Type, seen map[reflect.Type]bool) error {
// subtree. required means nonempty string/non-nil pointer or collection; false if t == nil || seen[t] {
// and zero are valid scalars. Use project Validate for semantic constraints. return nil
// Slices/maps use JSON syntax, avoiding ambiguous comma and empty semantics. }
func OverlayEnv[T any](base T, options EnvOptions) (T, error) { seen[t] = true
return projectOverlay(base, options, false) switch t.Kind() {
case reflect.Pointer, reflect.Slice, reflect.Array, reflect.Map:
return rejectEnvironmentTags(t.Elem(), seen)
case reflect.Struct:
for i := 0; i < t.NumField(); i++ {
f := t.Field(i)
if f.PkgPath != "" {
continue
}
_, env := f.Tag.Lookup("env")
_, def := f.Tag.Lookup("envDefault")
if env || def {
return &Error{Field: f.Name, Reason: "env/envDefault tags were removed; use Default() and explicit file references; see CONFIG.md"}
}
if f.Tag.Get("json") != "-" {
if err := rejectEnvironmentTags(f.Type, seen); err != nil {
return err
}
}
}
}
return nil
} }
var textType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem() var textType = reflect.TypeOf((*encoding.TextUnmarshaler)(nil)).Elem()
var durationType = reflect.TypeOf(time.Duration(0)) var durationType = reflect.TypeOf(time.Duration(0))
func projectOverlay[T any](base T, options EnvOptions, defaults bool) (T, error) { func parseTextValue(v reflect.Value, raw string) (err error) {
var zero T
if options.Lookup == nil {
options.Lookup = os.LookupEnv
}
if options.Empty == "" {
options.Empty = "allow"
}
if options.Empty != "allow" && options.Empty != "ignore" && options.Empty != "reject" {
return zero, errors.New("config: empty policy must be allow, ignore or reject")
}
v := reflect.ValueOf(&base).Elem()
if v.Kind() != reflect.Struct {
return zero, errors.New("config: project configuration must be a struct")
}
result := reflect.New(v.Type()).Elem()
result.Set(v)
seen := map[string]bool{}
var walk func(reflect.Value, string, int) error
walk = func(value reflect.Value, prefix string, depth int) error {
if depth > 32 {
return errors.New("config: nesting exceeds 32")
}
for i := 0; i < value.NumField(); i++ {
field := value.Type().Field(i)
dst := value.Field(i)
if !dst.CanSet() || field.PkgPath != "" {
continue
}
tag := strings.Split(field.Tag.Get("env"), ",")
if tag[0] == "-" {
continue
}
jsonName := strings.Split(field.Tag.Get("json"), ",")[0]
if jsonName == "-" && tag[0] == "" {
continue
}
if jsonName == "" {
jsonName = field.Name
}
key := prefix + envName(jsonName)
if tag[0] != "" {
key = tag[0]
}
empty, required := options.Empty, false
for _, opt := range tag[1:] {
switch opt {
case "required":
required = true
case "secret", "":
case "empty=allow":
empty = "allow"
case "empty=ignore":
empty = "ignore"
case "empty=reject":
empty = "reject"
default:
return fmt.Errorf("config: invalid env tag for %s", key)
}
}
if dst.Kind() == reflect.Struct && !dst.Addr().Type().Implements(textType) {
if err := walk(dst, key+"_", depth+1); err != nil {
return err
}
continue
}
if dst.Kind() == reflect.Pointer && dst.Type().Elem().Kind() == reflect.Struct && !dst.Type().Implements(textType) {
// Clone before traversing so a failed overlay never edits defaults.
child := reflect.New(dst.Type().Elem())
if !dst.IsNil() {
child.Elem().Set(dst.Elem())
}
if err := walk(child.Elem(), key+"_", depth+1); err != nil {
return err
}
if !dst.IsNil() || !child.Elem().IsZero() {
dst.Set(child)
}
if required && !defaults && dst.IsNil() {
return fmt.Errorf("config: %s is required", key)
}
continue
}
// Unexported runtime capabilities are skipped unless explicitly tagged.
if dst.Kind() == reflect.Func || dst.Kind() == reflect.Chan || dst.Kind() == reflect.Interface {
if tag[0] != "" {
return fmt.Errorf("config: %s has unsupported environment type", key)
}
continue
}
if seen[key] {
return fmt.Errorf("config: duplicate environment key %s", key)
}
seen[key] = true
raw, present := options.Lookup(key)
selectedKey := key
if !present && !defaults {
for _, alias := range options.Aliases[key] {
if raw, present = options.Lookup(alias); present {
selectedKey = alias
break
}
}
}
if defaults {
raw, present = field.Tag.Lookup("envDefault")
present = present && dst.IsZero()
}
if present && raw == "" && !defaults {
if empty == "reject" {
return fmt.Errorf("config: %s must not be empty", selectedKey)
}
if empty == "ignore" {
present = false
}
}
if present {
parsed := reflect.New(dst.Type()).Elem()
if err := parseEnvValue(parsed, raw); err != nil {
return fmt.Errorf("config: %s must be a valid %s", selectedKey, dst.Type())
}
dst.Set(parsed)
}
if required && !defaults && emptyRequired(dst) {
return fmt.Errorf("config: %s is required", key)
}
}
return nil
}
if err := walk(result, options.Prefix, 0); err != nil {
return zero, err
}
return result.Interface().(T), nil
}
func emptyRequired(v reflect.Value) bool {
switch v.Kind() {
case reflect.String:
return v.Len() == 0
case reflect.Pointer, reflect.Map, reflect.Slice:
return v.IsNil()
}
return false
}
func parseEnvValue(v reflect.Value, raw string) (err error) {
defer func() { defer func() {
if recover() != nil { if recover() != nil {
err = errors.New("invalid custom value") err = errors.New("invalid custom value")
@@ -223,7 +107,7 @@ func parseEnvValue(v reflect.Value, raw string) (err error) {
}() }()
if v.Kind() == reflect.Pointer { if v.Kind() == reflect.Pointer {
v.Set(reflect.New(v.Type().Elem())) v.Set(reflect.New(v.Type().Elem()))
return parseEnvValue(v.Elem(), raw) return parseTextValue(v.Elem(), raw)
} }
if v.CanAddr() && v.Addr().Type().Implements(textType) { if v.CanAddr() && v.Addr().Type().Implements(textType) {
return v.Addr().Interface().(encoding.TextUnmarshaler).UnmarshalText([]byte(raw)) return v.Addr().Interface().(encoding.TextUnmarshaler).UnmarshalText([]byte(raw))
@@ -266,37 +150,6 @@ func parseEnvValue(v reflect.Value, raw string) (err error) {
v.SetFloat(x) v.SetFloat(x)
} }
return err return err
case reflect.Slice, reflect.Map:
var value any
if err := apierrors.DecodeCatalogJSON([]byte(raw), &value); err != nil {
return err
}
value, err := durations(value, v.Type())
if err != nil {
return err
}
encoded, err := json.Marshal(value)
if err != nil {
return err
}
decoder := json.NewDecoder(bytes.NewReader(encoded))
decoder.DisallowUnknownFields()
return decoder.Decode(v.Addr().Interface())
} }
return errors.New("unsupported environment value") return errors.New("expected scalar text")
}
func envName(s string) string {
r := []rune(s)
var out strings.Builder
for i, c := range r {
if i > 0 && unicode.IsUpper(c) && (unicode.IsLower(r[i-1]) || unicode.IsDigit(r[i-1]) || (i+1 < len(r) && unicode.IsLower(r[i+1]))) {
out.WriteByte('_')
}
if c == '-' {
c = '_'
}
out.WriteRune(unicode.ToUpper(c))
}
return out.String()
} }

View File

@@ -42,20 +42,3 @@ func TestConfigFilePreservesDefaultsWithoutSharingMutableData(t *testing.T) {
t.Fatal("decode error leaked contents", err) t.Fatal("decode error leaked contents", err)
} }
} }
func TestEnvCollectionPrecisionAndDurationMaps(t *testing.T) {
type cfg struct {
IDs []int64 `json:"ids"`
Timers map[string]time.Duration `json:"timers"`
}
lookup := func(key string) (string, bool) {
v, ok := map[string]string{"IDS": "[9007199254740993]", "TIMERS": `{"backoff":"250ms"}`}[key]
return v, ok
}
c, err := OverlayEnv(cfg{}, EnvOptions{Lookup: lookup})
if err != nil || c.IDs[0] != 9007199254740993 || c.Timers["backoff"] != 250*time.Millisecond {
t.Fatal(c, err)
}
if _, err = OverlayEnv(cfg{}, EnvOptions{Lookup: func(key string) (string, bool) { return `{"x":"1s","x":"2s"}`, key == "TIMERS" }}); err == nil {
t.Fatal("duplicate map key accepted")
}
}

View File

@@ -1,124 +0,0 @@
package config
import (
"errors"
"os"
"path/filepath"
"reflect"
"strings"
"testing"
"time"
)
type projectNested struct {
ReadTimeout time.Duration `json:"readTimeout"`
Names []string `json:"names"`
}
type projectConfig struct {
Nested *projectNested `json:"nested"`
Enabled bool `json:"enabled"`
Count int `json:"count" envDefault:"12"`
Password string `json:"-" env:"ETAX_PASSWORD,secret,required,empty=reject"`
Label string `json:"label" env:",empty=ignore"`
}
func (c projectConfig) Validate() error {
if c.Count < 0 {
return errors.New("count must be nonnegative")
}
return nil
}
func TestProjectConfigurationOrderAndValues(t *testing.T) {
f := filepath.Join(t.TempDir(), "config.yaml")
if err := os.WriteFile(f, []byte("count: 3\nenabled: true\nnested:\n readTimeout: 2s\n names: [file]\nlabel: keep\n"), 0600); err != nil {
t.Fatal(err)
}
env := map[string]string{"ETAX_PASSWORD": " password ", "APP_ENABLED": "false", "APP_COUNT": "0", "APP_NESTED_READ_TIMEOUT": "7s", "APP_NESTED_NAMES": "[]", "APP_LABEL": ""}
options := EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}
cfg, err := LoadProject(f, projectConfig{}, options)
if err != nil {
t.Fatal(err)
}
if cfg.Enabled || cfg.Count != 0 || cfg.Password != " password " || cfg.Nested.ReadTimeout != 7*time.Second || cfg.Nested.Names == nil || len(cfg.Nested.Names) != 0 || cfg.Label != "keep" {
t.Fatal("overlay mismatch")
}
delete(env, "APP_COUNT")
cfg, err = LoadProject("", projectConfig{}, options)
if err != nil || cfg.Count != 12 {
t.Fatal(cfg.Count, err)
}
env["APP_COUNT"] = "-1"
if _, err = LoadProject(f, projectConfig{}, options); err == nil {
t.Fatal("project Validate not called")
}
}
func TestProjectOverlayIsTransactionalAndRedacted(t *testing.T) {
base := projectConfig{Nested: &projectNested{ReadTimeout: time.Second, Names: []string{"old"}}, Password: "original"}
env := map[string]string{"APP_NESTED_READ_TIMEOUT": "2s", "APP_NESTED_NAMES": "[\"new\"]", "APP_COUNT": "secret-value"}
_, err := OverlayEnv(base, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }})
if err == nil || strings.Contains(err.Error(), "secret-value") {
t.Fatal(err)
}
if base.Nested.ReadTimeout != time.Second || !reflect.DeepEqual(base.Nested.Names, []string{"old"}) {
t.Fatal("default pointer modified")
}
delete(env, "APP_COUNT")
env["ETAX_PASSWORD"] = ""
if _, err = OverlayEnv(base, EnvOptions{Prefix: "APP_", Lookup: func(k string) (string, bool) { v, ok := env[k]; return v, ok }}); err == nil {
t.Fatal("empty secret accepted")
}
type required struct {
Token string `env:"TOKEN,required"`
}
if _, err = OverlayEnv(required{}, EnvOptions{Lookup: func(string) (string, bool) { return "", false }}); err == nil {
t.Fatal("required absent")
}
type duplicate struct {
A string `env:"SAME"`
B string `env:"SAME"`
}
if _, err = OverlayEnv(duplicate{}, EnvOptions{}); err == nil {
t.Fatal("duplicate env binding accepted")
}
}
func TestEnvironmentCollectionAndEmptySemantics(t *testing.T) {
type sample struct {
Values []time.Duration
Flag bool
Text string
Optional *int
}
cases := []struct {
raw string
valid bool
}{{`["1s","2s"]`, true}, {`["invalid-secret"]`, false}}
for _, tc := range cases {
cfg, err := OverlayEnv(sample{}, EnvOptions{Lookup: func(k string) (string, bool) {
if k == "VALUES" {
return tc.raw, true
}
return "", false
}})
if (err == nil) != tc.valid {
t.Fatal(err)
}
if tc.valid && !reflect.DeepEqual(cfg.Values, []time.Duration{time.Second, 2 * time.Second}) {
t.Fatal(cfg)
}
if err != nil && strings.Contains(err.Error(), "secret") {
t.Fatal(err)
}
}
for _, mode := range []string{"allow", "ignore", "reject"} {
cfg, err := OverlayEnv(sample{Text: "old"}, EnvOptions{Empty: mode, Lookup: func(k string) (string, bool) { return "", k == "TEXT" }})
if mode == "reject" {
if err == nil {
t.Fatal("empty not rejected")
}
} else if err != nil || (mode == "allow" && cfg.Text != "") || (mode == "ignore" && cfg.Text != "old") {
t.Fatal(cfg, err)
}
}
}

210
config/references.go Normal file
View File

@@ -0,0 +1,210 @@
package config
import (
"errors"
"fmt"
"strings"
"unicode/utf8"
)
// WithLookupEnv supplies an explicit-reference lookup for tests or an embedding
// host. Ordinary projects need no loader options. The function must support
// concurrent loads; each load caches each referenced name independently.
func WithLookupEnv(lookup func(string) (string, bool)) LoaderOption {
return func(l *Loader) error {
if l == nil || l.sealed || lookup == nil {
return errors.New("config: expected a non-nil environment lookup on a new loader")
}
l.lookup = lookup
return nil
}
}
// resolvedText is textual input bound according to the target scalar type. It
// never becomes an object, collection or null by interpreting environment data.
type resolvedText string
const maxExpandedBytes = 4 << 20
type environmentValue struct {
value string
present bool
}
type referenceResolver struct {
lookup func(string) (string, bool)
cache map[string]environmentValue
doc Document
nodes, bytes, cacheBytes int
}
func resolveDocument(doc Document, lookup func(string) (string, bool)) (Document, error) {
r := referenceResolver{lookup: lookup, cache: map[string]environmentValue{}, doc: Document{Positions: doc.Positions, environments: map[string][]string{}}}
v, err := r.walk(doc.Value, "", 0)
if err != nil {
return Document{}, err
}
r.doc.Value = v
return r.doc, nil
}
func (r *referenceResolver) walk(v any, path string, depth int) (any, error) {
r.nodes++
if depth > maxConfigDepth || r.nodes > maxConfigNodes {
return nil, r.doc.diagnostic(path, "configuration nesting or node limit exceeded", nil)
}
switch x := v.(type) {
case string:
return r.text(x, path, false)
case Text:
return r.text(string(x), path, true)
case map[string]any:
out := make(map[string]any, len(x))
for _, key := range sortedKeys(x) {
value, err := r.walk(x[key], joinField(path, key), depth+1)
if err != nil {
return nil, err
}
out[key] = value
}
return out, nil
case []any:
out := make([]any, len(x))
for i, value := range x {
var err error
out[i], err = r.walk(value, fmt.Sprintf("%s[%d]", path, i), depth+1)
if err != nil {
return nil, err
}
}
return out, nil
default:
return v, nil
}
}
func (r *referenceResolver) fail(path, name, reason string) error {
d := r.doc.diagnostic(path, reason, nil)
d.Env = name
return d
}
func validEnvironmentName(name string) bool {
if len(name) == 0 {
return false
}
for i := 0; i < len(name); i++ {
c := name[i]
if c == '_' || c >= 'A' && c <= 'Z' || c >= 'a' && c <= 'z' || i > 0 && c >= '0' && c <= '9' {
continue
}
return false
}
return true
}
func (r *referenceResolver) lookupValue(name, path string) (environmentValue, error) {
if v, ok := r.cache[name]; ok {
return v, nil
}
value, present := r.lookup(name)
if len(value) > maxConfigBytes || r.cacheBytes+len(value) > maxExpandedBytes {
return environmentValue{}, r.fail(path, name, "environment value exceeds configuration size limit")
}
if !utf8.ValidString(value) {
return environmentValue{}, r.fail(path, name, "environment value is not valid UTF-8")
}
r.cacheBytes += len(value)
v := environmentValue{value, present}
r.cache[name] = v
return v, nil
}
func (r *referenceResolver) text(raw, path string, xmlText bool) (any, error) {
var out strings.Builder
var names []string
seen := map[string]bool{}
if !strings.Contains(raw, "$") {
if len(raw) > maxConfigBytes || r.bytes+len(raw) > maxExpandedBytes {
return nil, r.fail(path, "", "expanded configuration exceeds size limit")
}
r.bytes += len(raw)
if xmlText {
return Text(raw), nil
}
return raw, nil
}
appendText := func(s string) error {
if out.Len()+len(s) > maxConfigBytes || r.bytes+len(s) > maxExpandedBytes {
return r.fail(path, strings.Join(names, ", "), "expanded configuration exceeds size limit")
}
out.WriteString(s)
r.bytes += len(s)
return nil
}
// Scan only the file's decoded scalar. Inserted environment values are never
// scanned again, including dollars, quotes, separators or newline characters.
for i := 0; i < len(raw); {
j := strings.IndexByte(raw[i:], '$')
if j < 0 {
if err := appendText(raw[i:]); err != nil {
return nil, err
}
break
}
j += i
if err := appendText(raw[i:j]); err != nil {
return nil, err
}
switch {
case j+1 < len(raw) && raw[j+1] == '$':
if err := appendText("$"); err != nil {
return nil, err
}
i = j + 2
case j+1 < len(raw) && raw[j+1] == '{':
end := strings.IndexByte(raw[j+2:], '}')
if end < 0 {
return nil, r.fail(path, "", "unclosed environment reference; use ${NAME}, ${NAME:-fallback} or $${...} for a literal")
}
end += j + 2
expression := raw[j+2 : end]
name, fallback, hasDefault := strings.Cut(expression, ":-")
if !validEnvironmentName(name) || strings.Contains(fallback, "${") {
return nil, r.fail(path, "", "invalid environment reference; use ${NAME} or ${NAME:-fallback}; nested references are not supported")
}
r.nodes++
if r.nodes > maxConfigNodes {
return nil, r.fail(path, name, "environment reference count exceeds configuration limit")
}
if !seen[name] {
names = append(names, name)
seen[name] = true
}
value, err := r.lookupValue(name, path)
if err != nil {
return nil, err
}
selected := value.value
if !value.present || selected == "" {
if !hasDefault {
return nil, r.fail(path, name, "required environment variable is unset or empty")
}
selected = fallback
}
if err := appendText(selected); err != nil {
return nil, err
}
i = end + 1
default:
if err := appendText("$"); err != nil {
return nil, err
}
i = j + 1
}
}
if len(names) > 0 {
r.doc.environments[path] = names
return resolvedText(out.String()), nil
}
if xmlText {
return Text(out.String()), nil
}
return out.String(), nil
}

230
config/references_test.go Normal file
View File

@@ -0,0 +1,230 @@
package config
import (
"encoding/json"
"errors"
"fmt"
"os"
"strings"
"sync"
"testing"
"time"
)
type referenceConfig struct {
Port int `json:"port"`
BatchSize int `json:"batchSize"`
Enabled bool `json:"enabled"`
Password string `json:"password"`
Address string `json:"address"`
Timeout time.Duration `json:"timeout"`
Optional *int `json:"optional"`
Labels map[string]string `json:"labels"`
Numbers []int64 `json:"numbers"`
Dynamic any `json:"dynamic"`
}
func (c referenceConfig) Validate() error {
if c.Port < 0 {
return Invalid("port", "must not be negative")
}
return nil
}
func TestExplicitReferencesAcrossFormats(t *testing.T) {
// Delimiters, quotes and newlines belong to the credential, never file syntax.
secret := " DDSFLKJlsf:saf:-123\"\n${DO_NOT_READ}&<>\\ "
sources := map[string]string{
"json": `{"port":"${PORT:-3306}","enabled":"${ENABLED:-true}","password":"${PASSWORD}","address":"https://${HOST}:${PORT:-3306}/v1","timeout":"${TIMEOUT:-3s}","optional":"${ZERO}","numbers":["${BIG}"],"dynamic":"${ZERO}","labels":{"literal":"$${NOT_READ}","empty":"${OPTIONAL:-}","dollars":"$$cost"}}`,
"yaml": "port: \"${PORT:-3306}\"\nenabled: \"${ENABLED:-true}\"\npassword: \"${PASSWORD}\"\naddress: \"https://${HOST}:${PORT:-3306}/v1\"\ntimeout: \"${TIMEOUT:-3s}\"\noptional: \"${ZERO}\"\nnumbers: [\"${BIG}\"]\ndynamic: \"${ZERO}\"\nlabels:\n literal: \"$${NOT_READ}\"\n empty: \"${OPTIONAL:-}\"\n dollars: \"$$cost\"\n",
"xml": `<config><port>${PORT:-3306}</port><enabled>${ENABLED:-true}</enabled><password>${PASSWORD}</password><address>https://${HOST}:${PORT:-3306}/v1</address><timeout>${TIMEOUT:-3s}</timeout><optional>${ZERO}</optional><numbers type="array"><item>${BIG}</item></numbers><dynamic>${ZERO}</dynamic><labels><literal>$${NOT_READ}</literal><empty>${OPTIONAL:-}</empty><dollars>$$cost</dollars></labels></config>`,
}
for ext, source := range sources {
t.Run(ext, func(t *testing.T) {
calls := map[string]int{}
values := map[string]string{"PORT": "0", "ENABLED": "false", "PASSWORD": secret, "HOST": "example.test", "ZERO": "0", "BIG": "9007199254740993"}
l := testLoader(func(name string) (string, bool) { calls[name]++; v, ok := values[name]; return v, ok })
got, err := LoadProject(configFile(t, ext, source), referenceConfig{Port: 77, BatchSize: 100, Enabled: true, Password: "old"}, l)
if err != nil {
t.Fatal(err)
}
if got.Port != 0 || got.BatchSize != 100 || got.Enabled || got.Password != secret || got.Address != "https://example.test:0/v1" || got.Timeout != 3*time.Second || got.Optional == nil || *got.Optional != 0 || got.Numbers[0] != 9007199254740993 || got.Dynamic != "0" {
t.Fatal("binding changed values")
}
if got.Labels["literal"] != "${NOT_READ}" || got.Labels["empty"] != "" || got.Labels["dollars"] != "$cost" {
t.Fatal(got.Labels)
}
for name, n := range calls {
if n != 1 {
t.Fatal("not cached", name, n)
}
}
if calls["DO_NOT_READ"] != 0 || calls["NOT_READ"] != 0 {
t.Fatal("recursive expansion")
}
})
}
}
func TestReferenceFallbackAndFixedValues(t *testing.T) {
for _, present := range []bool{false, true} {
l := testLoader(func(string) (string, bool) { return "", present })
got, err := LoadProject(configFile(t, "yaml", "port: \"${PORT:-3306}\"\npassword: \"${PASSWORD:-DDSFLKJlsf:saf:-123}\""), referenceConfig{}, l)
if err != nil || got.Port != 3306 || got.Password != "DDSFLKJlsf:saf:-123" {
t.Fatal(got, err)
}
for _, ext := range []string{"json", "yaml", "xml"} {
raw := `{"password":"${PASSWORD}"}`
if ext == "yaml" {
raw = "password: '${PASSWORD}'"
}
if ext == "xml" {
raw = "<config><password>${PASSWORD}</password></config>"
}
path := configFile(t, ext, raw)
_, err = LoadProject(path, referenceConfig{Password: "initial"}, l)
var d *Error
if !errors.As(err, &d) || d.Env != "PASSWORD" || d.File != path || d.Field != "password" || d.Line < 1 {
t.Fatal("missing required-reference diagnostic", err)
}
}
}
calls := 0
l := testLoader(func(string) (string, bool) { calls++; return "999", true })
got, err := LoadProject(configFile(t, "json", `{"port":0,"enabled":false,"password":""}`), referenceConfig{Port: 77, BatchSize: 100, Enabled: true, Password: "old"}, l)
if err != nil || got.Port != 0 || got.BatchSize != 100 || got.Enabled || got.Password != "" || calls != 0 {
t.Fatal(got, err, calls)
}
got, err = LoadProject(configFile(t, "yaml", "password: '${VALUE}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return " ", true }))
if err != nil || got.Password != " " {
t.Fatal("trimmed credential", err)
}
}
func TestReferenceDiagnosticsAndScalarLimits(t *testing.T) {
for _, value := range []string{"private-not-int", "92233720368547758080", "false", "null", "-1"} {
path := configFile(t, "yaml", "port: '${PORT}'")
_, err := LoadProject(path, referenceConfig{}, testLoader(func(string) (string, bool) { return value, true }))
var d *Error
if !errors.As(err, &d) || d.File != path || d.Env != "PORT" || d.Field != "port" || d.Line != 1 || d.Column < 1 || strings.Contains(err.Error(), value) {
t.Fatalf("invalid diagnostic: %v", err)
}
}
for _, raw := range []string{"${BAD-123}", "${}", "${X", "${1BAD}", "${BAD NAME}", "${X:-${Y}}"} {
encoded, _ := json.Marshal(map[string]string{"password": raw})
if _, err := LoadProject(configFile(t, "json", string(encoded)), referenceConfig{}, testLoader(noEnvironment)); err == nil {
t.Fatal("invalid syntax accepted", raw)
}
}
for _, value := range []string{string([]byte{0xff}), strings.Repeat("x", maxConfigBytes+1)} {
if _, err := LoadProject(configFile(t, "yaml", "password: '${SECRET}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return value, true })); err == nil {
t.Fatal("unbounded/invalid environment")
}
}
path := configFile(t, "yaml", "labels:\n a: '${BIG}'\n b: '${BIG}'\n c: '${BIG}'\n d: '${BIG}'\n e: '${BIG}'")
if _, err := LoadProject(path, referenceConfig{}, testLoader(func(string) (string, bool) { return strings.Repeat("x", maxConfigBytes), true })); err == nil {
t.Fatal("expansion budget bypassed")
}
// An environment value cannot inject an array/object or coerce dynamic null.
for _, raw := range []string{"[1,2]", "{\"x\":1}", "null"} {
_, err := LoadProject(configFile(t, "yaml", "numbers: '${VALUE}'"), referenceConfig{}, testLoader(func(string) (string, bool) { return raw, true }))
if err == nil {
t.Fatal("structure injected")
}
}
}
func TestReferenceCacheIsolationAndRemovedRules(t *testing.T) {
path := configFile(t, "yaml", "password: '${TOKEN}'")
for i := 0; i < 24; i++ {
t.Run(fmt.Sprint(i), func(t *testing.T) {
t.Parallel()
v := fmt.Sprint(i)
l := testLoader(func(string) (string, bool) { return v, true })
for n := 0; n < 4; n++ {
got, err := LoadProject(path, referenceConfig{}, l)
if err != nil || got.Password != v {
t.Fatal("load leaked", err)
}
}
})
}
}
func TestReferenceLookupPerLoadAndConcurrentLoader(t *testing.T) {
var mu sync.Mutex
reads := 0
l := testLoader(func(string) (string, bool) { mu.Lock(); defer mu.Unlock(); reads++; return "secret", true })
path := configFile(t, "yaml", "password: '${TOKEN}${TOKEN}'")
var wg sync.WaitGroup
for i := 0; i < 24; i++ {
wg.Add(1)
go func() {
defer wg.Done()
got, err := LoadProject(path, referenceConfig{}, l)
if err != nil || got.Password != "secretsecret" {
t.Error(err)
}
}()
}
wg.Wait()
if reads != 24 {
t.Fatal(reads)
}
}
func TestRemovedEnvTagsAndFieldSuggestion(t *testing.T) {
type old struct {
Value int `json:"value" envDefault:"9"`
}
if _, err := LoadProject("", old{}); err == nil || !strings.Contains(err.Error(), "removed") {
t.Fatal(err)
}
type oldSecret struct {
Token string `json:"-" env:"TOKEN"`
}
if _, err := DecodeFile(configFile(t, "json", `{}`), oldSecret{}); err == nil {
t.Fatal("old env tag ignored")
}
_, err := LoadProject(configFile(t, "yaml", "batchSiz: 10"), referenceConfig{})
if err == nil || !strings.Contains(err.Error(), `did you mean "batchSize"`) {
t.Fatal(err)
}
t.Setenv("GO_OMNI_BATCH_SIZE", "999")
got, err := LoadProject("", referenceConfig{BatchSize: 100})
if err != nil || got.BatchSize != 100 {
t.Fatal("implicit overlay", err)
}
}
func FuzzConfigurationReferences(f *testing.F) {
for _, s := range []string{"${X}", "$${X}", "${X:-DDS:abc:-123}", "${X:-}", "https://${X}/${X}", "${X:-${Y}}"} {
f.Add(s, " secret${NO}\n\" ")
}
f.Fuzz(func(t *testing.T, raw, value string) {
if len(raw) > 8192 || len(value) > 8192 {
return
}
doc := Document{Value: map[string]any{"password": raw}}
resolved, err := resolveDocument(doc, func(string) (string, bool) { return value, true })
if err != nil {
return
}
_, _ = bindDocument(resolved, referenceConfig{})
})
}
func BenchmarkExplicitConfiguration(b *testing.B) {
for _, refs := range []bool{false, true} {
b.Run(fmt.Sprint(refs), func(b *testing.B) {
source := `{"port":3306,"enabled":false,"password":"fixed","batchSize":0}`
if refs {
source = `{"port":"${PORT:-3306}","enabled":"${ENABLED:-false}","password":"${PASSWORD}","batchSize":0}`
}
path := b.TempDir() + "/config.json"
if err := os.WriteFile(path, []byte(source), 0600); err != nil {
b.Fatal(err)
}
l := testLoader(func(k string) (string, bool) { return "private", k == "PASSWORD" })
b.ReportAllocs()
b.ResetTimer()
for i := 0; i < b.N; i++ {
if _, err := LoadProject(path, referenceConfig{BatchSize: 100}, l); err != nil {
b.Fatal(err)
}
}
})
}
}

51
config/suggestions.go Normal file
View File

@@ -0,0 +1,51 @@
package config
import (
"reflect"
"strconv"
"strings"
)
// Suggest only a unique, nearby declared field. Input values are never used.
func unknownFieldReason(name string, fields map[string]reflect.StructField) string {
if len(name) < 3 || len(name) > 128 {
return "unknown field"
}
best, match, tied := 3, "", false
for candidate := range fields {
if len(candidate) > 128 {
continue
}
d := fieldDistance(strings.ToLower(name), strings.ToLower(candidate))
if d < best {
best, match, tied = d, candidate, false
} else if d == best {
tied = true
}
}
if match != "" && !tied && best <= 2 {
return "unknown field; did you mean " + strconv.Quote(match) + "?"
}
return "unknown field"
}
func fieldDistance(a, b string) int {
prev := make([]int, len(b)+1)
for i := range prev {
prev[i] = i
}
for i := 0; i < len(a); i++ {
left, diagonal := i+1, prev[0]
prev[0] = left
for j := 0; j < len(b); j++ {
cost := 0
if a[i] != b[j] {
cost = 1
}
old := prev[j+1]
left = min(left+1, old+1, diagonal+cost)
prev[j+1] = left
diagonal = old
}
}
return prev[len(b)]
}

Some files were not shown because too many files have changed in this diff Show More