diff --git a/ADVANCED.md b/ADVANCED.md index 23bc03d..00babfb 100644 --- a/ADVANCED.md +++ b/ADVANCED.md @@ -11,15 +11,17 @@ | `openapi` | 路由与 OpenAPI 3.0.3 同时注册、DTO schema、响应信封 | 显式声明请求/响应；不自动推断任意 Handler 行为，不附带 Swagger UI | | `auth` | RS256/EdDSA JWT 验证与公钥轮换、可撤销 Session、Policy 接口 | 登录验证和业务授权规则由接入方提供；不自动获取 token 指定的密钥 | | `telemetry` | HTTP 入站/出站、Redis、SQL spans，HTTP 指标和 traceId 日志关联 | 注入 OTel exporter/reader 后才外发；SQL Query span 不覆盖后续遍历；不接管全局 provider | -| `config` / `logx` | JSON/YAML 文件、环境变量覆盖、动态日志级别 | 热更新只开放日志级别；连接池、监听、认证等静态变更要求重启 | +| `config` / `logx` | JSON/YAML 文件、环境变量覆盖、动态日志级别、轮转/脱敏/堆栈/慢请求 | 热更新只开放日志级别；连接池、监听、认证等静态变更要求重启 | | `resilience` | 失败阈值、熔断冷却、单探针恢复、上游并发隔离 | 一个实例对应一个依赖；默认内存状态，不做跨进程全局熔断 | | `middleware/redis` | 跨副本限流、幂等 claim/回放、容量限制 | 专用 Redis namespace；TTL 窗口内保护不等于永久业务去重 | | `queue` / `queue/redis` | 持久消息接口、Streams 接管、有限重试、失败消息管理 | 至少一次交付，处理器必须幂等；持久性取决于 Redis 部署 | -| `storage` | Local、S3 兼容（含 R2 配置）、OSS 官方 SDK 适配、临时下载链接 | 云平台专属功能单独扩展；本批没有云账号实网验收 | +| `storage` | Local、S3/R2、OSS、临时下载、受限直传、云端分片 | 云平台专属功能单独扩展；本批没有云账号实网验收 | | `database/migrate` | MySQL 显式部署迁移、摘要预览、失败 dirty 阻断 | 不随应用启动自动改表；无自动 down/force；已应用文件摘要需部署方留存校对 | | `audit` | 可注入 Sink、字段白名单、JSON 行串行输出、错误返回 | 不是防篡改存储；与业务同事务的审计要实现事务型 Sink | | `pagination` | page/pageSize、最大偏移与页长、统一结果 | offset 分页，不承诺跨页快照，不实现游标分页 | +日志的配置、脱敏边界、轮转和慢请求记录见 [logx 使用说明](logx/README.md)。框架保留通用 Context/结构化字段扩展，不新增固定业务身份上下文。 + ## 对象存储 业务只持有 `storage.Store`，调用 `Put`、`Get`、`Delete`。构造函数属于启动组合处： @@ -51,9 +53,9 @@ files, err := oss.New(client, bucket, 32<<20) 上方是互斥初始化片段，实际代码需逐次处理错误。import 路径分别为 `go-omni/storage/local`、`go-omni/storage/s3`、`go-omni/storage/oss`（完整 module 前缀 `git.yuntuo.hk/k2safe/`）；`ossSdk` 与 `ossCredentials` 来自 `github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss` 及其 `credentials` 子包。S3 凭据来自 `github.com/minio/minio-go/v7/pkg/credentials`。 -统一约定：key 是对象标识，不是磁盘路径；拒绝绝对路径、`..`、空路径段和反斜杠。Put 必须传已知长度，调用方保证与内容一致。默认单对象 32 MiB，流式读写；OSS 本适配器只做 simple upload，最大配置 5 GiB。Get 返回的 `Body` 由调用方关闭，读取过程也应有 context 截止。Delete 缺失对象成功；Get 缺失对象返回 `storage.ErrNotFound`。底层客户端/传输连接池由应用管理，不会在单次操作后关闭。 +统一约定：key 是对象标识，不是磁盘路径；拒绝绝对路径、`..`、空路径段和反斜杠。Put 必须传已知长度，并检查实际读取长度。默认单对象 32 MiB，流式读写；OSS 简单 Put 上限 5 GiB，更大使用独立分片能力。Get 返回的 `Body` 由调用方关闭，读取过程也应有 context 截止。Delete 缺失对象成功；Get 缺失对象返回 `storage.ErrNotFound`。底层客户端/传输连接池由应用管理，不会在单次操作后关闭。 -业务需要临时链接时显式依赖额外接口 `storage.Presigner`。S3/OSS 实现 `PresignGet`；Local 用 `storage.NewSigner(files, downloadBaseURL, secret)` 提供 HTTP 下载 Handler。链接 TTL 为 1 秒到 1 小时，私钥/临时凭据可能让云链接提前失效。签发链接前必须完成业务授权；对象 key 及 bucket 本身不是租户权限。签名链接不要写日志。首版未封装前端直传、分片续传、生命周期、ACL、CDN 自定义域名和版本管理。 +业务需要临时链接时显式依赖额外接口 `storage.Presigner`。S3/OSS 实现 `PresignGet`；Local 用 `storage.NewSigner(files, downloadBaseURL, secret)` 提供 HTTP 下载 Handler。链接 TTL 为 1 秒到 1 小时，私钥/临时凭据可能让云链接提前失效。签发链接前必须完成业务授权；对象 key 及 bucket 本身不是租户权限。签名链接不要写日志。受限直传和云端分片现已提供可选接口，见 [上传与续传说明](storage/README.md)。生命周期、ACL、CDN 自定义域名和版本管理仍由平台/部署管理。 R2 实现 S3 的子集，配置入口使用标准账户 endpoint；法域专属 endpoint 可显式配置。兼容性依据：[R2 S3 API](https://developers.cloudflare.com/r2/api/s3/api/)、[R2 签名链接](https://developers.cloudflare.com/r2/api/s3/presigned-urls/)。OSS 使用其[官方 Go SDK v2](https://www.alibabacloud.com/help/en/oss/developer-reference/manual-for-go-sdk-v2/)，没有假定全部 S3 行为与 OSS 等价。 diff --git a/API.md b/API.md new file mode 100644 index 0000000..979dd10 --- /dev/null +++ b/API.md @@ -0,0 +1,169 @@ +# API 契约与项目生成 + +本阶段目标：从 `.api` 定义生成可运行的 Go 项目；定义、绑定、校验与 OpenAPI 保持一致；重复生成不覆盖业务实现；业务代码继续独立于网络层。实现入口是 `cmd/go-omni/apigen`，运行时契约编译器是 `contract`。 + +## 与 go-zero 的对应关系 + +调研参考了 go-zero 官方的 [API DSL](https://go-zero.dev/reference/api-dsl/)、[参数规则](https://go-zero.dev/reference/api-dsl/parameter/)、[goctl API](https://go-zero.dev/reference/cli-guide/api/) 和 [项目结构](https://go-zero.dev/concepts/project-structure/)。借鉴其声明接口后生成项目的工作方式：统一的类型定义、路由声明、分组、前缀和命名中间件。实现独立编写，不引入 go-zero 或 Gin 运行时依赖。 + +| 使用体验 | go-omni 实现 | +| --- | --- | +| 新建接口项目 | `go-omni api new ` | +| 按契约生成、更新 | `go-omni api generate --api user.api --dir .` | +| 熟悉的命令别名 | `api go -api user.api -dir .` | +| 校验契约 | `api validate --api user.api` | +| CI 检查生成产物是否过期 | `api generate --api user.api --dir . --check` | +| 业务逻辑目录 | `internal/service/`，沿用 Omni 的 Service 分层 | +| 依赖装配 | `internal/servicecontext` | +| 接口文档 | 自动生成 `docs/openapi.json`，同时提供 `/openapi.json` | + +这是明确限定的 go-zero 风格方言，**不是完整 goctl 兼容实现**。不支持的声明和校验标签会在生成前报错。 + +## 立即试用 + +源码开发时，从 go-omni 仓库运行，显式引用本地框架： + +```sh +go run ./cmd/go-omni api new \ + --module example.com/omniDemo --framework-dir . ../omniDemo + +cd ../omniDemo +go mod tidy +go run ./cmd/server +``` + +访问 `http://127.0.0.1:8080/api/greet/Alice` 可得到带 `data.message` 的统一响应。`api new` 的问候示例包含可运行逻辑；自定义契约新生成的 Service 会明确返回 501，等待业务实现。 + +用现成的多分组定义生成项目，在 go-omni 仓库执行： + +```sh +go run ./cmd/go-omni new --api examples/contracts/user.api \ + --module example.com/userService --framework-dir . \ + --log-output file --log-format json ../userService +``` + +`api generate` 在新目录中也会生成完整项目。默认只监听 localhost；`GO_OMNI_HTTP_ADDR` 修改地址，`GO_OMNI_TRANSPORT=native` 切换网络层。配置顺序为默认值 → `etc/config.yaml` → 环境变量。文件日志支持现有轮转配置。远程试用使用与 CLI 一致的固定版本，见 [发布说明](RELEASE.md)。 + +API 项目现在支持组合数据库、存储和日志： + +```sh +go-omni new --api user.api --module example.com/userService \ + --database mysql --storage local --log-output file ./userService +``` + +- `--database none|memory|mysql`：API 默认 `none`，`memory` 是不创建数据库连接的别名；MySQL 生成连接池、配置、事务入口和 readiness 探针。 +- `--storage none|local|s3|r2|oss`：默认 none，生成统一 Files / Uploads / Downloads / UploadLinks 依赖；本地签名对象端点为 `/storage/object`。 +- `--log-output stdout|stderr|file` 与 `--log-format json|text`：配置和无配置文件启动使用一致默认值。 + +初次选项记录在 `.goOmni/apiManifest.json`，以后直接 `api generate` 会继承。已有项目通过参数切换选项会报错，因为初始化代码属于业务方；可在新目录生成对照后手动迁移。`--force` 不允许覆盖 API 项目。旧式 CRUD 脚手架仍保留。 + +MySQL 密钥只从 `GO_OMNI_DB_DSN` 获取，非秘密池配置位于 YAML，环境变量优先。存储使用原有 `GO_OMNI_STORAGE_*`；API 项目不会额外生成公开的签名发放接口，鉴权和对象权限由业务实现。初始化失败会清理已打开资源，退出清理可重复调用。 + +HTTP 请求类型不能推导数据库表结构；迁移和 Repository 由业务自己维护。完整案例见 [用户管理示例](examples/userService/README.md)，覆盖真实 MySQL、事务审计、鉴权、文件导出和日志。 + +## 接口定义 + +```go +syntax = "v1" + +type UpdateUserReq { + UserID int64 `path:"userId,range=[1:1000000]"` + Notify bool `form:"notify,default=false"` + Name string `json:"name" binding:"required,max=50"` + Age *int `json:"age,optional,range=[0:150]"` +} + +type UserResp { + UserID int64 `json:"userId"` + Name string `json:"name"` +} + +@server ( + group: user + prefix: /api/v1 +) +service user-api { + @doc "Update a user" + @handler updateUser + put /users/:userId (UpdateUserReq) returns (UserResp) +} +``` + +类型和 Go 字段使用导出命名；JSON、query、path 名称使用 lowerCamelCase；Header 使用 HTTP 标准名称。响应类型只定义 `data`，统一响应的 `code`、`message`、`requestId` 由框架添加。 + +支持： + +- 可省略的 `syntax = "v1"`；`info` 中的 title、version 和描述性元数据。 +- 单独的 `type X { ... }` 与 `type (...)`；基础标量、指针、嵌套命名类型、切片、字符串键 map。嵌套对象只使用 JSON 字段。 +- 相对 `.api` 导入及 `import (...)`，重复导入去重；拒绝循环和越过入口目录的导入。单文件 1 MiB、总计 8 MiB、最多 64 个文件，限制导入/类型深度及 schema 展开规模。 +- 同一 service 名称的多个块；`@server` 的 group、prefix、middleware，group 可以写 `admin/user`。 +- 每个路由必须有 `@handler`；可选 `@doc "..."`；GET/POST/PUT/PATCH/DELETE/HEAD/OPTIONS。 +- 可省略请求或响应；有响应默认 200，无响应默认 204。Omni 扩展 `@status 201`、`@status 202`；204 不允许响应类型，HEAD 不允许响应体，GET/HEAD 不允许 JSON 请求体。 +- 每个请求字段只使用一个来源：`json`、`path`、`form`、`header`。本版 `form` 明确表示 URL query，不解析 multipart 或表单 body。query 切片用重复键；header/path 使用标量。 +- 来源标签中的 `optional` / `omitempty`、标量 `default`、标量 `options=a|b`、数值 `range=[1:100]`（支持开区间和单边界）。 +- `binding` 的 `required`、`min`、`max`、`len`、`email`。长度按 Unicode 字符或切片元素数；范围边界必须有限且位于 ±2^53 内。 + +暂不支持匿名嵌入、别名、递归类型、外部 Go 类型、自定义 JSON 编解码、`[]byte`、泛型、`json:",string"`、任意 validator 标签、go-zero 的 JWT/签名/SSE 声明，以及客户端 SDK 生成。认证请用命名中间件接入。`@doc` 块等未列出的语法也会明确报错。 + +## 参数语义 + +无 optional/default 的字段按“是否出现”判断必填，`false`、`0`、`""` 是已提供的值。额外声明 `binding:"required"` 才会拒绝标量零值。默认值只在字段缺失时使用，不替换显式的空值。 + +业务需要区分“没传”和“传了零值”时使用 `*T` 加 optional。指针允许 JSON null；非指针拒绝 null；`binding:"required"` 拒绝指针 null。缺失与 null 都会得到 nil，本版没有三态 Nullable 类型。响应端的 `optional` 不等于 Go 编码器的 `omitempty`；需要省略零值时显式使用 `omitempty`。 + +JSON 必须是一个 object，Content-Type 为 application/json。默认最多 1 MiB，包含 chunked/未知长度和尾部空白；超限返回 413。重复 JSON key、未知字段、多个 JSON 值、标量 query/header 重复值、格式和约束错误返回 400；错误 Content-Type 返回 415。失败不会部分修改目标 DTO；错误详情只保留在内部 cause，公共响应不回显原始参数。 + +`contract.Binder` 在服务启动时编译，随后并发复用，不会每次请求重新分析标签。CLI 用相同编译器产生文档，因此来源、必填、默认、nullable、枚举、范围和长度具有同一来源。响应类型和约束由业务实现遵守，不在每次响应上再次校验；非 nullable 集合请返回空集合而不是 nil。 + +## 生成目录和业务边界 + +```text +api/ # 开发者维护的契约及导入 +cmd/server/main.go # 启动、日志、优雅退出 +etc/config.yaml +internal/settings/ +internal/servicecontext/ # 依赖初始化和释放 +internal/types/typesGen.go +internal/handler/routesGen.go +internal/handler/openapiGen.go +internal/handler//handlersGen.go +internal/service//service.go +internal/service//.go +internal/repository//repository.go +internal/middleware/registryGen.go +internal/middleware/.go +docs/openapi.json +.goOmni/apiManifest.json +``` + +Service 方法接收标准 `context.Context` 与生成的请求 DTO；Handler 调用 Service 并完成响应包装。网络适配器仍经 Omni/httpx 接入。Repository 是业务所有的扩展位置，不自动生成虚构 CRUD 查询。 + +命名中间件按声明顺序组合，第一个在最外层。新 hook 初始返回 501，必须实现后才放行；registry 自动维护，hook 文件只创建一次。缺失 hook 在注册任何路由前报错。文档通过 `x-omni-middleware` 标出命名 hook，不把名称误当成已实现的认证协议。 + +## 安全更新与 CI + +在生成项目中安装/使用与框架版本相符的 go-omni CLI： + +```sh +go-omni api validate --api api/user.api +go-omni api generate --api api/user.api --dir . +go-omni api generate --api api/user.api --dir . --check +go test ./... +``` + +`Gen.go` 和 OpenAPI 由生成器管理，manifest 保存其 SHA-256。手动修改后更新会报冲突。Service、Repository、main、配置、依赖装配和中间件实现按字节保留；只补缺失文件。删除接口/分组时只移除旧的生成文件，保留业务代码；已删除类型仍被旧业务引用时，需要明确迁移该业务文件，编译器会提示。 + +`--check` 只读，检查产物是否过期，以及所需骨架是否缺失。契约复制到项目后，请编辑项目内的 API；外部定义与已存在副本不一致时会拒绝覆盖。一个项目由一个 API 入口管理，可以通过导入拆分文件。 + +生成器拒绝输出路径穿越、受管理路径上的符号链接、大小写路径冲突和无所有权的覆盖。生成前统一校验/格式化，锁文件防止两个生成器同时修改；文件以原子替换写入，普通写入失败会回滚。本版不是整个目录的崩溃原子事务：进程被强杀或机器断电后，应按 Git 恢复生成产物和 manifest，再重试。请将 manifest 与生成产物一并提交，也不要在生成过程中编辑这些文件。 + +生成器不自动安装依赖、不访问数据库、不推送仓库、不替业务实现权限规则。 + +## 验证 + +```sh +go test -race -p=2 ./contract ./openapi ./cmd/go-omni/... +python3 scripts/verifyAPI.py +``` + +Go 测试覆盖契约拒绝规则、多源绑定、零值/缺失、默认/范围/嵌套集合、文档约束、幂等更新、只读检查、业务保留、删除路由、路径边界及生成项目编译。Python 验收会在临时目录生成项目，分别启动 nethttp/native，验证真实 HTTP 请求、中间件、日志 requestId 和优雅退出；结束后清理自己创建的进程和临时文件。结果记录在 `performance/apiGenerationLocal.json`。 diff --git a/CHANGELOG.md b/CHANGELOG.md index 3483486..540dd82 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,16 @@ # Changelog -## Unreleased +## v0.7.0-preview + +- 新增 `.api` 契约驱动生成器：DTO、严格多来源绑定与校验、路由、分组 Handler / Service / Repository、命名中间件及 OpenAPI；生成文件哈希管理、业务文件保护和只读 `--check`。 +- `new --api` / `api new` / `api generate` 可以组合 MySQL、日志以及 Local / S3 / R2 / OSS。保存初始基础设施选项，重生成自动沿用；显式切换已有项目选项会拒绝，避免静默丢失初始化代码。 +- 新增 `examples/userService` 独立业务示例与 `verifyAPIWorkflow.py`：用户 CRUD / 分页、管理员 Token 鉴权、MySQL 事务审计、故障回滚、文件导出、requestId 日志、数据库 readiness 和优雅退出；支持从远程版本验收。 +- 日志新增文件轮转、结构化敏感字段脱敏、异常堆栈及慢请求配置；存储新增统一上传策略、直传签名和云端分片能力。 +- 新增缓存、HTTP 客户端、限流 / 幂等 / 超时、后台任务、Redis 队列、事务 outbox、分布式锁、审计、迁移与 telemetry 等可选模块；业务身份与 Orbit 保持由应用自行组合。 +- `database.ApplyEnv` 支持文件/默认配置叠加环境变量；数据库配置使用 lowerCamelCase，DSN 不参与 JSON/YAML 序列化。 +- 修复 native 在业务层提前拒绝请求体为 413 时等待继续读取 body 的问题，拒绝后关闭未读连接。 + +以下是自上一版本以来的网络层变化与历史验收记录： - 新增 `TestNginxHTTP2Proxy` 与 `scripts/verifyHTTP2Proxy.py`：验证两段 TLS/h2、回源连接复用、证书校验拒绝、超时/客户端取消恢复，以及 GOAWAY 下的新旧实例切换和请求排空。Linux amd64、Nginx 1.30.5 普通与 race 模式各 3 轮通过；保留源码/镜像/日志校验信息和实际执行记录。 - HTTP/2 代理验收的编译与测试阶段顺序运行，均限制 2 核、2 GiB、256 个进程/线程；集成测试禁用外部网络并使用只读根文件系统。总时限、SIGTERM、SIGHUP 清理实测通过；native 默认协议保持 HTTP/1.1。 diff --git a/MODULES.md b/MODULES.md index 86e010e..405554e 100644 --- a/MODULES.md +++ b/MODULES.md @@ -2,7 +2,7 @@ 第二批的共享 Redis 中间件、可靠队列、多平台存储及其他模块见 [扩展模块](ADVANCED.md)。下文的单进程约束指本页原有默认实现。 -这批能力位于 main，尚未包含在 `v0.6.0-preview` 标签中。统一示例见 +这批能力从 `v0.7.0-preview` 提供。统一示例见 [examples/modules](examples/modules/README.md)。业务入口继续使用 `omni.New`，中间件依赖 `httpx`，不依赖具体服务端网络实现。各模块按需创建、注入 Service，再注册到应用关闭流程。 diff --git a/README.md b/README.md index 0114409..a463fa9 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@ GoOmni 是一套面向业务复用的 Go 后端基础框架。当前默认使用 远程 module 地址是 `git.yuntuo.hk/k2safe/go-omni`。业务项目可以直接引用指定版本： ```sh -go get git.yuntuo.hk/k2safe/go-omni@v0.6.0-preview +go get git.yuntuo.hk/k2safe/go-omni@v0.7.0-preview ``` `v0.4.0` 移除了 Gin 和 Gin 专属的 `WithMode` 配置；`omni.New()` 直接使用标准库路由，已有业务 Handler / Service 无需修改。 @@ -13,7 +13,7 @@ go get git.yuntuo.hk/k2safe/go-omni@v0.6.0-preview 创建新业务项目： ```sh -go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.6.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.7.0-preview go-omni new --module example.com/billing ./billing cd billing go mod tidy @@ -27,6 +27,17 @@ go run ./cmd/server 生成项目包含完整用户 CRUD、数据库 Repository、ServiceContext、建表脚本和就绪探针。 先应用生成的 `schema.sql`，再配置 `GO_OMNI_DB_DSN`；启动时会检查连接与表结构，退出时释放连接池。 +## API 定义与项目生成 + +当前源码新增 **API 定义驱动项目生成**：写 `.api` 定义请求、响应和路由，自动生成项目目录、DTO、Handler、Service 骨架及 OpenAPI。重复生成保留业务代码，支持只读 CI 检查，完整语法和示例见 [API 开发指南](API.md)。 + +```sh +go run ./cmd/go-omni api new --framework-dir . \ + --module example.com/omniDemo ../omniDemo +``` + +API 生成支持 `--database mysql --storage local --log-output file` 组合初始化。上面的命令使用本地源码；远程安装见 [发布说明](RELEASE.md)，完整业务案例见 [userService](examples/userService/README.md)。 + ## 目录结构 ```text @@ -41,6 +52,7 @@ go-omni/ ├── metrics/ # 请求指标与状态统计 ├── config/ # 配置管理 ├── validate/ # 参数校验 +├── contract/ # API 字段契约、严格多源绑定与文档约束 ├── auth/ # 认证和权限接口 ├── database/ # database/sql 生命周期和事务 ├── cache/ # 有界内存缓存、Redis 适配与合并回源 @@ -62,7 +74,7 @@ go-omni/ main 分支新增缓存、出站 HTTP、限流/幂等/超时中间件、后台任务和 `go-omni add` 模块脚手架。 使用方式、默认容量和进程内能力的适用范围见 [业务基础模块](MODULES.md)， -可以直接运行 [组合示例](examples/modules/README.md)。这些新增能力尚未包含在上面的历史版本标签中。 +可以直接运行 [组合示例](examples/modules/README.md)。这些能力从 `v0.7.0-preview` 提供。 扩展模块还包括 JWT/Session、Policy、文件配置及安全热更新、Redis 共享中间件、持久队列、多平台对象存储、OpenAPI、OpenTelemetry、熔断、迁移、审计和分页。 使用与边界见 [扩展模块](ADVANCED.md)，可运行 [文件服务组合示例](examples/platform/README.md)。go-omni 不集成 Orbit；需要流程编排的业务项目独立接入。 @@ -167,7 +179,7 @@ app.GET("/metrics", requestMetrics.Handler()) 接口返回请求总数、错误数、当前处理中请求、累计耗时和 HTTP 状态分布。它是进程内快照，生产环境可由业务侧定时采集到监控系统。 -`WithLogger(true)` 会启用 GoOmni 的统一 HTTP 访问日志；日志内容和业务日志共用 `requestId`。应用启动时可以调用 `logx.Setup` 配置 JSON/文本格式、级别、服务名和输出文件。 +`WithLogger(true)` 会启用 GoOmni 的统一 HTTP 访问日志；日志内容和业务日志共用 `requestId`。应用启动时可以调用 `logx.Setup` 配置 JSON/文本格式、级别、服务名和输出文件。文件轮转、敏感字段脱敏、异常堆栈与慢请求配置见 [logx 文档](logx/README.md)；文件直传与云端分片上传见 [storage 文档](storage/README.md)。 创建新业务项目时可通过 `go-omni new --log-output file --storage local` 按需生成配置、装配代码与可运行示例，当前源码使用方式见 [CLI 文档](cmd/go-omni/README.md)。 应用配置可以统一从 `GO_OMNI_*` 环境变量加载，再交给 `omni.New`： diff --git a/RELEASE.md b/RELEASE.md new file mode 100644 index 0000000..9d955ea --- /dev/null +++ b/RELEASE.md @@ -0,0 +1,63 @@ +# v0.7.0-preview + +这一版把 API 定义、项目生成、基础模块初始化和完整业务示例接通。 +预览版用于固定版本试用；API 方言有明确范围，HTTP/2 仍是独立实验适配器。 + +## 安装与开始 + +```sh +GOPRIVATE=git.yuntuo.hk go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.7.0-preview + +go-omni api new --module example.com/demo ./demo +cd demo +GOPRIVATE=git.yuntuo.hk go mod tidy +go run ./cmd/server +``` + +需要该 Git 仓库的读取权限时，通过自己的 Git 凭据管理器配置；不要将凭据写入 module path 或 go.mod。 +CLI 会将自身下载版本固定到生成项目的 go.mod。业务项目正常使用远程模块，不需要本地 replace。 + +使用已有定义，并一次选择基础模块： + +```sh +go-omni new --api user.api --module example.com/userService \ + --database mysql --storage local --log-output file ./userService +``` + +MySQL 与存储需要按生成 README 设置环境变量。任意契约生成的业务方法先返回 501， +由业务方实现；可直接参考 [完整用户示例](examples/userService/README.md)。 + +## 主要变化 + +- `.api` 生成项目、绑定/校验、路由、业务骨架和 OpenAPI；支持分组、导入、命名中间件。 +- 同一入口初始化 MySQL、日志、Local / S3 / R2 / OSS；标准 context 与 httpx 保持业务/网络分离。 +- 生成器保护手写业务文件，校验自有文件哈希，支持只读 --check；基础设施选项持久化到 manifest。 +- 完整用户示例包含数据库迁移、CRUD / 分页、事务审计、鉴权、文件导出与 requestId 日志。 +- 日志文件轮转/结构化脱敏、上传策略/直传/分片，以及此前 main 分支的通用业务模块一并提供。 + +所有变化及网络层验收记录见 [CHANGELOG](CHANGELOG.md)。 + +## 升级注意 + +1. 将 CLI 和项目框架依赖固定为相同版本。旧版 CRUD 项目继续使用现有目录；不能直接用 API 生成器接管已有 Go 目录。 +2. Gen.go 由生成器维护，业务实现/配置/迁移由项目维护。新增 API 后执行 generate 和 go test；修改或删除类型需要同步处理自己的业务代码。 +3. `.goOmni/apiManifest.json` 需随代码提交。切换数据库或存储选项时，在新目录生成对照后迁移；重新生成不会改写现有基础设施文件。 +4. 配置使用 lowerCamelCase。DSN、Token、存储密钥由环境变量提供；`database.Config` 的 DSN 不会被 JSON/YAML 序列化。 +5. API 的 none/memory 数据库选项仅表示不打开连接，不会从 HTTP DTO 自动推导内存仓库或数据表。 + +## 验证入口 + +```sh +GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=2 -count=1 -timeout=300s ./... +go vet ./... +go build ./... +python3 scripts/verifyAPIWorkflow.py +python3 scripts/verifyAPIWorkflow.py --version v0.7.0-preview \ + --report performance/apiWorkflowRemote.json +``` + +最后一项在全新 module cache 安装远程 CLI，并验证无本地 replace 的项目生成、 +真实 MySQL、nethttp/native 启动与业务流程。容器只使用本机临时资源，结束时清理。 +本轮验收见 [API 业务闭环报告](performance/apiWorkflowAcceptance.md)。 + +OSS/R2 云端仍需项目自己的账号做上线环境验收；当前云端适配器检查不等同于这些账号的实网测试。 diff --git a/app/app.go b/app/app.go index f3e99d4..17e3468 100644 --- a/app/app.go +++ b/app/app.go @@ -65,13 +65,14 @@ func configure(options ...config.Option) config.Config { func newApp(router httpx.Router, settings config.Config) *App { settings.TrustedProxies = append([]string(nil), settings.TrustedProxies...) + settings.Log.RedactKeys = append([]string(nil), settings.Log.RedactKeys...) resolver, err := httpx.NewClientIPResolver(settings.TrustedProxies...) if err != nil { panic("go-omni: " + err.Error()) } middlewares := []httpx.Middleware{middleware.RequestID(settings.RequestIDHeader)} if settings.EnableLogger { - middlewares = append(middlewares, middleware.AccessLog()) + middlewares = append(middlewares, middleware.AccessLog(middleware.WithSlowRequestThreshold(settings.SlowRequestThreshold))) } middlewares = append(middlewares, middleware.Recovery()) return &App{router: router, config: settings, middlewares: middlewares, clientIPResolver: resolver} @@ -93,6 +94,7 @@ func (a *App) ServeHTTP(writer http.ResponseWriter, request *http.Request) { func (a *App) Config() config.Config { settings := a.config settings.TrustedProxies = append([]string(nil), settings.TrustedProxies...) + settings.Log.RedactKeys = append([]string(nil), settings.Log.RedactKeys...) return settings } diff --git a/cmd/go-omni/README.md b/cmd/go-omni/README.md index 32385b5..10d96a8 100644 --- a/cmd/go-omni/README.md +++ b/cmd/go-omni/README.md @@ -1,16 +1,79 @@ # go-omni CLI +## API 契约驱动项目 + +```sh +# 创建包含可运行问候接口的新项目 +go run ./cmd/go-omni api new --module example.com/demo --framework-dir . ../demo + +# 从已有定义生成对应目录、类型、路由和业务骨架 +go run ./cmd/go-omni api generate --api examples/contracts/user.api \ + --dir ../userService --module example.com/userService --framework-dir . + +# 校验定义；也接受 goctl 风格的 api go -api ... -dir ... +go run ./cmd/go-omni api validate --api examples/contracts/user.api +``` + +`new --api ` 是契约项目的另一个入口。可以组合 `--database mysql --storage local --log-output file`，生成配置、依赖初始化与资源关闭。API 默认无数据库、无对象存储；`--database memory` 等同于不打开连接。`--force` 不允许覆盖 API 项目。初始配置选择写入 manifest，重新生成自动沿用。 + +后续在生成项目内执行 `go-omni api generate --api api/user.api --dir .`。 +`--check` 只读检查生成产物是否过期。生成器记录所有权，保留 Service / Repository / 配置 / main 的手写内容，遇到生成文件被手动修改则报冲突。新 Service 与中间件占位实现返回 501。 + +支持范围、参数语义、目录及安全更新策略见 [API 开发指南](../../API.md)。本版借鉴 go-zero 的工作流，不宣称兼容全部 goctl 语法；仍通过 Omni/httpx 保留网络层替换能力。 + +## 日志与对象存储按需装配 + +新增 `--log-output stdout|stderr|file`、`--log-format json|text`、`--storage none|local|s3|r2|oss`。 +指定任意一个新选项后，生成严格的配置文件、初始化代码和项目自身的测试；不指定则保留原有轻量骨架。 +对象存储独立于 `--database memory|mysql`。只有选择文件存储才会生成文件 Handler / Service 和对应适配器。 + +开发阶段可以使用本地源码；固定远程版本的安装和验收见 [发布说明](../../RELEASE.md)： + +```sh +# 在 go-omni 仓库中运行；目标目录须为空。 +go run ./cmd/go-omni new --framework-dir . \ + --module example.com/fileService --log-output file --log-format json \ + --storage local ../fileService +``` + +生成项目包含 `etc/config.yaml`、`internal/settings`、`internal/servicecontext/storage.go`、 +文件 Service/Handler、测试和 `MODULES.md`。按生成的 MODULES.md 配置示例 Token 和本地签名密钥后， +`go mod tidy && go run ./cmd/server` 即可体验。云存储分别选择 `--storage s3`、`r2`、`oss`， +再设置已有桶/凭据等环境变量；不自动创建桶或修改云资源。 + +配置优先级为默认值 → YAML/JSON → 环境变量；密钥仅允许环境变量。Service 依赖统一存储接口， +组合处负责创建客户端与有界连接池，退出时关闭自己创建的资源。文件接口提供服务端上传、直传签名、 +临时下载链接及统一错误处理；日志示例携带 requestId，文件输出默认带轮转。没有固定业务身份上下文。 + +新模板要求 `v0.7.0-preview` 或更新版本；明确指定旧版本并启用新模块会被拒绝， +避免模板引用旧框架不存在的接口。本地 `--framework-dir` 验证不受此限制。`--force` 可重新生成兼容配置， +但如果切换配置会遗留引用已移除依赖的旧文件，则拒绝操作，要求使用新目录；不会自动删除修改过的代码。 + +验证当前生成器： + +```sh +GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race ./cmd/go-omni/... +python3 scripts/verifyScaffoldModules.py +``` + +前者覆盖两种数据库与五种文件存储选择、日志方式/格式，以及生成项目的配置、资源关闭和接口测试。 +后者实际编译启动服务，验证日志、上传/下载/直传链接、错误状态、nethttp/native 下的 SIGINT/SIGTERM 和监听端口释放。 +云端选项的测试使用本地替身，不代表 OSS/R2 实网验收；本批不访问业务服务器。报告见 +[脚手架模块验收](../../performance/scaffoldModulesLocal.json)。 + +## 其他命令与远程安装 + main 分支还提供 `go-omni add orderItems --dir /path/to/project`，向已有 GoOmni 项目增加 Handler / Service / Repository / Model 与模块注册入口。模块名为 lowerCamelCase， 目标目录存在时拒绝覆盖，已有启动代码保持原样。生成模块带可替换的内存 CRUD Repository， 导入后使用 `orderItems.Register(application, repository, authMiddleware)` 注册。 -该命令尚未包含在下方历史标签中；当前源码可用 `go build -o /tmp/go-omni ./cmd/go-omni` 构建。 +该命令从 `v0.7.0-preview` 提供；源码可用 `go build -o /tmp/go-omni ./cmd/go-omni` 构建。 边界和验证方式见 [业务基础模块](../../MODULES.md)。 从远程安装，生成一个带 Handler / Service / Repository / Model 分层的服务骨架： ```sh -go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.6.0-preview +go install git.yuntuo.hk/k2safe/go-omni/cmd/go-omni@v0.7.0-preview go-omni new --module example.com/billing ./billing cd billing go mod tidy diff --git a/cmd/go-omni/api.go b/cmd/go-omni/api.go new file mode 100644 index 0000000..8928182 --- /dev/null +++ b/cmd/go-omni/api.go @@ -0,0 +1,127 @@ +package main + +import ( + "errors" + "fmt" + "strings" + + "git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/apigen" +) + +func runAPI(args []string) error { + if len(args) == 0 || args[0] == "help" || args[0] == "--help" || args[0] == "-h" { + usage() + return nil + } + command := args[0] + if command == "go" { + command = "generate" + } + if command == "check" { + command = "validate" + } + if command != "new" && command != "generate" && command != "validate" { + return fmt.Errorf("unknown api command %q", command) + } + o := apigen.Options{} + for i := 1; i < len(args); i++ { + arg := args[i] + if arg == "--help" || arg == "-h" { + usage() + return nil + } + if arg == "--check" { + if command != "generate" { + return errors.New("--check belongs to api generate") + } + o.Check = true + continue + } + if !strings.HasPrefix(arg, "-") { + if command == "new" && o.Dir == "" { + o.Dir = arg + continue + } + return fmt.Errorf("unexpected argument %q", arg) + } + name, value, inline := strings.Cut(arg, "=") + // goctl-style -api/-dir aliases ease migration; long forms are canonical. + if name == "-api" { + name = "--api" + } + if name == "-dir" { + name = "--dir" + } + switch name { + case "--api", "--dir", "--module", "--framework-dir", "--framework-version", "--log-output", "--log-format", "--database", "--storage": + default: + return fmt.Errorf("unknown api option %q", name) + } + if !inline { + if i+1 >= len(args) || strings.HasPrefix(args[i+1], "-") { + return fmt.Errorf("%s requires a value", name) + } + i++ + value = args[i] + } + if strings.TrimSpace(value) == "" { + return fmt.Errorf("%s requires a value", name) + } + if command == "validate" && name != "--api" { + return fmt.Errorf("api validate does not accept %s", name) + } + switch name { + case "--api": + o.API = value + case "--dir": + o.Dir = value + case "--module": + o.Module = value + case "--framework-dir": + o.FrameworkDir = value + case "--framework-version": + o.FrameworkVersion = value + case "--database": + o.Database = value + case "--storage": + o.Storage = value + case "--log-output": + o.LogOutput = value + case "--log-format": + o.LogFormat = value + } + } + if command == "validate" { + if o.API == "" { + return errors.New("--api is required") + } + spec, err := apigen.Parse(o.API) + if err != nil { + return err + } + if _, err = apigen.Compile(spec); err != nil { + return err + } + fmt.Printf("valid API: %d types, %d routes\n", len(spec.Types), len(spec.Routes)) + return nil + } + var out apigen.Output + var err error + if command == "new" { + out, err = apigen.New(o) + } else { + if o.Dir == "" { + o.Dir = "." + } + out, err = apigen.Generate(o) + } + if err != nil { + return err + } + if o.Check { + fmt.Println("API generated files are current") + } else { + fmt.Printf("API project ready: %d generated, %d created, %d preserved, %d removed\n", len(out.Generated), len(out.Created), len(out.Preserved), len(out.Removed)) + } + return nil +} diff --git a/cmd/go-omni/api_test.go b/cmd/go-omni/api_test.go new file mode 100644 index 0000000..3e5ab1b --- /dev/null +++ b/cmd/go-omni/api_test.go @@ -0,0 +1,44 @@ +package main + +import ( + "os" + "path/filepath" + "strings" + "testing" +) + +func TestAPICommands(t *testing.T) { + root, err := filepath.Abs("../..") + if err != nil { + t.Fatal(err) + } + project := filepath.Join(t.TempDir(), "demo") + if err = runAPI([]string{"new", project, "--module=example.com/demo", "--framework-dir", root}); err != nil { + t.Fatal(err) + } + api := filepath.Join(project, "api/service.api") + if err = runAPI([]string{"validate", "-api", api}); err != nil { + t.Fatal(err) + } + if err = runAPI([]string{"go", "-api", api, "-dir", project, "--check"}); err != nil { + t.Fatal(err) + } + if err = runNew([]string{"--force", project}); err == nil { + t.Fatal("CRUD starter overwrote API project") + } + target := filepath.Join(t.TempDir(), "other") + if err = runNew([]string{"--api", api, "--module", "example.com/other", "--framework-dir", root, target}); err != nil { + t.Fatal(err) + } + if _, err = os.Stat(filepath.Join(target, "internal/service/greet/greet.go")); err != nil { + t.Fatal(err) + } + for _, args := range [][]string{{"validate"}, {"generate", "--api"}, {"new", "--api=x", "somewhere"}, {"generate", "--dir="}, {"generate", "--force"}, {"validate", "--api", api, "--dir", project}, {"validate", "--check"}} { + if err = runAPI(args); err == nil { + t.Fatalf("invalid args accepted: %v", args) + } + } + if err = runNew([]string{"--api", api, "--force", target}); err == nil || !strings.Contains(err.Error(), "does not accept --force") { + t.Fatalf("incompatible profile: %v", err) + } +} diff --git a/cmd/go-omni/apigen/apigen_test.go b/cmd/go-omni/apigen/apigen_test.go new file mode 100644 index 0000000..00a46aa --- /dev/null +++ b/cmd/go-omni/apigen/apigen_test.go @@ -0,0 +1,360 @@ +package apigen + +import ( + "context" + "encoding/json" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" +) + +func frameworkRoot(t *testing.T) string { + t.Helper() + p, err := filepath.Abs("../../..") + if err != nil { + t.Fatal(err) + } + return p +} +func writeFile(t *testing.T, name, body string) { + t.Helper() + if err := os.MkdirAll(filepath.Dir(name), 0755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(name, []byte(body), 0644); err != nil { + t.Fatal(err) + } +} +func readFile(t *testing.T, name string) string { + t.Helper() + b, err := os.ReadFile(name) + if err != nil { + t.Fatal(err) + } + return string(b) +} +func parseText(t *testing.T, source string) (*Spec, error) { + t.Helper() + file := filepath.Join(t.TempDir(), "service.api") + writeFile(t, file, source) + return Parse(file) +} + +func TestCompileExample(t *testing.T) { + s, err := Parse(filepath.Join(frameworkRoot(t), "examples/contracts/user.api")) + if err != nil { + t.Fatal(err) + } + c, err := Compile(s) + if err != nil { + t.Fatal(err) + } + if len(c.Groups) != 2 || len(s.Routes) != 3 || len(s.Sources) != 2 { + t.Fatalf("wrong spec: %+v", s) + } + var doc map[string]any + if err = json.Unmarshal(c.Document, &doc); err != nil { + t.Fatal(err) + } + route := doc["paths"].(map[string]any)["/api/v1/users/{userId}"].(map[string]any)["put"].(map[string]any) + if len(route["parameters"].([]any)) != 3 { + t.Fatal("missing mixed parameters") + } + schema := route["requestBody"].(map[string]any)["content"].(map[string]any)["application/json"].(map[string]any)["schema"].(map[string]any) + if schema["additionalProperties"] != false || len(schema["required"].([]any)) != 2 { + t.Fatalf("wrong body schema: %+v", schema) + } + if strings.Contains(string(c.Document), `"user_id"`) { + t.Fatal("snake case wire name") + } +} +func TestInvalidDefinitions(t *testing.T) { + cases := []string{ + `syntax="v2"`, + `@server ( jwt: Auth ) service user { @handler go get /go }`, + "type Request { Name string `json:\"name\" binding:\"typo\"` }\nservice user { @handler run\n post /go (Request) }", + "type A { Other *A `json:\"other,optional\"` }\nservice user { @handler run\n post /go (A) }", + "type A { Other Missing `json:\"other\"` }\nservice user { @handler run\n post /go (A) }", + "type A { UserID int `json:\"user_id\"` }\nservice user { @handler run\n post /go (A) }", + "service user { @handler go\n get /go\n @handler other\n get /go\n}", + "service user { @handler go\n get /:id\n}", + "service user { @handler service\n get /go\n}", + "type A { ID int `path:\"id\"` }\nservice user { @handler go\n get /go (A)\n}", + "type A { Name string `json:\"name\"` }\nservice user { @handler go\n get /go (A)\n}", + "service user { @handler go\n @status 201\n post /go\n}", + "type A { Name string `header:\"Name\"` }\nservice user { @handler go\n get /go returns (A)\n}", + "@server (group: ../../bad)\nservice user { @handler go\n get /go\n}", + "service user { @handler go\n get /health/ready\n}", + "type A { Name string `json:\"name\"`\n Name int `json:\"other\"` }\nservice user { @handler go\n post /go (A)\n}", + "service user { @handler go\n get /go\n @handler Go\n get /other\n}", + "type A { Value []byte `json:\"value\"` }\nservice user { @handler go\n post /go (A)\n}", + } + for i, source := range cases { + s, err := parseText(t, source) + if err == nil { + _, err = Compile(s) + } + if err == nil { + t.Fatalf("case %d accepted: %s", i, source) + } + } +} +func TestImports(t *testing.T) { + root := t.TempDir() + entry := filepath.Join(root, "api", "entry.api") + writeFile(t, entry, "import \"types.api\"\nservice user { @handler ping\n get /ping\n}") + writeFile(t, filepath.Join(root, "api/types.api"), `import "entry.api"`) + if _, err := Parse(entry); err == nil || !strings.Contains(err.Error(), "circular") { + t.Fatalf("cycle: %v", err) + } + writeFile(t, entry, "import \"../outside.api\"\nservice user { @handler ping\n get /ping\n}") + writeFile(t, filepath.Join(root, "outside.api"), `type X { Value string `+"`json:\"value\"`"+` }`) + if _, err := Parse(entry); err == nil { + t.Fatal("escaped import root") + } + writeFile(t, entry, "import (\n \"shared.api\"\n \"./shared.api\"\n)\nservice user { @handler ping\n get /ping\n}") + writeFile(t, filepath.Join(root, "api/shared.api"), `type X { Value string `+"`json:\"value\"`"+` }`) + s, err := Parse(entry) + if err != nil || len(s.Types) != 1 { + t.Fatalf("duplicate import: %+v %v", s, err) + } +} + +func TestSharedPrefixConflict(t *testing.T) { + source := "type A { ID int `path:\"id\"` }\ntype B { Name string `path:\"name\"` }\nservice user {\n @handler first\n get /users/:id (A)\n @handler second\n get /users/:name/items (B)\n}" + s, err := parseText(t, source) + if err != nil { + t.Fatal(err) + } + if _, err = Compile(s); err == nil { + t.Fatal("shared prefix conflict was accepted") + } +} + +func FuzzParser(f *testing.F) { + for _, s := range []string{starterAPI, "syntax = \"v1\"", "type (", "@server (prefix: /api)"} { + f.Add(s) + } + f.Fuzz(func(t *testing.T, source string) { + if len(source) > 16384 { + t.Skip() + } + tokens, err := lex("fuzz.api", source) + if err != nil { + return + } + spec := &Spec{Title: "Fuzz", Version: "1"} + p := parser{tokens: tokens, spec: spec} + if p.parse() == nil { + _, _ = Compile(spec) + } + }) +} +func TestRegenerationAndConflicts(t *testing.T) { + dir := filepath.Join(t.TempDir(), "app") + if _, err := New(Options{Dir: dir, Module: "example.com/test", FrameworkDir: frameworkRoot(t)}); err != nil { + t.Fatal(err) + } + o := Options{Dir: dir, API: filepath.Join(dir, "api/service.api")} + if _, err := Generate(o); err != nil { + t.Fatal(err) + } + manifestBefore := readFile(t, filepath.Join(dir, manifestPath)) + typesBefore := readFile(t, filepath.Join(dir, "internal/types/typesGen.go")) + o.Check = true + if _, err := Generate(o); err != nil { + t.Fatal(err) + } + servicePath := filepath.Join(dir, "internal/service/greet/greet.go") + service := readFile(t, servicePath) + "\n// user edit must survive\n" + writeFile(t, servicePath, service) + for _, file := range []string{"cmd/server/main.go", "etc/config.yaml", "internal/servicecontext/context.go", "internal/repository/greet/repository.go"} { + body := readFile(t, filepath.Join(dir, file)) + if strings.HasSuffix(file, ".go") { + body += "\n// handwritten extension\n" + } else { + body += "\n# handwritten extension\n" + } + writeFile(t, filepath.Join(dir, file), body) + } + api := readFile(t, o.API) + api += "\n@server (group: extra)\nservice greet-api {\n @handler ping\n get /ping\n}\n" + writeFile(t, o.API, api) + if _, err := Generate(o); err == nil { + t.Fatal("check accepted stale output") + } + if readFile(t, filepath.Join(dir, manifestPath)) != manifestBefore { + t.Fatal("--check wrote manifest") + } + o.Check = false + if _, err := Generate(o); err != nil { + t.Fatal(err) + } + if readFile(t, servicePath) != service { + t.Fatal("business code overwritten") + } + // Retiring a group removes only owned handlers, keeping its business files. + writeFile(t, o.API, strings.Split(api, "\n@server (group: extra)")[0]) + if _, err := Generate(o); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(dir, "internal/handler/extra/handlersGen.go")); !os.IsNotExist(err) { + t.Fatal("stale handler remains") + } + if _, err := os.Stat(filepath.Join(dir, "internal/service/extra/ping.go")); err != nil { + t.Fatal("business stub removed") + } + writeFile(t, filepath.Join(dir, "internal/types/typesGen.go"), typesBefore+"\n// edited generated file\n") + before := snapshot(t, dir) + if _, err := Generate(o); err == nil || !strings.Contains(err.Error(), "was edited") { + t.Fatalf("did not protect edited output: %v", err) + } + after := snapshot(t, dir) + if !sameFiles(before, after) { + t.Fatal("failed regeneration changed files") + } +} +func TestOutputBoundaries(t *testing.T) { + for _, kind := range []string{"symlink", "unowned", "lock", "caseCollision"} { + t.Run(kind, func(t *testing.T) { + root := t.TempDir() + dir := filepath.Join(root, "app") + api := filepath.Join(root, "input/service.api") + writeFile(t, api, starterAPI) + switch kind { + case "symlink": + if err := os.MkdirAll(dir, 0755); err != nil { + t.Fatal(err) + } + outside := filepath.Join(root, "outside") + os.MkdirAll(outside, 0755) + if err := os.Symlink(outside, filepath.Join(dir, "internal")); err != nil { + t.Fatal(err) + } + case "unowned": + writeFile(t, filepath.Join(dir, "internal/types/typesGen.go"), "package types // mine\n") + case "lock": + writeFile(t, filepath.Join(dir, ".goOmni/api.lock"), "someone else") + case "caseCollision": + writeFile(t, api, starterAPI+"\n@server (group: gReet)\nservice greet-api {\n @handler extra\n get /extra\n}\n") + } + before := snapshot(t, root) + if _, err := Generate(Options{Dir: dir, API: api, Module: "example.com/test", FrameworkDir: frameworkRoot(t)}); err == nil { + t.Fatal("unsafe output accepted") + } + if !sameFiles(before, snapshot(t, root)) { + t.Fatal("conflict changed files") + } + }) + } +} +func snapshot(t *testing.T, root string) map[string]string { + t.Helper() + out := map[string]string{} + err := filepath.WalkDir(root, func(path string, d os.DirEntry, err error) error { + if err != nil { + return err + } + if d.Type().IsRegular() { + data, e := os.ReadFile(path) + if e != nil { + return e + } + rel, _ := filepath.Rel(root, path) + out[rel] = string(data) + } + return nil + }) + if err != nil { + t.Fatal(err) + } + return out +} +func sameFiles(a, b map[string]string) bool { + if len(a) != len(b) { + return false + } + for k, v := range a { + if b[k] != v { + return false + } + } + return true +} + +func TestGeneratedProjectsBuildAndRun(t *testing.T) { + if testing.Short() { + t.Skip("consumer module integration") + } + root := frameworkRoot(t) + for _, starter := range []bool{true, false} { + name := "contract" + if starter { + name = "starter" + } + t.Run(name, func(t *testing.T) { + dir := filepath.Join(t.TempDir(), "project") + o := Options{Dir: dir, Module: "example.com/consumer", FrameworkDir: root, LogOutput: "file", LogFormat: "text"} + var err error + if starter { + _, err = New(o) + } else { + o.API = filepath.Join(root, "examples/contracts/user.api") + _, err = Generate(o) + } + if err != nil { + t.Fatal(err) + } + source := consumerContractTest + if starter { + source = consumerStarterTest + } else { + writeFile(t, filepath.Join(dir, "internal/service/user/updateUser.go"), `package user +import("context";"example.com/consumer/internal/types") +func(s *Service) UpdateUser(ctx context.Context,req *types.UpdateUserReq)(*types.UserResp,error){return &types.UserResp{UserID:req.UserID,Name:req.Name,Active:req.Active},nil} +`) + } + writeFile(t, filepath.Join(dir, "contract_test.go"), source) + o.API = filepath.Join(dir, "api/service.api") + if !starter { + o.API = filepath.Join(dir, "api/user.api") + } + if _, err = Generate(o); err != nil { + t.Fatal(err) + } + ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, "go", "test", "-mod=mod", "-race", "-count=1", "./...") + cmd.Dir = dir + cmd.Env = append(os.Environ(), "GOWORK=off", "GOFLAGS=") + output, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("consumer: %v\n%s", err, output) + } + t.Log(string(output)) + }) + } +} + +const consumerStarterTest = `package consumer_test +import("context";"io";"net/http/httptest";"strings";"testing";omni "git.yuntuo.hk/k2safe/go-omni";"git.yuntuo.hk/k2safe/go-omni/logx";"example.com/consumer/internal/handler";"example.com/consumer/internal/servicecontext";"example.com/consumer/internal/settings") +func TestGreeting(t *testing.T){cfg,err:=settings.Load("etc/config.yaml");if err!=nil{t.Fatal(err)};cfg.Server.Log.Writer=io.Discard;cfg.Server.Log.File="";cfg.Server.Log.Rotation=logx.RotationConfig{};logger,err:=logx.New(cfg.Server.Log);if err!=nil{t.Fatal(err)};defer logger.Close();deps,err:=servicecontext.New(context.Background(),logger);if err!=nil{t.Fatal(err)};defer deps.Close();app:=omni.New();defer app.Close();if err=handler.RegisterRoutes(app,deps);err!=nil{t.Fatal(err)};w:=httptest.NewRecorder();app.Engine().ServeHTTP(w,httptest.NewRequest("GET","/api/greet/Alice",nil));if w.Code!=200||!strings.Contains(w.Body.String(),"Hello Alice")||w.Header().Get("X-Request-ID")==""{t.Fatalf("%d %s",w.Code,w.Body.String())}} +` +const consumerContractTest = `package consumer_test +import("context";"encoding/json";"io";"net/http/httptest";"strings";"testing";omni "git.yuntuo.hk/k2safe/go-omni";"git.yuntuo.hk/k2safe/go-omni/logx";"example.com/consumer/internal/handler";"example.com/consumer/internal/servicecontext") +func TestContract(t *testing.T){cfg:=logx.Default();cfg.Writer=io.Discard;logger,err:=logx.New(cfg);if err!=nil{t.Fatal(err)};defer logger.Close();deps,err:=servicecontext.New(context.Background(),logger);if err!=nil{t.Fatal(err)};defer deps.Close();app:=omni.New();defer app.Close();if err=handler.RegisterRoutes(app,deps);err!=nil{t.Fatal(err)} + for _,tc:=range []struct{method,path,body string;status int}{ + {"PUT","/api/v1/users/7?notify=true",` + "`" + `{"name":"Alice","active":false,"age":0}` + "`" + `,200}, + {"PUT","/api/v1/users/7",` + "`" + `{"name":"Alice"}` + "`" + `,400}, + {"PUT","/api/v1/users/7?notify=true¬ify=false",` + "`" + `{"name":"Alice","active":false}` + "`" + `,400}, + {"PUT","/api/v1/users/7",` + "`" + `{"name":"Alice","active":false,"age":151}` + "`" + `,400}, + {"GET","/api/v1/users?page=0","",400}, + {"GET","/api/v1/users?page=1","",501}, + {"DELETE","/admin/v1/users/7","",501}, + }{req:=httptest.NewRequest(tc.method,tc.path,strings.NewReader(tc.body));req.Header.Set("Content-Type","application/json");w:=httptest.NewRecorder();app.Engine().ServeHTTP(w,req);if w.Code!=tc.status{t.Fatalf("%s: %d %s",tc.path,w.Code,w.Body.String())};if tc.status==200{var v struct{Data struct{UserID int64 ` + "`json:\"userId\"`" + `;Name string ` + "`json:\"name\"`" + `}};if err:=json.Unmarshal(w.Body.Bytes(),&v);err!=nil||v.Data.UserID!=7||v.Data.Name!="Alice"{t.Fatal(w.Body.String())}}} + delete(deps.Middlewares,"Auth");if err:=handler.RegisterRoutes(omni.New(),deps);err==nil{t.Fatal("missing auth hook accepted")} +} +` diff --git a/cmd/go-omni/apigen/compile.go b/cmd/go-omni/apigen/compile.go new file mode 100644 index 0000000..d69146d --- /dev/null +++ b/cmd/go-omni/apigen/compile.go @@ -0,0 +1,332 @@ +package apigen + +import ( + "bytes" + "encoding/json" + "errors" + "fmt" + "go/ast" + goparser "go/parser" + gotoken "go/token" + "reflect" + "regexp" + "sort" + "strings" + + "git.yuntuo.hk/k2safe/go-omni/contract" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/openapi" +) + +type Compiled struct { + Spec *Spec + Bindings map[string]*contract.Binder + Document []byte + Groups []string +} + +var exportedName = regexp.MustCompile(`^[A-Z][A-Za-z0-9]*$`) +var goName = regexp.MustCompile(`^[a-zA-Z][A-Za-z0-9]*$`) +var lowerName = regexp.MustCompile(`^[a-z][A-Za-z0-9]*$`) +var serviceName = regexp.MustCompile(`^[a-zA-Z][a-zA-Z0-9-]*$`) +var staticSegment = regexp.MustCompile(`^[A-Za-z0-9._~-]+$`) + +func Compile(spec *Spec) (*Compiled, error) { + if spec == nil || len(spec.Routes) == 0 || !serviceName.MatchString(spec.Service) { + return nil, errors.New("api: valid service and routes required") + } + definitions := map[string]Type{} + for _, t := range spec.Types { + if !exportedName.MatchString(t.Name) { + return nil, fmt.Errorf("%s: type names must be UpperCamelCase", t.Position) + } + if _, ok := definitions[t.Name]; ok { + return nil, fmt.Errorf("%s: duplicate type %s", t.Position, t.Name) + } + definitions[t.Name] = t + } + resolved := map[string]reflect.Type{} + resolving := map[string]bool{} + var resolve func(string, int) (reflect.Type, error) + var expression func(ast.Expr, int) (reflect.Type, error) + primitives := map[string]reflect.Type{"string": reflect.TypeOf(""), "bool": reflect.TypeOf(false), "int": reflect.TypeOf(int(0)), "int8": reflect.TypeOf(int8(0)), "int16": reflect.TypeOf(int16(0)), "int32": reflect.TypeOf(int32(0)), "int64": reflect.TypeOf(int64(0)), "uint": reflect.TypeOf(uint(0)), "uint8": reflect.TypeOf(uint8(0)), "uint16": reflect.TypeOf(uint16(0)), "uint32": reflect.TypeOf(uint32(0)), "uint64": reflect.TypeOf(uint64(0)), "float32": reflect.TypeOf(float32(0)), "float64": reflect.TypeOf(float64(0))} + expression = func(e ast.Expr, depth int) (reflect.Type, error) { + if depth > 32 { + return nil, errors.New("type nesting exceeds 32") + } + switch e := e.(type) { + case *ast.Ident: + if t, ok := primitives[e.Name]; ok { + return t, nil + } + return resolve(e.Name, depth+1) + case *ast.StarExpr: + if _, double := e.X.(*ast.StarExpr); double { + return nil, errors.New("multiple pointer levels unsupported") + } + t, err := expression(e.X, depth+1) + if err != nil { + return nil, err + } + return reflect.PointerTo(t), nil + case *ast.ArrayType: + if e.Len != nil { + return nil, errors.New("use slices, not fixed arrays") + } + t, err := expression(e.Elt, depth+1) + if err != nil { + return nil, err + } + return reflect.SliceOf(t), nil + case *ast.MapType: + key, ok := e.Key.(*ast.Ident) + if !ok || key.Name != "string" { + return nil, errors.New("map keys must be string") + } + t, err := expression(e.Value, depth+1) + if err != nil { + return nil, err + } + return reflect.MapOf(reflect.TypeOf(""), t), nil + default: + return nil, errors.New("unsupported type expression") + } + } + resolve = func(name string, depth int) (reflect.Type, error) { + if depth > 32 { + return nil, errors.New("type nesting exceeds 32") + } + if t, ok := resolved[name]; ok { + return t, nil + } + if resolving[name] { + return nil, fmt.Errorf("recursive type %s is unsupported", name) + } + typ, ok := definitions[name] + if !ok { + return nil, fmt.Errorf("unknown type %s", name) + } + resolving[name] = true + defer delete(resolving, name) + var fields []reflect.StructField + seen := map[string]bool{} + for _, f := range typ.Fields { + if !exportedName.MatchString(f.Name) || seen[f.Name] { + return nil, fmt.Errorf("%s: invalid or duplicate field %s", name, f.Name) + } + seen[f.Name] = true + expr, err := goparser.ParseExpr(f.Type) + if err != nil { + return nil, err + } + t, err := expression(expr, depth+1) + if err != nil { + return nil, fmt.Errorf("%s.%s: %w", name, f.Name, err) + } + fields = append(fields, reflect.StructField{Name: f.Name, Type: t, Tag: reflect.StructTag(f.Tag)}) + } + t := reflect.StructOf(fields) + resolved[name] = t + return t, nil + } + c := &Compiled{Spec: spec, Bindings: map[string]*contract.Binder{}} + for _, t := range spec.Types { + rt, err := resolve(t.Name, 0) + if err != nil { + return nil, err + } + b, err := contract.ForType(rt) + if err != nil { + return nil, fmt.Errorf("%s: %w", t.Position, err) + } + c.Bindings[t.Name] = b + } + api, err := openapi.New(spec.Title, spec.Version) + if err != nil { + return nil, err + } + handlers, paths, shapes, groups := map[string]bool{}, map[string]bool{}, map[string]string{}, map[string]bool{} + middlewareSymbols := map[string]string{} + for _, r := range spec.Routes { + if !goName.MatchString(r.Handler) { + return nil, fmt.Errorf("%s: invalid handler name", r.Position) + } + if strings.EqualFold(r.Handler, "service") { + return nil, errors.New("handler name service conflicts with the service constructor file") + } + if r.Group == "" { + return nil, errors.New("group required") + } + for _, part := range strings.Split(r.Group, "/") { + if !lowerName.MatchString(part) || gotoken.Lookup(part).IsKeyword() { + return nil, fmt.Errorf("invalid lowerCamelCase group %q", r.Group) + } + } + h := r.Group + "/" + strings.ToLower(r.Handler) + if handlers[h] { + return nil, fmt.Errorf("duplicate handler %s in group %s", r.Handler, r.Group) + } + handlers[h] = true + groups[r.Group] = true + mseen := map[string]bool{} + for _, m := range r.Middleware { + if !goName.MatchString(m) || mseen[m] { + return nil, fmt.Errorf("invalid or duplicate middleware %q", m) + } + mseen[m] = true + if strings.EqualFold(m, "defaults") || strings.EqualFold(m, "registryGen") { + return nil, fmt.Errorf("reserved middleware name %q", m) + } + if prior, ok := middlewareSymbols[strings.ToLower(m)]; ok && prior != m { + return nil, fmt.Errorf("middleware names %s and %s collide", prior, m) + } + middlewareSymbols[strings.ToLower(m)] = m + } + shape, err := routeShape(r.Path) + if err != nil { + return nil, fmt.Errorf("%s: %w", r.Position, err) + } + for _, prior := range spec.Routes { + if prior.Method != r.Method || prior.Path == r.Path { + continue + } + if conflictPrefix(prior.Path, r.Path) { + return nil, fmt.Errorf("conflicting parameter names at a shared route prefix: %s and %s", prior.Path, r.Path) + } + } + if prior, exists := shapes[shape]; exists && prior != r.Path { + return nil, fmt.Errorf("ambiguous route parameter names: %s and %s", prior, r.Path) + } + shapes[shape] = r.Path + pathKey := r.Method + " " + shape + if paths[pathKey] { + return nil, fmt.Errorf("duplicate route %s", pathKey) + } + paths[pathKey] = true + if r.Path == "/health/live" || r.Path == "/health/ready" || r.Path == "/openapi.json" { + return nil, fmt.Errorf("route %s is reserved by the generated server", r.Path) + } + if r.Method != strings.ToLower(r.Method) { + return nil, errors.New("HTTP methods must be lowercase") + } + if r.Status != 200 && r.Status != 201 && r.Status != 202 && r.Status != 204 { + return nil, errors.New("@status supports 200, 201, 202 or 204") + } + if (r.Status == 204) != (r.Response == "") { + return nil, errors.New("204 requires no response; other success statuses require a response type") + } + if r.Method == "head" && r.Response != "" { + return nil, errors.New("HEAD must not declare a response body") + } + op := openapi.Operation{Method: r.Method, Path: r.Path, OperationID: operationID(r), Summary: r.Summary, Responses: map[int]openapi.Response{}} + if r.Request != "" { + b, ok := c.Bindings[r.Request] + if !ok { + return nil, fmt.Errorf("unknown request %s", r.Request) + } + op.Parameters = b.Parameters() + op.Request = b.BodySchema() + if op.Request != nil && (r.Method == "get" || r.Method == "head") { + return nil, errors.New("GET/HEAD requests cannot declare JSON bodies; use form/path/header tags") + } + } + var response *openapi.Schema + if r.Response != "" { + b, ok := c.Bindings[r.Response] + if !ok { + return nil, fmt.Errorf("unknown response %s", r.Response) + } + s, err := b.ResponseSchema() + if err != nil { + return nil, err + } + response = openapi.Envelope(s) + response.Required = append(response.Required, "data") + } + op.Responses[r.Status] = openapi.Response{Description: "Success", Schema: response} + errorSchema := &openapi.Schema{Type: "object", Required: []string{"code", "message"}, Properties: map[string]*openapi.Schema{"code": {Type: "string"}, "message": {Type: "string"}, "requestId": {Type: "string"}}} + for _, code := range []int{400, 413, 415, 500, 501} { + op.Responses[code] = openapi.Response{Description: "Error", Schema: errorSchema} + } + if err = api.Handle(discardRoutes{}, op, func(*httpx.Context) error { return nil }); err != nil { + return nil, fmt.Errorf("%s: %w", r.Position, err) + } + } + for group := range groups { + c.Groups = append(c.Groups, group) + } + sort.Strings(c.Groups) + c.Document, err = api.JSON() + if err != nil { + return nil, err + } + // Show named hooks in documentation without claiming they implement a + // particular authentication scheme. Applications own that policy. + var doc map[string]any + decoder := json.NewDecoder(bytes.NewReader(c.Document)) + decoder.UseNumber() + if err = decoder.Decode(&doc); err != nil { + return nil, err + } + for _, r := range spec.Routes { + if len(r.Middleware) > 0 { + path := r.Path + for _, part := range strings.Split(r.Path, "/") { + if strings.HasPrefix(part, ":") { + path = strings.ReplaceAll(path, part, "{"+part[1:]+"}") + } + } + doc["paths"].(map[string]any)[path].(map[string]any)[r.Method].(map[string]any)["x-omni-middleware"] = r.Middleware + } + } + c.Document, err = json.MarshalIndent(doc, "", " ") + return c, err +} + +type discardRoutes struct{} + +func (discardRoutes) Handle(string, string, httpx.Handler, ...httpx.Middleware) {} +func operationID(r Route) string { + return strings.ReplaceAll(r.Group, "/", "_") + "_" + upper(r.Handler) +} +func upper(s string) string { return strings.ToUpper(s[:1]) + s[1:] } +func lower(s string) string { return strings.ToLower(s[:1]) + s[1:] } +func routeShape(path string) (string, error) { + if path == "/" { + return path, nil + } + if !strings.HasPrefix(path, "/") { + return "", errors.New("route must start with /") + } + parts := strings.Split(path[1:], "/") + seen := map[string]bool{} + for i, part := range parts { + if strings.HasPrefix(part, ":") { + name := part[1:] + if !lowerName.MatchString(name) || seen[name] { + return "", errors.New("invalid/duplicate path parameter") + } + seen[name] = true + parts[i] = ":" + } else if !staticSegment.MatchString(part) || part == "." || part == ".." { + return "", errors.New("invalid route segment") + } + } + return "/" + strings.Join(parts, "/"), nil +} + +func conflictPrefix(left, right string) bool { + a, b := strings.Split(left, "/"), strings.Split(right, "/") + for i := 0; i < len(a) && i < len(b); i++ { + if strings.HasPrefix(a[i], ":") && strings.HasPrefix(b[i], ":") { + if a[i] != b[i] { + return true + } + continue + } + if a[i] != b[i] { + return false + } + } + return false +} diff --git a/cmd/go-omni/apigen/generate.go b/cmd/go-omni/apigen/generate.go new file mode 100644 index 0000000..13baae8 --- /dev/null +++ b/cmd/go-omni/apigen/generate.go @@ -0,0 +1,458 @@ +package apigen + +import ( + "errors" + "fmt" + "go/format" + "os" + "path/filepath" + "runtime/debug" + "sort" + "strconv" + "strings" + + "git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/scaffold" + "golang.org/x/mod/modfile" + "golang.org/x/mod/module" + "golang.org/x/mod/semver" +) + +const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" +const minimumVersion = "v0.7.0-preview" +const generatedHeader = "// Code generated by go-omni api. DO NOT EDIT.\n" + +type Options struct { + Database, Storage string + API, Dir, Module, FrameworkDir, FrameworkVersion, LogOutput, LogFormat string + Check bool +} +type Output struct{ Generated, Created, Preserved, Removed []string } + +// Generate creates or safely updates one contract-driven project. It renders +// and validates every output before applying an ownership-checked transaction. +func Generate(o Options) (Output, error) { + if o.API == "" || o.Dir == "" { + return Output{}, errors.New("--api and --dir are required") + } + spec, err := Parse(o.API) + if err != nil { + return Output{}, err + } + compiled, err := Compile(spec) + if err != nil { + return Output{}, err + } + return generate(o, compiled, false) +} + +const starterAPI = "syntax = \"v1\"\n\ninfo (\n title: \"Greeting API\"\n version: \"1.0.0\"\n)\n\ntype GreetReq {\n Name string `path:\"name\" binding:\"required,max=80\"`\n}\ntype GreetResp {\n Message string `json:\"message\"`\n}\n\n@server (\n group: greet\n prefix: /api\n)\nservice greet-api {\n @doc \"Greet a visitor\"\n @handler greet\n get /greet/:name (GreetReq) returns (GreetResp)\n}\n" + +// New creates a runnable greeting project, including its editable API file. +func New(o Options) (Output, error) { + if o.API != "" || o.Check { + return Output{}, errors.New("api new does not accept --api or --check") + } + if o.Dir == "" { + return Output{}, errors.New("project directory required") + } + if entries, err := os.ReadDir(o.Dir); err == nil && len(entries) > 0 { + return Output{}, errors.New("api new requires an empty directory") + } else if err != nil && !errors.Is(err, os.ErrNotExist) { + return Output{}, err + } + tokens, err := lex("service.api", starterAPI) + if err != nil { + return Output{}, err + } + spec := &Spec{Version: "1.0.0", Entry: "service.api", Sources: map[string][]byte{"service.api": []byte(starterAPI)}} + p := parser{tokens: tokens, spec: spec} + if err = p.parse(); err != nil { + return Output{}, err + } + c, err := Compile(spec) + if err != nil { + return Output{}, err + } + return generate(o, c, true) +} + +func generate(o Options, c *Compiled, starter bool) (Output, error) { + directory, err := filepath.Abs(o.Dir) + if err != nil { + return Output{}, err + } + o.Dir = directory + if err := resolveProfile(&o); err != nil { + return Output{}, err + } + seeds := map[string][]byte{} + moduleData, readErr := os.ReadFile(filepath.Join(directory, "go.mod")) + if readErr == nil { + mf, err := modfile.Parse("go.mod", moduleData, nil) + if err != nil || mf.Module == nil { + return Output{}, errors.New("invalid project go.mod") + } + if o.Module != "" && o.Module != mf.Module.Mod.Path { + return Output{}, errors.New("--module differs from existing go.mod") + } + o.Module = mf.Module.Mod.Path + found := false + requiredVersion := "" + for _, r := range mf.Require { + if r.Mod.Path == frameworkModule { + found = true + requiredVersion = r.Mod.Version + if o.FrameworkVersion != "" && o.FrameworkVersion != r.Mod.Version { + return Output{}, errors.New("existing framework version differs; update go.mod explicitly") + } + } + } + if !found { + return Output{}, errors.New("existing go.mod must require go-omni") + } + if semver.Compare(requiredVersion, minimumVersion) < 0 { + localCurrent := false + for _, r := range mf.Replace { + if r.Old.Path == frameworkModule && r.New.Version == "" { + path := r.New.Path + if !filepath.IsAbs(path) { + path = filepath.Join(directory, path) + } + if _, err := os.Stat(filepath.Join(path, "contract", "contract.go")); err == nil { + localCurrent = true + } + } + } + if !localCurrent { + return Output{}, fmt.Errorf("existing framework dependency predates API support; use %s or a current local replacement", minimumVersion) + } + } + if o.FrameworkDir != "" { + matched := false + for _, r := range mf.Replace { + if r.Old.Path == frameworkModule && r.New.Version == "" { + replacement := r.New.Path + if !filepath.IsAbs(replacement) { + replacement = filepath.Join(directory, replacement) + } + matched = canonical(replacement) == canonical(o.FrameworkDir) + } + } + if !matched { + return Output{}, errors.New("existing framework replacement differs; update go.mod explicitly") + } + } + } else if !errors.Is(readErr, os.ErrNotExist) { + return Output{}, readErr + } else { + if o.Module == "" { + o.Module = filepath.Base(directory) + } + data, err := moduleFile(o) + if err != nil { + return Output{}, err + } + seeds["go.mod"] = data + } + if err = module.CheckImportPath(o.Module); err != nil { + return Output{}, err + } + generated, base, err := renderProject(o, c, starter) + if err != nil { + return Output{}, err + } + for name, data := range base { + seeds[name] = data + } + // The API source remains developer-owned. A relocated project regenerates + // from its own api/ tree, including all imports. + entry := "api/" + c.Spec.Entry + for name, data := range c.Spec.Sources { + seeds["api/"+name] = data + } + if o.API != "" { + source, _ := filepath.Abs(o.API) + sourceReal, _ := filepath.EvalSymlinks(source) + targetReal := canonical(directory) + rel, e := filepath.Rel(targetReal, sourceReal) + if e == nil && rel != ".." && !strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + entry = filepath.ToSlash(rel) + for name := range c.Spec.Sources { + delete(seeds, "api/"+name) + } + } + } + // Source collisions must never leave a copied contract different from the + // generated code. On updates, use the developer-owned API in the project. + for name, data := range seeds { + if strings.HasSuffix(name, ".api") { + current, e := os.ReadFile(filepath.Join(directory, filepath.FromSlash(name))) + if e == nil && string(current) != string(data) { + return Output{}, fmt.Errorf("%s already differs; regenerate from that project API instead", name) + } + } + } + seeds["README.md"] = []byte(projectREADME(o.Module, entry) + infrastructureREADME(o)) + for _, files := range []map[string][]byte{generated, seeds} { + for name, data := range files { + if strings.HasSuffix(name, ".go") { + data, err = format.Source(data) + if err != nil { + return Output{}, fmt.Errorf("format %s: %w", name, err) + } + files[name] = data + } + } + } + return apply(directory, o.Module, entry, generated, seeds, o.Check, o.profile()) +} + +func moduleFile(o Options) ([]byte, error) { + if err := module.CheckImportPath(o.Module); err != nil { + return nil, err + } + root := o.FrameworkDir + version := o.FrameworkVersion + if root == "" && version == "" { + wd, _ := os.Getwd() + for wd != "" { + data, err := os.ReadFile(filepath.Join(wd, "go.mod")) + if err == nil { + mf, e := modfile.Parse("go.mod", data, nil) + if e == nil && mf.Module != nil && mf.Module.Mod.Path == frameworkModule { + root = wd + } + break + } + parent := filepath.Dir(wd) + if parent == wd { + break + } + wd = parent + } + } + if version == "" { + if info, ok := debug.ReadBuildInfo(); ok && info.Main.Path == frameworkModule && info.Main.Sum != "" { + version = info.Main.Version + } else { + version = minimumVersion + } + } + if !semver.IsValid(version) || semver.Major(version) != "v0" && semver.Major(version) != "v1" { + return nil, errors.New("invalid framework version") + } + if root == "" && semver.Compare(version, minimumVersion) < 0 { + return nil, fmt.Errorf("API generation needs %s or newer; use --framework-dir for unpublished source", minimumVersion) + } + text := "module " + o.Module + "\n\ngo 1.22.0\n\nrequire " + frameworkModule + " " + version + "\n" + if root != "" { + root = canonical(root) + data, err := os.ReadFile(filepath.Join(root, "go.mod")) + if err != nil { + return nil, err + } + mf, err := modfile.Parse("go.mod", data, nil) + if err != nil || mf.Module == nil || mf.Module.Mod.Path != frameworkModule { + return nil, errors.New("--framework-dir must contain go-omni") + } + if _, err = os.Stat(filepath.Join(root, "contract", "contract.go")); err != nil { + return nil, errors.New("local framework lacks API contract support") + } + relative, err := filepath.Rel(canonical(o.Dir), root) + if err != nil { + return nil, err + } + text += "\nreplace " + frameworkModule + " => " + strconv.Quote(filepath.ToSlash(relative)) + "\n" + } + return []byte(text), nil +} +func canonical(path string) string { + path, _ = filepath.Abs(path) + candidate := path + var tail []string + for { + if resolved, err := filepath.EvalSymlinks(candidate); err == nil { + for i := len(tail) - 1; i >= 0; i-- { + resolved = filepath.Join(resolved, tail[i]) + } + return resolved + } + parent := filepath.Dir(candidate) + if parent == candidate { + return path + } + tail = append(tail, filepath.Base(candidate)) + candidate = parent + } +} + +func renderProject(o Options, c *Compiled, starter bool) (map[string][]byte, map[string][]byte, error) { + generated, seeds := map[string][]byte{}, map[string][]byte{} + put := func(files map[string][]byte, name, content string) { files[name] = []byte(content) } + types := "package types\n\n" + for _, t := range c.Spec.Types { + types += "type " + t.Name + " struct {\n" + for _, f := range t.Fields { + types += fmt.Sprintf("%s %s `%s`\n", f.Name, f.Type, f.Tag) + } + types += "}\n" + } + put(generated, "internal/types/typesGen.go", generatedHeader+types) + put(generated, "docs/openapi.json", string(c.Document)+"\n") + put(generated, "internal/handler/openapiGen.go", generatedHeader+fmt.Sprintf("package handler\nimport (\"net/http\"; %q)\nconst OpenAPIDocument = %q\nfunc OpenAPI(c *httpx.Context) error { return c.WriteResponse(200,http.Header{\"Content-Type\": {\"application/json; charset=utf-8\"}},[]byte(OpenAPIDocument)) }\n", frameworkModule+"/httpx", string(c.Document)+"\n")) + imports := fmt.Sprintf("import (\"fmt\"; %q; %q; %q;", frameworkModule+"/httpx", frameworkModule+"/openapi", o.Module+"/internal/servicecontext") + routes := "func RegisterRoutes(router openapi.Registrar, deps *servicecontext.Context) error {\n if router==nil || deps==nil {return fmt.Errorf(\"router and dependencies required\")}\n" + middlewareSet := map[string]bool{} + for _, r := range c.Spec.Routes { + for _, m := range r.Middleware { + middlewareSet[m] = true + } + } + var middlewareNames []string + for m := range middlewareSet { + middlewareNames = append(middlewareNames, m) + } + sort.Strings(middlewareNames) + for _, m := range middlewareNames { + routes += fmt.Sprintf("if deps.Middlewares[%q]==nil {return fmt.Errorf(%q)}\n", m, "missing middleware "+m) + } + for i, group := range c.Groups { + imports += fmt.Sprintf("h%d %q; s%d %q;", i, o.Module+"/internal/handler/"+group, i, o.Module+"/internal/service/"+group) + routes += fmt.Sprintf("h%d := h%d.New(s%d.New(deps))\n", i, i, i) + groupRoutes := []Route{} + for _, r := range c.Spec.Routes { + if r.Group == group { + groupRoutes = append(groupRoutes, r) + } + } + packageName := filepath.Base(group) + text := "package " + packageName + "\nimport(\"context\"; \"fmt\";" + needsTypes, needsBinder := false, false + for _, r := range groupRoutes { + needsTypes = needsTypes || r.Request != "" || r.Response != "" + needsBinder = needsBinder || r.Request != "" + } + text += fmt.Sprintf("%q;", frameworkModule+"/httpx") + if needsTypes { + text += fmt.Sprintf("%q;", o.Module+"/internal/types") + } + if needsBinder { + text += fmt.Sprintf("%q;", frameworkModule+"/contract") + } + text += ")\n" + text += "type Service interface {\n" + for _, r := range groupRoutes { + text += signature(r) + "\n" + } + text += "}\ntype Handler struct {service Service}\nfunc New(s Service)*Handler {return &Handler{service:s}}\n" + for _, r := range groupRoutes { + name := upper(r.Handler) + if r.Request != "" { + text += fmt.Sprintf("var %sRequest = contract.MustNew[types.%s]()\n", lower(r.Handler), r.Request) + } + text += fmt.Sprintf("func(h *Handler) %s(c *httpx.Context)error {\n", name) + call := "c.Context()" + if r.Request != "" { + text += fmt.Sprintf("var req types.%s\nif err:=%sRequest.Bind(c.Request(),httpx.RouteParams(c.Context()),&req);err!=nil{return err}\n", r.Request, lower(r.Handler)) + call += ", &req" + } + if r.Response != "" { + text += fmt.Sprintf("resp,err:=h.service.%s(%s);if err!=nil{return err};if resp==nil{return fmt.Errorf(\"%s returned a nil response\")}\n", name, call, name) + method := map[int]string{200: "OK", 201: "Created", 202: "Accepted"}[r.Status] + text += "return c." + method + "(resp)\n}\n" + } else { + text += fmt.Sprintf("if err:=h.service.%s(%s);err!=nil{return err};return c.NoContent()\n}\n", name, call) + } + routes += fmt.Sprintf("router.Handle(%q,%q,h%d.%s", strings.ToUpper(r.Method), r.Path, i, name) + for _, m := range r.Middleware { + routes += fmt.Sprintf(",deps.Middlewares[%q]", m) + } + routes += ")\n" + service := fmt.Sprintf("package %s\nimport(\"context\"; apierrors %q;", packageName, frameworkModule+"/errors") + if r.Request != "" || r.Response != "" { + service += fmt.Sprintf("%q;", o.Module+"/internal/types") + } + service += ")\n" + service += "// " + name + " implements the business operation. This file is never overwritten.\nfunc(s *Service) " + signature(r) + " {\n" + if starter && r.Handler == "greet" { + service = strings.Replace(service, "apierrors "+strconv.Quote(frameworkModule+"/errors")+";", "\"fmt\";", 1) + service += "s.deps.Logger.InfoContext(ctx,\"greet.called\")\nreturn &types.GreetResp{Message:fmt.Sprintf(\"Hello %s\",req.Name)},nil\n}" + } else { + service += "// TODO: implement business behavior and use s.deps for dependencies.\n" + if r.Response != "" { + service += "return nil, " + } else { + service += "return " + } + service += "apierrors.New(501,\"NOT_IMPLEMENTED\",\"operation is not implemented\")\n}" + } + put(seeds, "internal/service/"+group+"/"+lower(r.Handler)+".go", service) + } + if !hasResponse(groupRoutes) { + text = strings.Replace(text, "\"fmt\";", "", 1) + } + put(generated, "internal/handler/"+group+"/handlersGen.go", generatedHeader+text) + put(seeds, "internal/service/"+group+"/service.go", fmt.Sprintf("package %s\nimport %q\ntype Service struct {deps *servicecontext.Context}\nfunc New(deps *servicecontext.Context)*Service{return &Service{deps:deps}}\n", packageName, o.Module+"/internal/servicecontext")) + put(seeds, "internal/repository/"+group+"/repository.go", "// Package "+packageName+" owns persistence contracts for this business group.\n// Define repository interfaces and implementations here; HTTP DTOs do not\n// determine a database schema. This file is never overwritten.\npackage "+packageName+"\n") + } + // Avoid unused imports when no route uses a named middleware. + imports = strings.Replace(imports, strconv.Quote(frameworkModule+"/httpx")+";", "", 1) + put(generated, "internal/handler/routesGen.go", generatedHeader+"package handler\n"+imports+")\n"+routes+"return nil\n}\n") + registry := "package middleware\nimport " + strconv.Quote(frameworkModule+"/httpx") + "\nfunc Defaults()map[string]httpx.Middleware{return map[string]httpx.Middleware{\n" + for _, m := range middlewareNames { + registry += fmt.Sprintf("%q:%s,\n", m, upper(m)) + put(seeds, "internal/middleware/"+lower(m)+".go", fmt.Sprintf("package middleware\nimport(%q; apierrors %q)\n// %s is a developer-owned hook. Implement it before enabling this route.\nfunc %s(next httpx.Handler)httpx.Handler{return func(c *httpx.Context)error {return apierrors.New(501,\"MIDDLEWARE_NOT_IMPLEMENTED\",\"middleware is not implemented\")}}\n", frameworkModule+"/httpx", frameworkModule+"/errors", upper(m), upper(m))) + } + put(generated, "internal/middleware/registryGen.go", generatedHeader+registry+"}}\n") + for name, source := range baseTemplates { + content := strings.ReplaceAll(source, "MODULE", o.Module) + content = strings.ReplaceAll(content, "LOGFORMAT", o.LogFormat) + content = strings.ReplaceAll(content, "LOGOUTPUT", o.LogOutput) + if name == "etc/config.yaml" && o.LogOutput == "file" { + content = strings.Replace(content, "output: file", "output: stderr\n file: ./var/log/server.log\n rotation:\n enabled: true\n maxSizeMb: 100\n maxBackups: 10\n maxAgeDays: 7\n compress: true", 1) + } + put(seeds, name, content) + } + + infra, err := scaffold.APIInfrastructure(o.Module, o.Database, o.Storage, o.LogOutput, o.LogFormat) + if err != nil { + return nil, nil, err + } + for name, data := range infra { + seeds[name] = data + } + if o.Storage == "local" { + for _, r := range c.Spec.Routes { + if r.Path == "/storage/object" { + return nil, nil, errors.New("/storage/object is reserved by local storage") + } + } + main := string(seeds["cmd/server/main.go"]) + main = strings.Replace(main, " if err=handler.RegisterRoutes", ` for _,method:=range []string{"GET","HEAD","PUT"}{app.Handle(method,"/storage/object",func(c *httpx.Context)error{deps.FileHTTP.ServeHTTP(c.Raw(),c.Request());return nil})} + if err=handler.RegisterRoutes`, 1) + main = strings.Replace(main, `"git.yuntuo.hk/k2safe/go-omni/logx"`, `"git.yuntuo.hk/k2safe/go-omni/logx"; "git.yuntuo.hk/k2safe/go-omni/httpx"`, 1) + seeds["cmd/server/main.go"] = []byte(main) + } + return generated, seeds, nil +} +func hasResponse(routes []Route) bool { + for _, r := range routes { + if r.Response != "" { + return true + } + } + return false +} +func signature(r Route) string { + s := upper(r.Handler) + "(ctx context.Context" + if r.Request != "" { + s += ",req *types." + r.Request + } + s += ") " + if r.Response != "" { + s += "(*types." + r.Response + ",error)" + } else { + s += "error" + } + return s +} diff --git a/cmd/go-omni/apigen/parse.go b/cmd/go-omni/apigen/parse.go new file mode 100644 index 0000000..1a62dae --- /dev/null +++ b/cmd/go-omni/apigen/parse.go @@ -0,0 +1,460 @@ +// Package apigen implements go-omni's deliberately bounded go-zero-style API +// dialect. It generates Omni handlers, never go-zero or transport dependencies. +package apigen + +import ( + "errors" + "fmt" + "io" + "os" + "path/filepath" + "strconv" + "strings" + "text/scanner" +) + +type Spec struct { + Title, Version, Service string + Types []Type + Routes []Route + Sources map[string][]byte // relative to the entry API's directory + Entry string +} +type Type struct { + Name string + Fields []Field + Position string +} +type Field struct{ Name, Type, Tag string } +type Route struct { + Method, Path, Request, Response, Handler, Group, Summary string + Middleware []string + Status int + Position string +} +type token struct { + text, position string + kind rune +} +type parser struct { + tokens []token + at int + spec *Spec + imports []string +} + +func Parse(path string) (*Spec, error) { + entry, err := filepath.Abs(path) + if err != nil { + return nil, err + } + entry, err = filepath.EvalSymlinks(entry) + if err != nil { + return nil, err + } + root := filepath.Dir(entry) + spec := &Spec{Version: "1.0.0", Sources: map[string][]byte{}, Entry: filepath.Base(entry)} + visiting, visited := map[string]bool{}, map[string]bool{} + total := 0 + var visit func(string, int) error + visit = func(name string, depth int) error { + if depth > 32 || len(visited)+len(visiting) >= 64 { + return errors.New("api: import limit exceeded") + } + real, err := filepath.EvalSymlinks(name) + if err != nil { + return err + } + rel, err := filepath.Rel(root, real) + if err != nil || rel == ".." || strings.HasPrefix(rel, ".."+string(filepath.Separator)) { + return errors.New("api: imports must stay within the entry API directory") + } + if filepath.Ext(real) != ".api" { + return errors.New("api: expected .api file") + } + if visiting[real] { + return fmt.Errorf("api: circular import %s", rel) + } + if visited[real] { + return nil + } + visiting[real] = true + defer delete(visiting, real) + f, err := os.Open(real) + if err != nil { + return err + } + data, err := io.ReadAll(io.LimitReader(f, (1<<20)+1)) + _ = f.Close() + if err != nil { + return err + } + total += len(data) + if len(data) > 1<<20 || total > 8<<20 { + return errors.New("api: source size limit exceeded") + } + tokens, err := lex(rel, string(data)) + if err != nil { + return err + } + p := &parser{tokens: tokens, spec: spec} + if err = p.parse(); err != nil { + return err + } + spec.Sources[filepath.ToSlash(rel)] = data + for _, imp := range p.imports { + if filepath.IsAbs(imp) || strings.Contains(imp, "\\") { + return errors.New("api: imports must use relative slash paths") + } + if err = visit(filepath.Join(filepath.Dir(real), filepath.FromSlash(imp)), depth+1); err != nil { + return err + } + } + visited[real] = true + return nil + } + if err = visit(entry, 0); err != nil { + return nil, err + } + if spec.Service == "" || len(spec.Routes) == 0 { + return nil, errors.New("api: at least one service route required") + } + if spec.Title == "" { + spec.Title = spec.Service + } + return spec, nil +} +func lex(name, source string) ([]token, error) { + var s scanner.Scanner + s.Init(strings.NewReader(source)) + s.Filename = name + s.Mode = scanner.ScanIdents | scanner.ScanInts | scanner.ScanStrings | scanner.ScanRawStrings | scanner.ScanComments | scanner.SkipComments + s.Whitespace ^= 1 << '\n' + var scanErr error + s.Error = func(s *scanner.Scanner, message string) { + if scanErr == nil { + scanErr = fmt.Errorf("%s: %s", s.Position, message) + } + } + var out []token + for { + kind := s.Scan() + out = append(out, token{s.TokenText(), s.Position.String(), kind}) + if kind == scanner.EOF { + break + } + } + return out, scanErr +} +func (p *parser) peek() string { return p.tokens[p.at].text } +func (p *parser) next() token { + t := p.tokens[p.at] + if t.kind != scanner.EOF { + p.at++ + } + return t +} +func (p *parser) skip() { + for p.peek() == "\n" || p.peek() == ";" { + p.next() + } +} +func (p *parser) err(message string) error { + return fmt.Errorf("%s: %s (got %q)", p.tokens[p.at].position, message, p.peek()) +} +func (p *parser) expect(s string) error { + if p.peek() != s { + return p.err("expected " + strconv.Quote(s)) + } + p.next() + return nil +} +func (p *parser) ident() (string, error) { + if p.tokens[p.at].kind != scanner.Ident { + return "", p.err("expected identifier") + } + return p.next().text, nil +} +func (p *parser) quoted() (string, error) { + if p.tokens[p.at].kind != scanner.String { + return "", p.err("expected quoted string") + } + return strconv.Unquote(p.next().text) +} +func (p *parser) parse() error { + server := map[string]string{} + syntaxSeen := false + for p.skip(); p.tokens[p.at].kind != scanner.EOF; p.skip() { + switch p.peek() { + case "syntax": + if syntaxSeen { + return p.err("duplicate syntax") + } + syntaxSeen = true + p.next() + if err := p.expect("="); err != nil { + return err + } + v, err := p.quoted() + if err != nil { + return err + } + if v != "v1" { + return p.err("only syntax v1 supported") + } + case "info": + p.next() + m, err := p.metadata() + if err != nil { + return err + } + for k, v := range m { + switch k { + case "title": + p.spec.Title = v + case "version": + p.spec.Version = v + case "desc", "description", "author", "email": + default: + return p.err("unsupported info property " + k) + } + } + case "import": + p.next() + if p.peek() == "(" { + p.next() + for p.skip(); p.peek() != ")"; p.skip() { + v, err := p.quoted() + if err != nil { + return err + } + p.imports = append(p.imports, v) + } + p.next() + } else { + v, err := p.quoted() + if err != nil { + return err + } + p.imports = append(p.imports, v) + } + case "type": + p.next() + if p.peek() == "(" { + p.next() + for p.skip(); p.peek() != ")"; p.skip() { + if err := p.parseType(); err != nil { + return err + } + } + p.next() + } else { + if err := p.parseType(); err != nil { + return err + } + } + case "@": + p.next() + if err := p.expect("server"); err != nil { + return err + } + if len(server) != 0 { + return p.err("duplicate server block") + } + m, err := p.metadata() + if err != nil { + return err + } + server = m + for k := range m { + if k != "prefix" && k != "group" && k != "middleware" { + return p.err("unsupported @server option " + k + "; wire authentication through named middleware") + } + } + case "service": + if err := p.parseService(server); err != nil { + return err + } + server = map[string]string{} + default: + return p.err("unsupported declaration") + } + } + if len(server) > 0 { + return p.err("@server must precede a service") + } + return nil +} +func (p *parser) metadata() (map[string]string, error) { + if err := p.expect("("); err != nil { + return nil, err + } + out := map[string]string{} + for p.skip(); p.peek() != ")"; p.skip() { + key, err := p.ident() + if err != nil { + return nil, err + } + if err = p.expect(":"); err != nil { + return nil, err + } + value := "" + if p.tokens[p.at].kind == scanner.String { + value, err = p.quoted() + if err != nil { + return nil, err + } + } else { + for p.peek() != "\n" && p.peek() != ";" && p.peek() != ")" && p.tokens[p.at].kind != scanner.EOF { + value += p.next().text + } + } + if value == "" { + return nil, p.err("metadata value required") + } + if _, exists := out[key]; exists { + return nil, p.err("duplicate metadata " + key) + } + out[key] = value + } + p.next() + return out, nil +} +func (p *parser) parseType() error { + pos := p.tokens[p.at].position + name, err := p.ident() + if err != nil { + return err + } + p.skip() + if err = p.expect("{"); err != nil { + return err + } + typ := Type{Name: name, Position: pos} + for p.skip(); p.peek() != "}"; p.skip() { + name, err := p.ident() + if err != nil { + return err + } + expression := "" + for p.peek() != "\n" && p.peek() != ";" && p.peek() != "}" && p.tokens[p.at].kind != scanner.RawString && p.tokens[p.at].kind != scanner.EOF { + expression += p.next().text + } + if expression == "" { + return p.err("field type required; embedded types are not supported") + } + if p.tokens[p.at].kind != scanner.RawString { + return p.err("field source tag required") + } + tag, err := strconv.Unquote(p.next().text) + if err != nil { + return err + } + typ.Fields = append(typ.Fields, Field{Name: name, Type: expression, Tag: tag}) + } + p.next() + p.spec.Types = append(p.spec.Types, typ) + return nil +} +func (p *parser) parseService(server map[string]string) error { + p.next() + name, err := p.ident() + if err != nil { + return err + } + for p.peek() == "-" { + p.next() + part, e := p.ident() + if e != nil { + return e + } + name += "-" + part + } + if p.spec.Service != "" && p.spec.Service != name { + return p.err("one service name per entry API is supported") + } + p.spec.Service = name + p.skip() + if err = p.expect("{"); err != nil { + return err + } + for p.skip(); p.peek() != "}"; p.skip() { + route := Route{Group: server["group"], Status: 200, Position: p.tokens[p.at].position} + if route.Group == "" { + route.Group = "api" + } + if server["middleware"] != "" { + route.Middleware = strings.Split(server["middleware"], ",") + } + seen := map[string]bool{} + for p.peek() == "@" { + p.next() + annotation, e := p.ident() + if e != nil { + return e + } + if seen[annotation] { + return p.err("duplicate route annotation") + } + seen[annotation] = true + switch annotation { + case "handler": + route.Handler, e = p.ident() + case "doc": + route.Summary, e = p.quoted() + case "status": + route.Status, e = strconv.Atoi(p.next().text) + default: + return p.err("unsupported route annotation " + annotation) + } + if e != nil { + return e + } + p.skip() + } + if route.Handler == "" { + return p.err("@handler required") + } + route.Method, err = p.ident() + if err != nil { + return err + } + path := "" + for p.peek() != "\n" && p.peek() != ";" && p.peek() != "(" && p.peek() != "returns" && p.peek() != "}" && p.tokens[p.at].kind != scanner.EOF { + path += p.next().text + } + route.Path = server["prefix"] + path + if p.peek() == "(" { + route.Request, err = p.routeType() + if err != nil { + return err + } + } + if p.peek() == "returns" { + p.next() + route.Response, err = p.routeType() + if err != nil { + return err + } + } + if p.peek() != "\n" && p.peek() != ";" && p.peek() != "}" { + return p.err("unexpected route suffix") + } + if route.Response == "" && !seen["status"] { + route.Status = 204 + } + p.spec.Routes = append(p.spec.Routes, route) + } + p.next() + return nil +} +func (p *parser) routeType() (string, error) { + if err := p.expect("("); err != nil { + return "", err + } + name, err := p.ident() + if err != nil { + return "", err + } + return name, p.expect(")") +} diff --git a/cmd/go-omni/apigen/profile.go b/cmd/go-omni/apigen/profile.go new file mode 100644 index 0000000..614bbdd --- /dev/null +++ b/cmd/go-omni/apigen/profile.go @@ -0,0 +1,73 @@ +package apigen + +import ( + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" +) + +type projectProfile struct { + Database string `json:"database"` + Storage string `json:"storage"` + LogOutput string `json:"logOutput"` + LogFormat string `json:"logFormat"` +} + +func (o Options) profile() projectProfile { + return projectProfile{o.Database, o.Storage, o.LogOutput, o.LogFormat} +} + +// Infrastructure files belong to the developer. Persist the original selection +// so a plain regeneration cannot silently change adapters or repair a missing +// file with settings from another profile. +func resolveProfile(o *Options) error { + defaults := projectProfile{"none", "none", "stderr", "json"} + data, err := os.ReadFile(filepath.Join(o.Dir, manifestPath)) + if err != nil && !errors.Is(err, os.ErrNotExist) { + return err + } + existing := err == nil + if existing { + var previous manifest + if err = json.Unmarshal(data, &previous); err != nil { + return err + } + if previous.Profile != nil { + defaults = *previous.Profile + } + } + if o.Database == "memory" { + o.Database = "none" + } + for _, item := range []struct { + name string + value *string + fallback string + }{ + {"database", &o.Database, defaults.Database}, {"storage", &o.Storage, defaults.Storage}, + {"log-output", &o.LogOutput, defaults.LogOutput}, {"log-format", &o.LogFormat, defaults.LogFormat}, + } { + if *item.value == "" { + *item.value = item.fallback + } else if existing && *item.value != item.fallback { + return fmt.Errorf("--%s differs from the original project profile; infrastructure is developer-owned, generate a new project to compare and migrate it explicitly", item.name) + } + } + if o.Database != "none" && o.Database != "mysql" { + return errors.New("--database must be none, memory or mysql") + } + switch o.Storage { + case "none", "local", "s3", "r2", "oss": + default: + return errors.New("--storage must be none, local, s3, r2 or oss") + } + if o.LogOutput != "stderr" && o.LogOutput != "stdout" && o.LogOutput != "file" { + return errors.New("--log-output must be stdout, stderr or file") + } + if o.LogFormat != "json" && o.LogFormat != "text" { + return errors.New("--log-format must be json or text") + } + return nil +} diff --git a/cmd/go-omni/apigen/profile_test.go b/cmd/go-omni/apigen/profile_test.go new file mode 100644 index 0000000..b096dcb --- /dev/null +++ b/cmd/go-omni/apigen/profile_test.go @@ -0,0 +1,98 @@ +package apigen + +import ( + "context" + "os" + "os/exec" + "path/filepath" + "strings" + "testing" + "time" +) + +func TestInfrastructureProfileOwnership(t *testing.T) { + dir := filepath.Join(t.TempDir(), "project") + o := Options{Dir: dir, Module: "example.com/profile", FrameworkDir: frameworkRoot(t), Database: "mysql", Storage: "local", LogOutput: "file"} + if _, err := New(o); err != nil { + t.Fatal(err) + } + o = Options{Dir: dir, API: filepath.Join(dir, "api/service.api"), Check: true} + if _, err := Generate(o); err != nil { + t.Fatal(err) + } + before := snapshot(t, dir) + for _, changed := range []Options{ + {Database: "none"}, {Storage: "s3"}, {LogOutput: "stdout"}, {LogFormat: "text"}, + } { + changed.Dir, changed.API = dir, o.API + if _, err := Generate(changed); err == nil { + t.Fatal("profile switch accepted") + } + if !sameFiles(before, snapshot(t, dir)) { + t.Fatal("profile rejection changed project") + } + } + // Restoring a missing seed must use the saved profile, not current defaults. + if err := os.Remove(filepath.Join(dir, "internal/servicecontext/storage.go")); err != nil { + t.Fatal(err) + } + if _, err := Generate(o); err == nil { + t.Fatal("check accepted missing infrastructure") + } + o.Check = false + if _, err := Generate(o); err != nil { + t.Fatal(err) + } + if !sameFiles(before, snapshot(t, dir)) { + t.Fatal("profile reconstruction changed project") + } +} + +func TestAPIInfrastructureConsumers(t *testing.T) { + if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") == "" { + t.Skip("consumer module integration") + } + for _, db := range []string{"none", "mysql"} { + for _, storage := range []string{"none", "local", "s3", "r2", "oss"} { + t.Run(db+"/"+storage, func(t *testing.T) { + dir := filepath.Join(t.TempDir(), "project") + if _, err := New(Options{Dir: dir, Module: "example.com/profile", FrameworkDir: frameworkRoot(t), Database: db, Storage: storage, LogOutput: "file"}); err != nil { + t.Fatal(err) + } + if _, err := Generate(Options{Dir: dir, API: filepath.Join(dir, "api/service.api"), Check: true}); err != nil { + t.Fatal(err) + } + test := `package settings +import("testing";"os") +func TestConfiguration(t *testing.T){ + t.Setenv("GO_OMNI_DB_DSN","test:test@tcp(127.0.0.1:1)/test") + t.Setenv("GO_OMNI_STORAGE_SIGNING_KEY","01234567890123456789012345678901") + t.Setenv("GO_OMNI_STORAGE_ACCESS_KEY","test");t.Setenv("GO_OMNI_STORAGE_SECRET_KEY","test") + t.Setenv("GO_OMNI_STORAGE_BUCKET","test");t.Setenv("GO_OMNI_STORAGE_ENDPOINT","ENDPOINT") + t.Setenv("GO_OMNI_STORAGE_ACCOUNT_ID","01234567890123456789012345678901") + cfg,err:=Load("../../etc/config.yaml");if err!=nil{t.Fatal(err)} + if cfg.Server.Log.File==""{t.Fatal("file logger absent")} + t.Setenv("GO_OMNI_LOG_OUTPUT","stdout");cfg,err=Load("");if err!=nil||cfg.Server.Log.File!=""{t.Fatalf("override: %v",err)} + _=os.Getenv("unused") +}` + endpoint := "" + if storage == "s3" { + endpoint = "127.0.0.1:9000" + } + if storage == "oss" { + endpoint = "https://oss-cn-hangzhou.aliyuncs.com" + } + writeFile(t, filepath.Join(dir, "internal/settings/profile_test.go"), strings.ReplaceAll(test, `"ENDPOINT"`, `"`+endpoint+`"`)) + ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second) + defer cancel() + cmd := exec.CommandContext(ctx, "go", "test", "-mod=mod", "-race", "-count=1", "./...") + cmd.Dir = dir + cmd.Env = append(os.Environ(), "GOWORK=off", "GOFLAGS=") + out, err := cmd.CombinedOutput() + if err != nil { + t.Fatalf("%v\n%s", err, out) + } + }) + } + } +} diff --git a/cmd/go-omni/apigen/templates.go b/cmd/go-omni/apigen/templates.go new file mode 100644 index 0000000..fb04010 --- /dev/null +++ b/cmd/go-omni/apigen/templates.go @@ -0,0 +1,160 @@ +package apigen + +import "fmt" + +var baseTemplates = map[string]string{ + "cmd/server/main.go": `package main +import ( + "context" + "errors" + "flag" + "log" + "os" + "os/signal" + "syscall" + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/logx" + "MODULE/internal/handler" + "MODULE/internal/servicecontext" + "MODULE/internal/settings" +) +func main(){ + path:=flag.String("config","etc/config.yaml","configuration file; empty uses defaults and environment") + flag.Parse() + if err:=run(*path);err!=nil{log.Fatal(err)} +} +func run(path string)(runErr error){ + ctx,stop:=signal.NotifyContext(context.Background(),os.Interrupt,syscall.SIGTERM) + defer stop() + cfg,err:=settings.Load(path);if err!=nil{return err} + logger,err:=logx.New(cfg.Server.Log);if err!=nil{return err} + defer func(){runErr=errors.Join(runErr,logger.Close())}() + logx.SetDefault(logger) + deps,err:=servicecontext.New(ctx,logger,cfg);if err!=nil{return err} + defer func(){runErr=errors.Join(runErr,deps.Close())}() + app:=omni.New(omni.WithConfig(cfg.Server)) + app.OnShutdown(logger.Close) + app.OnShutdown(deps.Close) + if err=handler.RegisterRoutes(app,deps);err!=nil{return err} + app.GET("/openapi.json",handler.OpenAPI) + app.GET("/health/live",deps.Health.LivenessHandler()) + app.GET("/health/ready",deps.Health.ReadinessHandler()) + logger.InfoContext(ctx,"service.starting","address",cfg.Server.Address) + return app.RunContext(ctx,cfg.Server.Address) +} +`, + ".gitignore": "/server\n/var/\n.env\n.env.*\n!.env.example\n/.goOmni/api.lock\n", +} + +func projectREADME(module, entry string) string { + return fmt.Sprintf(`# %s + +This project is generated from an editable API contract. Business code uses +standard context.Context; generated HTTP handlers use Omni's transport-neutral httpx. + +## Run + +`+"```sh"+` +go mod tidy +go run ./cmd/server +`+"```"+` + +GET /health/live and /health/ready check the running service. GET /openapi.json +serves the same document as docs/openapi.json. Bind address defaults to localhost. +Set GO_OMNI_TRANSPORT=native to select the native transport without changing services. +The greeting starter works immediately; other new operation stubs return 501 until +their business logic is implemented. Authentication hooks also fail closed with 501. + +## Develop + +1. Edit %s and its relative imports. +2. Run the same go-omni CLI version as the framework dependency: + +`+"```sh"+` +go-omni api validate --api %s +go-omni api generate --api %s --dir . +go-omni api generate --api %s --dir . --check +go test ./... +`+"```"+` + +The --check command is read-only and suitable for CI. It fails on stale or missing +generated files and missing business scaffolds. go.mod, configuration, main, +Service, Repository, servicecontext, middleware hook bodies and API files are +created once and then preserved byte for byte. Gen.go files and OpenAPI are owned +by the generator; manual edits cause a conflict instead of being overwritten. +Commit .goOmni/apiManifest.json alongside the generated files. Do not edit it. +Avoid editing files while generation is running; api.lock serializes generators. + +Routes and types that disappear are removed from generated output. Handwritten +business files remain: remove/adapt obsolete implementations explicitly if an +API change makes their old types or signatures fail to compile. Every regeneration +must be followed by go test ./... . Do not delete service files to change contracts. + +## Layout + +`+"```text"+` +api/ # editable contract and imported types +cmd/server/main.go # startup, lifecycle and route registration +etc/config.yaml # lowerCamelCase configuration +internal/settings/ # defaults -> file -> environment +internal/servicecontext/ # dependency initialization and cleanup +internal/types/typesGen.go # generated request/response types +internal/handler// # generated binding and service adapters +internal/service// # developer-owned business operations +internal/repository// # developer-owned persistence contracts +internal/middleware/ # named hooks and generated registry +docs/openapi.json # generated OpenAPI 3.0.3 +.goOmni/apiManifest.json # ownership and content hashes +`+"```"+` + +The API response describes data only. Omni adds code, message and requestId. +Do not define the envelope again in each response. Handlers reject nil successful +responses. Body DTOs use application/json and have a 1 MiB limit by default. +Use contract.Binder.Bind's optional limit in a custom adapter for larger payloads. + +Fields require exactly one source tag: json, path, form (URL query only), header. +All are required by presence unless optional, omitempty or default is specified. +Zero and false are valid present values; binding:"required" rejects scalar zero. +Use *T with optional when business logic needs to distinguish missing from zero. +Pointers may carry null; non-pointer null is rejected. Unknown/duplicate JSON +fields and duplicate scalar query/header values are rejected. Arrays in the query +use repeated keys. Enum/range/default and nested fields share the documentation +and binding compiler. Responses are expected to honor their declared constraints; +they are not validated on every response. Do not return nil slices/maps for a +non-nullable collection (initialize empty collections instead). + +Named middleware is composed in declaration order (first is outermost). Implement +the generated hooks or replace entries in servicecontext; never bypass an auth +hook merely to make the server start. Middleware names do not imply a JWT policy. + +This CLI supports a documented go-zero-style subset, not all goctl features. +Database tables and repositories cannot be inferred from an HTTP contract. Add +business repositories through servicecontext. Database and storage options are +composed at creation; the original profile is retained on regeneration. +`, module, entry, entry, entry, entry) +} + +func infrastructureREADME(o Options) string { + return fmt.Sprintf(` +## Infrastructure + +Selected database: %s; storage: %s; logging: %s/%s. +MySQL uses database/sql through deps.DB, including InTx and a readiness probe. +Set GO_OMNI_DB_DSN; pool settings live in etc/config.yaml and GO_OMNI_DB_*. +No tables are invented or migrated automatically from request/response DTOs. +The none/memory selection opens no database; implement repositories per business. + +With storage enabled, services receive deps.Files, Uploads, Downloads and +UploadLinks capability interfaces. Local signed GET/HEAD/PUT uses /storage/object; +set GO_OMNI_STORAGE_SIGNING_KEY (32+ bytes) and GO_OMNI_STORAGE_PUBLIC_URL. +Cloud credentials use GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY; set bucket and endpoint +or R2 accountId as appropriate. Signed URLs grant temporary access: authenticate +and authorize before issuing them. No public upload-link route is added. +Storage failures during startup close an already-open database; Close is idempotent. + +Database/storage/logging selection is recorded in .goOmni/apiManifest.json. +Changing selection through generation flags is rejected on an existing project; +edit your owned configuration and initialization explicitly, or generate a fresh +project for comparison. New routes still regenerate without repeating these flags. +`, o.Database, o.Storage, o.LogOutput, o.LogFormat) +} diff --git a/cmd/go-omni/apigen/write.go b/cmd/go-omni/apigen/write.go new file mode 100644 index 0000000..fc82688 --- /dev/null +++ b/cmd/go-omni/apigen/write.go @@ -0,0 +1,295 @@ +package apigen + +import ( + "bytes" + "crypto/sha256" + "encoding/hex" + "encoding/json" + "errors" + "fmt" + "os" + "path/filepath" + "sort" + "strings" +) + +const manifestPath = ".goOmni/apiManifest.json" + +type manifest struct { + Profile *projectProfile `json:"profile,omitempty"` + Version int `json:"version"` + Module string `json:"module"` + Entry string `json:"entry"` + Generated map[string]string `json:"generated"` +} +type change struct { + name string + before, after []byte + existed, remove bool + mode os.FileMode +} + +func apply(root, module, entry string, generated, seeds map[string][]byte, check bool, profile projectProfile) (Output, error) { + var result Output + if info, err := os.Lstat(root); err == nil { + if info.Mode()&os.ModeSymlink != 0 || !info.IsDir() { + return result, errors.New("api: output root must be a real directory") + } + } else if !errors.Is(err, os.ErrNotExist) { + return result, err + } + root = canonical(root) + names := map[string]bool{} + caseNames := map[string]string{} + for _, files := range []map[string][]byte{generated, seeds} { + for name := range files { + if names[name] { + return result, fmt.Errorf("duplicate output %s", name) + } + names[name] = true + folded := strings.ToLower(name) + if old, ok := caseNames[folded]; ok { + return result, fmt.Errorf("case-insensitive path collision: %s and %s", old, name) + } + caseNames[folded] = name + if err := safePath(root, name); err != nil { + return result, err + } + } + } + if err := safePath(root, manifestPath); err != nil { + return result, err + } + if !check { + if err := os.MkdirAll(filepath.Join(root, ".goOmni"), 0755); err != nil { + return result, err + } + lock := filepath.Join(root, ".goOmni", "api.lock") + f, err := os.OpenFile(lock, os.O_WRONLY|os.O_CREATE|os.O_EXCL, 0600) + if err != nil { + return result, fmt.Errorf("api: acquire generation lock (remove stale api.lock only after checking no generator is running): %w", err) + } + _, _ = fmt.Fprintf(f, "pid=%d\n", os.Getpid()) + _ = f.Close() + defer os.Remove(lock) + } + old := manifest{Generated: map[string]string{}} + oldData, err := os.ReadFile(filepath.Join(root, manifestPath)) + if err == nil { + if err = json.Unmarshal(oldData, &old); err != nil { + return result, fmt.Errorf("invalid ownership manifest: %w", err) + } + if old.Profile != nil && *old.Profile != profile { + return result, errors.New("api: profile changed during generation; retry") + } + if old.Version != 1 || old.Module != module || old.Entry != entry { + return result, errors.New("api: manifest version/module/entry differs; one API entry owns this project") + } + } else if !errors.Is(err, os.ErrNotExist) { + return result, err + } + if oldData == nil { + if err = filepath.WalkDir(root, func(path string, d os.DirEntry, walkErr error) error { + if errors.Is(walkErr, os.ErrNotExist) { + return nil + } + if walkErr != nil { + return walkErr + } + if d.IsDir() && d.Name() == ".git" { + return filepath.SkipDir + } + if !d.IsDir() && strings.HasSuffix(d.Name(), ".go") { + return fmt.Errorf("api: %s belongs to an existing Go layout; generate into a new directory", path) + } + return nil + }); err != nil { + return result, err + } + } + next := manifest{Profile: &profile, Version: 1, Module: module, Entry: entry, Generated: map[string]string{}} + for name, data := range generated { + next.Generated[name] = digest(data) + } + var changes []change + // Check ALL previous outputs, even stale ones, before writing any new file. + for name, hash := range old.Generated { + if !ownedPath(name) { + return result, fmt.Errorf("api: invalid owned path %q", name) + } + if err = safePath(root, name); err != nil { + return result, err + } + data, e := os.ReadFile(filepath.Join(root, filepath.FromSlash(name))) + if errors.Is(e, os.ErrNotExist) { + continue + } + if e != nil { + return result, e + } + if digest(data) != hash { + return result, fmt.Errorf("api: generated file %s was edited; move custom code into developer-owned files before regenerating", name) + } + if _, exists := generated[name]; !exists { + info, _ := os.Stat(filepath.Join(root, filepath.FromSlash(name))) + changes = append(changes, change{name: name, before: data, existed: true, remove: true, mode: info.Mode().Perm()}) + result.Removed = append(result.Removed, name) + } + } + paths := make([]string, 0, len(names)) + for name := range names { + paths = append(paths, name) + } + sort.Strings(paths) + for _, name := range paths { + target := filepath.Join(root, filepath.FromSlash(name)) + current, err := os.ReadFile(target) + exists := err == nil + if err != nil && !errors.Is(err, os.ErrNotExist) { + return result, err + } + content, isGenerated := generated[name] + if !isGenerated { + content = seeds[name] + } + if exists && !isGenerated { + result.Preserved = append(result.Preserved, name) + continue + } + if exists && isGenerated { + if _, owned := old.Generated[name]; !owned { + return result, fmt.Errorf("api: refusing to adopt existing unowned file %s", name) + } + } + if isGenerated { + result.Generated = append(result.Generated, name) + } else { + result.Created = append(result.Created, name) + } + if exists && bytes.Equal(current, content) { + continue + } + mode := os.FileMode(0644) + if exists { + info, e := os.Stat(target) + if e != nil { + return result, e + } + mode = info.Mode().Perm() + } + changes = append(changes, change{name: name, before: current, after: content, existed: exists, mode: mode}) + } + data, err := json.MarshalIndent(next, "", " ") + if err != nil { + return result, err + } + data = append(data, '\n') + if !bytes.Equal(oldData, data) { + changes = append(changes, change{name: manifestPath, before: oldData, after: data, existed: oldData != nil, mode: 0644}) + } + sort.Strings(result.Removed) + if check { + if len(changes) > 0 { + var stale []string + for _, c := range changes { + stale = append(stale, c.name) + } + sort.Strings(stale) + return result, fmt.Errorf("api: generated project is stale: %s", strings.Join(stale, ", ")) + } + return result, nil + } + // A rollback restores bytes/modes if an ordinary filesystem write fails. + // File replacement is atomic; the whole directory is not crash-atomic. + var applied []change + for _, c := range changes { + if err = safePath(root, c.name); err == nil { + current, e := os.ReadFile(filepath.Join(root, filepath.FromSlash(c.name))) + if c.existed { + if e != nil || !bytes.Equal(current, c.before) { + err = fmt.Errorf("file changed during generation: %s", c.name) + } + } else if !errors.Is(e, os.ErrNotExist) { + err = fmt.Errorf("file appeared during generation: %s", c.name) + } + } + if err == nil { + if c.remove { + err = os.Remove(filepath.Join(root, filepath.FromSlash(c.name))) + } else { + err = atomicWrite(root, c.name, c.after, c.mode) + } + } + if err != nil { + for i := len(applied) - 1; i >= 0; i-- { + previous := applied[i] + var rollback error + if previous.existed { + rollback = atomicWrite(root, previous.name, previous.before, previous.mode) + } else { + rollback = os.Remove(filepath.Join(root, filepath.FromSlash(previous.name))) + } + err = errors.Join(err, rollback) + } + return result, err + } + applied = append(applied, c) + } + return result, nil +} +func digest(data []byte) string { sum := sha256.Sum256(data); return hex.EncodeToString(sum[:]) } +func ownedPath(name string) bool { + if name == "docs/openapi.json" || name == "internal/types/typesGen.go" || name == "internal/handler/routesGen.go" || name == "internal/handler/openapiGen.go" || name == "internal/middleware/registryGen.go" { + return true + } + return strings.HasPrefix(name, "internal/handler/") && strings.HasSuffix(name, "/handlersGen.go") +} +func safePath(root, name string) error { + if filepath.IsAbs(name) || strings.Contains(name, "\\") || filepath.ToSlash(filepath.Clean(name)) != name || name == "." || strings.HasPrefix(name, "../") { + return fmt.Errorf("api: unsafe output path %q", name) + } + path := root + parts := strings.Split(name, "/") + for i, part := range parts { + path = filepath.Join(path, part) + info, err := os.Lstat(path) + if errors.Is(err, os.ErrNotExist) { + continue + } + if err != nil { + return err + } + if info.Mode()&os.ModeSymlink != 0 { + return fmt.Errorf("api: refusing symlink %s", path) + } + if i < len(parts)-1 && !info.IsDir() { + return fmt.Errorf("api: expected directory %s", path) + } + if i == len(parts)-1 && !info.Mode().IsRegular() { + return fmt.Errorf("api: expected regular file %s", path) + } + } + return nil +} +func atomicWrite(root, name string, data []byte, mode os.FileMode) error { + target := filepath.Join(root, filepath.FromSlash(name)) + if err := os.MkdirAll(filepath.Dir(target), 0755); err != nil { + return err + } + f, err := os.CreateTemp(filepath.Dir(target), ".apiGen-*") + if err != nil { + return err + } + defer os.Remove(f.Name()) + if err = f.Chmod(mode); err == nil { + _, err = f.Write(data) + } + if err == nil { + err = f.Sync() + } + err = errors.Join(err, f.Close()) + if err != nil { + return err + } + return os.Rename(f.Name(), target) +} diff --git a/cmd/go-omni/main.go b/cmd/go-omni/main.go index 6d866b5..fdb2d79 100644 --- a/cmd/go-omni/main.go +++ b/cmd/go-omni/main.go @@ -7,6 +7,7 @@ import ( "path/filepath" "strings" + "git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/apigen" "git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/scaffold" ) @@ -17,6 +18,11 @@ func main() { } switch os.Args[1] { + case "api": + if err := runAPI(os.Args[2:]); err != nil { + fmt.Fprintln(os.Stderr, "go-omni api:", err) + os.Exit(1) + } case "migrate": if err := runMigrate(os.Args[2:]); err != nil { fmt.Fprintln(os.Stderr, "go-omni migrate:", err) @@ -44,9 +50,43 @@ func main() { func runNew(args []string) error { options := scaffold.Options{} var target string + var apiFile string for index := 0; index < len(args); index++ { arg := args[index] switch { + case arg == "--api" || strings.HasPrefix(arg, "--api="): + _, value, inline := strings.Cut(arg, "=") + if !inline { + if index+1 >= len(args) || strings.HasPrefix(args[index+1], "-") { + return errors.New("--api requires a file") + } + index++ + value = args[index] + } + if value == "" { + return errors.New("--api requires a file") + } + apiFile = value + case arg == "--log-output" || arg == "--log-format" || arg == "--storage" || strings.HasPrefix(arg, "--log-output=") || strings.HasPrefix(arg, "--log-format=") || strings.HasPrefix(arg, "--storage="): + name, value, hasValue := strings.Cut(arg, "=") + if !hasValue { + if index+1 >= len(args) || strings.HasPrefix(args[index+1], "--") { + return fmt.Errorf("%s requires a value", name) + } + index++ + value = args[index] + } + if strings.TrimSpace(value) == "" { + return fmt.Errorf("%s requires a value", name) + } + switch name { + case "--log-output": + options.LogOutput = value + case "--log-format": + options.LogFormat = value + case "--storage": + options.Storage = value + } case arg == "--force": options.Force = true case arg == "--database": @@ -98,10 +138,26 @@ func runNew(args []string) error { if target == "" { return errors.New("project directory is required") } - if options.Module == "" { + if options.Module == "" && apiFile == "" { options.Module = filepath.Base(filepath.Clean(target)) } options.Dir = target + if apiFile != "" { + if options.Force { + return errors.New("--api protects existing business code and does not accept --force") + } + _, err := apigen.Generate(apigen.Options{API: apiFile, Dir: target, Module: options.Module, FrameworkDir: options.FrameworkDir, FrameworkVersion: options.FrameworkVersion, LogOutput: options.LogOutput, LogFormat: options.LogFormat, Database: options.Database, Storage: options.Storage}) + if err != nil { + return err + } + fmt.Printf("created API project %s; implement internal/service operations\n", filepath.Clean(target)) + return nil + } + if _, err := os.Lstat(filepath.Join(target, ".goOmni", "apiManifest.json")); err == nil { + return errors.New("this is an API project; use api generate instead of the CRUD/module starter") + } else if !errors.Is(err, os.ErrNotExist) { + return err + } if err := scaffold.Generate(options); err != nil { return err } @@ -115,10 +171,17 @@ func usage() { fmt.Fprintln(os.Stderr, "Usage:") fmt.Fprintln(os.Stderr, " go-omni migrate [--dir migrations] [--apply] [--dsn-env GO_OMNI_DB_DSN]") fmt.Fprintln(os.Stderr, " go-omni new [--module ] [--force] ") + fmt.Fprintln(os.Stderr, " go-omni new --api [--module ] ") + fmt.Fprintln(os.Stderr, " go-omni api new [--module ] Create a runnable API project") + fmt.Fprintln(os.Stderr, " go-omni api generate --api --dir [--check]") + fmt.Fprintln(os.Stderr, " go-omni api validate --api Validate definitions and OpenAPI") fmt.Fprintln(os.Stderr, " go-omni add [--dir ] Generate an isolated business module") fmt.Fprintln(os.Stderr, " --framework-version Pin a remote framework version (defaults to CLI version)") fmt.Fprintln(os.Stderr, " --framework-dir Use a local framework checkout") - fmt.Fprintln(os.Stderr, " --database Select generated storage (default memory)") + fmt.Fprintln(os.Stderr, " --database API defaults to none; CRUD defaults to memory") + fmt.Fprintln(os.Stderr, " --log-output Generate configured logging (default stderr)") + fmt.Fprintln(os.Stderr, " --log-format Select structured log format (default json)") + fmt.Fprintln(os.Stderr, " --storage Select optional storage adapter (default none)") } func runAdd(args []string) error { diff --git a/cmd/go-omni/main_test.go b/cmd/go-omni/main_test.go index 3f5fad4..9dafdcd 100644 --- a/cmd/go-omni/main_test.go +++ b/cmd/go-omni/main_test.go @@ -73,3 +73,28 @@ func TestNewMySQLDatabaseOptions(t *testing.T) { } } } + +func TestModuleOptionCLI(t *testing.T) { + for _, flags := range [][]string{ + {"--log-output", "file", "--log-format", "json", "--storage", "local"}, + {"--log-output=stdout", "--log-format=text", "--storage=none"}, + } { + project := filepath.Join(t.TempDir(), "demo") + args := append([]string{"--module", "example.com/demo"}, flags...) + if err := runNew(append(args, project)); err != nil { + t.Fatal(err) + } + if _, err := os.Stat(filepath.Join(project, "etc/config.yaml")); err != nil { + t.Fatal(err) + } + } + for _, args := range [][]string{ + {"--storage"}, {"--storage="}, {"--storage", "--force"}, + {"--log-output"}, {"--log-output="}, {"--log-format="}, + {"--storage", "unknown", "demo"}, {"--log-format", "xml", "demo"}, + } { + if err := runNew(args); err == nil { + t.Fatal("invalid module option accepted", args) + } + } +} diff --git a/cmd/go-omni/scaffold/apiInfrastructure.go b/cmd/go-omni/scaffold/apiInfrastructure.go new file mode 100644 index 0000000..377d71c --- /dev/null +++ b/cmd/go-omni/scaffold/apiInfrastructure.go @@ -0,0 +1,26 @@ +package scaffold + +import "fmt" + +// APIInfrastructure renders shared infrastructure without inventing business +// tables or routes. The API generator applies these as developer-owned seeds. +func APIInfrastructure(module, database, storage, logOutput, logFormat string) (map[string][]byte, error) { + data := templateData{Module: module, Database: database, Storage: storage, LogOutput: logOutput, LogFormat: logFormat, API: true, Modules: true} + files := map[string]string{ + "internal/settings/config.go": "templates/modules-settings.go.tmpl", + "etc/config.yaml": "templates/modules-config.yaml.tmpl", + "internal/servicecontext/context.go": "templates/api-context.go.tmpl", + } + if storage != "none" { + files["internal/servicecontext/storage.go"] = "templates/modules-storage.go.tmpl" + } + result := make(map[string][]byte, len(files)) + for name, source := range files { + content, err := render(source, data) + if err != nil { + return nil, fmt.Errorf("render %s: %w", name, err) + } + result[name] = []byte(content) + } + return result, nil +} diff --git a/cmd/go-omni/scaffold/moduleProfiles_test.go b/cmd/go-omni/scaffold/moduleProfiles_test.go new file mode 100644 index 0000000..a1a47eb --- /dev/null +++ b/cmd/go-omni/scaffold/moduleProfiles_test.go @@ -0,0 +1,143 @@ +package scaffold + +import ( + "context" + "os" + "os/exec" + "path/filepath" + "strconv" + "strings" + "testing" + "time" +) + +type moduleProfile struct{ database, storage, output, format string } + +var moduleProfiles = []moduleProfile{ + {"memory", "none", "stdout", "json"}, + {"memory", "local", "file", "json"}, + {"memory", "s3", "stdout", "text"}, + {"memory", "r2", "stderr", "json"}, + {"memory", "oss", "stderr", "text"}, + {"mysql", "none", "file", "text"}, + {"mysql", "local", "stdout", "json"}, + {"mysql", "s3", "file", "text"}, + {"mysql", "r2", "stderr", "text"}, + {"mysql", "oss", "file", "json"}, +} + +func generateProfile(t *testing.T, p moduleProfile) string { + t.Helper() + project := filepath.Join(t.TempDir(), "project") + if err := Generate(Options{Dir: project, Module: "example.com/profile", FrameworkDir: frameworkRoot(t), Database: p.database, Storage: p.storage, LogOutput: p.output, LogFormat: p.format}); err != nil { + t.Fatal(err) + } + return project +} +func TestGenerateModuleProfiles(t *testing.T) { + for _, p := range moduleProfiles { + t.Run(p.database+"/"+p.storage, func(t *testing.T) { + project := generateProfile(t, p) + for _, name := range []string{"etc/config.yaml", "internal/settings/config.go", "MODULES.md", ".gitignore"} { + if _, err := os.Stat(filepath.Join(project, name)); err != nil { + t.Fatal(err) + } + } + _, err := os.Stat(filepath.Join(project, "internal/handler/files.go")) + if p.storage == "none" && !os.IsNotExist(err) { + t.Fatal("unselected storage was emitted") + } + if p.storage != "none" && err != nil { + t.Fatal(err) + } + module, err := os.ReadFile(filepath.Join(project, "go.mod")) + if err != nil { + t.Fatal(err) + } + pieces := strings.Split(string(module), "=> ") + if len(pieces) != 2 { + t.Fatal("local replace missing") + } + relative, err := strconv.Unquote(strings.TrimSpace(pieces[1])) + if err != nil { + t.Fatal(err) + } + if _, err = os.Stat(filepath.Join(project, relative, "go.mod")); err != nil { + t.Fatal("replacement does not resolve", err) + } + cfg, _ := os.ReadFile(filepath.Join(project, "etc/config.yaml")) + if strings.Contains(string(cfg), "secretKey:") || strings.Contains(string(cfg), "filesToken:") { + t.Fatal("secret defaults emitted") + } + }) + } +} +func TestModuleOptionsRejectBeforeWriting(t *testing.T) { + for _, options := range []Options{ + {LogOutput: "invalid"}, {LogFormat: "xml"}, {Storage: "ftp"}, + {Storage: "local", FrameworkVersion: "v0.6.0-preview"}, + {Module: "example.com/x\nrequire malicious v1.0.0"}, + } { + options.Dir = filepath.Join(t.TempDir(), "project") + if options.Module == "" { + options.Module = "example.com/profile" + } + if err := Generate(options); err == nil { + t.Fatalf("invalid options accepted: %+v", options) + } + if _, err := os.Stat(options.Dir); !os.IsNotExist(err) { + t.Fatal("invalid options touched target") + } + } +} +func TestCanonicalPathBeforeDirectoryExists(t *testing.T) { + parent := t.TempDir() + alias := filepath.Join(t.TempDir(), "alias") + if err := os.Symlink(parent, alias); err != nil { + t.Skip(err) + } + path := filepath.Join(alias, "new", "nested") + if got, want := canonicalPath(path), filepath.Join(canonicalPath(parent), "new", "nested"); got != want { + t.Fatalf("%s != %s", got, want) + } +} + +// Compile consuming modules, then run their generated tests. No provider account +// is used: cloud constructors are checked offline, file workflows use Local. +func TestModuleProfilesIntegration(t *testing.T) { + if os.Getenv("GO_OMNI_SCAFFOLD_INTEGRATION") != "1" { + t.Skip("set GO_OMNI_SCAFFOLD_INTEGRATION=1 for generated-profile tests") + } + for _, p := range moduleProfiles { + t.Run(p.database+"/"+p.storage, func(t *testing.T) { + project := generateProfile(t, p) + ctx, cancel := context.WithTimeout(context.Background(), 2*time.Minute) + defer cancel() + cmd := exec.CommandContext(ctx, "go", "test", "-mod=mod", "-race", "-p=2", "-count=1", "-timeout=60s", "./...") + cmd.Dir = project + cmd.Env = append(os.Environ(), "GOWORK=off", "GOMAXPROCS=2", "GO_OMNI_TEST_MYSQL_DSN=") + if output, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("generated %s/%s: %v\n%s", p.database, p.storage, err, output) + } + }) + } +} + +func TestForceRejectsRemovedDependenciesWithoutTouchingCode(t *testing.T) { + project := generateProfile(t, moduleProfile{"mysql", "local", "file", "json"}) + path := filepath.Join(project, "cmd/server/main.go") + before, err := os.ReadFile(path) + if err != nil { + t.Fatal(err) + } + if err = Generate(Options{Dir: project, Module: "example.com/profile", FrameworkDir: frameworkRoot(t), Force: true, Storage: "none"}); err == nil { + t.Fatal("incompatible force overwrite accepted") + } + after, _ := os.ReadFile(path) + if string(after) != string(before) { + t.Fatal("failed profile switch changed code") + } + if err = Generate(Options{Dir: project, Module: "example.com/profile", FrameworkDir: frameworkRoot(t), Force: true, Database: "mysql", Storage: "local", LogOutput: "stdout"}); err != nil { + t.Fatal("compatible force overwrite rejected", err) + } +} diff --git a/cmd/go-omni/scaffold/scaffold.go b/cmd/go-omni/scaffold/scaffold.go index 52d32dc..a538f58 100644 --- a/cmd/go-omni/scaffold/scaffold.go +++ b/cmd/go-omni/scaffold/scaffold.go @@ -12,6 +12,9 @@ import ( "strconv" "strings" "text/template" + + "golang.org/x/mod/module" + "golang.org/x/mod/semver" ) //go:embed templates moduleTemplates @@ -24,6 +27,9 @@ type Options struct { FrameworkDir string FrameworkVersion string Database string + LogOutput string + LogFormat string + Storage string Force bool } @@ -48,6 +54,31 @@ func Generate(options Options) error { if strings.TrimSpace(options.Module) == "" { return errors.New("module path is required") } + if err := module.CheckImportPath(options.Module); err != nil { + return fmt.Errorf("invalid module path: %w", err) + } + modules := options.LogOutput != "" || options.LogFormat != "" || options.Storage != "" + logOutput, logFormat, storage := options.LogOutput, options.LogFormat, options.Storage + if logOutput == "" { + logOutput = "stderr" + } + if logFormat == "" { + logFormat = "json" + } + if storage == "" { + storage = "none" + } + if logOutput != "stdout" && logOutput != "stderr" && logOutput != "file" { + return fmt.Errorf("unsupported log output %q: choose stdout, stderr or file", logOutput) + } + if logFormat != "json" && logFormat != "text" { + return fmt.Errorf("unsupported log format %q: choose json or text", logFormat) + } + switch storage { + case "none", "local", "s3", "r2", "oss": + default: + return fmt.Errorf("unsupported object storage %q: choose none, local, s3, r2 or oss", storage) + } database := strings.ToLower(strings.TrimSpace(options.Database)) if database == "" { database = "memory" @@ -58,6 +89,9 @@ func Generate(options Options) error { version := strings.TrimSpace(options.FrameworkVersion) if version == "" { version = currentFrameworkVersion() + if modules && version == defaultFrameworkVersion { + version = modulesFrameworkVersion + } } if !validFrameworkVersion(version) { return fmt.Errorf("framework version %q must be a complete v0 or v1 semantic version", version) @@ -70,20 +104,20 @@ func Generate(options Options) error { if err := checkTarget(directory, options.Force); err != nil { return err } - if err := os.MkdirAll(directory, 0o755); err != nil { - return fmt.Errorf("create project directory: %w", err) - } - // An explicit remote version opts out of automatic local development mode. replace := "" if options.FrameworkDir != "" || options.FrameworkVersion == "" { replace = frameworkReplace(directory, options.FrameworkDir) } + if modules && replace == "" && semver.Compare(version, modulesFrameworkVersion) < 0 && version != installedFrameworkVersion() { + return fmt.Errorf("module templates require %s or newer; use --framework-dir for unpublished source", modulesFrameworkVersion) + } data := templateData{ Module: options.Module, FrameworkVersion: version, FrameworkReplace: replace, Database: database, + Modules: modules, LogOutput: logOutput, LogFormat: logFormat, Storage: storage, } projectFiles := make(map[string]string, len(files)+3) for path, source := range files { @@ -94,11 +128,42 @@ func Generate(options Options) error { projectFiles["schema.sql"] = "templates/schema.sql.tmpl" projectFiles["internal/servicecontext/databaseIntegration_test.go"] = "templates/databaseIntegration_test.go.tmpl" } + if modules { + projectFiles["cmd/server/main.go"] = "templates/modules-main.go.tmpl" + projectFiles["internal/settings/config.go"] = "templates/modules-settings.go.tmpl" + projectFiles["internal/settings/config_test.go"] = "templates/modules-settings_test.go.tmpl" + projectFiles["etc/config.yaml"] = "templates/modules-config.yaml.tmpl" + projectFiles[".gitignore"] = "templates/modules-gitignore.tmpl" + projectFiles["MODULES.md"] = "templates/modules-README.md.tmpl" + if storage != "none" { + projectFiles["internal/servicecontext/storage.go"] = "templates/modules-storage.go.tmpl" + projectFiles["internal/servicecontext/storage_test.go"] = "templates/modules-storage_test.go.tmpl" + projectFiles["internal/service/files.go"] = "templates/modules-filesService.go.tmpl" + projectFiles["internal/handler/files.go"] = "templates/modules-filesHandler.go.tmpl" + projectFiles["internal/handler/files_test.go"] = "templates/modules-filesHandler_test.go.tmpl" + } + } + if options.Force { + // Do not leave old generated Go files that reference dependencies omitted + // by this profile, and do not silently delete potentially edited code. + for _, path := range []string{"internal/repository/mysql.go", "internal/servicecontext/databaseIntegration_test.go", "schema.sql", "internal/settings/config.go", "internal/settings/config_test.go", "etc/config.yaml", "MODULES.md", "internal/servicecontext/storage.go", "internal/servicecontext/storage_test.go", "internal/service/files.go", "internal/handler/files.go", "internal/handler/files_test.go"} { + if _, emitted := projectFiles[path]; emitted { + continue + } + if _, err := os.Lstat(filepath.Join(directory, path)); err == nil { + return fmt.Errorf("existing %s conflicts with the selected profile; generate into a new directory instead of removing existing code", path) + } else if !errors.Is(err, os.ErrNotExist) { + return err + } + } + } paths := make([]string, 0, len(projectFiles)) for path := range projectFiles { paths = append(paths, path) } sort.Strings(paths) + // Render and format the whole project before touching the target directory. + contents := make(map[string]string, len(paths)) for _, path := range paths { content, err := render(projectFiles[path], data) if err != nil { @@ -111,6 +176,9 @@ func Generate(options Options) error { } content = string(formatted) } + contents[path] = content + } + for _, path := range paths { target := filepath.Join(directory, filepath.FromSlash(path)) if err := os.MkdirAll(filepath.Dir(target), 0o755); err != nil { return fmt.Errorf("create %s: %w", filepath.Dir(path), err) @@ -122,7 +190,7 @@ func Generate(options Options) error { return fmt.Errorf("check %s: %w", target, err) } } - if err := os.WriteFile(target, []byte(content), 0o644); err != nil { + if err := os.WriteFile(target, []byte(contents[path]), 0o644); err != nil { return fmt.Errorf("write %s: %w", target, err) } } @@ -130,10 +198,15 @@ func Generate(options Options) error { } type templateData struct { + API bool Module string FrameworkVersion string FrameworkReplace string Database string + Modules bool + LogOutput string + LogFormat string + Storage string } func render(name string, data templateData) (string, error) { @@ -202,11 +275,28 @@ func canonicalPath(path string) string { if err != nil { return path } - resolved, err := filepath.EvalSymlinks(abs) - if err == nil { - return resolved + // Resolve the nearest existing ancestor too: generation now renders before + // creating directories, and macOS /var commonly resolves through /private. + candidate := abs + var tail []string + for { + resolved, err := filepath.EvalSymlinks(candidate) + if err == nil { + for i := len(tail) - 1; i >= 0; i-- { + resolved = filepath.Join(resolved, tail[i]) + } + return resolved + } + if !errors.Is(err, os.ErrNotExist) { + return abs + } + parent := filepath.Dir(candidate) + if parent == candidate { + return abs + } + tail = append(tail, filepath.Base(candidate)) + candidate = parent } - return abs } func findFrameworkRoot() string { diff --git a/cmd/go-omni/scaffold/templates/README.md.tmpl b/cmd/go-omni/scaffold/templates/README.md.tmpl index f4e6a68..5984dba 100644 --- a/cmd/go-omni/scaffold/templates/README.md.tmpl +++ b/cmd/go-omni/scaffold/templates/README.md.tmpl @@ -1,10 +1,13 @@ # GoOmni service This service was generated with `go-omni new`. +{{if .Modules}} +**本项目启用了模块化配置，启动步骤和环境变量请先看 [MODULES.md](MODULES.md)。** 配置来自 `etc/config.yaml`，文件模块需要配置环境密钥后启动。 +{{end}} Framework dependency: `git.yuntuo.hk/k2safe/go-omni {{.FrameworkVersion}}`. -Storage: `{{.Database}}`. +Database: `{{.Database}}`. {{if eq .Database "mysql"}} 先创建业务数据库，再执行项目的建表脚本（替换示例数据库名、账号和地址）： diff --git a/cmd/go-omni/scaffold/templates/api-context.go.tmpl b/cmd/go-omni/scaffold/templates/api-context.go.tmpl new file mode 100644 index 0000000..354e3e1 --- /dev/null +++ b/cmd/go-omni/scaffold/templates/api-context.go.tmpl @@ -0,0 +1,69 @@ +package servicecontext +import ( + "context" + "errors" + "sync" + {{if ne .Storage "none"}}"net/http" + "git.yuntuo.hk/k2safe/go-omni/storage"{{end}} + {{if eq .Database "mysql"}}"time" + "git.yuntuo.hk/k2safe/go-omni/database" + "github.com/go-sql-driver/mysql"{{end}} + "git.yuntuo.hk/k2safe/go-omni/health" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/logx" + "{{.Module}}/internal/settings" + "{{.Module}}/internal/middleware" +) +// Context owns infrastructure, while Logger is borrowed from main. +// Add business repositories here; HTTP contracts do not define database tables. +type Context struct { + Logger *logx.Logger + Health *health.Checker + Middlewares map[string]httpx.Middleware + {{if eq .Database "mysql"}}DB *database.DB{{end}} + {{if ne .Storage "none"}} + Files storage.Store + Uploads *storage.Uploader + Downloads storage.Presigner + UploadLinks storage.UploadPresigner + FileHTTP http.Handler + storageClose func()error + {{end}} + closeMu sync.Mutex + closed bool + closeOnce sync.Once + closeErr error +} +// Pass the settings loaded by main. The optional form supports lightweight +// tests without infrastructure; selected adapters still require valid settings. +func New(ctx context.Context,logger *logx.Logger,configs ...settings.Config)(result *Context,err error){ + if ctx==nil{ctx=context.Background()} + if err=ctx.Err();err!=nil{return nil,err} + if len(configs)>1{return nil,errors.New("expected at most one configuration")} + cfg:=settings.Default();if len(configs)==1{cfg=configs[0]} + _=cfg + checker,err:=health.New();if err!=nil{return nil,err} + c:=&Context{Logger:logger,Health:checker,Middlewares:middleware.Defaults()} + defer func(){if err!=nil{err=errors.Join(err,c.Close())}}() + {{if eq .Database "mysql"}} + if err=cfg.Validate();err!=nil{return nil,err} + dsn,parseErr:=mysql.ParseDSN(cfg.Database.DSN) + if parseErr!=nil || dsn.DBName=="" {return nil,errors.New("GO_OMNI_DB_DSN must contain a valid MySQL database name")} + dsn.ParseTime=true;dsn.Loc=time.UTC + cfg.Database.DSN=dsn.FormatDSN() + c.DB,err=database.Open(ctx,cfg.Database);if err!=nil{return nil,err} + if err=checker.Add(health.Probe{Name:"database",Timeout:cfg.Database.PingTimeout,Check:c.DB.Health});err!=nil{return nil,err} + {{end}} + {{if ne .Storage "none"}}if err=c.InitStorage(ctx,cfg.Storage);err!=nil{return nil,err}{{end}} + return c,nil +} +// Close is idempotent, including startup failure and graceful shutdown. +func(c *Context)Close()error{ + if c==nil{return nil} + c.closeMu.Lock();defer c.closeMu.Unlock();c.closed=true + c.closeOnce.Do(func(){ + {{if ne .Storage "none"}}if c.storageClose!=nil{c.closeErr=errors.Join(c.closeErr,c.storageClose())}{{end}} + {{if eq .Database "mysql"}}if c.DB!=nil{c.closeErr=errors.Join(c.closeErr,c.DB.Close())}{{end}} + }) + return c.closeErr +} diff --git a/cmd/go-omni/scaffold/templates/internal-service-user.go.tmpl b/cmd/go-omni/scaffold/templates/internal-service-user.go.tmpl index ea124cb..7ee5009 100644 --- a/cmd/go-omni/scaffold/templates/internal-service-user.go.tmpl +++ b/cmd/go-omni/scaffold/templates/internal-service-user.go.tmpl @@ -23,7 +23,10 @@ func (s *Users) Create(ctx context.Context, input model.UserInput) (model.User, if err != nil { return model.User{}, err } - return s.services.Users.Create(ctx, input) + user, err := s.services.Users.Create(ctx, input) + if err != nil { s.services.Logger.ErrorContext(ctx,"user.create.failed","error",err); return model.User{},err } + s.services.Logger.InfoContext(ctx,"user.created","userId",user.ID) + return user,nil } func (s *Users) Get(ctx context.Context, id int64) (model.User, error) { diff --git a/cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl b/cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl index ade1abc..bde7108 100644 --- a/cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl +++ b/cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl @@ -2,6 +2,10 @@ package servicecontext import ( "context" + {{if and .Modules (ne .Storage "none")}}"errors" + "net/http" + "sync" + "git.yuntuo.hk/k2safe/go-omni/storage"{{end}} {{if eq .Database "mysql"}}"fmt" "time" "git.yuntuo.hk/k2safe/go-omni/database" @@ -17,6 +21,18 @@ type Context struct { Logger *logx.Logger Users repository.UserRepository Health *health.Checker + {{if and .Modules (ne .Storage "none")}} + Files storage.Store + Uploads *storage.Uploader + Downloads storage.Presigner + UploadLinks storage.UploadPresigner + FileHTTP http.Handler + storageClose func() error + closeMu sync.Mutex + closed bool + closeOnce sync.Once + closeErr error + {{end}} {{if eq .Database "mysql"}}DB *database.DB{{end}} } @@ -61,6 +77,15 @@ func New(ctx context.Context, logger *logx.Logger) (*Context, error) { // Close is safe to call on startup failure and again during application exit. func (c *Context) Close() error { - {{if eq .Database "mysql"}}if c != nil && c.DB != nil { return c.DB.Close() }{{end}} - return nil + {{if and .Modules (ne .Storage "none")}} + if c==nil{return nil} + c.closeMu.Lock();defer c.closeMu.Unlock() + c.closed=true + c.closeOnce.Do(func(){ + if c.storageClose!=nil {c.closeErr=errors.Join(c.closeErr,c.storageClose())} + {{if eq .Database "mysql"}}if c.DB!=nil{c.closeErr=errors.Join(c.closeErr,c.DB.Close())}{{end}} + }) + return c.closeErr + {{else}}{{if eq .Database "mysql"}}if c != nil && c.DB != nil { return c.DB.Close() }{{end}}{{end}} + {{if not (and .Modules (ne .Storage "none"))}}return nil{{end}} } diff --git a/cmd/go-omni/scaffold/templates/modules-README.md.tmpl b/cmd/go-omni/scaffold/templates/modules-README.md.tmpl new file mode 100644 index 0000000..da7e9fb --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-README.md.tmpl @@ -0,0 +1,97 @@ +# 按需装配的日志与文件模块 + +本项目选用日志输出 `{{.LogOutput}}`、格式 `{{.LogFormat}}`、对象存储 `{{.Storage}}`。数据库选择 `{{.Database}}` 独立于对象存储。 + +配置入口为 `etc/config.yaml`，通过 `go run ./cmd/server -config /path/to/config.yaml` 指定其他 JSON/YAML 文件；`-config ''` 只用代码默认值和环境变量。优先级：默认值 → 文件 → 非空环境变量。未知字段、无效参数会在监听前拒绝。 + +```sh +go mod tidy +{{if ne .Storage "none"}}# 示例文件接口使用单一共享 Token，不是用户/租户权限系统。 +export GO_OMNI_FILES_TOKEN="$(openssl rand -hex 32)" +{{if eq .Storage "local"}}# 本地签名密钥需要稳定保存；换密钥后旧链接失效。 +export GO_OMNI_STORAGE_SIGNING_KEY="$(openssl rand -hex 32)" +{{else}}# 配置当前存储平台已有的桶和凭据，不会自动创建桶。 +export GO_OMNI_STORAGE_BUCKET='your-bucket' +export GO_OMNI_STORAGE_ACCESS_KEY='your-access-key' +export GO_OMNI_STORAGE_SECRET_KEY='your-secret-key' +{{if eq .Storage "s3"}}export GO_OMNI_STORAGE_ENDPOINT='s3.us-east-1.amazonaws.com' +export GO_OMNI_STORAGE_REGION='us-east-1' +{{else if eq .Storage "r2"}}export GO_OMNI_STORAGE_ACCOUNT_ID='your-32-character-account-id' +{{else}}export GO_OMNI_STORAGE_REGION='cn-hangzhou' +{{end}}{{end}}{{end}} +go run ./cmd/server +``` + +{{if eq .Database "mysql"}}还需按 README 配置独立 MySQL 数据库并应用 schema.sql；应用不自动迁移。{{end}} + +默认仅监听 `127.0.0.1:8080`。`GO_OMNI_HTTP_ADDR` 覆盖地址；本地签名 URL 的外部地址通过 `GO_OMNI_STORAGE_PUBLIC_URL` 单独配置，始终以 `/storage/object` 结尾，不使用请求 Host 自动拼接可信链接。 + +## 日志 + +业务示例在创建用户/上传文件成功时通过 logx 写结构化日志。Context 原样传递 requestId，框架不自动增加用户、租户等业务字段。文件模式已配置轮转：100 MiB、10 份、7 天、压缩。调整 `server.log` 即可更改；`redactKeys` 可增加业务敏感键。 + +```sh +curl -H 'Content-Type: application/json' -H 'X-Request-ID: example-request' \ + -d '{"name":"Example","email":"example@example.com"}' http://127.0.0.1:8080/users +``` + +`GO_OMNI_LOG_LEVEL`、`GO_OMNI_LOG_FORMAT`、`GO_OMNI_LOG_FILE` 覆盖对应项；显式设置 `GO_OMNI_LOG_OUTPUT=stdout` 且未设置 LOG_FILE 时，切换到控制台并关闭文件轮转。慢请求阈值为 500ms，可通过 `GO_OMNI_SLOW_REQUEST_THRESHOLD` 调整。字段脱敏不扫描 msg/error 中的自由文本；不要把凭据或签名 URL 拼到日志中。 + +{{if ne .Storage "none"}}## 文件示例 + +Handler → Service → storage 接口；底层构造与连接池集中在 `internal/servicecontext/storage.go`。示例接口要求 `Authorization: Bearer $GO_OMNI_FILES_TOKEN`，它授权这个示例的全部文件，实际业务应替换认证和对象归属检查。存储适配器不承担业务授权。 + +```sh +printf 'hello go-omni' > /tmp/goOmniExample.txt +curl -H "Authorization: Bearer $GO_OMNI_FILES_TOKEN" \ + -H 'Content-Type: text/plain' -H 'X-Request-ID: file-example' \ + --data-binary @/tmp/goOmniExample.txt http://127.0.0.1:8080/files +``` + +保存响应中 `data.key`，再请求下载链接（把示例 key 替换为返回值）： + +```sh +curl -H "Authorization: Bearer $GO_OMNI_FILES_TOKEN" -H 'Content-Type: application/json' \ + -d '{"key":"uploads/returned-key"}' http://127.0.0.1:8080/files/downloadLink +``` + +返回 `data.url`，在有效期内 GET 即可下载，无需再携带 Token。签发前会检查对象存在；URL 默认 5 分钟有效，不写日志，响应设置 no-store。 + +需要直传时先申请链接： + +```sh +curl -H "Authorization: Bearer $GO_OMNI_FILES_TOKEN" -H 'Content-Type: application/json' \ + -d '{"size":13,"contentType":"text/plain"}' http://127.0.0.1:8080/files/uploadLink +``` + +响应包括随机 `data.key` 和 `data.request`（method/url/headers/expiresAt）。按返回的 method 和 headers 用原始文件字节上传到 URL；不能额外包 FormData。浏览器自动填写 Content-Length，其余签名 Header 原样发送；云桶需配置允许业务域名的 CORS。云端直传只绑定声明类型，实际内容需上传后验证；本地和 API 中转上传会检查允许类型及前缀嗅探结果。未完成内容验证前不要把暂存文件当作可信文件公开。 + +| 参数 | 默认值 / 环境变量 | +| --- | --- | +| 大小 | 32 MiB / GO_OMNI_STORAGE_MAX_BYTES | +| 类型 | text/plain、image/png、image/jpeg、application/pdf / GO_OMNI_STORAGE_CONTENT_TYPES（逗号分隔） | +| 链接有效期 | 5m / GO_OMNI_STORAGE_LINK_TTL（1s..1h） | +| 请求超时 | 30s / GO_OMNI_STORAGE_REQUEST_TIMEOUT（>0 且 <=5m） | +| 本地目录 | var/files / GO_OMNI_STORAGE_ROOT | +| 本地外部 URL | http://127.0.0.1:8080/storage/object / GO_OMNI_STORAGE_PUBLIC_URL | +| 云配置 | GO_OMNI_STORAGE_ENDPOINT、REGION、BUCKET、ACCOUNT_ID、SECURE | +| 环境密钥 | GO_OMNI_FILES_TOKEN、GO_OMNI_STORAGE_SIGNING_KEY、GO_OMNI_STORAGE_ACCESS_KEY、SECRET_KEY、SESSION_TOKEN | + +全部 JSON/YAML 字段使用 lowerCamelCase。密钥字段仅允许环境变量，配置文件中写入会报未知字段；不要把运行环境文件提交到仓库。S3 的 endpoint 为 host[:port]，OSS 的可选 endpoint 为 HTTPS origin；R2 从 accountId 派生官方 endpoint，region 为 auto。仅 S3/MinIO 的本地测试可设置 SECURE=false。 + +错误响应：未授权 401、无 Content-Length 411、超限 413、类型不允许 415、长度不符/非法 key 400、对象不存在 404、存储超时 504，其余内部错误统一隐藏细节。底层签名入口遵循存储协议返回 201/403 等状态，流式传输且访问日志保留实际状态。 + +云客户端只在组合处创建，使用专用有界连接池；配置正确不代表桶和网络已经可用，首次真实调用仍可能失败。不创建桶、不修改桶权限、不初始化任何业务数据库。更换平台时替换组合处适配器，业务 Service 接口保持不变。 +{{end}} + +## 生命周期与验证 + +启动失败会清理已创建资源；SIGINT/SIGTERM 后先等待 HTTP 请求退出，再关闭服务依赖，最后关闭日志。只关闭项目自己创建的资源，不关闭全局默认 HTTP Transport。 + +```sh +go test -race ./... +go build -o server ./cmd/server +./server +``` + +生成的测试覆盖配置优先级/校验{{if ne .Storage "none"}}、资源关闭幂等性和借用日志所有权、上传与临时链接、签名篡改、限额/类型错误。业务测试使用临时 Local，不访问云账号；云端配置需另做实网验收{{end}}。基础用户 CRUD 示例仍未启用认证，部署前按业务需要替换。 diff --git a/cmd/go-omni/scaffold/templates/modules-config.yaml.tmpl b/cmd/go-omni/scaffold/templates/modules-config.yaml.tmpl new file mode 100644 index 0000000..b50baf7 --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-config.yaml.tmpl @@ -0,0 +1,46 @@ +# Non-secret settings only. Environment variables override these values. +server: + address: '127.0.0.1:8080' + enableLogger: true + slowRequestThreshold: 500ms + shutdownTimeout: 10s + log: + service: {{printf "%q" .Module}} + level: info + format: {{.LogFormat}} +{{if eq .LogOutput "file"}} file: './var/log/server.log' + rotation: + enabled: true + maxSizeMb: 100 + maxBackups: 10 + maxAgeDays: 7 + compress: true +{{else}} output: {{.LogOutput}} +{{end}} redactKeys: [] +{{if ne .Storage "none"}}storage: + provider: {{.Storage}} + maxBytes: 33554432 + contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf'] + linkTtl: 5m + requestTimeout: 30s +{{if eq .Storage "local"}} root: './var/files' + publicUrl: 'http://127.0.0.1:8080/storage/object' +{{else}} bucket: '' # set GO_OMNI_STORAGE_BUCKET + secure: true +{{if eq .Storage "r2"}} accountId: '' # set GO_OMNI_STORAGE_ACCOUNT_ID + region: auto +{{else if eq .Storage "s3"}} endpoint: '' # set GO_OMNI_STORAGE_ENDPOINT (host[:port]) + region: us-east-1 +{{else}} region: cn-hangzhou + endpoint: '' # empty uses the official regional endpoint +{{end}}{{end}}{{end}} + +{{if and .API (eq .Database "mysql")}}database: + driver: mysql + maxOpenConns: 10 + maxIdleConns: 5 + connMaxLifetime: 30m + connMaxIdleTime: 5m + pingTimeout: 5s +# GO_OMNI_DB_DSN supplies the secret connection string. +{{end}} diff --git a/cmd/go-omni/scaffold/templates/modules-filesHandler.go.tmpl b/cmd/go-omni/scaffold/templates/modules-filesHandler.go.tmpl new file mode 100644 index 0000000..037fc0f --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-filesHandler.go.tmpl @@ -0,0 +1,86 @@ +package handler + +import ( + "context" + "crypto/subtle" + "errors" + "net/http" + "time" + + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/storage" + "{{.Module}}/internal/service" +) + +type Files struct {service *service.Files; maxBytes int64; timeout time.Duration} +func NewFiles(service *service.Files,maxBytes int64,timeout time.Duration)*Files{return &Files{service:service,maxBytes:maxBytes,timeout:timeout}} + +// RegisterRoutes protects example control endpoints with one shared demo token. +// Replace this gate with project authentication and object ownership checks. +// Signed object URLs carry their own short-lived authorization. +func (h *Files) RegisterRoutes(application *omni.App,token string,signed http.Handler)error { + if len(token)<32{return errors.New("file example requires a token of at least 32 bytes")} + bounded:=func(next omni.Handler)omni.Handler{return func(c *omni.Context)error{ + ctx,cancel:=context.WithTimeout(c.Context(),h.timeout);defer cancel();c.WithContext(ctx);return next(c) + }} + authorized:=func(next omni.Handler)omni.Handler{return func(c *omni.Context)error{ + if len(c.Request().Header.Values("Authorization"))!=1 || subtle.ConstantTimeCompare([]byte(c.Header("Authorization")),[]byte("Bearer "+token))!=1{return omni.Unauthorized("UNAUTHORIZED","invalid file access token")} + return next(c) + }} + group:=application.Group("/files",authorized,bounded) + group.POST("",h.Upload) + group.POST("/uploadLink",h.UploadLink) + group.POST("/downloadLink",h.DownloadLink) + if signed!=nil {application.Any("/storage/object",func(c *omni.Context)error{ + signed.ServeHTTP(contextWriter{c},c.Request());return nil + },bounded)} + return nil +} +func (h *Files) Upload(c *omni.Context)error { + size:=c.Request().ContentLength + if size<0{return omni.NewError(http.StatusLengthRequired,"LENGTH_REQUIRED","Content-Length is required")} + if size>h.maxBytes{return omni.NewError(http.StatusRequestEntityTooLarge,"FILE_TOO_LARGE","file exceeds upload limit")} + body:=http.MaxBytesReader(c.Raw(),c.Request().Body,h.maxBytes) + defer body.Close() + info,err:=h.service.Upload(c.Context(),body,size,c.Header("Content-Type")) + if err!=nil{return fileError(err)} + return c.Created(info) +} +func (h *Files) UploadLink(c *omni.Context)error { + var input struct{Size int64 `json:"size" binding:"gte=0"`;ContentType string `json:"contentType" binding:"required,max=256"`} + if err:=c.BindJSON(&input);err!=nil{return err} + link,err:=h.service.UploadLink(c.Context(),input.Size,input.ContentType) + if err!=nil{return fileError(err)} + c.SetResponseHeader("Cache-Control","no-store") + return c.OK(link) +} +func (h *Files) DownloadLink(c *omni.Context)error { + var input struct{Key string `json:"key" binding:"required,max=1024"`} + if err:=c.BindJSON(&input);err!=nil{return err} + if err:=storage.Check(c.Context(),input.Key);err!=nil{return omni.BadRequest("INVALID_KEY","invalid object key")} + link,err:=h.service.DownloadLink(c.Context(),input.Key) + if err!=nil{return fileError(err)} + c.SetResponseHeader("Cache-Control","no-store") + return c.OK(map[string]string{"url":link}) +} +func fileError(err error)error { + var limit *http.MaxBytesError + switch { + case errors.Is(err,storage.ErrTooLarge),errors.As(err,&limit):return omni.NewError(413,"FILE_TOO_LARGE","file exceeds upload limit").WithCause(err) + case errors.Is(err,storage.ErrContentType):return omni.NewError(415,"CONTENT_TYPE_NOT_ALLOWED","file content type is not allowed").WithCause(err) + case errors.Is(err,storage.ErrSizeMismatch):return omni.BadRequest("SIZE_MISMATCH","file size does not match Content-Length").WithCause(err) + case errors.Is(err,storage.ErrNotFound):return omni.NotFound("FILE_NOT_FOUND","file was not found").WithCause(err) + case errors.Is(err,context.DeadlineExceeded):return omni.NewError(504,"STORAGE_TIMEOUT","storage operation timed out").WithCause(err) + default:return err + } +} + +// Track raw streaming handler responses through Omni so access logs retain the +// actual status (including failed signatures), without buffering object bodies. +type contextWriter struct{ctx *omni.Context} +func (w contextWriter) Header()http.Header{return w.ctx.Raw().Header()} +func (w contextWriter) WriteHeader(status int){_ = w.ctx.WriteResponse(status,nil,nil)} +func (w contextWriter) Write(data []byte)(int,error){ + if !w.ctx.Responded(){if err:=w.ctx.WriteResponse(http.StatusOK,nil,nil);err!=nil{return 0,err}} + return w.ctx.Raw().Write(data) +} diff --git a/cmd/go-omni/scaffold/templates/modules-filesHandler_test.go.tmpl b/cmd/go-omni/scaffold/templates/modules-filesHandler_test.go.tmpl new file mode 100644 index 0000000..3d10131 --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-filesHandler_test.go.tmpl @@ -0,0 +1,75 @@ +package handler_test + +import ( + "bytes" + "context" + "encoding/json" + "io" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "testing" + "time" + + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/logx" + "git.yuntuo.hk/k2safe/go-omni/storage" + "git.yuntuo.hk/k2safe/go-omni/storage/local" + "{{.Module}}/internal/handler" + "{{.Module}}/internal/service" +) + +// Exercise business code using Local even when production selects a cloud +// adapter. These tests do not use cloud accounts or network credentials. +func TestFileUploadAndLinks(t *testing.T){ + files,err:=local.New(t.TempDir(),64);if err!=nil{t.Fatal(err)} + policy:=storage.UploadPolicy{MaxBytes:64,ContentTypes:[]string{"text/plain"}} + signer,err:=storage.NewUploadSigner(files,"http://localhost/storage/object",bytes.Repeat([]byte{1},32),policy);if err!=nil{t.Fatal(err)} + uploader,err:=storage.NewUploader(files,policy);if err!=nil{t.Fatal(err)} + var logs bytes.Buffer + logger,err:=logx.New(logx.Config{Writer:&logs});if err!=nil{t.Fatal(err)} + logx.SetDefault(logger);defer logx.SetDefault(nil) + app:=omni.New(omni.WithLogger(true)) + token:=strings.Repeat("t",32) + svc:=service.NewFiles(files,uploader,signer,signer,logger,time.Minute) + if err=handler.NewFiles(svc,64,time.Second).RegisterRoutes(app,token,signer);err!=nil{t.Fatal(err)} + request:=func(method,path,body,contentType string,authorized bool)*httptest.ResponseRecorder{ + r:=httptest.NewRequest(method,path,strings.NewReader(body));r.Header.Set("Content-Type",contentType);r.Header.Set("X-Request-ID","upload-test") + if authorized{r.Header.Set("Authorization","Bearer "+token)} + w:=httptest.NewRecorder();app.ServeHTTP(w,r);return w + } + if w:=request("POST","/files","hello","text/plain",false);w.Code!=401{t.Fatal(w.Code)} + w:=request("POST","/files","hello","text/plain",true) + var saved struct{Data storage.Info} + if w.Code!=201 || json.Unmarshal(w.Body.Bytes(),&saved)!=nil || saved.Data.Size!=5{t.Fatal(w.Code,w.Body.String())} + raw,_:=json.Marshal(map[string]string{"key":saved.Data.Key}) + w=request("POST","/files/downloadLink",string(raw),"application/json",true) + var download struct{Data struct{URL string}} + if w.Code!=200 || json.Unmarshal(w.Body.Bytes(),&download)!=nil{t.Fatal(w.Code,w.Body.String())} + w=request("GET",download.Data.URL,"","",false) + if w.Code!=200 || w.Body.String()!="hello"{t.Fatal(w.Code,w.Body.String())} + u,_:=url.Parse(download.Data.URL);q:=u.Query();q.Set("signature","tampered");u.RawQuery=q.Encode() + if w=request("GET",u.String(),"","",false);w.Code!=403{t.Fatal(w.Code)} + for _,tc:=range []struct{path,body,contentType string;status int}{ + {"/files",strings.Repeat("x",65),"text/plain",413}, + {"/files","hello","image/png",415}, + {"/files","hello","text/plain",415}, + {"/files/downloadLink",`{"key":"../bad"}`,"application/json",400}, + {"/files/downloadLink",`{"key":"missing"}`,"application/json",404}, + {"/files/uploadLink",`{"size":65,"contentType":"text/plain"}`,"application/json",413}, + {"/files/uploadLink",`{"size":5,"contentType":"image/png"}`,"application/json",415}, + }{if w=request("POST",tc.path,tc.body,tc.contentType,true);w.Code!=tc.status{t.Fatal(tc.path,w.Code,w.Body.String())}} + w=request("POST","/files/uploadLink",`{"size":5,"contentType":"text/plain"}`,"application/json",true) + var direct struct{Data service.FileUploadLink} + if w.Code!=200 || json.Unmarshal(w.Body.Bytes(),&direct)!=nil{t.Fatal(w.Code,w.Body.String())} + if w=request("PUT",direct.Data.Request.URL,"world","text/plain",false);w.Code!=201{t.Fatal(w.Code,w.Body.String())} + object,err:=files.Get(context.Background(),direct.Data.Key);if err!=nil{t.Fatal(err)};body,err:=io.ReadAll(object.Body);object.Body.Close();if err!=nil || string(body)!="world"{t.Fatal(err,string(body))} + decoder:=json.NewDecoder(&logs);sawBusiness,sawDenied:=false,false + for {var entry map[string]any;err:=decoder.Decode(&entry);if err==io.EOF{break};if err!=nil{t.Fatal(err)} + if entry["requestId"]!="upload-test"{t.Fatal("lost request correlation")} + if entry["msg"]=="file.uploaded"{sawBusiness=true} + if entry["msg"]=="http.request" && entry["path"]=="/storage/object" && entry["status"]==float64(http.StatusForbidden){sawDenied=true} + } + if !sawBusiness || !sawDenied{t.Fatal("business log or signed-handler status missing")} +} diff --git a/cmd/go-omni/scaffold/templates/modules-filesService.go.tmpl b/cmd/go-omni/scaffold/templates/modules-filesService.go.tmpl new file mode 100644 index 0000000..2670175 --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-filesService.go.tmpl @@ -0,0 +1,61 @@ +package service + +import ( + "context" + "crypto/rand" + "encoding/hex" + "io" + "time" + + "git.yuntuo.hk/k2safe/go-omni/logx" + "git.yuntuo.hk/k2safe/go-omni/storage" +) + +// Files borrows provider-neutral capabilities; it never creates or closes clients. +// Authentication/ownership rules belong to this project, outside storage adapters. +type Files struct { + store storage.Store + uploader *storage.Uploader + downloads storage.Presigner + uploadLinks storage.UploadPresigner + logger *logx.Logger + ttl time.Duration +} +func NewFiles(store storage.Store,uploader *storage.Uploader,downloads storage.Presigner,uploadLinks storage.UploadPresigner,logger *logx.Logger,ttl time.Duration)*Files{ + return &Files{store:store,uploader:uploader,downloads:downloads,uploadLinks:uploadLinks,logger:logger.With("component","files"),ttl:ttl} +} +func newFileKey()(string,error){ + var id [16]byte + if _,err:=rand.Read(id[:]);err!=nil{return "",err} + return "uploads/"+hex.EncodeToString(id[:]),nil +} +func (s *Files) Upload(ctx context.Context,body io.Reader,size int64,contentType string)(storage.Info,error){ + key,err:=newFileKey();if err!=nil{return storage.Info{},err} + info,err:=s.uploader.Put(ctx,key,body,size,contentType) + if err!=nil{s.logger.WarnContext(ctx,"file.upload.failed","error",err);return storage.Info{},err} + s.logger.InfoContext(ctx,"file.uploaded","key",info.Key,"size",info.Size,"contentType",info.ContentType) + return info,nil +} +type FileUploadLink struct { + Key string `json:"key"` + Request storage.UploadRequest `json:"request"` +} +func (s *Files) UploadLink(ctx context.Context,size int64,contentType string)(FileUploadLink,error){ + key,err:=newFileKey();if err!=nil{return FileUploadLink{},err} + // Check the same policy before issuing a local or cloud capability. + // Cloud byte-level checks must happen in a post-upload validation workflow. + request,err:=s.uploadLinks.PresignPut(ctx,key,size,contentType,s.ttl) + if err!=nil{return FileUploadLink{},err} + s.logger.InfoContext(ctx,"file.uploadLink.issued","key",key) + return FileUploadLink{Key:key,Request:request},nil +} +func (s *Files) DownloadLink(ctx context.Context,key string)(string,error){ + if err:=storage.Check(ctx,key);err!=nil{return "",err} + // Confirm the object exists without buffering its contents. + object,err:=s.store.Get(ctx,key);if err!=nil{return "",err} + if err=object.Body.Close();err!=nil{return "",err} + link,err:=s.downloads.PresignGet(ctx,key,s.ttl) + if err!=nil{return "",err} + s.logger.InfoContext(ctx,"file.downloadLink.issued","key",key) + return link.String(),nil +} diff --git a/cmd/go-omni/scaffold/templates/modules-gitignore.tmpl b/cmd/go-omni/scaffold/templates/modules-gitignore.tmpl new file mode 100644 index 0000000..80d278b --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-gitignore.tmpl @@ -0,0 +1,6 @@ +/var/ +/server +.env +.env.* +*.local.yaml +*.local.json diff --git a/cmd/go-omni/scaffold/templates/modules-main.go.tmpl b/cmd/go-omni/scaffold/templates/modules-main.go.tmpl new file mode 100644 index 0000000..a762f12 --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-main.go.tmpl @@ -0,0 +1,50 @@ +package main + +import ( + "context" + "errors" + "flag" + "log" + "os" + "os/signal" + "syscall" + + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/logx" + "{{.Module}}/internal/handler" + "{{.Module}}/internal/service" + "{{.Module}}/internal/servicecontext" + "{{.Module}}/internal/settings" +) + +func main(){ + path:=flag.String("config","etc/config.yaml","JSON/YAML settings; empty uses defaults and environment") + flag.Parse() + if err:=run(*path);err!=nil{log.Fatal(err)} +} +func run(path string)(runErr error){ + ctx,stop:=signal.NotifyContext(context.Background(),os.Interrupt,syscall.SIGTERM) + defer stop() + cfg,err:=settings.Load(path) + if err!=nil{return err} + logger,err:=logx.New(cfg.Server.Log) + if err!=nil{return err} + defer func(){runErr=errors.Join(runErr,logger.Close())}() + logx.SetDefault(logger) + services,err:=servicecontext.New(ctx,logger) + if err!=nil{return err} + defer func(){runErr=errors.Join(runErr,services.Close())}() + {{if ne .Storage "none"}}if err=services.InitStorage(ctx,cfg.Storage);err!=nil{return err}{{end}} + application:=omni.New(omni.WithConfig(cfg.Server)) + application.OnShutdown(logger.Close) + application.OnShutdown(services.Close) + application.GET("/health/live",services.Health.LivenessHandler()) + application.GET("/health/ready",services.Health.ReadinessHandler()) + handler.NewUsers(service.NewUsers(services)).RegisterRoutes(application) + {{if ne .Storage "none"}} + files:=service.NewFiles(services.Files,services.Uploads,services.Downloads,services.UploadLinks,logger,cfg.Storage.LinkTTL) + if err=handler.NewFiles(files,cfg.Storage.MaxBytes,cfg.Storage.RequestTimeout).RegisterRoutes(application,cfg.Storage.FilesToken,services.FileHTTP);err!=nil{return err} + {{end}} + logger.InfoContext(ctx,"service.starting","address",cfg.Server.Address) + return application.RunContext(ctx,cfg.Server.Address) +} diff --git a/cmd/go-omni/scaffold/templates/modules-settings.go.tmpl b/cmd/go-omni/scaffold/templates/modules-settings.go.tmpl new file mode 100644 index 0000000..41b8742 --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-settings.go.tmpl @@ -0,0 +1,140 @@ +// Package settings owns this project's configuration; it adds no business identity model. +package settings + +import ( + "errors" + "os" + "strings" + "time" + {{if ne .Storage "none"}}"fmt" + {{if ne .Storage "r2"}}"net/url"{{else}}"regexp"{{end}} + "strconv" + "git.yuntuo.hk/k2safe/go-omni/storage"{{end}} + {{if and .API (eq .Database "mysql")}}"git.yuntuo.hk/k2safe/go-omni/database"{{end}} + "git.yuntuo.hk/k2safe/go-omni/config" + "git.yuntuo.hk/k2safe/go-omni/logx" +) + +type Config struct { + Server config.Config `json:"server"` + {{if and .API (eq .Database "mysql")}}Database database.Config `json:"database"`{{end}} + {{if ne .Storage "none"}}Storage StorageConfig `json:"storage"`{{end}} +} + +func Default() Config { + server:=config.Default() + server.Address="127.0.0.1:8080" + server.EnableLogger=true + server.SlowRequestThreshold=500*time.Millisecond + server.Log.Service={{printf "%q" .Module}} + server.Log.Format={{printf "%q" .LogFormat}} + {{if eq .LogOutput "file"}}server.Log.File="./var/log/server.log" + server.Log.Rotation=logx.RotationConfig{Enabled:true,MaxSizeMB:100,MaxBackups:10,MaxAgeDays:7,Compress:true} + {{else}}server.Log.Output={{printf "%q" .LogOutput}}{{end}} + return Config{Server:server{{if and .API (eq .Database "mysql")}},Database:database.DefaultConfig(){{end}}{{if ne .Storage "none"}},Storage:StorageConfig{ + Provider:{{printf "%q" .Storage}}, Root:"./var/files", PublicURL:"http://127.0.0.1:8080/storage/object", + Secure:true, MaxBytes:32<<20, ContentTypes:[]string{"text/plain","image/png","image/jpeg","application/pdf"}, + LinkTTL:5*time.Minute, RequestTimeout:30*time.Second, + {{if eq .Storage "s3"}}Region:"us-east-1",{{end}} + {{if eq .Storage "r2"}}Region:"auto",{{end}} + {{if eq .Storage "oss"}}Region:"cn-hangzhou",{{end}} + }{{end}}} +} + +// Load uses defaults -> strict YAML/JSON -> nonempty environment variables. +// Pass an empty path to use only defaults and environment. +func Load(path string)(Config,error) { + cfg:=Default() + var err error + if path!="" { cfg,err=config.DecodeFile(path,cfg);if err!=nil{return Config{},err} } + cfg.Server,err=config.ApplyEnv(cfg.Server,"") + if err!=nil{return Config{},err} + // A stream explicitly selected by environment overrides a file profile. + if output:=strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT"));output!="" && strings.TrimSpace(os.Getenv("GO_OMNI_LOG_FILE"))=="" { + cfg.Server.Log.File="";cfg.Server.Log.Rotation=logx.RotationConfig{} + } + {{if ne .Storage "none"}} + if err=cfg.Storage.applyEnv();err!=nil{return Config{},err} + {{end}} + {{if and .API (eq .Database "mysql")}}cfg.Database,err=database.ApplyEnv(cfg.Database,"");if err!=nil{return Config{},err}{{end}} + if err=cfg.Validate();err!=nil{return Config{},err} + return cfg,nil +} +func (c Config) Validate()error { + {{if and .API (eq .Database "mysql")}} + if err:=c.Database.Validate();err!=nil{return err} + if c.Database.Driver!="mysql" || c.Database.MaxOpenConns<1 {return errors.New("database requires driver mysql and a positive maxOpenConns")} + {{end}} + if err:=c.Server.Validate();err!=nil{return err} + if c.Server.Log.Format!="json" && c.Server.Log.Format!="text" {return errors.New("log.format must be json or text")} + switch strings.ToLower(c.Server.Log.Level) {case "debug","info","warn","warning","error":default:return errors.New("invalid log.level")} + {{if ne .Storage "none"}}return c.Storage.Validate(){{else}}return nil{{end}} +} + +{{if ne .Storage "none"}} +type StorageConfig struct { + Provider string `json:"provider"` + Root string `json:"root"` + PublicURL string `json:"publicUrl"` + Endpoint string `json:"endpoint"` + Region string `json:"region"` + Bucket string `json:"bucket"` + AccountID string `json:"accountId"` + Secure bool `json:"secure"` + MaxBytes int64 `json:"maxBytes"` + ContentTypes []string `json:"contentTypes"` + LinkTTL time.Duration `json:"linkTtl"` + RequestTimeout time.Duration `json:"requestTimeout"` + // Secrets are environment-only and cannot be populated by config files. + FilesToken string `json:"-"` + SigningKey string `json:"-"` + AccessKey string `json:"-"` + SecretKey string `json:"-"` + SessionToken string `json:"-"` +} +func (s StorageConfig) Policy()storage.UploadPolicy{return storage.UploadPolicy{MaxBytes:s.MaxBytes,ContentTypes:s.ContentTypes}} +func (s *StorageConfig) applyEnv()error { + values:=map[string]*string{ + "GO_OMNI_STORAGE_ROOT":&s.Root,"GO_OMNI_STORAGE_PUBLIC_URL":&s.PublicURL, + "GO_OMNI_STORAGE_ENDPOINT":&s.Endpoint,"GO_OMNI_STORAGE_REGION":&s.Region, + "GO_OMNI_STORAGE_BUCKET":&s.Bucket,"GO_OMNI_STORAGE_ACCOUNT_ID":&s.AccountID, + "GO_OMNI_STORAGE_SIGNING_KEY":&s.SigningKey,"GO_OMNI_FILES_TOKEN":&s.FilesToken, + "GO_OMNI_STORAGE_ACCESS_KEY":&s.AccessKey,"GO_OMNI_STORAGE_SECRET_KEY":&s.SecretKey, + "GO_OMNI_STORAGE_SESSION_TOKEN":&s.SessionToken, + } + for name,target:=range values {if value:=strings.TrimSpace(os.Getenv(name));value!=""{*target=value}} + for name,target:=range map[string]*time.Duration{"GO_OMNI_STORAGE_LINK_TTL":&s.LinkTTL,"GO_OMNI_STORAGE_REQUEST_TIMEOUT":&s.RequestTimeout} { + if value:=strings.TrimSpace(os.Getenv(name));value!="" {v,err:=time.ParseDuration(value);if err!=nil{return fmt.Errorf("%s must be a duration",name)};*target=v} + } + if value:=os.Getenv("GO_OMNI_STORAGE_MAX_BYTES");value!=""{v,err:=strconv.ParseInt(value,10,64);if err!=nil{return errors.New("GO_OMNI_STORAGE_MAX_BYTES must be an integer")};s.MaxBytes=v} + if value:=os.Getenv("GO_OMNI_STORAGE_SECURE");value!=""{v,err:=strconv.ParseBool(value);if err!=nil{return errors.New("GO_OMNI_STORAGE_SECURE must be a boolean")};s.Secure=v} + if value:=strings.TrimSpace(os.Getenv("GO_OMNI_STORAGE_CONTENT_TYPES"));value!="" {s.ContentTypes=strings.Split(value,",");for i:=range s.ContentTypes{s.ContentTypes[i]=strings.TrimSpace(s.ContentTypes[i])}} + return nil +} +func (s StorageConfig) Validate()error { + if s.Provider!={{printf "%q" .Storage}} {return errors.New("storage.provider must match the adapter selected at generation")} + if err:=s.Policy().Validate();err!=nil{return err} + if s.LinkTTLtime.Hour{return errors.New("storage.linkTtl must be 1s..1h")} + if s.RequestTimeout<=0 || s.RequestTimeout>5*time.Minute{return errors.New("storage.requestTimeout must be >0 and <=5m")} + {{if not .API}}if len(s.FilesToken)<32 || strings.ContainsAny(s.FilesToken," \t\r\n") {return errors.New("set GO_OMNI_FILES_TOKEN to a token of at least 32 bytes without whitespace")}{{end}} + {{if eq .Storage "local"}} + if s.Root=="" {return errors.New("storage.root is required")} + u,err:=url.Parse(s.PublicURL) + if err!=nil || u.Host=="" || u.Hostname()=="" || (u.Scheme!="http" && u.Scheme!="https") || u.User!=nil || u.RawQuery!="" || u.Fragment!="" || u.Path!="/storage/object" {return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object")} + if len(s.SigningKey)<32 {return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes")} + {{else}} + if s.Bucket=="" || s.Region=="" || s.AccessKey=="" || s.SecretKey=="" {return errors.New("storage bucket, region and GO_OMNI_STORAGE_ACCESS_KEY/SECRET_KEY are required")} + {{if eq .Storage "r2"}} + if !regexp.MustCompile(`^[a-fA-F0-9]{32}$`).MatchString(s.AccountID) || s.Region!="auto" || !s.Secure {return errors.New("R2 requires a 32-hex accountId, region auto and secure=true")} + if s.Endpoint!=""{return errors.New("this R2 profile derives endpoint from accountId; customize servicecontext for a jurisdiction-specific endpoint")} + {{else if eq .Storage "s3"}} + u,err:=url.Parse("https://"+s.Endpoint) + if err!=nil || s.Endpoint=="" || u.Hostname()=="" || u.User!=nil || u.Path!="" || u.RawQuery!="" || u.Fragment!="" {return errors.New("S3 endpoint must be host[:port] without scheme or path")} + {{else if eq .Storage "oss"}} + if s.Endpoint!="" {u,err:=url.Parse(s.Endpoint);if err!=nil || u.Scheme!="https" || u.Hostname()=="" || u.User!=nil || u.Path!="" || u.RawQuery!="" || u.Fragment!="" {return errors.New("OSS endpoint must be an HTTPS origin")}} + if !s.Secure{return errors.New("OSS requires secure=true")} + {{end}} + {{end}} + return nil +} +{{end}} diff --git a/cmd/go-omni/scaffold/templates/modules-settings_test.go.tmpl b/cmd/go-omni/scaffold/templates/modules-settings_test.go.tmpl new file mode 100644 index 0000000..3689990 --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-settings_test.go.tmpl @@ -0,0 +1,38 @@ +package settings + +import ( + "os" + "path/filepath" + "strings" + "testing" +) +func environment(t *testing.T){ + t.Helper() + for _,entry:=range os.Environ(){key,_,_:=strings.Cut(entry,"=");if strings.HasPrefix(key,"GO_OMNI_"){t.Setenv(key,"")}} + {{if ne .Storage "none"}} + t.Setenv("GO_OMNI_FILES_TOKEN",strings.Repeat("t",32)) + t.Setenv("GO_OMNI_STORAGE_SIGNING_KEY",strings.Repeat("s",32)) + t.Setenv("GO_OMNI_STORAGE_ACCESS_KEY","fixtureAccess") + t.Setenv("GO_OMNI_STORAGE_SECRET_KEY","fixtureSecret") + t.Setenv("GO_OMNI_STORAGE_BUCKET","testbucket") + {{if eq .Storage "s3"}}t.Setenv("GO_OMNI_STORAGE_ENDPOINT","127.0.0.1:9000"){{end}} + {{if eq .Storage "r2"}}t.Setenv("GO_OMNI_STORAGE_ACCOUNT_ID",strings.Repeat("a",32)){{end}} + {{end}} +} +func TestSettingsPrecedenceAndStrictness(t *testing.T){ + environment(t) + path:=filepath.Join(t.TempDir(),"config.yaml") + if err:=os.WriteFile(path,[]byte("server:\n address: '127.0.0.1:8123'\n log:\n level: debug\n"),0600);err!=nil{t.Fatal(err)} + t.Setenv("GO_OMNI_HTTP_ADDR","127.0.0.1:9123") + t.Setenv("GO_OMNI_LOG_OUTPUT","stdout") + cfg,err:=Load(path);if err!=nil{t.Fatal(err)} + if cfg.Server.Address!="127.0.0.1:9123" || cfg.Server.Log.Level!="debug" || cfg.Server.Log.Output!="stdout" || cfg.Server.Log.File!="" || cfg.Server.Log.Rotation.Enabled {t.Fatal("configuration precedence failed")} + for _,body:=range []string{"unexpected: true","server:\n slowRequestThreshold: -1s","server:\n log:\n format: invalid"{{if ne .Storage "none"}},"storage:\n filesToken: forbidden","storage:\n maxBytes: -1","storage:\n provider: unsupported"{{end}}}{ + if err:=os.WriteFile(path,[]byte(body),0600);err!=nil{t.Fatal(err)} + if _,err:=Load(path);err==nil{t.Fatal("invalid configuration accepted")} + } + {{if ne .Storage "none"}} + t.Setenv("GO_OMNI_FILES_TOKEN","") + if _,err:=Load("");err==nil{t.Fatal("missing file token accepted")} + {{end}} +} diff --git a/cmd/go-omni/scaffold/templates/modules-storage.go.tmpl b/cmd/go-omni/scaffold/templates/modules-storage.go.tmpl new file mode 100644 index 0000000..fa772ef --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-storage.go.tmpl @@ -0,0 +1,79 @@ +package servicecontext + +import ( + "context" + "errors" + {{if eq .Storage "local"}} + "git.yuntuo.hk/k2safe/go-omni/storage/local" + {{else}} + "net" + "net/http" + "time" + {{if eq .Storage "oss"}} + ossStore "git.yuntuo.hk/k2safe/go-omni/storage/oss" + ossSDK "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss" + "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials" + {{else}} + "git.yuntuo.hk/k2safe/go-omni/storage/s3" + "github.com/minio/minio-go/v7/pkg/credentials" + {{end}}{{end}} + "git.yuntuo.hk/k2safe/go-omni/storage" + "{{.Module}}/internal/settings" +) + +// InitStorage is called once, before serving requests. The composition root +// owns provider clients; business services receive only capability interfaces. +func (c *Context) InitStorage(ctx context.Context,cfg settings.StorageConfig)error { + if c==nil || ctx==nil{return errors.New("storage initialization requires context")} + c.closeMu.Lock();defer c.closeMu.Unlock() + if c.closed{return errors.New("service dependencies are closed")} + if err:=ctx.Err();err!=nil{return err} + if c.Files!=nil{return errors.New("storage already initialized")} + if err:=cfg.Validate();err!=nil{return err} + {{if eq .Storage "local"}} + files,err:=local.New(cfg.Root,cfg.MaxBytes) + if err!=nil{return err} + signer,err:=storage.NewUploadSigner(files,cfg.PublicURL,[]byte(cfg.SigningKey),cfg.Policy()) + if err!=nil{return err} + uploader,err:=storage.NewUploader(files,cfg.Policy()) + if err!=nil{return err} + c.Files,c.Uploads,c.Downloads,c.UploadLinks,c.FileHTTP=files,uploader,signer,signer,signer + {{else}} + // Dedicated bounded pool, never the process-wide default transport. + transport:=&http.Transport{ + Proxy:http.ProxyFromEnvironment, + DialContext:(&net.Dialer{Timeout:5*time.Second,KeepAlive:30*time.Second}).DialContext, + ForceAttemptHTTP2:true, MaxIdleConns:32,MaxIdleConnsPerHost:8,MaxConnsPerHost:32, + IdleConnTimeout:90*time.Second,TLSHandshakeTimeout:5*time.Second, + ResponseHeaderTimeout:cfg.RequestTimeout, + } + initialized:=false + defer func(){if !initialized{transport.CloseIdleConnections()}}() + {{if eq .Storage "oss"}} + sdkConfig:=ossSDK.LoadDefaultConfig().WithRegion(cfg.Region). + WithSignatureVersion(ossSDK.SignatureVersionV4). + WithAdditionalHeaders([]string{"content-length"}). + WithCredentialsProvider(credentials.NewStaticCredentialsProvider(cfg.AccessKey,cfg.SecretKey,cfg.SessionToken)). + WithHttpClient(&http.Client{Transport:transport,Timeout:cfg.RequestTimeout}) + if cfg.Endpoint!=""{sdkConfig.WithEndpoint(cfg.Endpoint)} + files,err:=ossStore.New(ossSDK.NewClient(sdkConfig),cfg.Bucket,cfg.MaxBytes) + {{else}} + creds:=credentials.NewStaticV4(cfg.AccessKey,cfg.SecretKey,cfg.SessionToken) + {{if eq .Storage "r2"}} + options,err:=s3.R2Config(cfg.AccountID,cfg.Bucket,creds) + if err!=nil{return err} + options.MaxBytes,options.Transport=cfg.MaxBytes,transport + {{else}} + options:=s3.Config{Endpoint:cfg.Endpoint,Region:cfg.Region,Bucket:cfg.Bucket,Secure:cfg.Secure,MaxBytes:cfg.MaxBytes,Credentials:creds,Transport:transport} + {{end}} + files,err:=s3.Open(options) + {{end}} + if err!=nil{return err} + uploader,err:=storage.NewUploader(files,cfg.Policy()) + if err!=nil{return err} + c.Files,c.Uploads,c.Downloads,c.UploadLinks=files,uploader,files,uploader + c.storageClose=func()error{transport.CloseIdleConnections();return nil} + initialized=true + {{end}} + return nil +} diff --git a/cmd/go-omni/scaffold/templates/modules-storage_test.go.tmpl b/cmd/go-omni/scaffold/templates/modules-storage_test.go.tmpl new file mode 100644 index 0000000..fc309ab --- /dev/null +++ b/cmd/go-omni/scaffold/templates/modules-storage_test.go.tmpl @@ -0,0 +1,38 @@ +package servicecontext + +import ( + "bytes" + "context" + "strings" + "testing" + + "git.yuntuo.hk/k2safe/go-omni/logx" + "{{.Module}}/internal/settings" +) +func TestStorageInitializationAndOwnership(t *testing.T){ + cfg:=settings.Default().Storage + cfg.FilesToken=strings.Repeat("t",32) + cfg.SigningKey=strings.Repeat("s",32) + cfg.Root=t.TempDir() + cfg.AccessKey,cfg.SecretKey,cfg.Bucket="fixtureAccess","fixtureSecret","testbucket" + {{if eq .Storage "s3"}}cfg.Endpoint="127.0.0.1:9000"{{end}} + {{if eq .Storage "r2"}}cfg.AccountID=strings.Repeat("a",32){{end}} + var output bytes.Buffer + logger,err:=logx.New(logx.Config{Writer:&output});if err!=nil{t.Fatal(err)};defer logger.Close() + c:=&Context{Logger:logger} + canceled,cancel:=context.WithCancel(context.Background());cancel() + if err=c.InitStorage(canceled,cfg);err==nil || c.Files!=nil{t.Fatal("canceled startup initialized storage")} + invalid:=cfg;invalid.FilesToken="" + if err=c.InitStorage(context.Background(),invalid);err==nil || c.Files!=nil{t.Fatal("invalid settings initialized storage")} + if err=c.InitStorage(context.Background(),cfg);err!=nil{t.Fatal(err)} + if c.Files==nil || c.Uploads==nil || c.Downloads==nil || c.UploadLinks==nil{t.Fatal("dependencies not assembled")} + if err=c.InitStorage(context.Background(),cfg);err==nil{t.Fatal("duplicate initialization accepted")} + cleanup:=c.storageClose;count:=0 + c.storageClose=func()error{count++;if cleanup!=nil{return cleanup()};return nil} + if err=c.Close();err!=nil{t.Fatal(err)};if err=c.Close();err!=nil{t.Fatal(err)} + logger.Info("still usable") + if count!=1 || !strings.Contains(output.String(),"still usable"){t.Fatal("resource ownership violated")} + closed:=&Context{Logger:logger} + if err=closed.Close();err!=nil{t.Fatal(err)} + if err=closed.InitStorage(context.Background(),cfg);err==nil || closed.Files!=nil{t.Fatal("closed context reopened resources")} +} diff --git a/cmd/go-omni/scaffold/version.go b/cmd/go-omni/scaffold/version.go index dbde2b9..b99ec0b 100644 --- a/cmd/go-omni/scaffold/version.go +++ b/cmd/go-omni/scaffold/version.go @@ -10,7 +10,19 @@ const frameworkModule = "git.yuntuo.hk/k2safe/go-omni" // Source builds can carry an unpublished VCS version (including +dirty). // Keep this fallback in sync with the next release; installed CLIs use their // downloaded module version, which also has a module checksum. -const defaultFrameworkVersion = "v0.6.0-preview" +const defaultFrameworkVersion = "v0.7.0-preview" + +// This target is not a claim that the release already exists. Source users +// select a local framework checkout until the module templates are published. +const modulesFrameworkVersion = "v0.7.0-preview" + +func installedFrameworkVersion() string { + info, _ := debug.ReadBuildInfo() + if info != nil && info.Main.Path == frameworkModule && info.Main.Sum != "" && validFrameworkVersion(info.Main.Version) { + return info.Main.Version + } + return "" +} var frameworkVersionPattern = regexp.MustCompile(`^v[01]\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)(-((0|[1-9][0-9]*|[0-9]*[A-Za-z-][0-9A-Za-z-]*)(\.(0|[1-9][0-9]*|[0-9]*[A-Za-z-][0-9A-Za-z-]*))*))?(\+[0-9A-Za-z-]+(\.[0-9A-Za-z-]+)*)?$`) diff --git a/config/config.go b/config/config.go index cb66d3f..698775d 100644 --- a/config/config.go +++ b/config/config.go @@ -24,14 +24,15 @@ type TLSConfig struct { // Config controls the default runtime behavior of an App. type Config struct { - Address string `json:"address"` - Transport string `json:"transport"` - TLS TLSConfig `json:"tls"` - RequestIDHeader string `json:"requestIdHeader"` - EnableLogger bool `json:"enableLogger"` - TrustedProxies []string `json:"trustedProxies"` - ShutdownTimeout time.Duration `json:"shutdownTimeout"` - Log logx.Config `json:"log"` + Address string `json:"address"` + Transport string `json:"transport"` + TLS TLSConfig `json:"tls"` + RequestIDHeader string `json:"requestIdHeader"` + EnableLogger bool `json:"enableLogger"` + SlowRequestThreshold time.Duration `json:"slowRequestThreshold"` + TrustedProxies []string `json:"trustedProxies"` + ShutdownTimeout time.Duration `json:"shutdownTimeout"` + Log logx.Config `json:"log"` } func Default() Config { @@ -46,6 +47,9 @@ func Default() Config { // Validate checks configuration values before an application starts. func (c Config) Validate() error { + if c.SlowRequestThreshold < 0 { + return &ConfigError{Field: "slowRequestThreshold", Message: "must be nonnegative"} + } if strings.TrimSpace(c.Address) == "" { return &ConfigError{Field: "address", Message: "is required"} } @@ -114,6 +118,11 @@ func WithLogger(enabled bool) Option { return func(cfg *Config) { cfg.EnableLogger = enabled } } +// WithSlowRequestThreshold enables access logs and warns after this duration. +func WithSlowRequestThreshold(threshold time.Duration) Option { + return func(cfg *Config) { cfg.SlowRequestThreshold = threshold; cfg.EnableLogger = true } +} + // WithTrustedProxies allows only these IPs/CIDRs to supply X-Forwarded-For. // An empty list disables forwarding-header trust. Invalid entries are rejected // by Config.Validate and panic when passed directly to App.New. @@ -126,15 +135,21 @@ func WithTrustedProxies(proxies ...string) Option { // environment. Later options can still override individual fields. func WithConfig(settings Config) Option { settings.TrustedProxies = append([]string(nil), settings.TrustedProxies...) + settings.Log.RedactKeys = append([]string(nil), settings.Log.RedactKeys...) return func(cfg *Config) { *cfg = settings cfg.TrustedProxies = append([]string(nil), settings.TrustedProxies...) + cfg.Log.RedactKeys = append([]string(nil), settings.Log.RedactKeys...) } } // WithLogConfig configures the logger used by the application runtime. func WithLogConfig(settings logx.Config) Option { - return func(cfg *Config) { cfg.Log = settings } + settings.RedactKeys = append([]string(nil), settings.RedactKeys...) + return func(cfg *Config) { + cfg.Log = settings + cfg.Log.RedactKeys = append([]string(nil), settings.RedactKeys...) + } } // WithShutdownTimeout controls how long RunContext waits for active requests. diff --git a/config/config_test.go b/config/config_test.go index 0e1e834..c7a2e85 100644 --- a/config/config_test.go +++ b/config/config_test.go @@ -14,6 +14,7 @@ func TestLoadFromEnvUsesDefaults(t *testing.T) { "GO_OMNI_LOG_LEVEL", "GO_OMNI_LOG_FORMAT", "GO_OMNI_LOG_OUTPUT", "GO_OMNI_LOG_FILE", "GO_OMNI_SERVICE", "GO_OMNI_ENVIRONMENT", "GO_OMNI_LOG_ADD_SOURCE", "GO_OMNI_SHUTDOWN_TIMEOUT", + "GO_OMNI_SLOW_REQUEST_THRESHOLD", } { t.Setenv(key, "") } @@ -80,6 +81,7 @@ func TestLoadFromEnvOverridesSettings(t *testing.T) { t.Setenv("GO_OMNI_ENVIRONMENT", "test") t.Setenv("GO_OMNI_LOG_ADD_SOURCE", "true") t.Setenv("GO_OMNI_SHUTDOWN_TIMEOUT", "3s") + t.Setenv("GO_OMNI_SLOW_REQUEST_THRESHOLD", "500ms") cfg, err := LoadFromEnv("") if err != nil { @@ -91,6 +93,9 @@ func TestLoadFromEnvOverridesSettings(t *testing.T) { if cfg.ShutdownTimeout != 3*time.Second || cfg.Log.Level != "debug" || cfg.Log.Format != "text" || cfg.Log.Output != "stdout" || cfg.Log.Service != "billing" || cfg.Log.Environment != "test" || !cfg.Log.AddSource { t.Fatalf("unexpected logger settings: %+v", cfg.Log) } + if cfg.SlowRequestThreshold != 500*time.Millisecond { + t.Fatal(cfg.SlowRequestThreshold) + } } func TestLoadFromEnvRejectsInvalidValues(t *testing.T) { @@ -142,3 +147,23 @@ func TestWithConfigAndWithLogConfig(t *testing.T) { t.Fatalf("log config was not replaced: %+v", cfg.Log) } } + +func TestApplyEnvPreservesProjectDefaults(t *testing.T) { + cfg := Default() + cfg.Address = "127.0.0.1:8123" + cfg.Log.RedactKeys = []string{"bankAccount"} + cfg.TrustedProxies = []string{"127.0.0.1"} + t.Setenv("OVERLAY_TEST_LOG_LEVEL", "debug") + got, err := ApplyEnv(cfg, "OVERLAY_TEST_") + if err != nil { + t.Fatal(err) + } + if got.Address != cfg.Address || got.Log.Level != "debug" { + t.Fatal(got) + } + got.Log.RedactKeys[0] = "changed" + got.TrustedProxies[0] = "192.0.2.1" + if cfg.Log.RedactKeys[0] != "bankAccount" || cfg.TrustedProxies[0] != "127.0.0.1" { + t.Fatal("caller-owned defaults mutated") + } +} diff --git a/config/env.go b/config/env.go index 7267680..8eb3782 100644 --- a/config/env.go +++ b/config/env.go @@ -16,6 +16,14 @@ func LoadFromEnv(prefix string) (Config, error) { return applyEnv(Default(), prefix) } +// ApplyEnv overlays nonempty environment values on a caller's defaults or file +// settings, then validates the result. Caller-owned slices are not retained. +func ApplyEnv(defaults Config, prefix string) (Config, error) { + defaults.TrustedProxies = append([]string(nil), defaults.TrustedProxies...) + defaults.Log.RedactKeys = append([]string(nil), defaults.Log.RedactKeys...) + return applyEnv(defaults, prefix) +} + func applyEnv(cfg Config, prefix string) (Config, error) { prefix = strings.TrimSpace(prefix) if prefix == "" { @@ -46,6 +54,12 @@ func applyEnv(cfg Config, prefix string) (Config, error) { if cfg.Log.AddSource, err = envBool(prefix+"LOG_ADD_SOURCE", cfg.Log.AddSource); err != nil { return Config{}, err } + if value := strings.TrimSpace(os.Getenv(prefix + "SLOW_REQUEST_THRESHOLD")); value != "" { + cfg.SlowRequestThreshold, err = time.ParseDuration(value) + if err != nil { + return Config{}, fmt.Errorf("%sSLOW_REQUEST_THRESHOLD must be a duration: %w", prefix, err) + } + } if value := strings.TrimSpace(os.Getenv(prefix + "SHUTDOWN_TIMEOUT")); value != "" { cfg.ShutdownTimeout, err = time.ParseDuration(value) if err != nil { diff --git a/config/file.go b/config/file.go index e29af26..4b98e79 100644 --- a/config/file.go +++ b/config/file.go @@ -182,6 +182,7 @@ func (r *Reloader) Snapshot() Config { defer r.mu.Unlock() cfg := r.current cfg.TrustedProxies = append([]string(nil), cfg.TrustedProxies...) + cfg.Log.RedactKeys = append([]string(nil), cfg.Log.RedactKeys...) return cfg } func (r *Reloader) Watch(ctx context.Context, interval time.Duration, onError func(error)) error { diff --git a/contract/README.md b/contract/README.md new file mode 100644 index 0000000..5e25a02 --- /dev/null +++ b/contract/README.md @@ -0,0 +1,30 @@ +# contract + +Generated API projects use this package to compile Go field tags once and bind +each request without depending on a specific HTTP server. `New[T]` validates a +struct and returns an immutable Binder. `ForType` provides the same compiler to +the CLI. `BodySchema`, `Parameters`, and `ResponseSchema` return detached OpenAPI +snapshots. `MustNew[T]` is intended for generated, already checked declarations. + +```go +type Request struct { + ID int64 `path:"id,range=[1:100]"` + Name string `json:"name" binding:"required,max=50"` +} + +var request = contract.MustNew[Request]() + +func Handle(c *httpx.Context) error { + var req Request + if err := request.Bind(c.Request(), httpx.RouteParams(c.Context()), &req); err != nil { + return err + } + return c.OK(req.Name) +} +``` + +The default body limit is 1 MiB. An optional positive final argument to Bind +overrides it. Application transport read timeouts still control slow clients. +See [API.md](../API.md) for source tags, validation rules, null/presence behavior, +strict decoding and supported type limits. This API leaves the existing +`Context.BindJSON/BindQuery/BindURI` behavior unchanged. diff --git a/contract/bind.go b/contract/bind.go new file mode 100644 index 0000000..a44f67e --- /dev/null +++ b/contract/bind.go @@ -0,0 +1,310 @@ +package contract + +import ( + "encoding/json" + "errors" + "fmt" + "io" + "math" + "mime" + "net/http" + "net/url" + "reflect" + "strconv" + + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" +) + +// Bind fills a fresh DTO transactionally. A JSON body, if declared, is required, +// strictly decoded, and limited to 1 MiB unless maxBytes is supplied. Query +// parameters never overwrite path/body/header values. form means URL query only. +func (b *Binder) Bind(r *http.Request, params map[string]string, target any, maxBytes ...int64) error { + if b == nil || r == nil { + return apierrors.Internal(errors.New("contract: nil binder or request")) + } + v := reflect.ValueOf(target) + if v.Kind() != reflect.Pointer || v.IsNil() || v.Elem().Type() != b.typ { + return apierrors.Internal(errors.New("contract: target type mismatch")) + } + limit := DefaultMaxBodyBytes + if len(maxBytes) > 1 || (len(maxBytes) == 1 && (maxBytes[0] <= 0 || maxBytes[0] == math.MaxInt64)) { + return apierrors.Internal(errors.New("contract: invalid body limit")) + } + if len(maxBytes) == 1 { + limit = maxBytes[0] + } + fresh := reflect.New(b.typ).Elem() + body := map[string]any{} + if len(b.root.schema.Properties) > 0 { + ct, _, err := mime.ParseMediaType(r.Header.Get("Content-Type")) + if err != nil || ct != "application/json" { + return apierrors.New(415, "UNSUPPORTED_MEDIA_TYPE", "Content-Type must be application/json") + } + if r.Body == nil { + return invalid(errors.New("body required")) + } + if r.ContentLength > limit { + return tooLarge() + } + reader := &io.LimitedReader{R: r.Body, N: limit + 1} + dec := json.NewDecoder(reader) + dec.UseNumber() + value, err := decodeValue(dec, 0) + if reader.N == 0 { + return tooLarge() + } + if err != nil { + var maxErr *http.MaxBytesError + if errors.As(err, &maxErr) { + return tooLarge() + } + return invalid(err) + } + _, err = dec.Token() + if reader.N == 0 { + return tooLarge() + } + if err != io.EOF { + return invalid(errors.New("exactly one JSON value required")) + } + var ok bool + body, ok = value.(map[string]any) + if !ok { + return invalid(errors.New("body must be an object")) + } + for key := range body { + if _, ok = b.root.schema.Properties[key]; !ok { + return invalid(fmt.Errorf("unknown body field %s", key)) + } + } + } + query := url.Values{} + if r.URL != nil { + var err error + query, err = url.ParseQuery(r.URL.RawQuery) + if err != nil { + return invalid(errors.New("malformed query")) + } + } + for _, f := range b.root.fields { + var value any + var present bool + if f.source == "json" { + value, present = body[f.name] + } else { + var raw []string + switch f.source { + case "path": + if s, ok := params[f.name]; ok { + raw = []string{s} + } + case "form": + raw = query[f.name] + case "header": + raw = r.Header.Values(f.name) + } + present = len(raw) > 0 + if present { + var err error + value, err = fromStrings(f.value, raw) + if err != nil { + return invalid(fmt.Errorf("%s: %w", f.name, err)) + } + } + } + if err := assignField(f, fresh.Field(f.index), value, present); err != nil { + return invalid(fmt.Errorf("%s: %w", f.name, err)) + } + } + v.Elem().Set(fresh) + return nil +} +func invalid(err error) error { + return apierrors.BadRequest("INVALID_REQUEST", "request parameters are invalid").WithCause(err) +} +func tooLarge() error { return apierrors.New(413, "BODY_TOO_LARGE", "request body exceeds limit") } + +func fromStrings(n *node, values []string) (any, error) { + n = base(n) + if n.typ.Kind() == reflect.Slice { + out := make([]any, 0, len(values)) + for _, s := range values { + v, err := parseScalar(s, base(n.elem).typ) + if err != nil { + return nil, err + } + out = append(out, v) + } + return out, nil + } + if len(values) != 1 { + return nil, errors.New("duplicate scalar parameter") + } + return parseScalar(values[0], n.typ) +} +func assignField(f field, dst reflect.Value, value any, present bool) error { + if !present { + if f.hasDefault { + value = f.fallback + } else if f.optional { + return nil + } else { + return errors.New("required field missing") + } + } + if err := setValue(f.value, dst, value); err != nil { + return err + } + return checkField(f, dst) +} +func setValue(n *node, dst reflect.Value, value any) error { + if value == nil { + if n.typ.Kind() == reflect.Pointer { + dst.SetZero() + return nil + } + return errors.New("null requires a pointer type") + } + if n.typ.Kind() == reflect.Pointer { + dst.Set(reflect.New(n.typ.Elem())) + return setValue(n.elem, dst.Elem(), value) + } + switch n.typ.Kind() { + case reflect.Struct: + object, ok := value.(map[string]any) + if !ok { + return errors.New("expected object") + } + for key := range object { + if _, ok = n.schema.Properties[key]; !ok { + return fmt.Errorf("unknown field %s", key) + } + } + for _, f := range n.fields { + item, exists := object[f.name] + if err := assignField(f, dst.Field(f.index), item, exists); err != nil { + return fmt.Errorf("%s: %w", f.name, err) + } + } + case reflect.Slice: + items, ok := value.([]any) + if !ok { + return errors.New("expected array") + } + dst.Set(reflect.MakeSlice(n.typ, len(items), len(items))) + for i, item := range items { + if err := setValue(n.elem, dst.Index(i), item); err != nil { + return fmt.Errorf("item %d: %w", i, err) + } + } + case reflect.Map: + items, ok := value.(map[string]any) + if !ok { + return errors.New("expected object") + } + dst.Set(reflect.MakeMapWithSize(n.typ, len(items))) + for key, item := range items { + v := reflect.New(n.typ.Elem()).Elem() + if err := setValue(n.elem, v, item); err != nil { + return fmt.Errorf("map value: %w", err) + } + dst.SetMapIndex(reflect.ValueOf(key).Convert(n.typ.Key()), v) + } + case reflect.String: + s, ok := value.(string) + if !ok { + return errors.New("expected string") + } + dst.SetString(s) + case reflect.Bool: + v, ok := value.(bool) + if !ok { + return errors.New("expected boolean") + } + dst.SetBool(v) + default: + num, ok := value.(json.Number) + if !ok { + return errors.New("expected number") + } + switch n.typ.Kind() { + case reflect.Float32, reflect.Float64: + v, err := strconv.ParseFloat(string(num), n.typ.Bits()) + if err != nil || math.IsInf(v, 0) || math.IsNaN(v) { + return errors.New("number out of range") + } + dst.SetFloat(v) + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + v, err := strconv.ParseUint(string(num), 10, n.typ.Bits()) + if err != nil { + return errors.New("invalid unsigned integer") + } + dst.SetUint(v) + default: + v, err := strconv.ParseInt(string(num), 10, n.typ.Bits()) + if err != nil { + return errors.New("invalid integer") + } + dst.SetInt(v) + } + } + return nil +} + +// Token decoding rejects duplicate keys at every level (encoding/json normally +// keeps the last value). Depth and body limits bound client-controlled work. +func decodeValue(dec *json.Decoder, depth int) (any, error) { + if depth > 64 { + return nil, errors.New("JSON nesting exceeds 64") + } + token, err := dec.Token() + if err != nil { + return nil, err + } + switch token { + case json.Delim('{'): + object := map[string]any{} + for dec.More() { + key, err := dec.Token() + if err != nil { + return nil, err + } + name, ok := key.(string) + if !ok { + return nil, errors.New("invalid object key") + } + if _, exists := object[name]; exists { + return nil, errors.New("duplicate JSON field") + } + value, err := decodeValue(dec, depth+1) + if err != nil { + return nil, err + } + object[name] = value + } + end, err := dec.Token() + if err != nil || end != json.Delim('}') { + return nil, errors.New("unterminated object") + } + return object, nil + case json.Delim('['): + items := []any{} + for dec.More() { + v, err := decodeValue(dec, depth+1) + if err != nil { + return nil, err + } + items = append(items, v) + } + end, err := dec.Token() + if err != nil || end != json.Delim(']') { + return nil, errors.New("unterminated array") + } + return items, nil + default: + if _, bad := token.(json.Delim); bad { + return nil, errors.New("unexpected JSON delimiter") + } + return token, nil + } +} diff --git a/contract/contract.go b/contract/contract.go new file mode 100644 index 0000000..89d0488 --- /dev/null +++ b/contract/contract.go @@ -0,0 +1,394 @@ +// Package contract compiles API field tags into a reusable request binder and +// OpenAPI schemas. Generated services and the CLI use the same compiler. +package contract + +import ( + "encoding/json" + "errors" + "fmt" + "reflect" + "regexp" + "strconv" + "strings" + + "git.yuntuo.hk/k2safe/go-omni/openapi" + "github.com/go-playground/validator/v10" +) + +const DefaultMaxBodyBytes int64 = 1 << 20 + +var wireName = regexp.MustCompile(`^[a-z][A-Za-z0-9]*$`) +var headerName = regexp.MustCompile(`^[A-Za-z][A-Za-z0-9-]*$`) +var valuesValidator = validator.New() + +// Binder is immutable after compilation and safe for concurrent requests. +type Binder struct { + root *node + typ reflect.Type +} +type node struct { + typ reflect.Type + schema *openapi.Schema + fields []field + elem *node +} +type field struct { + index int + name, source string + value *node + optional bool + fallback any + hasDefault bool + binding string +} + +func New[T any]() (*Binder, error) { return ForType(reflect.TypeOf((*T)(nil)).Elem()) } +func MustNew[T any]() *Binder { + b, err := New[T]() + if err != nil { + panic(err) + } + return b +} + +// ForType also lets generators check the exact runtime contract before writing files. +// The root must be a struct; recursive and custom-marshaling types are rejected. +func ForType(t reflect.Type) (*Binder, error) { + if t == nil || t.Kind() != reflect.Struct { + return nil, errors.New("contract: root must be a struct") + } + budget := 10000 + n, err := compile(t, true, map[reflect.Type]bool{}, 0, &budget) + if err != nil { + return nil, err + } + return &Binder{root: n, typ: t}, nil +} + +func compile(t reflect.Type, root bool, stack map[reflect.Type]bool, depth int, budget *int) (*node, error) { + *budget-- + if *budget < 0 { + return nil, errors.New("contract: schema expansion exceeds 10000 nodes") + } + if depth > 32 { + return nil, errors.New("contract: type nesting exceeds 32") + } + if stack[t] { + return nil, errors.New("contract: recursive types are not supported") + } + stack[t] = true + defer delete(stack, t) + n := &node{typ: t, schema: &openapi.Schema{}} + // Wire behavior must be described completely by these tags. + marshaler := reflect.TypeOf((*json.Marshaler)(nil)).Elem() + unmarshaler := reflect.TypeOf((*json.Unmarshaler)(nil)).Elem() + if t.Implements(marshaler) || reflect.PointerTo(t).Implements(marshaler) || t.Implements(unmarshaler) || reflect.PointerTo(t).Implements(unmarshaler) { + return nil, errors.New("contract: custom JSON marshalers are unsupported") + } + switch t.Kind() { + case reflect.Pointer: + e, err := compile(t.Elem(), false, stack, depth+1, budget) + if err != nil { + return nil, err + } + n.elem = e + *n.schema = *e.schema + n.schema.Nullable = true + case reflect.String: + n.schema.Type = "string" + case reflect.Bool: + n.schema.Type = "boolean" + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64, reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + n.schema.Type = "integer" + if t.Bits() <= 32 { + n.schema.Format = "int32" + } else { + n.schema.Format = "int64" + } + if t.Kind() >= reflect.Uint && t.Kind() <= reflect.Uint64 { + v := float64(0) + n.schema.Minimum = &v + if t.Bits() < 64 { + max := float64((uint64(1) << uint(t.Bits())) - 1) + n.schema.Maximum = &max + if t.Bits() == 32 { + n.schema.Format = "int64" + } + } else { + n.schema.Format = "" + } + } else if t.Bits() < 32 { + min, max := -float64(int64(1)< 0 { + n.schema.Enum = append(n.schema.Enum, nil) + } + if f.source == "path" && (f.optional || f.hasDefault || sf.Type.Kind() == reflect.Pointer) { + return f, errors.New("path fields must be required non-pointer values") + } + if f.hasDefault { + v := reflect.New(sf.Type).Elem() + if err = setValue(n, v, f.fallback); err != nil { + return f, err + } + if err = checkField(f, v); err != nil { + return f, fmt.Errorf("default violates validation: %w", err) + } + } + return f, nil +} + +// ParseTags strictly parses Go struct tags, including duplicate/unknown key checks +// performed by the compiler. It is also used to normalize generated Go JSON tags. +func ParseTags(raw string) (map[string]string, error) { + out := map[string]string{} + for strings.TrimSpace(raw) != "" { + raw = strings.TrimLeft(raw, " ") + i := strings.Index(raw, ":") + if i < 1 { + return nil, errors.New("invalid struct tag") + } + key := raw[:i] + raw = raw[i+1:] + if strings.ContainsAny(key, " \t\r\n\"`") || len(raw) == 0 || raw[0] != '"' { + return nil, errors.New("invalid struct tag") + } + end := 1 + for end < len(raw) { + if raw[end] == '\\' { + end += 2 + continue + } + if raw[end] == '"' { + break + } + end++ + } + if end >= len(raw) { + return nil, errors.New("unterminated struct tag") + } + v, err := strconv.Unquote(raw[:end+1]) + if err != nil { + return nil, err + } + if _, ok := out[key]; ok { + return nil, fmt.Errorf("duplicate tag %q", key) + } + out[key] = v + raw = raw[end+1:] + if raw != "" && raw[0] != ' ' { + return nil, errors.New("struct tags require spaces") + } + } + return out, nil +} + +func base(n *node) *node { + for n.typ.Kind() == reflect.Pointer { + n = n.elem + } + return n +} +func isNumber(k reflect.Kind) bool { + return k >= reflect.Int && k <= reflect.Float64 && k != reflect.Uintptr +} +func isScalar(k reflect.Kind) bool { return k == reflect.String || k == reflect.Bool || isNumber(k) } + +func (b *Binder) BodySchema() *openapi.Schema { + if len(b.root.schema.Properties) == 0 { + return nil + } + return copySchema(b.root.schema) +} + +// ResponseSchema rejects request-only fields to keep response docs honest. +func (b *Binder) ResponseSchema() (*openapi.Schema, error) { + for _, f := range b.root.fields { + if f.source != "json" { + return nil, errors.New("contract: response fields must use json tags") + } + } + return copySchema(b.root.schema), nil +} +func (b *Binder) Parameters() []openapi.Parameter { + var out []openapi.Parameter + for _, f := range b.root.fields { + if f.source == "json" { + continue + } + in := f.source + if in == "form" { + in = "query" + } + out = append(out, openapi.Parameter{Name: f.name, In: in, Required: !f.optional && !f.hasDefault, Schema: copySchema(f.value.schema)}) + } + return out +} +func copySchema(s *openapi.Schema) *openapi.Schema { + data, _ := json.Marshal(s) + var out openapi.Schema + _ = json.Unmarshal(data, &out) + return &out +} diff --git a/contract/contract_test.go b/contract/contract_test.go new file mode 100644 index 0000000..1afc91f --- /dev/null +++ b/contract/contract_test.go @@ -0,0 +1,244 @@ +package contract + +import ( + "encoding/json" + "net/http/httptest" + "reflect" + "strings" + "sync" + "testing" + + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" +) + +type address struct { + City string `json:"city" binding:"required,max=30"` +} +type mixed struct { + ID int64 `path:"userId,range=[1:100]"` + Page int `form:"page,default=1,range=[1:50]"` + Tags []string `form:"tag,optional" binding:"max=3"` + Client string `header:"X-Client"` + Name string `json:"name" binding:"required,max=20"` + Active bool `json:"active"` + Count *int `json:"count,optional"` + Contact address `json:"contact"` + Items []address `json:"items,optional"` + Labels map[string]address `json:"labels,optional"` +} + +const validBody = `{"name":"小明","active":false,"count":0,"contact":{"city":"杭州"}}` + +func bindMixed(body, query string) (mixed, error) { + b := MustNew[mixed]() + r := httptest.NewRequest("POST", "/users/7"+query, strings.NewReader(body)) + r.Header.Set("Content-Type", "application/json") + r.Header.Set("X-Client", "web") + var v mixed + return v, b.Bind(r, map[string]string{"userId": "7"}, &v) +} +func TestMixedSourcesAndPresence(t *testing.T) { + b := MustNew[mixed]() + r := httptest.NewRequest("POST", "/users/7?tag=a&tag=b&userId=99&name=query", strings.NewReader(validBody)) + r.Header.Set("Content-Type", "application/json; charset=utf-8") + r.Header.Set("X-Client", "web") + v := mixed{Name: "previous"} + if err := b.Bind(r, map[string]string{"userId": "7"}, &v); err != nil { + t.Fatal(err) + } + if v.ID != 7 || v.Page != 1 || v.Name != "小明" || v.Active || v.Count == nil || *v.Count != 0 || len(v.Tags) != 2 || v.Client != "web" || v.Contact.City != "杭州" { + t.Fatalf("bad binding: %+v", v) + } + r = httptest.NewRequest("POST", "/", strings.NewReader(strings.Replace(validBody, `,"count":0`, "", 1))) + r.Header.Set("Content-Type", "application/json") + r.Header.Set("X-Client", "web") + if err := b.Bind(r, map[string]string{"userId": "7"}, &v); err != nil || v.Count != nil { + t.Fatalf("missing != zero: %+v %v", v, err) + } +} +func TestRejectInvalidRequests(t *testing.T) { + cases := []struct{ name, body, query string }{ + {"missing", `{"name":"a","contact":{"city":"杭州"}}`, ""}, + {"empty", strings.Replace(validBody, `"小明"`, `""`, 1), ""}, + {"caseSensitive", strings.Replace(validBody, `"name"`, `"Name"`, 1), ""}, + {"unknown", strings.Replace(validBody, `"active":false`, `"active":false,"extra":1`, 1), ""}, + {"duplicate", strings.Replace(validBody, `"active":false`, `"active":false,"active":true`, 1), ""}, + {"nestedDuplicate", strings.Replace(validBody, `"city":"杭州"`, `"city":"杭州","city":"北京"`, 1), ""}, + {"nestedRequired", strings.Replace(validBody, `{"city":"杭州"}`, `{}`, 1), ""}, + {"nestedUnknown", strings.Replace(validBody, `{"city":"杭州"}`, `{"city":"杭州","other":0}`, 1), ""}, + {"trailing", validBody + ` {}`, ""}, + {"null", strings.Replace(validBody, `false`, `null`, 1), ""}, + {"numberOverflow", strings.Replace(validBody, `"count":0`, `"count":99999999999999999999`, 1), ""}, + {"arrayValidation", strings.TrimSuffix(validBody, "}") + `,"items":[{"city":""}]}`, ""}, + {"mapValidation", strings.TrimSuffix(validBody, "}") + `,"labels":{"home":{"city":""}}}`, ""}, + {"queryDuplicate", validBody, "?page=1&page=2"}, + {"queryDefaultNotOnEmpty", validBody, "?page="}, + {"range", validBody, "?page=51"}, + {"malformedQuery", validBody, "?page=%zz"}, + {"arrayLimit", validBody, "?tag=a&tag=b&tag=c&tag=d"}, + } + b := MustNew[mixed]() + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + r := httptest.NewRequest("POST", "/"+tc.query, strings.NewReader(tc.body)) + r.Header.Set("Content-Type", "application/json") + r.Header.Set("X-Client", "web") + v := mixed{Name: "unchanged"} + err := b.Bind(r, map[string]string{"userId": "7"}, &v) + if err == nil || apierrors.From(err).Status != 400 { + t.Fatalf("wanted 400: %v", err) + } + if v.Name != "unchanged" || v.Page != 0 { + t.Fatal("failed binding modified target") + } + }) + } +} +func TestBodyLimitsAndMediaType(t *testing.T) { + type request struct { + Name string `json:"name"` + } + b := MustNew[request]() + for _, tc := range []struct { + body, ct string + limit int64 + status int + }{ + {`{"name":"x"}`, "text/plain", 100, 415}, + {`{"name":"long"}`, "application/json", 10, 413}, + {`{"name":"x"}` + strings.Repeat(" ", 2048), "application/json", 1024, 413}, + {`{"name":"x"}`, "application/json", 12, 0}, + } { + r := httptest.NewRequest("POST", "/", strings.NewReader(tc.body)) + r.ContentLength = -1 + r.Header.Set("Content-Type", tc.ct) + var req request + err := b.Bind(r, nil, &req, tc.limit) + if tc.status == 0 { + if err != nil { + t.Fatal(err) + } + } else if err == nil || apierrors.From(err).Status != tc.status { + t.Fatalf("status %d wanted: %v", tc.status, err) + } + } +} +func TestSchemaUsesSameRules(t *testing.T) { + b := MustNew[mixed]() + s := b.BodySchema() + if len(s.Required) != 3 || s.Properties["count"].Nullable != true || s.Properties["name"].MaxLength == nil || *s.Properties["name"].MaxLength != 20 { + t.Fatalf("wrong schema: %+v", s) + } + encoded, err := json.Marshal(s) + if err != nil { + t.Fatal(err) + } + if !strings.Contains(string(encoded), `"additionalProperties":false`) { + t.Fatal(string(encoded)) + } + params := b.Parameters() + if len(params) != 4 || params[1].Required || params[1].Schema.Default != json.Number("1") { + t.Fatalf("wrong parameters: %+v", params) + } + // Returned documentation snapshots do not mutate concurrent binders. + s.Properties["name"].MaxLength = nil + if b.BodySchema().Properties["name"].MaxLength == nil { + t.Fatal("shared schema") + } + if _, err = b.ResponseSchema(); err == nil { + t.Fatal("request sources in response accepted") + } +} +func TestRulesAndConcurrentReuse(t *testing.T) { + type req struct { + Score float64 `json:"score,options=1.0|2,range=(0:2]"` + Email string `json:"email" binding:"email"` + Value *int `json:"value" binding:"required"` + } + b := MustNew[req]() + var wg sync.WaitGroup + for i := 0; i < 24; i++ { + wg.Add(1) + go func() { + defer wg.Done() + r := httptest.NewRequest("POST", "/", strings.NewReader(`{"score":1,"email":"a@example.com","value":3}`)) + r.Header.Set("Content-Type", "application/json") + var v req + if err := b.Bind(r, nil, &v); err != nil { + t.Error(err) + } + }() + } + wg.Wait() + for _, body := range []string{`{"score":0,"email":"a@example.com","value":1}`, `{"score":2,"email":"bad","value":1}`, `{"score":2,"email":"a@example.com","value":null}`, `{"score":2,"email":"a@example.com","value":0}`} { + r := httptest.NewRequest("POST", "/", strings.NewReader(body)) + r.Header.Set("Content-Type", "application/json") + var v req + if err := b.Bind(r, nil, &v); err == nil { + t.Fatal(body) + } + } +} +func TestInvalidContracts(t *testing.T) { + for _, tag := range []string{`json:"bad_name"`, `json:"a" form:"a"`, `json:"a,optionl"`, `json:"a,range=[5:2]"`, `json:"a,default=20,range=[1:10]"`, `json:"a,optional" binding:"required"`, `json:"a" binding:"unknown"`, `path:"a,optional"`, `json:"a" json:"b"`, `jsoon:"a"`, `json:"a,options=x"`} { + typ := reflect.StructOf([]reflect.StructField{{Name: "Value", Type: reflect.TypeOf(0), Tag: reflect.StructTag(tag)}}) + if _, err := ForType(typ); err == nil { + t.Fatalf("accepted %s", tag) + } + } +} + +func TestExactLargeIntegerConstraints(t *testing.T) { + type req struct { + Value int64 `json:"value,range=[:9007199254740992]"` + } + b := MustNew[req]() + for _, tc := range []struct { + body string + valid bool + }{{`{"value":9007199254740992}`, true}, {`{"value":9007199254740993}`, false}} { + r := httptest.NewRequest("POST", "/", strings.NewReader(tc.body)) + r.Header.Set("Content-Type", "application/json") + var v req + if err := b.Bind(r, nil, &v); (err == nil) != tc.valid { + t.Fatalf("%s: %v", tc.body, err) + } + } + type defaults struct { + Value uint64 `form:"value,default=18446744073709551615,options=18446744073709551615"` + } + d := MustNew[defaults]() + data, err := json.Marshal(d.Parameters()) + if err != nil || !strings.Contains(string(data), "18446744073709551615") { + t.Fatalf("lost precision: %s %v", data, err) + } +} + +func TestIntrinsicIntegerBounds(t *testing.T) { + type req struct { + Small uint8 `json:"small,range=[-1:500]"` + Wide uint64 `json:"wide"` + } + b := MustNew[req]() + s := b.BodySchema() + if *s.Properties["small"].Minimum != 0 || *s.Properties["small"].Maximum != 255 || s.Properties["wide"].Format != "" { + t.Fatalf("wrong integer bounds: %+v", s.Properties) + } +} + +func FuzzStrictBody(f *testing.F) { + for _, seed := range []string{validBody, `{}`, `{"name":"a","name":"b"}`, `[null]`, `{"x":[]}`} { + f.Add(seed) + } + binder := MustNew[mixed]() + f.Fuzz(func(t *testing.T, body string) { + if len(body) > 16384 { + t.Skip() + } + req := httptest.NewRequest("POST", "/", strings.NewReader(body)) + req.Header.Set("Content-Type", "application/json") + req.Header.Set("X-Client", "fuzz") + var value mixed + _ = binder.Bind(req, map[string]string{"userId": "1"}, &value, 16384) + }) +} diff --git a/contract/rules.go b/contract/rules.go new file mode 100644 index 0000000..997f62c --- /dev/null +++ b/contract/rules.go @@ -0,0 +1,259 @@ +package contract + +import ( + "encoding/json" + "errors" + "fmt" + "math" + "reflect" + "strconv" + "strings" + "unicode/utf8" + + "git.yuntuo.hk/k2safe/go-omni/openapi" +) + +func applyRange(s *openapi.Schema, value string) error { + if len(value) < 3 || (value[0] != '[' && value[0] != '(') || (value[len(value)-1] != ']' && value[len(value)-1] != ')') { + return errors.New("invalid range: use [min:max], (min:max), or mixed bounds") + } + lo, hi, ok := strings.Cut(value[1:len(value)-1], ":") + if !ok || (lo == "" && hi == "") { + return errors.New("invalid numeric range") + } + if lo != "" { + v, err := bound(lo) + if err != nil { + return err + } + if s.Minimum == nil || v > *s.Minimum || (v == *s.Minimum && value[0] == '(') { + s.Minimum = &v + s.ExclusiveMinimum = value[0] == '(' + } + } + if hi != "" { + v, err := bound(hi) + if err != nil { + return err + } + if s.Maximum == nil || v < *s.Maximum || (v == *s.Maximum && value[len(value)-1] == ')') { + s.Maximum = &v + s.ExclusiveMaximum = value[len(value)-1] == ')' + } + } + return checkBounds(s) +} +func bound(s string) (float64, error) { + v, err := strconv.ParseFloat(s, 64) + if err != nil || math.IsInf(v, 0) || math.IsNaN(v) || math.Abs(v) > 1<<53 { + return 0, errors.New("constraint bound must be finite and within ±2^53") + } + return v, nil +} +func checkBounds(s *openapi.Schema) error { + if s.Minimum != nil && s.Maximum != nil && (*s.Minimum > *s.Maximum || (*s.Minimum == *s.Maximum && (s.ExclusiveMinimum || s.ExclusiveMaximum))) { + return errors.New("empty numeric range") + } + if s.MinLength != nil && s.MaxLength != nil && *s.MinLength > *s.MaxLength { + return errors.New("empty string length range") + } + if s.MinItems != nil && s.MaxItems != nil && *s.MinItems > *s.MaxItems { + return errors.New("empty array length range") + } + return nil +} +func applyBinding(f *field) error { + if f.binding == "" { + return nil + } + n := base(f.value) + k := n.typ.Kind() + s := f.value.schema + seen := map[string]bool{} + for _, rule := range strings.Split(f.binding, ",") { + name, value, _ := strings.Cut(rule, "=") + if seen[name] { + return fmt.Errorf("duplicate binding rule %s", name) + } + seen[name] = true + switch name { + case "required": + if rule != name || f.optional || f.hasDefault { + return errors.New("required conflicts with optional/default") + } + if !isScalar(k) { + return errors.New("binding required supports scalars; other fields are already required by presence") + } + s.Nullable = false + switch k { + case reflect.String: + one := 1 + s.MinLength = &one + case reflect.Bool: + s.Not = &openapi.Schema{Enum: []any{false}} + default: + s.Not = &openapi.Schema{Enum: []any{0}} + } + case "email": + if rule != name || k != reflect.String { + return errors.New("email requires a string") + } + s.Format = "email" + case "min", "max", "len": + if k == reflect.String || k == reflect.Slice { + v, err := strconv.Atoi(value) + if err != nil || v < 0 { + return errors.New("length constraints must be nonnegative integers") + } + if k == reflect.String { + if name == "min" || name == "len" { + if s.MinLength == nil || v > *s.MinLength { + s.MinLength = &v + } + } + if name == "max" || name == "len" { + s.MaxLength = &v + } + } else { + if name == "min" || name == "len" { + s.MinItems = &v + } + if name == "max" || name == "len" { + s.MaxItems = &v + } + } + } else if isNumber(k) && name != "len" { + v, err := bound(value) + if err != nil { + return err + } + if name == "min" { + if s.Minimum == nil || v > *s.Minimum { + s.Minimum = &v + s.ExclusiveMinimum = false + } + } else { + if s.Maximum == nil || v < *s.Maximum { + s.Maximum = &v + s.ExclusiveMaximum = false + } + } + } else { + return fmt.Errorf("%s requires a number, string or slice", name) + } + default: + return fmt.Errorf("unsupported binding rule %q; use required,min,max,len,email or source-tag options/range", name) + } + } + return checkBounds(s) +} + +func parseScalar(raw string, t reflect.Type) (any, error) { + switch t.Kind() { + case reflect.String: + return raw, nil + case reflect.Bool: + if raw == "true" { + return true, nil + } + if raw == "false" { + return false, nil + } + return nil, errors.New("expected true or false") + case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64: + v, err := strconv.ParseInt(raw, 10, t.Bits()) + if err != nil { + return nil, errors.New("invalid integer") + } + return json.Number(strconv.FormatInt(v, 10)), nil + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + v, err := strconv.ParseUint(raw, 10, t.Bits()) + if err != nil { + return nil, errors.New("invalid unsigned integer") + } + return json.Number(strconv.FormatUint(v, 10)), nil + case reflect.Float32, reflect.Float64: + v, err := strconv.ParseFloat(raw, t.Bits()) + if err != nil || math.IsInf(v, 0) || math.IsNaN(v) { + return nil, errors.New("invalid finite number") + } + return json.Number(strconv.FormatFloat(v, 'g', -1, t.Bits())), nil + } + return nil, errors.New("expected scalar") +} + +func checkField(f field, v reflect.Value) error { + if v.Kind() == reflect.Pointer { + if v.IsNil() { + if strings.Contains(","+f.binding+",", ",required,") { + return errors.New("must not be null") + } + return nil + } + v = v.Elem() + } + if strings.Contains(","+f.binding+",", ",required,") && v.IsZero() { + return errors.New("must not be zero or empty") + } + s := f.value.schema + if s.Format == "email" && valuesValidator.Var(v.Interface(), "email") != nil { + return errors.New("invalid email") + } + if len(s.Enum) > 0 { + found := false + for _, item := range s.Enum { + option := reflect.New(v.Type()).Elem() + if setValue(base(f.value), option, item) == nil && reflect.DeepEqual(option.Interface(), v.Interface()) { + found = true + break + } + } + if !found { + return errors.New("not an allowed option") + } + } + if isNumber(v.Kind()) { + var number float64 + switch v.Kind() { + case reflect.Float32, reflect.Float64: + number = v.Float() + case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64: + number = float64(v.Uint()) + if v.Uint() > 1<<53 { + number = math.Inf(1) + } + default: + number = float64(v.Int()) + if v.Int() > 1<<53 { + number = math.Inf(1) + } else if v.Int() < -(1 << 53) { + number = math.Inf(-1) + } + } + if s.Minimum != nil && (number < *s.Minimum || (s.ExclusiveMinimum && number == *s.Minimum)) { + return errors.New("below minimum") + } + if s.Maximum != nil && (number > *s.Maximum || (s.ExclusiveMaximum && number == *s.Maximum)) { + return errors.New("above maximum") + } + } + if v.Kind() == reflect.String { + size := utf8.RuneCountInString(v.String()) + if s.MinLength != nil && size < *s.MinLength { + return errors.New("string too short") + } + if s.MaxLength != nil && size > *s.MaxLength { + return errors.New("string too long") + } + } + if v.Kind() == reflect.Slice { + size := v.Len() + if s.MinItems != nil && size < *s.MinItems { + return errors.New("too few items") + } + if s.MaxItems != nil && size > *s.MaxItems { + return errors.New("too many items") + } + } + return nil +} diff --git a/database/configOverlay_test.go b/database/configOverlay_test.go new file mode 100644 index 0000000..b50fbe7 --- /dev/null +++ b/database/configOverlay_test.go @@ -0,0 +1,35 @@ +package database + +import ( + "encoding/json" + "strings" + "testing" + "time" +) + +func TestApplyEnvPreservesFilePoolAndHidesDSN(t *testing.T) { + cfg := DefaultConfig() + cfg.MaxOpenConns = 3 + cfg.MaxIdleConns = 1 + cfg.PingTimeout = time.Second + t.Setenv("OMNI_TEST_DSN", "user:secret@tcp(localhost:3306)/test") + t.Setenv("OMNI_TEST_MAX_IDLE_CONNS", "0") + got, err := ApplyEnv(cfg, "OMNI_TEST_") + if err != nil { + t.Fatal(err) + } + if got.MaxOpenConns != 3 || got.MaxIdleConns != 0 || got.PingTimeout != time.Second { + t.Fatalf("lost file defaults: %+v", got) + } + data, err := json.Marshal(got) + if err != nil { + t.Fatal(err) + } + if strings.Contains(string(data), "secret") || !strings.Contains(string(data), "maxOpenConns") { + t.Fatalf("unsafe config encoding: %s", data) + } + t.Setenv("OMNI_TEST_MAX_OPEN_CONNS", "bad") + if _, err = ApplyEnv(cfg, "OMNI_TEST_"); err == nil { + t.Fatal("invalid override accepted") + } +} diff --git a/database/database.go b/database/database.go index 9986239..8e1f080 100644 --- a/database/database.go +++ b/database/database.go @@ -14,13 +14,13 @@ import ( ) type Config struct { - Driver string - DSN string - MaxOpenConns int - MaxIdleConns int - ConnMaxLifetime time.Duration - ConnMaxIdleTime time.Duration - PingTimeout time.Duration + Driver string `json:"driver"` + DSN string `json:"-"` + MaxOpenConns int `json:"maxOpenConns"` + MaxIdleConns int `json:"maxIdleConns"` + ConnMaxLifetime time.Duration `json:"connMaxLifetime"` + ConnMaxIdleTime time.Duration `json:"connMaxIdleTime"` + PingTimeout time.Duration `json:"pingTimeout"` } func DefaultConfig() Config { @@ -35,16 +35,25 @@ func DefaultConfig() Config { // ConfigFromEnv reads GO_OMNI_DB_* style settings. DSN is required. func ConfigFromEnv(prefix string) (Config, error) { + return ApplyEnv(DefaultConfig(), prefix) +} + +// ApplyEnv overlays nonempty environment settings on file/default configuration. +// DSN is environment-only when Config is decoded from JSON or YAML. +func ApplyEnv(cfg Config, prefix string) (Config, error) { prefix = strings.TrimSpace(prefix) if prefix == "" { prefix = "GO_OMNI_DB_" } - cfg := DefaultConfig() - cfg.Driver = strings.TrimSpace(os.Getenv(prefix + "DRIVER")) + if value := strings.TrimSpace(os.Getenv(prefix + "DRIVER")); value != "" { + cfg.Driver = value + } if cfg.Driver == "" { cfg.Driver = "mysql" } - cfg.DSN = strings.TrimSpace(os.Getenv(prefix + "DSN")) + if value := strings.TrimSpace(os.Getenv(prefix + "DSN")); value != "" { + cfg.DSN = value + } if cfg.DSN == "" { return Config{}, fmt.Errorf("%sDSN is required", prefix) } diff --git a/examples/contracts/README.md b/examples/contracts/README.md new file mode 100644 index 0000000..de5320d --- /dev/null +++ b/examples/contracts/README.md @@ -0,0 +1,15 @@ +# 接口定义示例 + +`user.api` 是入口，导入 `shared.api`。包含 JSON/path/query/header 混合请求、分页与重复 query 参数、嵌套对象、optional 指针、user 与 admin/user 分组以及 Auth/Audit 命名中间件。 + +在 go-omni 根目录执行： + +```sh +go run ./cmd/go-omni api validate --api examples/contracts/user.api +go run ./cmd/go-omni api generate --api examples/contracts/user.api \ + --dir ../userService --module example.com/userService --framework-dir . +``` + +项目中补充 `internal/service` 的业务实现和 `internal/middleware` 的 Auth/Audit hook，再启动服务。初始占位实现返回 501。 + +需要立即能返回成功结果的最小项目，用 `go-omni api new` 创建问候示例。完整语法、目录、更新策略见 [API.md](../../API.md)。 diff --git a/examples/contracts/shared.api b/examples/contracts/shared.api new file mode 100644 index 0000000..dff9942 --- /dev/null +++ b/examples/contracts/shared.api @@ -0,0 +1,11 @@ +syntax = "v1" + +type Address { + City string `json:"city" binding:"required,max=50"` +} + +type UserResp { + UserID int64 `json:"userId"` + Name string `json:"name"` + Active bool `json:"active"` +} diff --git a/examples/contracts/user.api b/examples/contracts/user.api new file mode 100644 index 0000000..1e763fb --- /dev/null +++ b/examples/contracts/user.api @@ -0,0 +1,58 @@ +syntax = "v1" + +info ( + title: "User API" + version: "1.0.0" +) + +import "shared.api" + +type ( + UpdateUserReq { + UserID int64 `path:"userId,range=[1:1000000]"` + Notify bool `form:"notify,default=false"` + Client string `header:"X-Client,optional"` + Name string `json:"name" binding:"required,max=50"` + Active bool `json:"active"` + Age *int `json:"age,optional,range=[0:150]"` + Address *Address `json:"address,optional"` + } + ListUsersReq { + Page int `form:"page,default=1,range=[1:1000]"` + PageSize int `form:"pageSize,default=20,range=[1:100]"` + Tags []string `form:"tag,optional" binding:"max=10"` + } + ListUsersResp { + Items []UserResp `json:"items"` + Page int `json:"page"` + Total int64 `json:"total"` + } + DeleteUserReq { + UserID int64 `path:"userId,range=[1:1000000]"` + } +) + +@server ( + group: user + prefix: /api/v1 +) +service user-api { + @doc "Update a user" + @handler updateUser + put /users/:userId (UpdateUserReq) returns (UserResp) + + @doc "List users" + @handler listUsers + get /users (ListUsersReq) returns (ListUsersResp) +} + +@server ( + group: admin/user + prefix: /admin/v1 + middleware: Auth,Audit +) +service user-api { + @doc "Delete a user; business-owned authentication and auditing required" + @handler deleteUser + delete /users/:userId (DeleteUserReq) +} diff --git a/examples/userService/.gitignore b/examples/userService/.gitignore new file mode 100644 index 0000000..3da1ea2 --- /dev/null +++ b/examples/userService/.gitignore @@ -0,0 +1,6 @@ +/server +/var/ +.env +.env.* +!.env.example +/.goOmni/api.lock diff --git a/examples/userService/.goOmni/apiManifest.json b/examples/userService/.goOmni/apiManifest.json new file mode 100644 index 0000000..4bd4240 --- /dev/null +++ b/examples/userService/.goOmni/apiManifest.json @@ -0,0 +1,19 @@ +{ + "profile": { + "database": "mysql", + "storage": "local", + "logOutput": "file", + "logFormat": "json" + }, + "version": 1, + "module": "example.com/userService", + "entry": "api/user.api", + "generated": { + "docs/openapi.json": "0675e4eb1278b36828aa31b12587b0d33d2f384112f67a43179eb72b52c3a0ea", + "internal/handler/openapiGen.go": "8ddc60c338630db915b9f074ade776f526eabf60fc99b33e425359fd415c4926", + "internal/handler/routesGen.go": "7f45030e7eb9f055c4b4d2ff12ad855d7c726aad4aee63c5b3c3f97559a64257", + "internal/handler/user/handlersGen.go": "06f3c2798ff323812ff3eb2d8b933a569af3ce1b86a67b66036374add98c0d7e", + "internal/middleware/registryGen.go": "a6122a5453dedc6a4415860108814c50e155128cfe5412b39dd33ce2d01b4b00", + "internal/types/typesGen.go": "3ab2ce9ffe467c3ea53ac3d1fec283ecebc87cef57fd592d08dca991fd964876" + } +} diff --git a/examples/userService/README.md b/examples/userService/README.md new file mode 100644 index 0000000..fd34183 --- /dev/null +++ b/examples/userService/README.md @@ -0,0 +1,79 @@ +# API 驱动的用户管理示例 + +这个独立 Go module 从 `api/user.api` 生成，随后在开发者拥有的文件中实现业务。 +使用 MySQL 持久化、事务审计、环境变量 Token 鉴权、文件日志和 local 存储；业务层只接收标准 context 和基础模块接口。 + +## 启动 + +先准备空的业务数据库，再设置环境变量（替换占位值，密钥不写进配置或 Git）： + +```sh +export GO_OMNI_DB_DSN='user:password@tcp(127.0.0.1:3306)/user_service?parseTime=true' +export USER_SERVICE_API_TOKEN='<至少32字节的随机值>' +export GO_OMNI_STORAGE_SIGNING_KEY='<另一个至少32字节的随机值>' +export GO_OMNI_STORAGE_PUBLIC_URL='http://127.0.0.1:8080/storage/object' +go mod tidy +# 在本目录运行与框架版本一致的 CLI；迁移必须显式执行。 +go-omni migrate --dir migrations --apply +go run ./cmd/server +``` + +源码示例的 go.mod 使用 `replace ... => ../..`。复制到独立仓库时，删除这条 replace， +保留已发布的固定依赖版本，再执行 go mod tidy。远程版本验证脚本会自动检查没有本地替换。 +`GO_OMNI_TRANSPORT=native` 可切换为 native，业务代码无需调整。 + +配置是默认值 → etc/config.yaml → 环境变量。日志默认写入 var/log/server.log 并轮转， +`GO_OMNI_LOG_OUTPUT=stderr` 可以切换为标准错误输出。MySQL 默认最多 10 条连接。 +应用检查 users 和 user_audit 表，缺失时拒绝启动，不自动改表。 + +## 接口 + +除健康检查、OpenAPI 和持有效签名的对象下载外，业务接口需要 `Authorization: Bearer `。 +这个 Token 代表示例的管理员；可通过同一个命名中间件槽位替换为 JWT/OIDC 和业务授权策略。 +框架不会推断租户模型或自动给接口放行。 + +| 接口 | 行为 | +| --- | --- | +| POST /api/v1/users | 创建，返回 201；邮箱规范化并唯一 | +| GET /api/v1/users/:id | 查询，缺失返回 404 | +| GET /api/v1/users?page=1&pageSize=20 | 分页，pageSize 最大 100，空结果返回 [] | +| PUT /api/v1/users/:id | 完整更新；邮箱冲突返回 409 | +| DELETE /api/v1/users/:id | 删除，返回 204 | +| POST /api/v1/users/:id/export | 私有文本导出，返回 5 分钟下载链接 | +| GET /health/ready | 数据库不可用时返回 503，恢复后重新就绪 | +| GET /openapi.json | 与 docs/openapi.json 一致 | + +参数不合法返回 400，认证失败 401，邮箱重复 409，内部故障只返回通用 500。 +OpenAPI 的业务数据类型来自契约；401/404/409 的业务错误码见上表。 + +创建、更新、删除都会在**同一个 MySQL 事务**写入 user_audit；审计写入失败时业务变更一起回滚。 +事务提交后才写成功业务日志，requestId 同时出现在 HTTP 响应、业务日志和审计记录。 +数据库异常日志不记录 SQL、DSN 或用户输入。 + +文件导出演示 `deps.Uploads` 和 `deps.Downloads`。对象使用随机 key，签名 URL 不进入日志。 +签名过期只限制下载，不删除文件；实际项目应给 exports/ 配置保留期限或清理任务。 +只签发下载链接，不对外开放任意对象上传或签名接口。 + +## 开发与验证 + +```sh +go-omni api generate --api api/user.api --dir . +go-omni api generate --api api/user.api --dir . --check +go test ./... +``` + +Gen.go 和 docs/openapi.json 由生成器维护；Service、Repository、迁移、鉴权、配置和 main 保留。 +修改 .api 的类型/方法签名后，需要同步修改自己的业务实现并运行测试。 + +在框架仓库运行： + +```sh +python3 scripts/verifyAPIWorkflow.py +# 已发布版本：全新 module cache 安装 CLI，无本地 replace 消费框架。 +python3 scripts/verifyAPIWorkflow.py --version v0.7.0-preview \ + --report performance/apiWorkflowRemote.json +``` + +验收创建只绑定 localhost 的临时 MySQL 容器（1 CPU / 768 MiB），使用随机密钥，结束时删除。 +覆盖 nethttp/native 的 CRUD、校验、鉴权、事务错误/异常/取消回滚、数据库停机恢复、 +重启持久化、签名文件下载、文件日志与优雅退出。不会连接业务服务器。 diff --git a/examples/userService/api/user.api b/examples/userService/api/user.api new file mode 100644 index 0000000..b16c1bf --- /dev/null +++ b/examples/userService/api/user.api @@ -0,0 +1,60 @@ +syntax = "v1" +info ( + title: "User service reference" + version: "1.0.0" +) +type ( + UserResp { + ID int64 `json:"id"` + Name string `json:"name"` + Email string `json:"email"` + CreatedAt string `json:"createdAt"` + UpdatedAt string `json:"updatedAt"` + } + CreateUserReq { + Name string `json:"name" binding:"required,max=100"` + Email string `json:"email" binding:"required,email,max=254"` + } + UserIDReq { + ID int64 `path:"id,range=[1:9007199254740991]"` + } + UpdateUserReq { + ID int64 `path:"id,range=[1:9007199254740991]"` + Name string `json:"name" binding:"required,max=100"` + Email string `json:"email" binding:"required,email,max=254"` + } + ListUsersReq { + Page int `form:"page,default=1,range=[1:1000000]"` + PageSize int `form:"pageSize,default=20,range=[1:100]"` + } + UserPageResp { + Items []UserResp `json:"items"` + Total int `json:"total"` + Page int `json:"page"` + PageSize int `json:"pageSize"` + } + ExportResp { + DownloadURL string `json:"downloadUrl"` + } +) +@server ( + group: user + prefix: /api/v1 + middleware: Auth +) +service user-api { + @handler createUser + @status 201 + post /users (CreateUserReq) returns (UserResp) + @handler getUser + get /users/:id (UserIDReq) returns (UserResp) + @handler listUsers + get /users (ListUsersReq) returns (UserPageResp) + @handler updateUser + put /users/:id (UpdateUserReq) returns (UserResp) + @handler deleteUser + delete /users/:id (UserIDReq) + @doc "Export one user as a private text object with a short-lived download link" + @handler exportUser + post /users/:id/export (UserIDReq) returns (ExportResp) +} diff --git a/examples/userService/cmd/server/main.go b/examples/userService/cmd/server/main.go new file mode 100644 index 0000000..df67594 --- /dev/null +++ b/examples/userService/cmd/server/main.go @@ -0,0 +1,58 @@ +package main + +import ( + "context" + "errors" + "example.com/userService/internal/handler" + "example.com/userService/internal/servicecontext" + "example.com/userService/internal/settings" + "flag" + omni "git.yuntuo.hk/k2safe/go-omni" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/logx" + "log" + "os" + "os/signal" + "syscall" +) + +func main() { + path := flag.String("config", "etc/config.yaml", "configuration file; empty uses defaults and environment") + flag.Parse() + if err := run(*path); err != nil { + log.Fatal(err) + } +} +func run(path string) (runErr error) { + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + cfg, err := settings.Load(path) + if err != nil { + return err + } + logger, err := logx.New(cfg.Server.Log) + if err != nil { + return err + } + defer func() { runErr = errors.Join(runErr, logger.Close()) }() + logx.SetDefault(logger) + deps, err := servicecontext.New(ctx, logger, cfg) + if err != nil { + return err + } + defer func() { runErr = errors.Join(runErr, deps.Close()) }() + app := omni.New(omni.WithConfig(cfg.Server)) + app.OnShutdown(logger.Close) + app.OnShutdown(deps.Close) + for _, method := range []string{"GET", "HEAD", "PUT"} { + app.Handle(method, "/storage/object", func(c *httpx.Context) error { deps.FileHTTP.ServeHTTP(c.Raw(), c.Request()); return nil }) + } + if err = handler.RegisterRoutes(app, deps); err != nil { + return err + } + app.GET("/openapi.json", handler.OpenAPI) + app.GET("/health/live", deps.Health.LivenessHandler()) + app.GET("/health/ready", deps.Health.ReadinessHandler()) + logger.InfoContext(ctx, "service.starting", "address", cfg.Server.Address) + return app.RunContext(ctx, cfg.Server.Address) +} diff --git a/examples/userService/docs/openapi.json b/examples/userService/docs/openapi.json new file mode 100644 index 0000000..82b65b8 --- /dev/null +++ b/examples/userService/docs/openapi.json @@ -0,0 +1,1249 @@ +{ + "components": { + "securitySchemes": { + "bearerAuth": { + "scheme": "bearer", + "type": "http" + } + } + }, + "info": { + "title": "User service reference", + "version": "1.0.0" + }, + "openapi": "3.0.3", + "paths": { + "/api/v1/users": { + "get": { + "operationId": "user_ListUsers", + "parameters": [ + { + "in": "query", + "name": "page", + "required": false, + "schema": { + "default": 1, + "format": "int64", + "maximum": 1000000, + "minimum": 1, + "type": "integer" + } + }, + { + "in": "query", + "name": "pageSize", + "required": false, + "schema": { + "default": 20, + "format": "int64", + "maximum": 100, + "minimum": 1, + "type": "integer" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "data": { + "additionalProperties": false, + "properties": { + "items": { + "items": { + "additionalProperties": false, + "properties": { + "createdAt": { + "type": "string" + }, + "email": { + "type": "string" + }, + "id": { + "format": "int64", + "type": "integer" + }, + "name": { + "type": "string" + }, + "updatedAt": { + "type": "string" + } + }, + "required": [ + "id", + "name", + "email", + "createdAt", + "updatedAt" + ], + "type": "object" + }, + "type": "array" + }, + "page": { + "format": "int64", + "type": "integer" + }, + "pageSize": { + "format": "int64", + "type": "integer" + }, + "total": { + "format": "int64", + "type": "integer" + } + }, + "required": [ + "items", + "total", + "page", + "pageSize" + ], + "type": "object" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message", + "data" + ], + "type": "object" + } + } + }, + "description": "Success" + }, + "400": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "413": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "415": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "500": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "501": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + } + }, + "summary": "", + "x-omni-middleware": [ + "Auth" + ] + }, + "post": { + "operationId": "user_CreateUser", + "requestBody": { + "content": { + "application/json": { + "schema": { + "additionalProperties": false, + "properties": { + "email": { + "format": "email", + "maxLength": 254, + "minLength": 1, + "type": "string" + }, + "name": { + "maxLength": 100, + "minLength": 1, + "type": "string" + } + }, + "required": [ + "name", + "email" + ], + "type": "object" + } + } + }, + "required": true + }, + "responses": { + "201": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "data": { + "additionalProperties": false, + "properties": { + "createdAt": { + "type": "string" + }, + "email": { + "type": "string" + }, + "id": { + "format": "int64", + "type": "integer" + }, + "name": { + "type": "string" + }, + "updatedAt": { + "type": "string" + } + }, + "required": [ + "id", + "name", + "email", + "createdAt", + "updatedAt" + ], + "type": "object" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message", + "data" + ], + "type": "object" + } + } + }, + "description": "Success" + }, + "400": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "413": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "415": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "500": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "501": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + } + }, + "summary": "", + "x-omni-middleware": [ + "Auth" + ] + } + }, + "/api/v1/users/{id}": { + "delete": { + "operationId": "user_DeleteUser", + "parameters": [ + { + "in": "path", + "name": "id", + "required": true, + "schema": { + "format": "int64", + "maximum": 9007199254740991, + "minimum": 1, + "type": "integer" + } + } + ], + "responses": { + "204": { + "description": "Success" + }, + "400": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "413": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "415": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "500": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "501": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + } + }, + "summary": "", + "x-omni-middleware": [ + "Auth" + ] + }, + "get": { + "operationId": "user_GetUser", + "parameters": [ + { + "in": "path", + "name": "id", + "required": true, + "schema": { + "format": "int64", + "maximum": 9007199254740991, + "minimum": 1, + "type": "integer" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "data": { + "additionalProperties": false, + "properties": { + "createdAt": { + "type": "string" + }, + "email": { + "type": "string" + }, + "id": { + "format": "int64", + "type": "integer" + }, + "name": { + "type": "string" + }, + "updatedAt": { + "type": "string" + } + }, + "required": [ + "id", + "name", + "email", + "createdAt", + "updatedAt" + ], + "type": "object" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message", + "data" + ], + "type": "object" + } + } + }, + "description": "Success" + }, + "400": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "413": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "415": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "500": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "501": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + } + }, + "summary": "", + "x-omni-middleware": [ + "Auth" + ] + }, + "put": { + "operationId": "user_UpdateUser", + "parameters": [ + { + "in": "path", + "name": "id", + "required": true, + "schema": { + "format": "int64", + "maximum": 9007199254740991, + "minimum": 1, + "type": "integer" + } + } + ], + "requestBody": { + "content": { + "application/json": { + "schema": { + "additionalProperties": false, + "properties": { + "email": { + "format": "email", + "maxLength": 254, + "minLength": 1, + "type": "string" + }, + "name": { + "maxLength": 100, + "minLength": 1, + "type": "string" + } + }, + "required": [ + "name", + "email" + ], + "type": "object" + } + } + }, + "required": true + }, + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "data": { + "additionalProperties": false, + "properties": { + "createdAt": { + "type": "string" + }, + "email": { + "type": "string" + }, + "id": { + "format": "int64", + "type": "integer" + }, + "name": { + "type": "string" + }, + "updatedAt": { + "type": "string" + } + }, + "required": [ + "id", + "name", + "email", + "createdAt", + "updatedAt" + ], + "type": "object" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message", + "data" + ], + "type": "object" + } + } + }, + "description": "Success" + }, + "400": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "413": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "415": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "500": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "501": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + } + }, + "summary": "", + "x-omni-middleware": [ + "Auth" + ] + } + }, + "/api/v1/users/{id}/export": { + "post": { + "operationId": "user_ExportUser", + "parameters": [ + { + "in": "path", + "name": "id", + "required": true, + "schema": { + "format": "int64", + "maximum": 9007199254740991, + "minimum": 1, + "type": "integer" + } + } + ], + "responses": { + "200": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "data": { + "additionalProperties": false, + "properties": { + "downloadUrl": { + "type": "string" + } + }, + "required": [ + "downloadUrl" + ], + "type": "object" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message", + "data" + ], + "type": "object" + } + } + }, + "description": "Success" + }, + "400": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "413": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "415": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "500": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + }, + "501": { + "content": { + "application/json": { + "schema": { + "properties": { + "code": { + "type": "string" + }, + "message": { + "type": "string" + }, + "requestId": { + "type": "string" + } + }, + "required": [ + "code", + "message" + ], + "type": "object" + } + } + }, + "description": "Error" + } + }, + "summary": "Export one user as a private text object with a short-lived download link", + "x-omni-middleware": [ + "Auth" + ] + } + } + } +} diff --git a/examples/userService/etc/config.yaml b/examples/userService/etc/config.yaml new file mode 100644 index 0000000..f8c5ed6 --- /dev/null +++ b/examples/userService/etc/config.yaml @@ -0,0 +1,37 @@ +# Non-secret settings only. Environment variables override these values. +server: + address: '127.0.0.1:8080' + enableLogger: true + slowRequestThreshold: 500ms + shutdownTimeout: 10s + log: + service: "example.com/userService" + level: info + format: json + file: './var/log/server.log' + rotation: + enabled: true + maxSizeMb: 100 + maxBackups: 10 + maxAgeDays: 7 + compress: true + redactKeys: [] +storage: + provider: local + maxBytes: 33554432 + contentTypes: ['text/plain', 'image/png', 'image/jpeg', 'application/pdf'] + linkTtl: 5m + requestTimeout: 30s + root: './var/files' + publicUrl: 'http://127.0.0.1:8080/storage/object' + + +database: + driver: mysql + maxOpenConns: 10 + maxIdleConns: 5 + connMaxLifetime: 30m + connMaxIdleTime: 5m + pingTimeout: 5s +# GO_OMNI_DB_DSN supplies the secret connection string. + diff --git a/examples/userService/go.mod b/examples/userService/go.mod new file mode 100644 index 0000000..87d685c --- /dev/null +++ b/examples/userService/go.mod @@ -0,0 +1,26 @@ +module example.com/userService + +go 1.22.0 + +require ( + git.yuntuo.hk/k2safe/go-omni v0.7.0-preview + github.com/go-sql-driver/mysql v1.9.3 +) + +require ( + filippo.io/edwards25519 v1.1.0 // indirect + github.com/gabriel-vasile/mimetype v1.4.3 // indirect + github.com/go-playground/locales v0.14.1 // indirect + github.com/go-playground/universal-translator v0.18.1 // indirect + github.com/go-playground/validator/v10 v10.20.0 // indirect + github.com/golang-jwt/jwt/v5 v5.3.1 // indirect + github.com/leodido/go-urn v1.4.0 // indirect + golang.org/x/crypto v0.31.0 // indirect + golang.org/x/net v0.33.0 // indirect + golang.org/x/sys v0.28.0 // indirect + golang.org/x/text v0.21.0 // indirect + gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect + gopkg.in/yaml.v3 v3.0.1 // indirect +) + +replace git.yuntuo.hk/k2safe/go-omni => ../.. diff --git a/examples/userService/go.sum b/examples/userService/go.sum new file mode 100644 index 0000000..1c7a338 --- /dev/null +++ b/examples/userService/go.sum @@ -0,0 +1,38 @@ +filippo.io/edwards25519 v1.1.0 h1:FNf4tywRC1HmFuKW5xopWpigGjJKiJSV0Cqo0cJWDaA= +filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/gabriel-vasile/mimetype v1.4.3 h1:in2uUcidCuFcDKtdcBxlR0rJ1+fsokWf+uqxgUFjbI0= +github.com/gabriel-vasile/mimetype v1.4.3/go.mod h1:d8uq/6HKRL6CGdk+aubisF/M5GcPfT7nKyLpA0lbSSk= +github.com/go-playground/assert/v2 v2.2.0 h1:JvknZsQTYeFEAhQwI4qEt9cyV5ONwRHC+lYKSsYSR8s= +github.com/go-playground/assert/v2 v2.2.0/go.mod h1:VDjEfimB/XKnb+ZQfWdccd7VUvScMdVu0Titje2rxJ4= +github.com/go-playground/locales v0.14.1 h1:EWaQ/wswjilfKLTECiXz7Rh+3BjFhfDFKv/oXslEjJA= +github.com/go-playground/locales v0.14.1/go.mod h1:hxrqLVvrK65+Rwrd5Fc6F2O76J/NuW9t0sjnWqG1slY= +github.com/go-playground/universal-translator v0.18.1 h1:Bcnm0ZwsGyWbCzImXv+pAJnYK9S473LQFuzCbDbfSFY= +github.com/go-playground/universal-translator v0.18.1/go.mod h1:xekY+UJKNuX9WP91TpwSH2VMlDf28Uj24BCp08ZFTUY= +github.com/go-playground/validator/v10 v10.20.0 h1:K9ISHbSaI0lyB2eWMPJo+kOS/FBExVwjEviJTixqxL8= +github.com/go-playground/validator/v10 v10.20.0/go.mod h1:dbuPbCMFw/DrkbEynArYaCwl3amGuJotoKCe95atGMM= +github.com/go-sql-driver/mysql v1.9.3 h1:U/N249h2WzJ3Ukj8SowVFjdtZKfu9vlLZxjPXV1aweo= +github.com/go-sql-driver/mysql v1.9.3/go.mod h1:qn46aNg1333BRMNU69Lq93t8du/dwxI64Gl8i5p1WMU= +github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY= +github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE= +github.com/leodido/go-urn v1.4.0 h1:WT9HwE9SGECu3lg4d/dIA+jxlljEa1/ffXKmRjqdmIQ= +github.com/leodido/go-urn v1.4.0/go.mod h1:bvxc+MVxLKB4z00jd1z+Dvzr47oO32F/QSNjSBOlFxI= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk= +github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= +golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U= +golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk= +golang.org/x/net v0.33.0 h1:74SYHlV8BIgHIFC/LrYkOGIwL19eTYXQ5wc6TBuO36I= +golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4= +golang.org/x/sys v0.28.0 h1:Fksou7UEQUWlKvIdsqzJmUmCX3cZuD2+P3XyyzwMhlA= +golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA= +golang.org/x/text v0.21.0 h1:zyQAAkrwaneQ066sspRyJaG9VNi/YJ1NfzcGB3hZ/qo= +golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc= +gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/examples/userService/internal/handler/openapiGen.go b/examples/userService/internal/handler/openapiGen.go new file mode 100644 index 0000000..2ea90a4 --- /dev/null +++ b/examples/userService/internal/handler/openapiGen.go @@ -0,0 +1,13 @@ +// Code generated by go-omni api. DO NOT EDIT. +package handler + +import ( + "git.yuntuo.hk/k2safe/go-omni/httpx" + "net/http" +) + +const OpenAPIDocument = "{\n \"components\": {\n \"securitySchemes\": {\n \"bearerAuth\": {\n \"scheme\": \"bearer\",\n \"type\": \"http\"\n }\n }\n },\n \"info\": {\n \"title\": \"User service reference\",\n \"version\": \"1.0.0\"\n },\n \"openapi\": \"3.0.3\",\n \"paths\": {\n \"/api/v1/users\": {\n \"get\": {\n \"operationId\": \"user_ListUsers\",\n \"parameters\": [\n {\n \"in\": \"query\",\n \"name\": \"page\",\n \"required\": false,\n \"schema\": {\n \"default\": 1,\n \"format\": \"int64\",\n \"maximum\": 1000000,\n \"minimum\": 1,\n \"type\": \"integer\"\n }\n },\n {\n \"in\": \"query\",\n \"name\": \"pageSize\",\n \"required\": false,\n \"schema\": {\n \"default\": 20,\n \"format\": \"int64\",\n \"maximum\": 100,\n \"minimum\": 1,\n \"type\": \"integer\"\n }\n }\n ],\n \"responses\": {\n \"200\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"data\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"items\": {\n \"items\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"createdAt\": {\n \"type\": \"string\"\n },\n \"email\": {\n \"type\": \"string\"\n },\n \"id\": {\n \"format\": \"int64\",\n \"type\": \"integer\"\n },\n \"name\": {\n \"type\": \"string\"\n },\n \"updatedAt\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"id\",\n \"name\",\n \"email\",\n \"createdAt\",\n \"updatedAt\"\n ],\n \"type\": \"object\"\n },\n \"type\": \"array\"\n },\n \"page\": {\n \"format\": \"int64\",\n \"type\": \"integer\"\n },\n \"pageSize\": {\n \"format\": \"int64\",\n \"type\": \"integer\"\n },\n \"total\": {\n \"format\": \"int64\",\n \"type\": \"integer\"\n }\n },\n \"required\": [\n \"items\",\n \"total\",\n \"page\",\n \"pageSize\"\n ],\n \"type\": \"object\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\",\n \"data\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Success\"\n },\n \"400\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"413\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"415\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"500\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"501\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n }\n },\n \"summary\": \"\",\n \"x-omni-middleware\": [\n \"Auth\"\n ]\n },\n \"post\": {\n \"operationId\": \"user_CreateUser\",\n \"requestBody\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"email\": {\n \"format\": \"email\",\n \"maxLength\": 254,\n \"minLength\": 1,\n \"type\": \"string\"\n },\n \"name\": {\n \"maxLength\": 100,\n \"minLength\": 1,\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"name\",\n \"email\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"required\": true\n },\n \"responses\": {\n \"201\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"data\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"createdAt\": {\n \"type\": \"string\"\n },\n \"email\": {\n \"type\": \"string\"\n },\n \"id\": {\n \"format\": \"int64\",\n \"type\": \"integer\"\n },\n \"name\": {\n \"type\": \"string\"\n },\n \"updatedAt\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"id\",\n \"name\",\n \"email\",\n \"createdAt\",\n \"updatedAt\"\n ],\n \"type\": \"object\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\",\n \"data\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Success\"\n },\n \"400\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"413\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"415\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"500\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"501\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n }\n },\n \"summary\": \"\",\n \"x-omni-middleware\": [\n \"Auth\"\n ]\n }\n },\n \"/api/v1/users/{id}\": {\n \"delete\": {\n \"operationId\": \"user_DeleteUser\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"id\",\n \"required\": true,\n \"schema\": {\n \"format\": \"int64\",\n \"maximum\": 9007199254740991,\n \"minimum\": 1,\n \"type\": \"integer\"\n }\n }\n ],\n \"responses\": {\n \"204\": {\n \"description\": \"Success\"\n },\n \"400\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"413\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"415\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"500\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"501\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n }\n },\n \"summary\": \"\",\n \"x-omni-middleware\": [\n \"Auth\"\n ]\n },\n \"get\": {\n \"operationId\": \"user_GetUser\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"id\",\n \"required\": true,\n \"schema\": {\n \"format\": \"int64\",\n \"maximum\": 9007199254740991,\n \"minimum\": 1,\n \"type\": \"integer\"\n }\n }\n ],\n \"responses\": {\n \"200\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"data\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"createdAt\": {\n \"type\": \"string\"\n },\n \"email\": {\n \"type\": \"string\"\n },\n \"id\": {\n \"format\": \"int64\",\n \"type\": \"integer\"\n },\n \"name\": {\n \"type\": \"string\"\n },\n \"updatedAt\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"id\",\n \"name\",\n \"email\",\n \"createdAt\",\n \"updatedAt\"\n ],\n \"type\": \"object\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\",\n \"data\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Success\"\n },\n \"400\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"413\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"415\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"500\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"501\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n }\n },\n \"summary\": \"\",\n \"x-omni-middleware\": [\n \"Auth\"\n ]\n },\n \"put\": {\n \"operationId\": \"user_UpdateUser\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"id\",\n \"required\": true,\n \"schema\": {\n \"format\": \"int64\",\n \"maximum\": 9007199254740991,\n \"minimum\": 1,\n \"type\": \"integer\"\n }\n }\n ],\n \"requestBody\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"email\": {\n \"format\": \"email\",\n \"maxLength\": 254,\n \"minLength\": 1,\n \"type\": \"string\"\n },\n \"name\": {\n \"maxLength\": 100,\n \"minLength\": 1,\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"name\",\n \"email\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"required\": true\n },\n \"responses\": {\n \"200\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"data\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"createdAt\": {\n \"type\": \"string\"\n },\n \"email\": {\n \"type\": \"string\"\n },\n \"id\": {\n \"format\": \"int64\",\n \"type\": \"integer\"\n },\n \"name\": {\n \"type\": \"string\"\n },\n \"updatedAt\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"id\",\n \"name\",\n \"email\",\n \"createdAt\",\n \"updatedAt\"\n ],\n \"type\": \"object\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\",\n \"data\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Success\"\n },\n \"400\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"413\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"415\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"500\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"501\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n }\n },\n \"summary\": \"\",\n \"x-omni-middleware\": [\n \"Auth\"\n ]\n }\n },\n \"/api/v1/users/{id}/export\": {\n \"post\": {\n \"operationId\": \"user_ExportUser\",\n \"parameters\": [\n {\n \"in\": \"path\",\n \"name\": \"id\",\n \"required\": true,\n \"schema\": {\n \"format\": \"int64\",\n \"maximum\": 9007199254740991,\n \"minimum\": 1,\n \"type\": \"integer\"\n }\n }\n ],\n \"responses\": {\n \"200\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"data\": {\n \"additionalProperties\": false,\n \"properties\": {\n \"downloadUrl\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"downloadUrl\"\n ],\n \"type\": \"object\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\",\n \"data\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Success\"\n },\n \"400\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"413\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"415\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"500\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n },\n \"501\": {\n \"content\": {\n \"application/json\": {\n \"schema\": {\n \"properties\": {\n \"code\": {\n \"type\": \"string\"\n },\n \"message\": {\n \"type\": \"string\"\n },\n \"requestId\": {\n \"type\": \"string\"\n }\n },\n \"required\": [\n \"code\",\n \"message\"\n ],\n \"type\": \"object\"\n }\n }\n },\n \"description\": \"Error\"\n }\n },\n \"summary\": \"Export one user as a private text object with a short-lived download link\",\n \"x-omni-middleware\": [\n \"Auth\"\n ]\n }\n }\n }\n}\n" + +func OpenAPI(c *httpx.Context) error { + return c.WriteResponse(200, http.Header{"Content-Type": {"application/json; charset=utf-8"}}, []byte(OpenAPIDocument)) +} diff --git a/examples/userService/internal/handler/routesGen.go b/examples/userService/internal/handler/routesGen.go new file mode 100644 index 0000000..6fbf326 --- /dev/null +++ b/examples/userService/internal/handler/routesGen.go @@ -0,0 +1,27 @@ +// Code generated by go-omni api. DO NOT EDIT. +package handler + +import ( + h0 "example.com/userService/internal/handler/user" + s0 "example.com/userService/internal/service/user" + "example.com/userService/internal/servicecontext" + "fmt" + "git.yuntuo.hk/k2safe/go-omni/openapi" +) + +func RegisterRoutes(router openapi.Registrar, deps *servicecontext.Context) error { + if router == nil || deps == nil { + return fmt.Errorf("router and dependencies required") + } + if deps.Middlewares["Auth"] == nil { + return fmt.Errorf("missing middleware Auth") + } + h0 := h0.New(s0.New(deps)) + router.Handle("POST", "/api/v1/users", h0.CreateUser, deps.Middlewares["Auth"]) + router.Handle("GET", "/api/v1/users/:id", h0.GetUser, deps.Middlewares["Auth"]) + router.Handle("GET", "/api/v1/users", h0.ListUsers, deps.Middlewares["Auth"]) + router.Handle("PUT", "/api/v1/users/:id", h0.UpdateUser, deps.Middlewares["Auth"]) + router.Handle("DELETE", "/api/v1/users/:id", h0.DeleteUser, deps.Middlewares["Auth"]) + router.Handle("POST", "/api/v1/users/:id/export", h0.ExportUser, deps.Middlewares["Auth"]) + return nil +} diff --git a/examples/userService/internal/handler/user/handlersGen.go b/examples/userService/internal/handler/user/handlersGen.go new file mode 100644 index 0000000..3ad35dd --- /dev/null +++ b/examples/userService/internal/handler/user/handlersGen.go @@ -0,0 +1,120 @@ +// Code generated by go-omni api. DO NOT EDIT. +package user + +import ( + "context" + "example.com/userService/internal/types" + "fmt" + "git.yuntuo.hk/k2safe/go-omni/contract" + "git.yuntuo.hk/k2safe/go-omni/httpx" +) + +type Service interface { + CreateUser(ctx context.Context, req *types.CreateUserReq) (*types.UserResp, error) + GetUser(ctx context.Context, req *types.UserIDReq) (*types.UserResp, error) + ListUsers(ctx context.Context, req *types.ListUsersReq) (*types.UserPageResp, error) + UpdateUser(ctx context.Context, req *types.UpdateUserReq) (*types.UserResp, error) + DeleteUser(ctx context.Context, req *types.UserIDReq) error + ExportUser(ctx context.Context, req *types.UserIDReq) (*types.ExportResp, error) +} +type Handler struct{ service Service } + +func New(s Service) *Handler { return &Handler{service: s} } + +var createUserRequest = contract.MustNew[types.CreateUserReq]() + +func (h *Handler) CreateUser(c *httpx.Context) error { + var req types.CreateUserReq + if err := createUserRequest.Bind(c.Request(), httpx.RouteParams(c.Context()), &req); err != nil { + return err + } + resp, err := h.service.CreateUser(c.Context(), &req) + if err != nil { + return err + } + if resp == nil { + return fmt.Errorf("CreateUser returned a nil response") + } + return c.Created(resp) +} + +var getUserRequest = contract.MustNew[types.UserIDReq]() + +func (h *Handler) GetUser(c *httpx.Context) error { + var req types.UserIDReq + if err := getUserRequest.Bind(c.Request(), httpx.RouteParams(c.Context()), &req); err != nil { + return err + } + resp, err := h.service.GetUser(c.Context(), &req) + if err != nil { + return err + } + if resp == nil { + return fmt.Errorf("GetUser returned a nil response") + } + return c.OK(resp) +} + +var listUsersRequest = contract.MustNew[types.ListUsersReq]() + +func (h *Handler) ListUsers(c *httpx.Context) error { + var req types.ListUsersReq + if err := listUsersRequest.Bind(c.Request(), httpx.RouteParams(c.Context()), &req); err != nil { + return err + } + resp, err := h.service.ListUsers(c.Context(), &req) + if err != nil { + return err + } + if resp == nil { + return fmt.Errorf("ListUsers returned a nil response") + } + return c.OK(resp) +} + +var updateUserRequest = contract.MustNew[types.UpdateUserReq]() + +func (h *Handler) UpdateUser(c *httpx.Context) error { + var req types.UpdateUserReq + if err := updateUserRequest.Bind(c.Request(), httpx.RouteParams(c.Context()), &req); err != nil { + return err + } + resp, err := h.service.UpdateUser(c.Context(), &req) + if err != nil { + return err + } + if resp == nil { + return fmt.Errorf("UpdateUser returned a nil response") + } + return c.OK(resp) +} + +var deleteUserRequest = contract.MustNew[types.UserIDReq]() + +func (h *Handler) DeleteUser(c *httpx.Context) error { + var req types.UserIDReq + if err := deleteUserRequest.Bind(c.Request(), httpx.RouteParams(c.Context()), &req); err != nil { + return err + } + if err := h.service.DeleteUser(c.Context(), &req); err != nil { + return err + } + return c.NoContent() +} + +var exportUserRequest = contract.MustNew[types.UserIDReq]() + +func (h *Handler) ExportUser(c *httpx.Context) error { + var req types.UserIDReq + if err := exportUserRequest.Bind(c.Request(), httpx.RouteParams(c.Context()), &req); err != nil { + return err + } + resp, err := h.service.ExportUser(c.Context(), &req) + if err != nil { + return err + } + if resp == nil { + return fmt.Errorf("ExportUser returned a nil response") + } + return c.OK(resp) +} diff --git a/examples/userService/internal/middleware/auth.go b/examples/userService/internal/middleware/auth.go new file mode 100644 index 0000000..4827c03 --- /dev/null +++ b/examples/userService/internal/middleware/auth.go @@ -0,0 +1,40 @@ +package middleware + +import ( + "crypto/sha256" + "crypto/subtle" + "errors" + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "os" + "strings" +) + +// Auth is the fail-closed registry default. Startup replaces it with configured +// authentication. This administrative example uses one environment-only key; +// production applications can inject JWT/OIDC through the same middleware slot. +func Auth(next httpx.Handler) httpx.Handler { + return func(c *httpx.Context) error { + return apierrors.Unauthorized("UNAUTHENTICATED", "authentication required") + } +} +func NewAuthFromEnv() (httpx.Middleware, error) { + token := os.Getenv("USER_SERVICE_API_TOKEN") + if len(token) < 32 || strings.ContainsAny(token, " \t\r\n") { + return nil, errors.New("USER_SERVICE_API_TOKEN requires at least 32 bytes without whitespace") + } + expected := sha256.Sum256([]byte(token)) + return func(next httpx.Handler) httpx.Handler { + return func(c *httpx.Context) error { + parts := strings.Fields(c.Header("Authorization")) + if len(parts) != 2 || !strings.EqualFold(parts[0], "Bearer") { + return apierrors.Unauthorized("UNAUTHENTICATED", "authentication required") + } + actual := sha256.Sum256([]byte(parts[1])) + if subtle.ConstantTimeCompare(expected[:], actual[:]) != 1 { + return apierrors.Unauthorized("UNAUTHENTICATED", "authentication required") + } + return next(c) + } + }, nil +} diff --git a/examples/userService/internal/middleware/auth_test.go b/examples/userService/internal/middleware/auth_test.go new file mode 100644 index 0000000..37c1f3f --- /dev/null +++ b/examples/userService/internal/middleware/auth_test.go @@ -0,0 +1,35 @@ +package middleware + +import ( + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "net/http/httptest" + "testing" +) + +func TestAuthFailsClosedAndDoesNotLeakKey(t *testing.T) { + t.Setenv("USER_SERVICE_API_TOKEN", "") + if _, err := NewAuthFromEnv(); err == nil { + t.Fatal("missing credential accepted") + } + token := "test-key-012345678901234567890123456789" + t.Setenv("USER_SERVICE_API_TOKEN", token) + middleware, err := NewAuthFromEnv() + if err != nil { + t.Fatal(err) + } + for _, header := range []string{"", "Basic " + token, "Bearer wrong", "Bearer " + token} { + called := false + r := httptest.NewRequest("GET", "/", nil) + r.Header.Set("Authorization", header) + c := httpx.NewContext(r, httptest.NewRecorder(), nil) + err = middleware(func(*httpx.Context) error { called = true; return nil })(c) + if header == "Bearer "+token { + if err != nil || !called { + t.Fatal("valid token rejected") + } + } else if err == nil || called || apierrors.From(err).Status != 401 { + t.Fatal("unauthenticated request reached service") + } + } +} diff --git a/examples/userService/internal/middleware/registryGen.go b/examples/userService/internal/middleware/registryGen.go new file mode 100644 index 0000000..06774d7 --- /dev/null +++ b/examples/userService/internal/middleware/registryGen.go @@ -0,0 +1,10 @@ +// Code generated by go-omni api. DO NOT EDIT. +package middleware + +import "git.yuntuo.hk/k2safe/go-omni/httpx" + +func Defaults() map[string]httpx.Middleware { + return map[string]httpx.Middleware{ + "Auth": Auth, + } +} diff --git a/examples/userService/internal/model/errors.go b/examples/userService/internal/model/errors.go new file mode 100644 index 0000000..3f6e7c8 --- /dev/null +++ b/examples/userService/internal/model/errors.go @@ -0,0 +1,11 @@ +package model + +import "errors" + +// Domain errors are translated into HTTP responses only by the handler layer. +var ( + ErrUserNotFound = errors.New("user not found") + ErrEmailTaken = errors.New("email already in use") + ErrInvalidUser = errors.New("invalid user") + ErrInvalidPage = errors.New("invalid pagination") +) diff --git a/examples/userService/internal/model/user.go b/examples/userService/internal/model/user.go new file mode 100644 index 0000000..376f3eb --- /dev/null +++ b/examples/userService/internal/model/user.go @@ -0,0 +1,25 @@ +package model + +import "time" + +// User is the business entity; it has no dependency on HTTP or a database driver. +type User struct { + ID int64 `json:"id"` + Name string `json:"name"` + Email string `json:"email"` + CreatedAt time.Time `json:"createdAt"` + UpdatedAt time.Time `json:"updatedAt"` +} + +// UserInput is shared by creation and full replacement (PUT). +type UserInput struct { + Name string + Email string +} + +type UserPage struct { + Items []User `json:"items"` + Total int `json:"total"` + Page int `json:"page"` + PageSize int `json:"pageSize"` +} diff --git a/examples/userService/internal/repository/user/mysql.go b/examples/userService/internal/repository/user/mysql.go new file mode 100644 index 0000000..d35eeb3 --- /dev/null +++ b/examples/userService/internal/repository/user/mysql.go @@ -0,0 +1,143 @@ +package user + +import ( + "context" + "database/sql" + "errors" + + "example.com/userService/internal/model" + "github.com/go-sql-driver/mysql" +) + +// SQLExecutor is implemented by both *sql.DB and *sql.Tx. A repository created +// with a transaction keeps all of its queries on that same transaction. +type SQLExecutor interface { + ExecContext(context.Context, string, ...any) (sql.Result, error) + QueryContext(context.Context, string, ...any) (*sql.Rows, error) + QueryRowContext(context.Context, string, ...any) *sql.Row +} + +// MySQLUsers is the persistent implementation of UserRepository. +// The schema is documented in schema.sql; migrations stay outside the framework. +type MySQLUsers struct { + db SQLExecutor +} + +func NewMySQLUsers(db SQLExecutor) *MySQLUsers { + return &MySQLUsers{db: db} +} + +// CheckSchema validates that the application's migration has been applied. +// Runtime startup never creates or alters tables. +func (r *MySQLUsers) CheckSchema(ctx context.Context) error { + rows, err := r.db.QueryContext(ctx, `SELECT id, name, email, created_at, updated_at FROM users LIMIT 0`) + if err != nil { + return err + } + return rows.Close() +} + +func (r *MySQLUsers) Create(ctx context.Context, input model.UserInput) (model.User, error) { + result, err := r.db.ExecContext(ctx, ` + INSERT INTO users (name, email, created_at, updated_at) + VALUES (?, ?, UTC_TIMESTAMP(6), UTC_TIMESTAMP(6))`, input.Name, input.Email) + if err != nil { + return model.User{}, mapMySQLError(err) + } + id, err := result.LastInsertId() + if err != nil { + return model.User{}, err + } + return r.Get(ctx, id) +} + +func (r *MySQLUsers) Get(ctx context.Context, id int64) (model.User, error) { + var user model.User + err := r.db.QueryRowContext(ctx, ` + SELECT id, name, email, created_at, updated_at + FROM users WHERE id = ?`, id).Scan( + &user.ID, &user.Name, &user.Email, &user.CreatedAt, &user.UpdatedAt, + ) + if errors.Is(err, sql.ErrNoRows) { + return model.User{}, model.ErrUserNotFound + } + if err != nil { + return model.User{}, err + } + return user, nil +} + +func (r *MySQLUsers) List(ctx context.Context, page, pageSize int) (model.UserPage, error) { + if page < 1 || pageSize < 1 || pageSize > 100 { + return model.UserPage{}, model.ErrInvalidPage + } + var total int + if err := r.db.QueryRowContext(ctx, `SELECT COUNT(*) FROM users`).Scan(&total); err != nil { + return model.UserPage{}, err + } + result := model.UserPage{Items: []model.User{}, Total: total, Page: page, PageSize: pageSize} + if total == 0 || page-1 > (total-1)/pageSize { + return result, nil + } + offset := (page - 1) * pageSize + rows, err := r.db.QueryContext(ctx, ` + SELECT id, name, email, created_at, updated_at + FROM users ORDER BY id LIMIT ? OFFSET ?`, pageSize, offset) + if err != nil { + return model.UserPage{}, err + } + defer rows.Close() + for rows.Next() { + var user model.User + if err := rows.Scan(&user.ID, &user.Name, &user.Email, &user.CreatedAt, &user.UpdatedAt); err != nil { + return model.UserPage{}, err + } + result.Items = append(result.Items, user) + } + if err := rows.Err(); err != nil { + return model.UserPage{}, err + } + return result, nil +} + +func (r *MySQLUsers) Update(ctx context.Context, id int64, input model.UserInput) (model.User, error) { + result, err := r.db.ExecContext(ctx, ` + UPDATE users SET name = ?, email = ?, updated_at = UTC_TIMESTAMP(6) + WHERE id = ?`, input.Name, input.Email, id) + if err != nil { + return model.User{}, mapMySQLError(err) + } + changed, err := result.RowsAffected() + if err != nil { + return model.User{}, err + } + if changed == 0 { + // MySQL may report zero affected rows when the replacement is identical; + // distinguish that from an ID that does not exist. + return r.Get(ctx, id) + } + return r.Get(ctx, id) +} + +func (r *MySQLUsers) Delete(ctx context.Context, id int64) error { + result, err := r.db.ExecContext(ctx, `DELETE FROM users WHERE id = ?`, id) + if err != nil { + return err + } + deleted, err := result.RowsAffected() + if err != nil { + return err + } + if deleted == 0 { + return model.ErrUserNotFound + } + return nil +} + +func mapMySQLError(err error) error { + var mysqlErr *mysql.MySQLError + if errors.As(err, &mysqlErr) && mysqlErr.Number == 1062 { + return model.ErrEmailTaken + } + return err +} diff --git a/examples/userService/internal/repository/user/repository.go b/examples/userService/internal/repository/user/repository.go new file mode 100644 index 0000000..c08d1eb --- /dev/null +++ b/examples/userService/internal/repository/user/repository.go @@ -0,0 +1,16 @@ +package user + +import ( + "context" + "example.com/userService/internal/model" +) + +// UserRepository describes the storage operations the business needs. +// Implementations must atomically enforce unique, normalized email addresses. +type UserRepository interface { + Create(context.Context, model.UserInput) (model.User, error) + Get(context.Context, int64) (model.User, error) + List(context.Context, int, int) (model.UserPage, error) + Update(context.Context, int64, model.UserInput) (model.User, error) + Delete(context.Context, int64) error +} diff --git a/examples/userService/internal/repository/user/store.go b/examples/userService/internal/repository/user/store.go new file mode 100644 index 0000000..a213602 --- /dev/null +++ b/examples/userService/internal/repository/user/store.go @@ -0,0 +1,41 @@ +package user + +import ( + "context" + "database/sql" + "errors" + "git.yuntuo.hk/k2safe/go-omni/database" + "time" +) + +type UnitOfWork interface { + UserRepository + WithTx(context.Context, func(UserRepository, func(string, int64, string) error) error) error +} +type Store struct { + *MySQLUsers + db *database.DB +} + +func NewStore(db *database.DB) *Store { return &Store{MySQLUsers: NewMySQLUsers(db.DB), db: db} } +func (s *Store) WithTx(ctx context.Context, work func(UserRepository, func(string, int64, string) error) error) error { + return s.db.InTx(ctx, nil, func(tx *sql.Tx) error { + audit := func(operation string, id int64, requestID string) error { + _, err := tx.ExecContext(ctx, "INSERT INTO user_audit (user_id,operation,request_id,created_at) VALUES (?,?,?,?)", id, operation, requestID, time.Now().UTC()) + return err + } + return work(NewMySQLUsers(tx), audit) + }) +} + +// CheckSchema validates columns; it never mutates database schema. +func (s *Store) CheckSchema(ctx context.Context) error { + if err := s.MySQLUsers.CheckSchema(ctx); err != nil { + return err + } + rows, err := s.db.QueryContext(ctx, "SELECT id,user_id,operation,request_id,created_at FROM user_audit LIMIT 0") + if err != nil { + return err + } + return errors.Join(rows.Err(), rows.Close()) +} diff --git a/examples/userService/internal/repository/user/transaction_test.go b/examples/userService/internal/repository/user/transaction_test.go new file mode 100644 index 0000000..a8846ec --- /dev/null +++ b/examples/userService/internal/repository/user/transaction_test.go @@ -0,0 +1,84 @@ +package user + +import ( + "context" + "errors" + "os" + "testing" + + "example.com/userService/internal/model" + "git.yuntuo.hk/k2safe/go-omni/database" +) + +// This suite runs only against the disposable database created by verifyAPIWorkflow. +func TestTransactions(t *testing.T) { + if os.Getenv("USER_SERVICE_TEST_DATABASE") != "1" { + t.Skip("isolated MySQL required") + } + cfg, err := database.ConfigFromEnv("") + if err != nil { + t.Fatal(err) + } + db, err := database.Open(context.Background(), cfg) + if err != nil { + t.Fatal(err) + } + defer db.Close() + store := NewStore(db) + for _, kind := range []string{"error", "panic", "cancel", "commit"} { + t.Run(kind, func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + email := kind + "@transaction.example" + var recovered any + func() { + defer func() { recovered = recover() }() + err = store.WithTx(ctx, func(r UserRepository, audit func(string, int64, string) error) error { + u, e := r.Create(ctx, model.UserInput{Name: "Transaction", Email: email}) + if e != nil { + return e + } + if e = audit("created", u.ID, "transactionCheck"); e != nil { + return e + } + switch kind { + case "error": + return errors.New("rollback sentinel") + case "panic": + panic("rollback sentinel") + case "cancel": + cancel() + } + return nil + }) + }() + if kind == "panic" && recovered == nil { + t.Fatal("panic lost") + } + if kind == "commit" && err != nil { + t.Fatal(err) + } + if kind != "commit" && kind != "panic" && err == nil { + t.Fatal("failed transaction committed") + } + var count int + if e := db.QueryRow("SELECT COUNT(*) FROM users WHERE email=?", email).Scan(&count); e != nil { + t.Fatal(e) + } + want := 0 + if kind == "commit" { + want = 1 + } + if count != want { + t.Fatalf("rows=%d want=%d", count, want) + } + }) + } + var audits int + if err = db.QueryRow("SELECT COUNT(*) FROM user_audit WHERE request_id='transactionCheck'").Scan(&audits); err != nil { + t.Fatal(err) + } + if audits != 1 { + t.Fatalf("audit transaction mismatch: %d", audits) + } +} diff --git a/examples/userService/internal/service/user/createUser.go b/examples/userService/internal/service/user/createUser.go new file mode 100644 index 0000000..690dbcb --- /dev/null +++ b/examples/userService/internal/service/user/createUser.go @@ -0,0 +1,16 @@ +package user + +import ( + "context" + "example.com/userService/internal/model" + repo "example.com/userService/internal/repository/user" + "example.com/userService/internal/types" +) + +func (s *Service) CreateUser(ctx context.Context, req *types.CreateUserReq) (*types.UserResp, error) { + input, err := normalize(req.Name, req.Email) + if err != nil { + return nil, s.failure(ctx, err) + } + return s.mutate(ctx, "created", func(r repo.UserRepository) (model.User, error) { return r.Create(ctx, input) }) +} diff --git a/examples/userService/internal/service/user/deleteUser.go b/examples/userService/internal/service/user/deleteUser.go new file mode 100644 index 0000000..886edb6 --- /dev/null +++ b/examples/userService/internal/service/user/deleteUser.go @@ -0,0 +1,16 @@ +package user + +import ( + "context" + "example.com/userService/internal/model" + repo "example.com/userService/internal/repository/user" + "example.com/userService/internal/types" +) + +func (s *Service) DeleteUser(ctx context.Context, req *types.UserIDReq) error { + _, err := s.mutate(ctx, "deleted", func(r repo.UserRepository) (model.User, error) { + err := r.Delete(ctx, req.ID) + return model.User{ID: req.ID}, err + }) + return err +} diff --git a/examples/userService/internal/service/user/exportUser.go b/examples/userService/internal/service/user/exportUser.go new file mode 100644 index 0000000..0e69cc6 --- /dev/null +++ b/examples/userService/internal/service/user/exportUser.go @@ -0,0 +1,40 @@ +package user + +import ( + "context" + "crypto/rand" + "encoding/hex" + "encoding/json" + "example.com/userService/internal/types" + "fmt" + "strings" + "time" +) + +func (s *Service) ExportUser(ctx context.Context, req *types.UserIDReq) (*types.ExportResp, error) { + u, err := s.GetUser(ctx, req) + if err != nil { + return nil, err + } + payload, err := json.Marshal(u) + if err != nil { + return nil, s.failure(ctx, err) + } + var nonce [16]byte + if _, err = rand.Read(nonce[:]); err != nil { + return nil, s.failure(ctx, err) + } + key := fmt.Sprintf("exports/users/%d/%s.txt", u.ID, hex.EncodeToString(nonce[:])) + if _, err = s.deps.Uploads.Put(ctx, key, strings.NewReader(string(payload)), int64(len(payload)), "text/plain"); err != nil { + return nil, s.failure(ctx, err) + } + url, err := s.deps.Downloads.PresignGet(ctx, key, 5*time.Minute) + if err != nil { + cleanup, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Second) + defer cancel() + _ = s.deps.Files.Delete(cleanup, key) + return nil, s.failure(ctx, err) + } + s.deps.Logger.InfoContext(ctx, "user.exported", "userId", u.ID) + return &types.ExportResp{DownloadURL: url.String()}, nil +} diff --git a/examples/userService/internal/service/user/getUser.go b/examples/userService/internal/service/user/getUser.go new file mode 100644 index 0000000..c5b81a4 --- /dev/null +++ b/examples/userService/internal/service/user/getUser.go @@ -0,0 +1,14 @@ +package user + +import ( + "context" + "example.com/userService/internal/types" +) + +func (s *Service) GetUser(ctx context.Context, req *types.UserIDReq) (*types.UserResp, error) { + u, err := s.deps.Users.Get(ctx, req.ID) + if err != nil { + return nil, s.failure(ctx, err) + } + return response(u), nil +} diff --git a/examples/userService/internal/service/user/listUsers.go b/examples/userService/internal/service/user/listUsers.go new file mode 100644 index 0000000..77d0645 --- /dev/null +++ b/examples/userService/internal/service/user/listUsers.go @@ -0,0 +1,18 @@ +package user + +import ( + "context" + "example.com/userService/internal/types" +) + +func (s *Service) ListUsers(ctx context.Context, req *types.ListUsersReq) (*types.UserPageResp, error) { + page, err := s.deps.Users.List(ctx, req.Page, req.PageSize) + if err != nil { + return nil, s.failure(ctx, err) + } + result := &types.UserPageResp{Items: []types.UserResp{}, Total: page.Total, Page: page.Page, PageSize: page.PageSize} + for _, item := range page.Items { + result.Items = append(result.Items, *response(item)) + } + return result, nil +} diff --git a/examples/userService/internal/service/user/service.go b/examples/userService/internal/service/user/service.go new file mode 100644 index 0000000..c6e5229 --- /dev/null +++ b/examples/userService/internal/service/user/service.go @@ -0,0 +1,65 @@ +package user + +import ( + "context" + "errors" + "example.com/userService/internal/model" + repo "example.com/userService/internal/repository/user" + "example.com/userService/internal/servicecontext" + "example.com/userService/internal/types" + apierrors "git.yuntuo.hk/k2safe/go-omni/errors" + "git.yuntuo.hk/k2safe/go-omni/logx" + "net/mail" + "strings" + "time" + "unicode/utf8" +) + +type Service struct{ deps *servicecontext.Context } + +func New(deps *servicecontext.Context) *Service { return &Service{deps: deps} } +func normalize(name, email string) (model.UserInput, error) { + input := model.UserInput{Name: strings.TrimSpace(name), Email: strings.ToLower(strings.TrimSpace(email))} + address, err := mail.ParseAddress(input.Email) + if input.Name == "" || utf8.RuneCountInString(input.Name) > 100 || len(input.Email) > 254 || err != nil || address.Address != input.Email { + return model.UserInput{}, model.ErrInvalidUser + } + return input, nil +} +func response(u model.User) *types.UserResp { + return &types.UserResp{ID: u.ID, Name: u.Name, Email: u.Email, CreatedAt: u.CreatedAt.UTC().Format(time.RFC3339Nano), UpdatedAt: u.UpdatedAt.UTC().Format(time.RFC3339Nano)} +} +func (s *Service) failure(ctx context.Context, err error) error { + switch { + case err == nil: + return nil + case errors.Is(err, model.ErrUserNotFound): + return apierrors.NotFound("USER_NOT_FOUND", "user not found") + case errors.Is(err, model.ErrEmailTaken): + return apierrors.New(409, "EMAIL_TAKEN", "email already in use") + case errors.Is(err, model.ErrInvalidUser): + return apierrors.BadRequest("INVALID_USER", "invalid user") + case errors.Is(err, model.ErrInvalidPage): + return apierrors.BadRequest("INVALID_PAGE", "invalid pagination") + default: + // Do not log SQL/DSN or user input; requestId correlates the failed operation. + s.deps.Logger.ErrorContext(ctx, "user.operationFailed", "category", "persistence") + return apierrors.Internal(err) + } +} +func (s *Service) mutate(ctx context.Context, operation string, fn func(repo.UserRepository) (model.User, error)) (*types.UserResp, error) { + var u model.User + err := s.deps.Users.WithTx(ctx, func(r repo.UserRepository, audit func(string, int64, string) error) error { + var err error + u, err = fn(r) + if err != nil { + return err + } + return audit(operation, u.ID, logx.RequestID(ctx)) + }) + if err != nil { + return nil, s.failure(ctx, err) + } + s.deps.Logger.InfoContext(ctx, "user."+operation, "userId", u.ID) + return response(u), nil +} diff --git a/examples/userService/internal/service/user/updateUser.go b/examples/userService/internal/service/user/updateUser.go new file mode 100644 index 0000000..a73483b --- /dev/null +++ b/examples/userService/internal/service/user/updateUser.go @@ -0,0 +1,16 @@ +package user + +import ( + "context" + "example.com/userService/internal/model" + repo "example.com/userService/internal/repository/user" + "example.com/userService/internal/types" +) + +func (s *Service) UpdateUser(ctx context.Context, req *types.UpdateUserReq) (*types.UserResp, error) { + input, err := normalize(req.Name, req.Email) + if err != nil { + return nil, s.failure(ctx, err) + } + return s.mutate(ctx, "updated", func(r repo.UserRepository) (model.User, error) { return r.Update(ctx, req.ID, input) }) +} diff --git a/examples/userService/internal/servicecontext/context.go b/examples/userService/internal/servicecontext/context.go new file mode 100644 index 0000000..e6a7f73 --- /dev/null +++ b/examples/userService/internal/servicecontext/context.go @@ -0,0 +1,122 @@ +package servicecontext + +import ( + "context" + "errors" + "example.com/userService/internal/middleware" + userrepo "example.com/userService/internal/repository/user" + "example.com/userService/internal/settings" + "git.yuntuo.hk/k2safe/go-omni/database" + "git.yuntuo.hk/k2safe/go-omni/health" + "git.yuntuo.hk/k2safe/go-omni/httpx" + "git.yuntuo.hk/k2safe/go-omni/logx" + "git.yuntuo.hk/k2safe/go-omni/storage" + "github.com/go-sql-driver/mysql" + "net/http" + "sync" + "time" +) + +// Context owns infrastructure, while Logger is borrowed from main. +// Add business repositories here; HTTP contracts do not define database tables. +type Context struct { + Users userrepo.UnitOfWork + Logger *logx.Logger + Health *health.Checker + Middlewares map[string]httpx.Middleware + DB *database.DB + + Files storage.Store + Uploads *storage.Uploader + Downloads storage.Presigner + UploadLinks storage.UploadPresigner + FileHTTP http.Handler + storageClose func() error + + closeMu sync.Mutex + closed bool + closeOnce sync.Once + closeErr error +} + +// Pass the settings loaded by main. The optional form supports lightweight +// tests without infrastructure; selected adapters still require valid settings. +func New(ctx context.Context, logger *logx.Logger, configs ...settings.Config) (result *Context, err error) { + if ctx == nil { + ctx = context.Background() + } + if err = ctx.Err(); err != nil { + return nil, err + } + if len(configs) > 1 { + return nil, errors.New("expected at most one configuration") + } + cfg := settings.Default() + if len(configs) == 1 { + cfg = configs[0] + } + _ = cfg + checker, err := health.New() + if err != nil { + return nil, err + } + c := &Context{Logger: logger, Health: checker, Middlewares: middleware.Defaults()} + defer func() { + if err != nil { + err = errors.Join(err, c.Close()) + } + }() + + if err = cfg.Validate(); err != nil { + return nil, err + } + dsn, parseErr := mysql.ParseDSN(cfg.Database.DSN) + if parseErr != nil || dsn.DBName == "" { + return nil, errors.New("GO_OMNI_DB_DSN must contain a valid MySQL database name") + } + dsn.ParseTime = true + dsn.Loc = time.UTC + cfg.Database.DSN = dsn.FormatDSN() + c.DB, err = database.Open(ctx, cfg.Database) + if err != nil { + return nil, err + } + if err = checker.Add(health.Probe{Name: "database", Timeout: cfg.Database.PingTimeout, Check: c.DB.Health}); err != nil { + return nil, err + } + + if err = c.InitStorage(ctx, cfg.Storage); err != nil { + return nil, err + } + c.Users = userrepo.NewStore(c.DB) + checkCtx, cancel := context.WithTimeout(ctx, cfg.Database.PingTimeout) + defer cancel() + if err = userrepo.NewStore(c.DB).CheckSchema(checkCtx); err != nil { + return nil, errors.New("user schema is missing or incompatible; apply migrations first") + } + authHook, authErr := middleware.NewAuthFromEnv() + if authErr != nil { + return nil, authErr + } + c.Middlewares["Auth"] = authHook + return c, nil +} + +// Close is idempotent, including startup failure and graceful shutdown. +func (c *Context) Close() error { + if c == nil { + return nil + } + c.closeMu.Lock() + defer c.closeMu.Unlock() + c.closed = true + c.closeOnce.Do(func() { + if c.storageClose != nil { + c.closeErr = errors.Join(c.closeErr, c.storageClose()) + } + if c.DB != nil { + c.closeErr = errors.Join(c.closeErr, c.DB.Close()) + } + }) + return c.closeErr +} diff --git a/examples/userService/internal/servicecontext/context_test.go b/examples/userService/internal/servicecontext/context_test.go new file mode 100644 index 0000000..e9e1a98 --- /dev/null +++ b/examples/userService/internal/servicecontext/context_test.go @@ -0,0 +1,81 @@ +package servicecontext + +import ( + "context" + "io" + "os" + "path/filepath" + "testing" + "time" + + "example.com/userService/internal/settings" + "git.yuntuo.hk/k2safe/go-omni/database" + "git.yuntuo.hk/k2safe/go-omni/logx" +) + +func TestStartupFailureAndIdempotentCleanup(t *testing.T) { + if os.Getenv("USER_SERVICE_TEST_DATABASE") != "1" { + t.Skip("isolated MySQL required") + } + t.Setenv("USER_SERVICE_API_TOKEN", "test-012345678901234567890123456789") + t.Setenv("GO_OMNI_STORAGE_SIGNING_KEY", "test-012345678901234567890123456789") + cfg, err := settings.Load("../../etc/config.yaml") + if err != nil { + t.Fatal(err) + } + observer, err := database.Open(context.Background(), cfg.Database) + if err != nil { + t.Fatal(err) + } + defer observer.Close() + count := func() int { + var n int + if err := observer.QueryRow("SELECT COUNT(*) FROM information_schema.processlist WHERE USER=SUBSTRING_INDEX(CURRENT_USER(),'@',1) AND DB=DATABASE()").Scan(&n); err != nil { + t.Fatal(err) + } + return n + } + baseline := count() + cfg.Server.Log.File = "" + cfg.Server.Log.Writer = io.Discard + cfg.Server.Log.Rotation = logx.RotationConfig{} + logger, err := logx.New(cfg.Server.Log) + if err != nil { + t.Fatal(err) + } + defer logger.Close() + root := t.TempDir() + file := filepath.Join(root, "notADirectory") + if err = os.WriteFile(file, []byte("x"), 0600); err != nil { + t.Fatal(err) + } + cfg.Storage.Root = file + if c, err := New(context.Background(), logger, cfg); err == nil { + c.Close() + t.Fatal("invalid storage root accepted") + } + deadline := time.Now().Add(3 * time.Second) + for count() != baseline { + if time.Now().After(deadline) { + t.Fatal("failed startup leaked MySQL connection") + } + time.Sleep(10 * time.Millisecond) + } + cfg.Storage.Root = filepath.Join(root, "files") + c, err := New(context.Background(), logger, cfg) + if err != nil { + t.Fatal(err) + } + if err = c.Close(); err != nil { + t.Fatal(err) + } + if err = c.Close(); err != nil { + t.Fatal(err) + } + if err = c.DB.Ping(); err == nil { + t.Fatal("closed application retained database") + } + if err = c.InitStorage(context.Background(), cfg.Storage); err == nil { + t.Fatal("closed dependencies reopened storage") + } +} diff --git a/examples/userService/internal/servicecontext/storage.go b/examples/userService/internal/servicecontext/storage.go new file mode 100644 index 0000000..dbbc5c1 --- /dev/null +++ b/examples/userService/internal/servicecontext/storage.go @@ -0,0 +1,49 @@ +package servicecontext + +import ( + "context" + "errors" + + "git.yuntuo.hk/k2safe/go-omni/storage/local" + + "example.com/userService/internal/settings" + "git.yuntuo.hk/k2safe/go-omni/storage" +) + +// InitStorage is called once, before serving requests. The composition root +// owns provider clients; business services receive only capability interfaces. +func (c *Context) InitStorage(ctx context.Context, cfg settings.StorageConfig) error { + if c == nil || ctx == nil { + return errors.New("storage initialization requires context") + } + c.closeMu.Lock() + defer c.closeMu.Unlock() + if c.closed { + return errors.New("service dependencies are closed") + } + if err := ctx.Err(); err != nil { + return err + } + if c.Files != nil { + return errors.New("storage already initialized") + } + if err := cfg.Validate(); err != nil { + return err + } + + files, err := local.New(cfg.Root, cfg.MaxBytes) + if err != nil { + return err + } + signer, err := storage.NewUploadSigner(files, cfg.PublicURL, []byte(cfg.SigningKey), cfg.Policy()) + if err != nil { + return err + } + uploader, err := storage.NewUploader(files, cfg.Policy()) + if err != nil { + return err + } + c.Files, c.Uploads, c.Downloads, c.UploadLinks, c.FileHTTP = files, uploader, signer, signer, signer + + return nil +} diff --git a/examples/userService/internal/settings/config.go b/examples/userService/internal/settings/config.go new file mode 100644 index 0000000..12d4107 --- /dev/null +++ b/examples/userService/internal/settings/config.go @@ -0,0 +1,193 @@ +// Package settings owns this project's configuration; it adds no business identity model. +package settings + +import ( + "errors" + "fmt" + "git.yuntuo.hk/k2safe/go-omni/config" + "git.yuntuo.hk/k2safe/go-omni/database" + "git.yuntuo.hk/k2safe/go-omni/logx" + "git.yuntuo.hk/k2safe/go-omni/storage" + "net/url" + "os" + "strconv" + "strings" + "time" +) + +type Config struct { + Server config.Config `json:"server"` + Database database.Config `json:"database"` + Storage StorageConfig `json:"storage"` +} + +func Default() Config { + server := config.Default() + server.Address = "127.0.0.1:8080" + server.EnableLogger = true + server.SlowRequestThreshold = 500 * time.Millisecond + server.Log.Service = "example.com/userService" + server.Log.Format = "json" + server.Log.File = "./var/log/server.log" + server.Log.Rotation = logx.RotationConfig{Enabled: true, MaxSizeMB: 100, MaxBackups: 10, MaxAgeDays: 7, Compress: true} + + return Config{Server: server, Database: database.DefaultConfig(), Storage: StorageConfig{ + Provider: "local", Root: "./var/files", PublicURL: "http://127.0.0.1:8080/storage/object", + Secure: true, MaxBytes: 32 << 20, ContentTypes: []string{"text/plain", "image/png", "image/jpeg", "application/pdf"}, + LinkTTL: 5 * time.Minute, RequestTimeout: 30 * time.Second, + }} +} + +// Load uses defaults -> strict YAML/JSON -> nonempty environment variables. +// Pass an empty path to use only defaults and environment. +func Load(path string) (Config, error) { + cfg := Default() + var err error + if path != "" { + cfg, err = config.DecodeFile(path, cfg) + if err != nil { + return Config{}, err + } + } + cfg.Server, err = config.ApplyEnv(cfg.Server, "") + if err != nil { + return Config{}, err + } + // A stream explicitly selected by environment overrides a file profile. + if output := strings.TrimSpace(os.Getenv("GO_OMNI_LOG_OUTPUT")); output != "" && strings.TrimSpace(os.Getenv("GO_OMNI_LOG_FILE")) == "" { + cfg.Server.Log.File = "" + cfg.Server.Log.Rotation = logx.RotationConfig{} + } + + if err = cfg.Storage.applyEnv(); err != nil { + return Config{}, err + } + + cfg.Database, err = database.ApplyEnv(cfg.Database, "") + if err != nil { + return Config{}, err + } + if err = cfg.Validate(); err != nil { + return Config{}, err + } + return cfg, nil +} +func (c Config) Validate() error { + + if err := c.Database.Validate(); err != nil { + return err + } + if c.Database.Driver != "mysql" || c.Database.MaxOpenConns < 1 { + return errors.New("database requires driver mysql and a positive maxOpenConns") + } + + if err := c.Server.Validate(); err != nil { + return err + } + if c.Server.Log.Format != "json" && c.Server.Log.Format != "text" { + return errors.New("log.format must be json or text") + } + switch strings.ToLower(c.Server.Log.Level) { + case "debug", "info", "warn", "warning", "error": + default: + return errors.New("invalid log.level") + } + return c.Storage.Validate() +} + +type StorageConfig struct { + Provider string `json:"provider"` + Root string `json:"root"` + PublicURL string `json:"publicUrl"` + Endpoint string `json:"endpoint"` + Region string `json:"region"` + Bucket string `json:"bucket"` + AccountID string `json:"accountId"` + Secure bool `json:"secure"` + MaxBytes int64 `json:"maxBytes"` + ContentTypes []string `json:"contentTypes"` + LinkTTL time.Duration `json:"linkTtl"` + RequestTimeout time.Duration `json:"requestTimeout"` + // Secrets are environment-only and cannot be populated by config files. + FilesToken string `json:"-"` + SigningKey string `json:"-"` + AccessKey string `json:"-"` + SecretKey string `json:"-"` + SessionToken string `json:"-"` +} + +func (s StorageConfig) Policy() storage.UploadPolicy { + return storage.UploadPolicy{MaxBytes: s.MaxBytes, ContentTypes: s.ContentTypes} +} +func (s *StorageConfig) applyEnv() error { + values := map[string]*string{ + "GO_OMNI_STORAGE_ROOT": &s.Root, "GO_OMNI_STORAGE_PUBLIC_URL": &s.PublicURL, + "GO_OMNI_STORAGE_ENDPOINT": &s.Endpoint, "GO_OMNI_STORAGE_REGION": &s.Region, + "GO_OMNI_STORAGE_BUCKET": &s.Bucket, "GO_OMNI_STORAGE_ACCOUNT_ID": &s.AccountID, + "GO_OMNI_STORAGE_SIGNING_KEY": &s.SigningKey, "GO_OMNI_FILES_TOKEN": &s.FilesToken, + "GO_OMNI_STORAGE_ACCESS_KEY": &s.AccessKey, "GO_OMNI_STORAGE_SECRET_KEY": &s.SecretKey, + "GO_OMNI_STORAGE_SESSION_TOKEN": &s.SessionToken, + } + for name, target := range values { + if value := strings.TrimSpace(os.Getenv(name)); value != "" { + *target = value + } + } + for name, target := range map[string]*time.Duration{"GO_OMNI_STORAGE_LINK_TTL": &s.LinkTTL, "GO_OMNI_STORAGE_REQUEST_TIMEOUT": &s.RequestTimeout} { + if value := strings.TrimSpace(os.Getenv(name)); value != "" { + v, err := time.ParseDuration(value) + if err != nil { + return fmt.Errorf("%s must be a duration", name) + } + *target = v + } + } + if value := os.Getenv("GO_OMNI_STORAGE_MAX_BYTES"); value != "" { + v, err := strconv.ParseInt(value, 10, 64) + if err != nil { + return errors.New("GO_OMNI_STORAGE_MAX_BYTES must be an integer") + } + s.MaxBytes = v + } + if value := os.Getenv("GO_OMNI_STORAGE_SECURE"); value != "" { + v, err := strconv.ParseBool(value) + if err != nil { + return errors.New("GO_OMNI_STORAGE_SECURE must be a boolean") + } + s.Secure = v + } + if value := strings.TrimSpace(os.Getenv("GO_OMNI_STORAGE_CONTENT_TYPES")); value != "" { + s.ContentTypes = strings.Split(value, ",") + for i := range s.ContentTypes { + s.ContentTypes[i] = strings.TrimSpace(s.ContentTypes[i]) + } + } + return nil +} +func (s StorageConfig) Validate() error { + if s.Provider != "local" { + return errors.New("storage.provider must match the adapter selected at generation") + } + if err := s.Policy().Validate(); err != nil { + return err + } + if s.LinkTTL < time.Second || s.LinkTTL > time.Hour { + return errors.New("storage.linkTtl must be 1s..1h") + } + if s.RequestTimeout <= 0 || s.RequestTimeout > 5*time.Minute { + return errors.New("storage.requestTimeout must be >0 and <=5m") + } + + if s.Root == "" { + return errors.New("storage.root is required") + } + u, err := url.Parse(s.PublicURL) + if err != nil || u.Host == "" || u.Hostname() == "" || (u.Scheme != "http" && u.Scheme != "https") || u.User != nil || u.RawQuery != "" || u.Fragment != "" || u.Path != "/storage/object" { + return errors.New("storage.publicUrl must be an absolute HTTP(S) URL ending in /storage/object") + } + if len(s.SigningKey) < 32 { + return errors.New("set GO_OMNI_STORAGE_SIGNING_KEY to a stable secret of at least 32 bytes") + } + + return nil +} diff --git a/examples/userService/internal/types/typesGen.go b/examples/userService/internal/types/typesGen.go new file mode 100644 index 0000000..aa151ad --- /dev/null +++ b/examples/userService/internal/types/typesGen.go @@ -0,0 +1,35 @@ +// Code generated by go-omni api. DO NOT EDIT. +package types + +type UserResp struct { + ID int64 `json:"id"` + Name string `json:"name"` + Email string `json:"email"` + CreatedAt string `json:"createdAt"` + UpdatedAt string `json:"updatedAt"` +} +type CreateUserReq struct { + Name string `json:"name" binding:"required,max=100"` + Email string `json:"email" binding:"required,email,max=254"` +} +type UserIDReq struct { + ID int64 `path:"id,range=[1:9007199254740991]"` +} +type UpdateUserReq struct { + ID int64 `path:"id,range=[1:9007199254740991]"` + Name string `json:"name" binding:"required,max=100"` + Email string `json:"email" binding:"required,email,max=254"` +} +type ListUsersReq struct { + Page int `form:"page,default=1,range=[1:1000000]"` + PageSize int `form:"pageSize,default=20,range=[1:100]"` +} +type UserPageResp struct { + Items []UserResp `json:"items"` + Total int `json:"total"` + Page int `json:"page"` + PageSize int `json:"pageSize"` +} +type ExportResp struct { + DownloadURL string `json:"downloadUrl"` +} diff --git a/examples/userService/migrations/000001_users.down.sql b/examples/userService/migrations/000001_users.down.sql new file mode 100644 index 0000000..e905ee3 --- /dev/null +++ b/examples/userService/migrations/000001_users.down.sql @@ -0,0 +1,2 @@ +DROP TABLE user_audit; +DROP TABLE users; diff --git a/examples/userService/migrations/000001_users.up.sql b/examples/userService/migrations/000001_users.up.sql new file mode 100644 index 0000000..2de3d81 --- /dev/null +++ b/examples/userService/migrations/000001_users.up.sql @@ -0,0 +1,18 @@ +CREATE TABLE users ( + id BIGINT NOT NULL AUTO_INCREMENT, + name VARCHAR(100) NOT NULL, + email VARCHAR(254) NOT NULL, + created_at DATETIME(6) NOT NULL, + updated_at DATETIME(6) NOT NULL, + PRIMARY KEY (id), + UNIQUE KEY uq_users_email (email) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; + +CREATE TABLE user_audit ( + id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY, + user_id BIGINT NOT NULL, + operation VARCHAR(16) NOT NULL, + request_id VARCHAR(128) NOT NULL, + created_at DATETIME(6) NOT NULL, + INDEX ix_user_audit_user (user_id) +) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; diff --git a/go.mod b/go.mod index 4ededc7..087d1fd 100644 --- a/go.mod +++ b/go.mod @@ -17,6 +17,7 @@ require ( go.opentelemetry.io/otel/trace v1.31.0 golang.org/x/mod v0.21.0 golang.org/x/net v0.33.0 + gopkg.in/natefinch/lumberjack.v2 v2.2.1 gopkg.in/yaml.v3 v3.0.1 ) diff --git a/go.sum b/go.sum index babf363..871ab2d 100644 --- a/go.sum +++ b/go.sum @@ -130,5 +130,7 @@ golang.org/x/time v0.5.0 h1:o7cqy6amK/52YcAKIPlM3a+Fpj35zvRj2TP+e1xFSfk= golang.org/x/time v0.5.0/go.mod h1:3BpzKBy/shNhVucY/MWOyx10tF3SFh9QdLuxbVysPQM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/natefinch/lumberjack.v2 v2.2.1 h1:bBRl1b0OH9s/DuPhuXpNl+VtCaJXFZ5/uEFST95x9zc= +gopkg.in/natefinch/lumberjack.v2 v2.2.1/go.mod h1:YD8tP3GAjkrDg1eZH7EGmyESg/lsYskCTPBJVb9jqSc= gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/internal/storagefixture/multipart.go b/internal/storagefixture/multipart.go new file mode 100644 index 0000000..eb089c5 --- /dev/null +++ b/internal/storagefixture/multipart.go @@ -0,0 +1,161 @@ +package storagefixture + +import ( + "crypto/md5" + "encoding/xml" + "fmt" + "io" + "net/http" + "sort" + "strconv" + "strings" + "sync" + "time" +) + +// MultipartHandler checks protocol sequencing, not provider authentication. +// The fixture is memory bounded and should only be mounted on httptest servers. +func MultipartHandler() http.Handler { + type part struct { + Number int `xml:"PartNumber"` + ETag string `xml:"ETag"` + Size int64 `xml:"Size"` + data []byte + } + type upload struct { + key, contentType string + parts map[int]part + } + var mu sync.Mutex + uploads := map[string]*upload{} + objects := map[string]value{} + sequence := 0 + simple := Handler() + return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + q := r.URL.Query() + id := q.Get("uploadId") + if _, start := q["uploads"]; !start && id == "" { + mu.Lock() + v, ok := objects[r.URL.Path] + if ok && r.Method == "DELETE" { + delete(objects, r.URL.Path) + mu.Unlock() + w.WriteHeader(204) + return + } + mu.Unlock() + if ok && (r.Method == "GET" || r.Method == "HEAD") { + w.Header().Set("Content-Length", strconv.Itoa(len(v.data))) + w.Header().Set("Content-Type", v.contentType) + w.Header().Set("ETag", v.etag) + w.Header().Set("Last-Modified", time.Now().UTC().Format(http.TimeFormat)) + if r.Method == "GET" { + w.Write(v.data) + } + return + } + simple.ServeHTTP(w, r) + return + } + mu.Lock() + defer mu.Unlock() + fail := func(code string, status int) { + w.Header().Set("Content-Type", "application/xml") + w.WriteHeader(status) + fmt.Fprintf(w, "%sfixture rejection", code) + } + if r.Header.Get("Authorization") == "" && q.Get("X-Amz-Signature") == "" && q.Get("x-oss-signature") == "" { + fail("AccessDenied", 403) + return + } + if _, ok := q["uploads"]; ok && r.Method == "POST" { + if len(uploads) >= 8 { + fail("TooManyUploads", 503) + return + } + sequence++ + id = strconv.Itoa(sequence) + uploads[id] = &upload{key: r.URL.Path, contentType: r.Header.Get("Content-Type"), parts: map[int]part{}} + fmt.Fprintf(w, "testbucketfile%s", id) + return + } + u, ok := uploads[id] + if !ok || u.key != r.URL.Path { + fail("NoSuchUpload", 404) + return + } + switch r.Method { + case "PUT": + number, _ := strconv.Atoi(q.Get("partNumber")) + if number < 1 || number > 3 { + fail("InvalidPart", 400) + return + } + data, err := io.ReadAll(io.LimitReader(r.Body, (6<<20)+1)) + if err != nil || len(data) > 6<<20 { + fail("EntityTooLarge", 413) + return + } + p := part{Number: number, ETag: fmt.Sprintf("\"%x\"", md5.Sum(data)), Size: int64(len(data)), data: data} + u.parts[number] = p + w.Header().Set("ETag", p.ETag) + case "GET": + marker, _ := strconv.Atoi(q.Get("part-number-marker")) + limit, _ := strconv.Atoi(q.Get("max-parts")) + if limit < 1 { + limit = 1000 + } + numbers := []int{} + for n := range u.parts { + if n > marker { + numbers = append(numbers, n) + } + } + sort.Ints(numbers) + var result struct { + XMLName xml.Name `xml:"ListPartsResult"` + Truncated bool `xml:"IsTruncated"` + Next int `xml:"NextPartNumberMarker"` + Parts []part `xml:"Part"` + } + if len(numbers) > limit { + result.Truncated = true + numbers = numbers[:limit] + result.Next = numbers[len(numbers)-1] + } + for _, n := range numbers { + result.Parts = append(result.Parts, u.parts[n]) + } + xml.NewEncoder(w).Encode(result) + case "POST": + var request struct { + Parts []part `xml:"Part"` + } + if xml.NewDecoder(io.LimitReader(r.Body, 65537)).Decode(&request) != nil || len(request.Parts) == 0 { + fail("InvalidPart", 400) + return + } + var data []byte + for _, p := range request.Parts { + actual, ok := u.parts[p.Number] + if !ok || strings.Trim(p.ETag, "\"") != strings.Trim(actual.ETag, "\"") { + fail("InvalidPart", 400) + return + } + if len(data)+len(actual.data) > 7<<20 { + fail("EntityTooLarge", 413) + return + } + data = append(data, actual.data...) + } + objects[u.key] = value{data: data, contentType: u.contentType, etag: fmt.Sprintf("\"%x\"", md5.Sum(data))} + delete(uploads, id) + fmt.Fprint(w, "testbucketfile\"completed\"") + case "DELETE": + delete(uploads, id) + w.WriteHeader(204) + default: + fail("InvalidRequest", 400) + } + }) +} diff --git a/logx/README.md b/logx/README.md index 6796356..ec15cfc 100644 --- a/logx/README.md +++ b/logx/README.md @@ -1,31 +1,62 @@ # logx -`logx` 是 GoOmni 的业务层结构化日志入口。业务代码不需要依赖具体网络库： +业务通过 `logx` 记录结构化日志，独立于 HTTP、队列和具体业务字段。普通 `context.Context` 继续由业务管理；框架不会约定或自动注入 userId、tenantId。 ```go -import "git.yuntuo.hk/k2safe/go-omni/logx" - -func (s *UserService) Create(ctx context.Context, input CreateInput) error { - logx.InfoContext(ctx, "user.create.start", "email", input.Email) - // ... - return nil -} +logx.InfoContext(ctx, "order.created", "orderId", orderID) +logx.ErrorContext(ctx, "order.failed", "error", err) +// 可选：附上本次记录位置的当前 goroutine 堆栈。 +logx.ErrorStackContext(ctx, "unexpected.failure", "error", err) +// 固定字段可使用子日志对象，无需重复传入。 +logger := logx.With("component", "invoice") +logger.InfoContext(ctx, "invoice.saved", "invoiceId", invoiceID) ``` -应用启动时可以替换默认输出： +`Debug/Info/Warn/Error` 有包级和实例方法，也有带 Context 的方法。`requestId` 自动跟随 HTTP Context；接入 telemetry 后可带 `traceId`。不要在 Service 中改用 `context.Background()`，否则会丢失取消信号和关联字段。 + +## 文件轮转 ```go closeLog, err := logx.Setup(logx.Config{ - Service: "billing-api", - Environment: "production", - Level: "info", - Format: "json", - File: "./var/log/billing-api.log", + Service: "billing-api", Environment: "production", + Level: "info", Format: "json", File: "./var/log/billing-api.log", + Rotation: logx.RotationConfig{ + Enabled: true, MaxSizeMB: 100, MaxBackups: 10, + MaxAgeDays: 7, Compress: true, + }, + RedactKeys: []string{"bankAccount", "identityNumber"}, }) -if err != nil { - return err -} +if err != nil { return err } defer closeLog() ``` -HTTP 请求的 `X-Request-ID` 会被放进标准 `context.Context`，之后的 `InfoContext`、`ErrorContext` 会自动带上 `requestId`。后续可以在不改变业务调用方式的前提下增加文件轮转、采样和 OpenTelemetry 导出。 +也可用 `omni.WithLogConfig(...)` 交给应用运行生命周期管理。子日志共用输出、级别和关闭状态。独立 `logx.New` 返回的实例由创建者负责关闭。 + +轮转显式开启；三个数值为 0 时分别取 100 MiB、10 份、7 天。超过大小后轮转，旧文件按份数/年龄清理，可 gzip 压缩；保留天数不表示每天定时切割。压缩和历史清理由底层异步执行，不承诺 Close 等待压缩完成。轮转使用 [lumberjack v2](https://github.com/natefinch/lumberjack/tree/v2.2.1)，**一个日志文件只能由一个进程拥有**。容器通常直接用 stdout/stderr，让运行平台管理日志文件。 + +`rotation.enabled` 与自定义 Writer 不能同时使用；只设轮转但未提供 File 会报错。开启轮转时会创建父目录，并在初始化阶段检查文件可打开。自定义 Writer 的并发和生命周期由接入方负责。普通文件输出仍然是追加写入；未开启轮转不自动清理。 + +## 结构化脱敏 + +默认匹配 password、passwd、secret、clientSecret、token、accessToken、refreshToken、authorization、cookie、setCookie、apiKey、privateKey、signedUrl。匹配忽略大小写、下划线和连字符；`RedactKeys` 追加业务自行定义的键，不替换默认列表。 + +覆盖普通属性、With 属性、slog Group/LogValuer、嵌套字符串键 map、切片和导出 struct 字段。敏感值变为 `[REDACTED]`，不修改调用者的数据；循环/过深结构截断。嵌套字节数组隐藏为 `[BINARY]`，非字符串键 map 隐藏为 `[UNSUPPORTED MAP]`，不调用对象自定义 JSON 序列化方法。 + +**脱敏按字段名处理，不解析自由文本。** 把密码拼进 msg/error/URL 字符串，或使用不在列表中的字段名，仍可能泄露。签名链接、完整请求体和认证头不应写入日志。业务隐私字段自行放进 `RedactKeys`；日志中不会自动添加身份/租户约定。 + +## HTTP 异常与慢请求 + +```go +application := omni.New( + omni.WithLogger(true), + omni.WithSlowRequestThreshold(500*time.Millisecond), +) +``` + +`WithSlowRequestThreshold` 同时开启访问日志，0 禁用慢请求识别。也可以在 JSON/YAML 中配 `enableLogger: true`、`slowRequestThreshold: "500ms"`，或使用 `GO_OMNI_SLOW_REQUEST_THRESHOLD` 配合 `GO_OMNI_ENABLE_LOGGER=true`。 + +每个请求记录一条 `http.request`：方法、路径、匹配路由、实际状态码、耗时、客户端 IP 和关联字段。正常请求 INFO，超过阈值 WARN（`slow: true`），5xx ERROR；返回的 error 会记录到结构化字段。计时覆盖 Handler 调用，不包含框架之外的代理排队，也不自动计入 Handler 返回后继续写入的自管流。 + +Recovery 自动记录单独的 ERROR `http.panic` 和当前堆栈，再走原有统一错误响应；如果响应已提交，不改写已发送状态。客户端不会收到日志堆栈。普通业务 error 默认不抓堆栈，需要时显式使用 `ErrorStackContext`，它记录调用位置，不回溯 error 最初创建位置。 + +运行时可通过 `logger.SetLevel("debug")` 调整级别；文件轮转和脱敏配置属于静态设置，变更后重建实例/重启。当前不包含日志采样、异步缓冲、日志平台导出或慢 SQL 记录。 diff --git a/logx/logx.go b/logx/logx.go index b2870f0..0c6fc82 100644 --- a/logx/logx.go +++ b/logx/logx.go @@ -10,6 +10,7 @@ import ( "io" "log/slog" "os" + "runtime/debug" "strings" "sync" "sync/atomic" @@ -21,14 +22,16 @@ import ( // standard stream. File takes precedence over Output and appends to the named // file. type Config struct { - Level string `json:"level"` - Format string `json:"format"` - Output string `json:"output"` - File string `json:"file"` - Service string `json:"service"` - Environment string `json:"environment"` - AddSource bool `json:"addSource"` - Writer io.Writer `json:"-"` + Level string `json:"level"` + Format string `json:"format"` + Output string `json:"output"` + File string `json:"file"` + Service string `json:"service"` + Environment string `json:"environment"` + AddSource bool `json:"addSource"` + Writer io.Writer `json:"-"` + Rotation RotationConfig `json:"rotation"` + RedactKeys []string `json:"redactKeys"` } // Default returns the default production-friendly logger configuration. @@ -62,13 +65,17 @@ var ( func init() { // Keep package-level logging useful without requiring every service to // repeat setup. Applications can replace this logger with Setup. - defaultLogger.Store(&Logger{logger: slog.New(slog.NewJSONHandler(os.Stderr, nil))}) + logger, _ := New(Default()) + defaultLogger.Store(logger) } // New creates an independent logger. The caller owns its lifecycle and should // call Close when the process is shutting down. func New(cfg Config) (*Logger, error) { cfg = normalizeConfig(cfg) + if err := cfg.Rotation.validate(cfg); err != nil { + return nil, err + } level, err := parseLevel(cfg.Level) if err != nil { return nil, err @@ -81,7 +88,7 @@ func New(cfg Config) (*Logger, error) { dynamicLevel := new(slog.LevelVar) dynamicLevel.Set(level) - opts := &slog.HandlerOptions{Level: dynamicLevel, AddSource: cfg.AddSource} + opts := &slog.HandlerOptions{Level: dynamicLevel, AddSource: cfg.AddSource, ReplaceAttr: redactor(cfg.RedactKeys)} var handler slog.Handler if cfg.Format == "text" { handler = slog.NewTextHandler(writer, opts) @@ -208,6 +215,19 @@ func (l *Logger) ErrorContext(ctx context.Context, msg string, args ...any) { l.logger.Log(ctx, slog.LevelError, msg, contextAttrs(ctx, args)...) } +// ErrorStackContext records the current goroutine's stack at the logging call, +// not the historical location where an error was originally constructed. +func (l *Logger) ErrorStackContext(ctx context.Context, msg string, args ...any) { + attrs := make([]any, 0, len(args)+2) + attrs = append(attrs, args...) + attrs = append(attrs, "stack", string(debug.Stack())) + l.ErrorContext(ctx, msg, attrs...) +} + +func ErrorStackContext(ctx context.Context, msg string, args ...any) { + defaultLogger.Load().ErrorStackContext(ctx, msg, args...) +} + func (l *Logger) Debug(msg string, args ...any) { l.DebugContext(context.Background(), msg, args...) } func (l *Logger) Info(msg string, args ...any) { l.InfoContext(context.Background(), msg, args...) } func (l *Logger) Warn(msg string, args ...any) { l.WarnContext(context.Background(), msg, args...) } @@ -324,6 +344,9 @@ func openWriter(cfg Config) (io.Writer, func() error, error) { return cfg.Writer, nil, nil } if cfg.File != "" { + if cfg.Rotation.Enabled { + return rotatingWriter(cfg) + } file, err := os.OpenFile(cfg.File, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0o644) if err != nil { return nil, nil, err diff --git a/logx/production_test.go b/logx/production_test.go new file mode 100644 index 0000000..853fa09 --- /dev/null +++ b/logx/production_test.go @@ -0,0 +1,157 @@ +package logx + +import ( + "bytes" + "compress/gzip" + "context" + "encoding/json" + "io" + "log/slog" + "os" + "path/filepath" + "strings" + "sync" + "testing" + "time" +) + +type credentialsValue struct{} + +func (credentialsValue) LogValue() slog.Value { + return slog.GroupValue(slog.String("token", "do-not-print"), slog.String("label", "keep")) +} +func TestStructuredRedaction(t *testing.T) { + for _, format := range []string{"json", "text"} { + t.Run(format, func(t *testing.T) { + var out bytes.Buffer + logger, err := New(Config{Writer: &out, Format: format, RedactKeys: []string{"bankAccount"}}) + if err != nil { + t.Fatal(err) + } + defer logger.Close() + type account struct { + Name string `json:"name"` + Password string `json:"password"` + Account string `json:"bankAccount"` + } + nested := map[string]any{"profile": account{"visible", "do-not-print", "do-not-print"}, "Authorization": "do-not-print", "items": []any{map[string]string{"API_KEY": "do-not-print", "label": "visible"}}} + logger.With("refreshToken", "do-not-print").WithGroup("metadata").Info("event", "body", nested, slog.Group("password", "value", "do-not-print"), "valuer", credentialsValue{}) + if strings.Contains(out.String(), "do-not-print") || !strings.Contains(out.String(), "visible") || !strings.Contains(out.String(), redacted) { + t.Fatal(out.String()) + } + if format == "json" { + var value map[string]any + if err := json.Unmarshal(out.Bytes(), &value); err != nil { + t.Fatal(err) + } + } + }) + } +} +func TestRedactionDoesNotMutateOrLoop(t *testing.T) { + var out bytes.Buffer + logger, _ := New(Config{Writer: &out}) + m := map[string]any{"password": "original"} + m["self"] = m + logger.Info("event", "map", m) + if m["password"] != "original" || strings.Contains(out.String(), "original") || !strings.Contains(out.String(), "TRUNCATED") { + t.Fatal(out.String()) + } +} +func TestRotationCompressRetentionAndClose(t *testing.T) { + dir := t.TempDir() + file := filepath.Join(dir, "logs", "app.log") + logger, err := New(Config{File: file, Rotation: RotationConfig{Enabled: true, MaxSizeMB: 1, MaxBackups: 2, MaxAgeDays: 1, Compress: true}}) + if err != nil { + t.Fatal(err) + } + payload := strings.Repeat("x", 600<<10) + for i := 0; i < 5; i++ { + logger.Info("rotation", "value", payload) + time.Sleep(3 * time.Millisecond) + } + deadline := time.Now().Add(3 * time.Second) + var backups []string + for time.Now().Before(deadline) { + backups, _ = filepath.Glob(filepath.Join(dir, "logs", "app-*.gz")) + plain, _ := filepath.Glob(filepath.Join(dir, "logs", "app-*.log")) + if len(backups) == 2 && len(plain) == 0 { + break + } + time.Sleep(10 * time.Millisecond) + } + if len(backups) != 2 { + t.Fatalf("backups: %v", backups) + } + for _, path := range backups { + f, err := os.Open(path) + if err != nil { + t.Fatal(err) + } + z, err := gzip.NewReader(f) + if err != nil { + t.Fatal(err) + } + data, err := io.ReadAll(z) + z.Close() + f.Close() + if err != nil || !json.Valid(bytes.TrimSpace(data)) { + t.Fatal("corrupt backup", err) + } + } + child := logger.With("component", "test") + if err = logger.Close(); err != nil { + t.Fatal(err) + } + before, _ := os.Stat(file) + child.Info("must not reopen") + after, _ := os.Stat(file) + if before.Size() != after.Size() { + t.Fatal("closed logger wrote") + } +} +func TestRotationConcurrentWritesAndValidation(t *testing.T) { + for _, cfg := range []Config{{Rotation: RotationConfig{Enabled: true}}, {File: "unused", Writer: io.Discard, Rotation: RotationConfig{Enabled: true}}, {Rotation: RotationConfig{MaxBackups: -1}}} { + if _, err := New(cfg); err == nil { + t.Fatal("invalid rotation accepted") + } + } + file := filepath.Join(t.TempDir(), "app.log") + logger, err := New(Config{File: file, Rotation: RotationConfig{Enabled: true}}) + if err != nil { + t.Fatal(err) + } + var wg sync.WaitGroup + for i := 0; i < 8; i++ { + wg.Add(1) + go func() { + defer wg.Done() + for j := 0; j < 100; j++ { + logger.Info("concurrent", "n", j) + } + }() + } + wg.Wait() + logger.Close() + raw, _ := os.ReadFile(file) + if len(bytes.Split(bytes.TrimSpace(raw), []byte("\n"))) != 800 { + t.Fatal("lost logs") + } +} + +func TestBusinessErrorStack(t *testing.T) { + var out bytes.Buffer + logger, _ := New(Config{Writer: &out, AddSource: true}) + ctx := WithRequestID(context.Background(), "request") + logger.ErrorStackContext(ctx, "failure", "password", "do-not-print") + var entry map[string]any + if err := json.Unmarshal(out.Bytes(), &entry); err != nil { + t.Fatal(err) + } + if !strings.Contains(entry["stack"].(string), "TestBusinessErrorStack") || entry["requestId"] != "request" || strings.Contains(out.String(), "do-not-print") { + t.Fatal(out.String()) + } + if entry["source"].(map[string]any)["file"] == nil { + t.Fatal("source schema changed") + } +} diff --git a/logx/redaction.go b/logx/redaction.go new file mode 100644 index 0000000..063dda5 --- /dev/null +++ b/logx/redaction.go @@ -0,0 +1,133 @@ +package logx + +import ( + "log/slog" + "reflect" + "strings" + "time" + "unicode" +) + +const redacted = "[REDACTED]" + +// Redaction applies to structured keys, including nested maps and structs. +// Free text (messages, errors, URLs embedded in strings) is not parsed. +func redactor(extra []string) func([]string, slog.Attr) slog.Attr { + keys := make(map[string]bool) + for _, key := range append([]string{"password", "passwd", "secret", "clientSecret", "token", "accessToken", "refreshToken", "authorization", "cookie", "setCookie", "apiKey", "privateKey", "signedUrl"}, extra...) { + keys[normalizedKey(key)] = true + } + return func(groups []string, a slog.Attr) slog.Attr { + for _, group := range groups { + if keys[normalizedKey(group)] { + return slog.String(a.Key, redacted) + } + } + if keys[normalizedKey(a.Key)] { + return slog.String(a.Key, redacted) + } + a.Value = a.Value.Resolve() + if a.Value.Kind() == slog.KindAny { + a.Value = slog.AnyValue(sanitize(reflect.ValueOf(a.Value.Any()), keys, 0)) + } + return a + } +} + +func normalizedKey(key string) string { + return strings.Map(func(r rune) rune { + if unicode.IsLetter(r) || unicode.IsDigit(r) { + return unicode.ToLower(r) + } + return -1 + }, key) +} + +func sanitize(v reflect.Value, keys map[string]bool, depth int) any { + if !v.IsValid() { + return nil + } + if depth >= 16 { + return "[TRUNCATED]" + } + if (v.Kind() == reflect.Pointer || v.Kind() == reflect.Interface) && v.IsNil() { + return nil + } + if v.CanInterface() { + switch x := v.Interface().(type) { + case *slog.Source: + return sanitize(reflect.ValueOf(map[string]any{"function": x.Function, "file": x.File, "line": x.Line}), keys, depth+1) + case time.Time: + return x + case error: + return x.Error() + case []slog.Attr: + out := make(map[string]any, len(x)) + for _, a := range x { + if keys[normalizedKey(a.Key)] { + out[a.Key] = redacted + } else { + out[a.Key] = sanitize(reflect.ValueOf(a.Value.Resolve().Any()), keys, depth+1) + } + } + return out + case slog.LogValuer: + value := slog.AnyValue(x).Resolve() + return sanitize(reflect.ValueOf(value.Any()), keys, depth+1) + } + } + switch v.Kind() { + case reflect.Interface, reflect.Pointer: + return sanitize(v.Elem(), keys, depth+1) + case reflect.Map: + if v.Type().Key().Kind() != reflect.String { + return "[UNSUPPORTED MAP]" + } + out := make(map[string]any, v.Len()) + it := v.MapRange() + for it.Next() { + key := it.Key().String() + if keys[normalizedKey(key)] { + out[key] = redacted + } else { + out[key] = sanitize(it.Value(), keys, depth+1) + } + } + return out + case reflect.Struct: + out := make(map[string]any) + for i := 0; i < v.NumField(); i++ { + field := v.Type().Field(i) + if !field.IsExported() { + continue + } + name := strings.Split(field.Tag.Get("json"), ",")[0] + if name == "-" { + continue + } + if name == "" { + name = field.Name + } + if keys[normalizedKey(name)] || keys[normalizedKey(field.Name)] { + out[name] = redacted + } else { + out[name] = sanitize(v.Field(i), keys, depth+1) + } + } + return out + case reflect.Slice, reflect.Array: + if v.Type().Elem().Kind() == reflect.Uint8 { + return "[BINARY]" + } + out := make([]any, v.Len()) + for i := range out { + out[i] = sanitize(v.Index(i), keys, depth+1) + } + return out + default: + if v.CanInterface() { + return v.Interface() + } + return nil + } +} diff --git a/logx/rotation.go b/logx/rotation.go new file mode 100644 index 0000000..b5f17e4 --- /dev/null +++ b/logx/rotation.go @@ -0,0 +1,83 @@ +package logx + +import ( + "errors" + "io" + "os" + "path/filepath" + "sync" + + "gopkg.in/natefinch/lumberjack.v2" +) + +// RotationConfig opts a file logger into size-based rotation. Zero limits use +// 100 MiB, 10 backups and 7 days. Each file must have a single process owner. +type RotationConfig struct { + Enabled bool `json:"enabled"` + MaxSizeMB int `json:"maxSizeMb"` + MaxBackups int `json:"maxBackups"` + MaxAgeDays int `json:"maxAgeDays"` + Compress bool `json:"compress"` +} + +func (r RotationConfig) validate(cfg Config) error { + if r.MaxSizeMB < 0 || r.MaxSizeMB > 1024*1024 || r.MaxBackups < 0 || r.MaxBackups > 10000 || r.MaxAgeDays < 0 || r.MaxAgeDays > 36500 { + return errors.New("logx: invalid rotation limits") + } + if r.Enabled && (cfg.File == "" || cfg.Writer != nil) { + return errors.New("logx: rotation requires a file and no custom writer") + } + return nil +} + +func rotatingWriter(cfg Config) (io.Writer, func() error, error) { + r := cfg.Rotation + if r.MaxSizeMB == 0 { + r.MaxSizeMB = 100 + } + if r.MaxBackups == 0 { + r.MaxBackups = 10 + } + if r.MaxAgeDays == 0 { + r.MaxAgeDays = 7 + } + if err := os.MkdirAll(filepath.Dir(cfg.File), 0700); err != nil { + return nil, nil, err + } + // Fail at setup, rather than silently dropping the first log record. + f, err := os.OpenFile(cfg.File, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0600) + if err != nil { + return nil, nil, err + } + if err = f.Close(); err != nil { + return nil, nil, err + } + w := &rotationWriter{writer: &lumberjack.Logger{Filename: cfg.File, MaxSize: r.MaxSizeMB, MaxBackups: r.MaxBackups, MaxAge: r.MaxAgeDays, Compress: r.Compress}} + return w, w.Close, nil +} + +// Lumberjack can reopen after Close; prevent children of a closed logger from +// resurrecting its file. The lock also serializes shutdown with pending writes. +type rotationWriter struct { + mu sync.Mutex + writer *lumberjack.Logger + closed bool +} + +func (w *rotationWriter) Write(p []byte) (int, error) { + w.mu.Lock() + defer w.mu.Unlock() + if w.closed { + return 0, io.ErrClosedPipe + } + return w.writer.Write(p) +} +func (w *rotationWriter) Close() error { + w.mu.Lock() + defer w.mu.Unlock() + if w.closed { + return nil + } + w.closed = true + return w.writer.Close() +} diff --git a/middleware/middleware.go b/middleware/middleware.go index ed5de5e..025505b 100644 --- a/middleware/middleware.go +++ b/middleware/middleware.go @@ -2,6 +2,7 @@ package middleware import ( "fmt" + "runtime/debug" "time" apierrors "git.yuntuo.hk/k2safe/go-omni/errors" @@ -37,6 +38,7 @@ func Recovery() httpx.Middleware { return func(ctx *httpx.Context) (err error) { defer func() { if recovered := recover(); recovered != nil { + logx.ErrorContext(ctx.Context(), "http.panic", "panic", recovered, "stack", string(debug.Stack()), "method", ctx.Request().Method, "route", httpx.RoutePattern(ctx.Context())) err = apierrors.Internal(fmt.Errorf("panic recovered: %v", recovered)) } }() @@ -48,7 +50,25 @@ func Recovery() httpx.Middleware { // AccessLog records one structured event after each request. It deliberately // lives in GoOmni instead of using a router-specific logger so business // applications get the same log format with every transport adapter. -func AccessLog() httpx.Middleware { +type AccessLogOption func(*accessLogConfig) +type accessLogConfig struct{ slowThreshold time.Duration } + +// WithSlowRequestThreshold logs slow requests at WARN. Zero disables detection. +func WithSlowRequestThreshold(threshold time.Duration) AccessLogOption { + return func(c *accessLogConfig) { + if threshold >= 0 { + c.slowThreshold = threshold + } + } +} + +func AccessLog(options ...AccessLogOption) httpx.Middleware { + settings := accessLogConfig{} + for _, option := range options { + if option != nil { + option(&settings) + } + } return func(next httpx.Handler) httpx.Handler { return func(ctx *httpx.Context) error { started := time.Now() @@ -57,14 +77,30 @@ func AccessLog() httpx.Middleware { if err != nil && !ctx.Responded() { status = apierrors.From(err).Status } - logx.InfoContext(ctx.Context(), "http.request", + duration := time.Since(started) + slow := settings.slowThreshold > 0 && duration >= settings.slowThreshold + attrs := []any{ "method", ctx.Request().Method, "path", ctx.Request().URL.Path, "route", httpx.RoutePattern(ctx.Context()), "status", status, - "durationMs", time.Since(started).Seconds()*1000, + "durationMs", duration.Seconds() * 1000, "clientIp", ctx.ClientIP(), - ) + } + if slow { + attrs = append(attrs, "slow", true) + } + if err != nil { + attrs = append(attrs, "error", err) + } + switch { + case status >= 500: + logx.ErrorContext(ctx.Context(), "http.request", attrs...) + case slow: + logx.WarnContext(ctx.Context(), "http.request", attrs...) + default: + logx.InfoContext(ctx.Context(), "http.request", attrs...) + } return err } } diff --git a/middleware/middleware_test.go b/middleware/middleware_test.go index da43b75..9737692 100644 --- a/middleware/middleware_test.go +++ b/middleware/middleware_test.go @@ -4,8 +4,11 @@ import ( "bytes" "encoding/json" "errors" + "io" "net/http/httptest" + "strings" "testing" + "time" apierrors "git.yuntuo.hk/k2safe/go-omni/errors" "git.yuntuo.hk/k2safe/go-omni/httpx" @@ -42,12 +45,19 @@ func TestAccessLogReportsCommittedStatus(t *testing.T) { ctx.WriteError(err) } var entry struct { + Message string `json:"msg"` Status int Route string RequestID string `json:"requestId"` } - if err := json.Unmarshal(output.Bytes(), &entry); err != nil { - t.Fatal(err) + decoder := json.NewDecoder(&output) + for { + if err := decoder.Decode(&entry); err != nil { + t.Fatal(err) + } + if entry.Message == "http.request" { + break + } } if entry.Status != tc.status || w.Code != tc.status || entry.Route != "/users/:id" || entry.RequestID != "logRequest" { t.Fatalf("log=%s response=%d; expected status %d and request metadata", &output, w.Code, tc.status) @@ -86,3 +96,48 @@ func TestAccessLogUsesVerifiedClientIP(t *testing.T) { t.Fatalf("expected verified client IP, got %q", entry.ClientIP) } } + +func TestSlowAndPanicLogs(t *testing.T) { + var out bytes.Buffer + logger, _ := logx.New(logx.Config{Writer: &out}) + logx.SetDefault(logger) + defer logx.SetDefault(nil) + ctx := httpx.NewContext(httptest.NewRequest("GET", "/panic?secret=hidden", nil), httptest.NewRecorder(), nil) + chain := middleware.RequestID("X-Request-ID")(middleware.AccessLog(middleware.WithSlowRequestThreshold(time.Nanosecond))(middleware.Recovery()(func(c *httpx.Context) error { panic("boom") }))) + err := chain(ctx) + if err == nil { + t.Fatal("panic not recovered") + } + decoder := json.NewDecoder(&out) + count := 0 + for { + var entry map[string]any + err := decoder.Decode(&entry) + if err == io.EOF { + break + } + if err != nil { + t.Fatal(err) + } + count++ + if entry["requestId"] == "" || entry["level"] != "ERROR" { + t.Fatal(entry) + } + if entry["msg"] == "http.panic" && !strings.Contains(entry["stack"].(string), "TestSlowAndPanicLogs") { + t.Fatal(entry) + } + if entry["msg"] == "http.request" && (entry["slow"] != true || entry["status"] != float64(500)) { + t.Fatal(entry) + } + } + if count != 2 { + t.Fatal(count) + } + out.Reset() + ctx = httpx.NewContext(httptest.NewRequest("GET", "/slow", nil), httptest.NewRecorder(), nil) + err = middleware.AccessLog(middleware.WithSlowRequestThreshold(time.Nanosecond))(func(c *httpx.Context) error { return c.OK("done") })(ctx) + var entry map[string]any + if err != nil || json.Unmarshal(out.Bytes(), &entry) != nil || entry["level"] != "WARN" || entry["slow"] != true { + t.Fatal(out.String(), err) + } +} diff --git a/omni.go b/omni.go index bfe55f2..85744b7 100644 --- a/omni.go +++ b/omni.go @@ -48,6 +48,10 @@ func WithRequestIDHeader(header string) Option { return config.WithRequestIDHead func WithLogger(enabled bool) Option { return config.WithLogger(enabled) } +func WithSlowRequestThreshold(threshold time.Duration) Option { + return config.WithSlowRequestThreshold(threshold) +} + func WithTrustedProxies(proxies ...string) Option { return config.WithTrustedProxies(proxies...) } func WithConfig(settings Config) Option { return config.WithConfig(settings) } diff --git a/openapi/openapi.go b/openapi/openapi.go index cf84f45..858fc4a 100644 --- a/openapi/openapi.go +++ b/openapi/openapi.go @@ -2,6 +2,7 @@ package openapi import ( + "bytes" "encoding/json" "errors" "fmt" @@ -15,14 +16,26 @@ import ( ) type Schema struct { - Type string `json:"type,omitempty"` - Format string `json:"format,omitempty"` - Description string `json:"description,omitempty"` - Properties map[string]*Schema `json:"properties,omitempty"` - Required []string `json:"required,omitempty"` - Items *Schema `json:"items,omitempty"` - AdditionalProperties *Schema `json:"additionalProperties,omitempty"` - Nullable bool `json:"nullable,omitempty"` + Type string `json:"type,omitempty"` + Format string `json:"format,omitempty"` + Description string `json:"description,omitempty"` + Properties map[string]*Schema `json:"properties,omitempty"` + Required []string `json:"required,omitempty"` + Items *Schema `json:"items,omitempty"` + AdditionalProperties *Schema `json:"additionalProperties,omitempty"` + AdditionalPropertiesAllowed *bool `json:"-"` + Nullable bool `json:"nullable,omitempty"` + Minimum *float64 `json:"minimum,omitempty"` + Maximum *float64 `json:"maximum,omitempty"` + ExclusiveMinimum bool `json:"exclusiveMinimum,omitempty"` + ExclusiveMaximum bool `json:"exclusiveMaximum,omitempty"` + MinLength *int `json:"minLength,omitempty"` + MaxLength *int `json:"maxLength,omitempty"` + MinItems *int `json:"minItems,omitempty"` + MaxItems *int `json:"maxItems,omitempty"` + Enum []any `json:"enum,omitempty"` + Default any `json:"default,omitempty"` + Not *Schema `json:"not,omitempty"` } type Parameter struct { Name string `json:"name"` @@ -135,7 +148,9 @@ func (a *API) Handle(router Registrar, op Operation, handler httpx.Handler, midd return err } var frozen any - if err = json.Unmarshal(encoded, &frozen); err != nil { + decoder := json.NewDecoder(bytes.NewReader(encoded)) + decoder.UseNumber() + if err = decoder.Decode(&frozen); err != nil { return err } a.mu.Lock() diff --git a/openapi/schemaJSON.go b/openapi/schemaJSON.go new file mode 100644 index 0000000..30742f5 --- /dev/null +++ b/openapi/schemaJSON.go @@ -0,0 +1,47 @@ +package openapi + +import ( + "bytes" + "encoding/json" +) + +// OpenAPI permits both a schema and a boolean for additionalProperties. Keep +// the existing schema field source-compatible and expose the boolean separately. +func (s Schema) MarshalJSON() ([]byte, error) { + type alias Schema + if s.AdditionalPropertiesAllowed == nil { + return json.Marshal(alias(s)) + } + return json.Marshal(struct { + alias + Allowed bool `json:"additionalProperties"` + }{alias(s), *s.AdditionalPropertiesAllowed}) +} +func (s *Schema) UnmarshalJSON(data []byte) error { + type alias Schema + var raw map[string]json.RawMessage + if err := json.Unmarshal(data, &raw); err != nil { + return err + } + var allowed *bool + if value, ok := raw["additionalProperties"]; ok && (string(value) == "true" || string(value) == "false") { + var v bool + _ = json.Unmarshal(value, &v) + allowed = &v + delete(raw, "additionalProperties") + var err error + data, err = json.Marshal(raw) + if err != nil { + return err + } + } + var a alias + decoder := json.NewDecoder(bytes.NewReader(data)) + decoder.UseNumber() + if err := decoder.Decode(&a); err != nil { + return err + } + *s = Schema(a) + s.AdditionalPropertiesAllowed = allowed + return nil +} diff --git a/openapi/schemaJSON_test.go b/openapi/schemaJSON_test.go new file mode 100644 index 0000000..b991a68 --- /dev/null +++ b/openapi/schemaJSON_test.go @@ -0,0 +1,48 @@ +package openapi + +import ( + "encoding/json" + "strings" + "testing" + + "git.yuntuo.hk/k2safe/go-omni/httpx" +) + +func TestSchemaAdditionalPropertiesAndExactNumbers(t *testing.T) { + for _, text := range []string{ + `{"type":"object","additionalProperties":false}`, + `{"type":"object","additionalProperties":true}`, + `{"type":"object","additionalProperties":{"type":"string"}}`, + `{"type":"integer","default":18446744073709551615,"enum":[18446744073709551615]}`, + } { + var s Schema + if err := json.Unmarshal([]byte(text), &s); err != nil { + t.Fatal(err) + } + b, err := json.Marshal(s) + if err != nil { + t.Fatal(err) + } + if strings.Contains(text, "18446744073709551615") && !strings.Contains(string(b), "18446744073709551615") { + t.Fatalf("lost precision: %s", b) + } + var roundTrip Schema + if err = json.Unmarshal(b, &roundTrip); err != nil { + t.Fatal(err) + } + } + api, _ := New("Exact numbers", "1") + s := &Schema{Type: "integer", Enum: []any{json.Number("18446744073709551615")}} + err := api.Handle(discardSchemaRoutes{}, Operation{Method: "get", Path: "/id", OperationID: "readId", Responses: map[int]Response{200: {Description: "ok", Schema: s}}}, func(*httpx.Context) error { return nil }) + if err != nil { + t.Fatal(err) + } + data, _ := api.JSON() + if !strings.Contains(string(data), "18446744073709551615") { + t.Fatal(string(data)) + } +} + +type discardSchemaRoutes struct{} + +func (discardSchemaRoutes) Handle(string, string, httpx.Handler, ...httpx.Middleware) {} diff --git a/performance/apiGenerationAcceptance.md b/performance/apiGenerationAcceptance.md new file mode 100644 index 0000000..844a73c --- /dev/null +++ b/performance/apiGenerationAcceptance.md @@ -0,0 +1,36 @@ +# API 契约生成阶段验收 + +日期：2026-09-23。目标已完成：从 `.api` 生成可运行项目、请求/响应 DTO、绑定校验、Handler、分组路由、Service / Repository 扩展位置与 OpenAPI；支持安全重复生成，保持业务与网络实现分离。 + +## 交付 + +- `api new` 可直接启动的问候项目；`new --api` / `api generate` 从自定义契约创建项目。 +- `api go -api ... -dir ...` 别名、`api validate`、只读 `api generate --check`。 +- 导入、分组、路径前缀、命名中间件、混合请求来源、类型与校验语义，详见 [API.md](../API.md)。 +- `contract` 同源绑定与文档编译器，包括严格 JSON、缺失/零值、嵌套集合、精确整数默认值及约束。 +- manifest 所有权、内容指纹、业务文件保留、过期生成文件清理、路径/符号链接/大小写保护及生成锁。 +- `examples/contracts` 多分组示例、框架/CLI/生成项目文档。 +- 实际联调发现并修复 native 在应用层提前返回 413 后仍等待未发送请求体的问题。现在放弃未读 body 并关闭连接，避免延迟响应或把 body 当成后续请求。Content-Length/chunked 两种 framing 均有回归测试；保留其他状态原有协议错误优先级。 + +## 检查结果 + +| 验证 | 结果 | +| --- | --- | +| `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=2 -count=1 -timeout=240s ./...` | 39 个有测试的包全部通过；包含原有 memory/mysql × none/local/s3/r2/oss 共 10 种生成组合 | +| 生成后的独立消费者模块 | starter 与多分组 contract 项目均通过编译、race 和请求测试 | +| `go vet ./...` | 通过 | +| `go build ./...` | 通过 | +| `git diff --check` | 通过 | +| Parser fuzz（2 workers，4s 目标时长） | 69,457 次输入执行，通过 | +| 严格 body binder fuzz（2 workers，4s 目标时长） | 109,477 次输入执行，通过 | +| `python3 scripts/verifyAPI.py` | starter/contract × nethttp/native，4 组共 54 项检查通过 | + +真实 HTTP 检查包括：就绪、文档与文件一致、问候接口、请求 ID、JSON/path/query/header 组合、false/0、必填和范围、重复标量、未知字段、尾随 JSON、413/415、认证 401、中间件顺序、成功 201/202/204、业务日志 requestId、SIGTERM 退出以及监听端口释放。更新契约后检查过期检测、重新生成和手写 Service 保留，再编译启动。 + +完整测试输出：[apiGenerationTests.txt](apiGenerationTests.txt)。真实服务报告及验收时源码 SHA-256：[apiGenerationLocal.json](apiGenerationLocal.json)。临时服务和消费者目录由验收脚本自动清理。 + +## 范围说明 + +本阶段是文档中列明的 go-zero 风格 `.api` 子集。未包含完整 goctl 语法、客户端 SDK、数据库模型生成、已有业务项目的自动迁移或线上发布。自定义 Service / 中间件初始为 501 占位，业务自行实现。 + +本轮使用本机临时服务、离线存储替身和已有测试。未连接业务服务器、未做云账号实网测试、未推送或创建发布标签。远程旧版本还不能使用这些新命令；当前通过本地框架 replace 验证。 diff --git a/performance/apiGenerationLocal.json b/performance/apiGenerationLocal.json new file mode 100644 index 0000000..b19d954 --- /dev/null +++ b/performance/apiGenerationLocal.json @@ -0,0 +1,115 @@ +{ + "passed": true, + "durationSeconds": 3.59, + "cases": [ + { + "project": "starter", + "transport": "nethttp", + "checks": [ + "readiness", + "servedOpenApiMatchesFile", + "greeting", + "requestId", + "gracefulShutdown", + "listenerReleased" + ], + "passed": true + }, + { + "project": "starter", + "transport": "native", + "checks": [ + "readiness", + "servedOpenApiMatchesFile", + "greeting", + "requestId", + "gracefulShutdown", + "listenerReleased" + ], + "passed": true + }, + { + "project": "contract", + "transport": "nethttp", + "checks": [ + "readiness", + "servedOpenApiMatchesFile", + "mixedSources", + "zeroAndFalse", + "pathRange", + "defaultAndQuery", + "requiredFields", + "unknownFields", + "duplicateScalar", + "trailingJson", + "rangeValidation", + "bodyLimit413", + "auth401", + "status201", + "status202", + "contentType415", + "middlewareComposition", + "noContent204", + "gracefulShutdown", + "listenerReleased", + "businessLogCorrelation" + ], + "passed": true + }, + { + "project": "contract", + "transport": "native", + "checks": [ + "readiness", + "servedOpenApiMatchesFile", + "mixedSources", + "zeroAndFalse", + "pathRange", + "defaultAndQuery", + "requiredFields", + "unknownFields", + "duplicateScalar", + "trailingJson", + "rangeValidation", + "bodyLimit413", + "auth401", + "status201", + "status202", + "contentType415", + "middlewareComposition", + "noContent204", + "gracefulShutdown", + "listenerReleased", + "businessLogCorrelation" + ], + "passed": true + } + ], + "sourceSha256": { + "cmd/go-omni/api.go": "59f3b8b45497a9f560d649ac5e7b622362f5f20edd81251d7dc76f21605fc90a", + "cmd/go-omni/apigen/apigen_test.go": "8e7c94795ffbc3ee062756a77cfeb274f3cd1a78f1dbc3b51ec4ceb14f478518", + "cmd/go-omni/apigen/compile.go": "6709da0b69e8d10d49a874447e00c93d0a819d8367b2188e9626bd0c7b3baef2", + "cmd/go-omni/apigen/generate.go": "373ec9834aa4ed71a5311c1ef40b78dc6c303aa884d7bb16a680058ca751b2bc", + "cmd/go-omni/apigen/parse.go": "e85fe261fdec2e3487851e265babd7af8c7f6153694ad6fb73177aac33f8ef95", + "cmd/go-omni/apigen/profile.go": "877939398384c44316e5e052d92c4a93fb3b74a6742df2414a0054913f3fca27", + "cmd/go-omni/apigen/profile_test.go": "5f3c091da241fe0f8b99ac19e9efd2c3177aea72e63a346f99fecc0f548aad1a", + "cmd/go-omni/apigen/templates.go": "9303a2fafe44cb7e763a2c5beb193c16fa0dc0c2dbde4cb6e14f2ba533d3e56e", + "cmd/go-omni/apigen/write.go": "1ce3871fd92f638d097c576583c169daa0ce2fa8010dbcd34a8cd7faa314a1a2", + "cmd/go-omni/main.go": "6f851d21792a9eb9fcbc5095e8cacb3dcee1c715a3980650c75381f0e46f51a9", + "contract/README.md": "ff3b56398cf676df401d84d1169dfff19fc83ee074b07fb4c16fdf5af2e8fe17", + "contract/bind.go": "8a6be9046c9a489a4bed78bca1a017f53e89ca8209b4c0f971b032709f0a8228", + "contract/contract.go": "e496e7a57a6e97c482d25bb7ad6c9d07186328d97095f4bdca84965956bd9912", + "contract/contract_test.go": "0b651bb4305baed8b78ac764786af11e3538858783c6e117c7294fde50ec0873", + "contract/rules.go": "3a8df741b4c5074fe8589acaf5bb584b9b779afbe0412f4ef1b8315c222706b7", + "examples/contracts/README.md": "a7ebe6983c8d89d8a745f924f3d6fb3aba72d31dccd197112c18f50bd17456a9", + "examples/contracts/shared.api": "cb0096bb1565916b13ae8d2a0bdad1e2ffe7410a417d25ac347e67c7b9d77dfc", + "examples/contracts/user.api": "59b98806a142b65f782e83a60f0723bc554833cec113f1c7fb20e09540870ae8", + "openapi/openapi.go": "ed04396c95fa7b5b1c13fdcd99d4cfc8617c20c54d7959d51f281b532d0b44f6", + "openapi/openapi_test.go": "c92f4a2f548f0304708ac157402fb8641273473e04c524226f2429c2b64b99ab", + "openapi/schemaJSON.go": "f400cbba429cd02dc4f5bc70810a99c5d5841fc2995e6380e5de7c0e1f79d3b0", + "openapi/schemaJSON_test.go": "e0e1c7c9380350e67d8e209ba4041e3e52fe2bd86af829ff770affa8814e9c28", + "scripts/verifyAPI.py": "8fb41860ccc4320b64e26dbb001a8bae588b95641332e84a34f2d7f00fa86c3e", + "transport/native/earlyReject_test.go": "58c7b67d7c2dbb8867dec6075b891acb2c1bbbf0ad886fa0cb061c7c5ba95b0e", + "transport/native/server.go": "3b2b33d2a4c0a25034f61c54797bd9e143d994b48e6a689bb3ebd2b8c08050ce" + } +} diff --git a/performance/apiGenerationTests.txt b/performance/apiGenerationTests.txt new file mode 100644 index 0000000..d826a43 --- /dev/null +++ b/performance/apiGenerationTests.txt @@ -0,0 +1,52 @@ +ok git.yuntuo.hk/k2safe/go-omni 1.364s +ok git.yuntuo.hk/k2safe/go-omni/app 2.158s +ok git.yuntuo.hk/k2safe/go-omni/audit 1.297s +ok git.yuntuo.hk/k2safe/go-omni/auth 1.341s +ok git.yuntuo.hk/k2safe/go-omni/cache 1.309s +ok git.yuntuo.hk/k2safe/go-omni/cache/redis 1.288s +ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni 1.723s +ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/apigen 5.678s +ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/scaffold 37.948s +ok git.yuntuo.hk/k2safe/go-omni/config 1.303s +ok git.yuntuo.hk/k2safe/go-omni/contract 1.327s +ok git.yuntuo.hk/k2safe/go-omni/database 1.287s +ok git.yuntuo.hk/k2safe/go-omni/database/migrate 1.329s +? git.yuntuo.hk/k2safe/go-omni/errors [no test files] +? git.yuntuo.hk/k2safe/go-omni/examples/basic [no test files] +ok git.yuntuo.hk/k2safe/go-omni/examples/modules 1.366s +ok git.yuntuo.hk/k2safe/go-omni/examples/platform 1.383s +ok git.yuntuo.hk/k2safe/go-omni/examples/reference 1.413s +? git.yuntuo.hk/k2safe/go-omni/examples/reference/handler [no test files] +? git.yuntuo.hk/k2safe/go-omni/examples/reference/model [no test files] +ok git.yuntuo.hk/k2safe/go-omni/examples/reference/repository 1.480s +? git.yuntuo.hk/k2safe/go-omni/examples/reference/service [no test files] +ok git.yuntuo.hk/k2safe/go-omni/health 1.301s +ok git.yuntuo.hk/k2safe/go-omni/httpc 1.395s +ok git.yuntuo.hk/k2safe/go-omni/httpx 1.324s +ok git.yuntuo.hk/k2safe/go-omni/internal/perfcheck 1.358s +? git.yuntuo.hk/k2safe/go-omni/internal/storagefixture [no test files] +? git.yuntuo.hk/k2safe/go-omni/internal/testutil [no test files] +ok git.yuntuo.hk/k2safe/go-omni/logx 1.671s +ok git.yuntuo.hk/k2safe/go-omni/metrics 1.671s +ok git.yuntuo.hk/k2safe/go-omni/middleware 1.355s +ok git.yuntuo.hk/k2safe/go-omni/middleware/redis 1.313s +ok git.yuntuo.hk/k2safe/go-omni/openapi 1.236s +ok git.yuntuo.hk/k2safe/go-omni/pagination 1.291s +? git.yuntuo.hk/k2safe/go-omni/queue [no test files] +ok git.yuntuo.hk/k2safe/go-omni/queue/redis 1.440s +ok git.yuntuo.hk/k2safe/go-omni/resilience 1.332s +? git.yuntuo.hk/k2safe/go-omni/response [no test files] +ok git.yuntuo.hk/k2safe/go-omni/storage 1.340s +ok git.yuntuo.hk/k2safe/go-omni/storage/local 1.328s +ok git.yuntuo.hk/k2safe/go-omni/storage/oss 1.465s +ok git.yuntuo.hk/k2safe/go-omni/storage/s3 10.988s +? git.yuntuo.hk/k2safe/go-omni/storage/storagetest [no test files] +ok git.yuntuo.hk/k2safe/go-omni/task 1.403s +ok git.yuntuo.hk/k2safe/go-omni/telemetry 1.348s +ok git.yuntuo.hk/k2safe/go-omni/transport 1.335s +ok git.yuntuo.hk/k2safe/go-omni/transport/http2 12.593s +? git.yuntuo.hk/k2safe/go-omni/transport/internal/routing [no test files] +ok git.yuntuo.hk/k2safe/go-omni/transport/native 1.999s +ok git.yuntuo.hk/k2safe/go-omni/transport/nethttp 1.320s +? git.yuntuo.hk/k2safe/go-omni/transport/transporttest [no test files] +? git.yuntuo.hk/k2safe/go-omni/validate [no test files] diff --git a/performance/apiWorkflowAcceptance.md b/performance/apiWorkflowAcceptance.md new file mode 100644 index 0000000..cbfe37c --- /dev/null +++ b/performance/apiWorkflowAcceptance.md @@ -0,0 +1,31 @@ +# API、基础模块与业务示例验收 + +本轮将 API 项目生成与数据库、存储、日志选项组合，新增可运行的用户管理示例。 +源码及 CLI 目标版本为 v0.7.0-preview。 + +## 本地结果 + +- 框架 `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=2 -count=1 -timeout=300s ./...`：39 个有测试包通过。 +- 包含 10 种 API 组合（none/mysql × none/local/s3/r2/oss）及既有 10 种 CRUD 模块组合；生成项目实际编译并运行测试。 +- `go vet ./...`、`go build ./...`、`git diff --check` 通过。 +- API 基础回归：starter 与契约项目，在 nethttp/native 共 54 项检查通过。 +- 独立用户示例：nethttp/native 共 30 项业务、存储和生命周期检查通过。 +- MySQL 真实事务：提交、错误回滚、panic 回滚、context 取消回滚；业务增/改/删在审计写入失败时整体回滚。 +- 通过 CLI 显式应用数据库迁移；数据库停机 readiness=503，恢复后重新就绪；进程重启数据仍在。 +- 28 份开发者拥有的文件在重生成后保持原字节，包括业务实现、配置、迁移和测试。 +- 存储初始化失败关闭此前打开的数据库连接，Close 可重复调用，关闭后不能重开存储。 + +数据库容器只绑定 localhost，限制 1 CPU / 768 MiB，使用随机临时密钥，退出删除容器及卷。 +所有服务器进程都是本机临时子进程；未使用业务服务器。OSS/R2 仍不包含云账号实测。 + +## 证据 + +- [完整框架测试](apiWorkflowTests.txt) +- [本地业务报告](apiWorkflowLocal.json) +- [API 回归](apiGenerationLocal.json) +- [模块脚手架回归](scaffoldModulesLocal.json) + +## 远程版本验收 + +发布标签后执行 `python3 scripts/verifyAPIWorkflow.py --version v0.7.0-preview --report performance/apiWorkflowRemote.json`。 +结果完成后记录于同目录 remote 报告；该检查从空 module cache 安装 CLI，并要求业务依赖没有本地 replace。 diff --git a/performance/apiWorkflowLocal.json b/performance/apiWorkflowLocal.json new file mode 100644 index 0000000..ab309eb --- /dev/null +++ b/performance/apiWorkflowLocal.json @@ -0,0 +1,151 @@ +{ + "passed": true, + "frameworkVersion": "local", + "noLocalReplace": false, + "durationSeconds": 19.79, + "regenerationPreservedHandwrittenFiles": 28, + "groups": [ + { + "transport": "nethttp", + "checks": [ + "authBeforeService", + "emptyPageAndQueryDefaults", + "contractValidation", + "createReadNormalize", + "uniqueEmailConflict", + "updateAndPagination", + "createUpdateDeleteRollbackOnAuditFailure", + "storageUploadAndSignedDownload", + "signedDownloadTamperRejected", + "openAPIMatchesContract", + "readinessOutageAndRecovery", + "persistenceAfterServiceRestart", + "deleteAndTransactionalAudit", + "gracefulStopAndListenerRelease", + "businessFileLogsAndSecretExclusion" + ] + }, + { + "transport": "native", + "checks": [ + "authBeforeService", + "emptyPageAndQueryDefaults", + "contractValidation", + "createReadNormalize", + "uniqueEmailConflict", + "updateAndPagination", + "createUpdateDeleteRollbackOnAuditFailure", + "storageUploadAndSignedDownload", + "signedDownloadTamperRejected", + "openAPIMatchesContract", + "readinessOutageAndRecovery", + "persistenceAfterServiceRestart", + "deleteAndTransactionalAudit", + "gracefulStopAndListenerRelease", + "businessFileLogsAndSecretExclusion" + ] + } + ], + "transactionTests": [ + "commit", + "errorRollback", + "panicRollback", + "cancellationRollback" + ], + "startupTests": [ + "storageFailureClosesDatabase", + "idempotentClose", + "closedContextCannotReopen" + ], + "migrationCLI": true, + "frameworkSum": null, + "sourceHashes": { + "cmd/go-omni/api.go": "59f3b8b45497a9f560d649ac5e7b622362f5f20edd81251d7dc76f21605fc90a", + "cmd/go-omni/api_test.go": "ee68b4b8b0155f4d123135b0706c6d17d8fc90e2bcfc1312aa4f9e7a9da2454b", + "cmd/go-omni/apigen/apigen_test.go": "8e7c94795ffbc3ee062756a77cfeb274f3cd1a78f1dbc3b51ec4ceb14f478518", + "cmd/go-omni/apigen/compile.go": "6709da0b69e8d10d49a874447e00c93d0a819d8367b2188e9626bd0c7b3baef2", + "cmd/go-omni/apigen/generate.go": "373ec9834aa4ed71a5311c1ef40b78dc6c303aa884d7bb16a680058ca751b2bc", + "cmd/go-omni/apigen/parse.go": "e85fe261fdec2e3487851e265babd7af8c7f6153694ad6fb73177aac33f8ef95", + "cmd/go-omni/apigen/profile.go": "877939398384c44316e5e052d92c4a93fb3b74a6742df2414a0054913f3fca27", + "cmd/go-omni/apigen/profile_test.go": "5f3c091da241fe0f8b99ac19e9efd2c3177aea72e63a346f99fecc0f548aad1a", + "cmd/go-omni/apigen/templates.go": "9303a2fafe44cb7e763a2c5beb193c16fa0dc0c2dbde4cb6e14f2ba533d3e56e", + "cmd/go-omni/apigen/write.go": "1ce3871fd92f638d097c576583c169daa0ce2fa8010dbcd34a8cd7faa314a1a2", + "cmd/go-omni/main.go": "6f851d21792a9eb9fcbc5095e8cacb3dcee1c715a3980650c75381f0e46f51a9", + "cmd/go-omni/main_test.go": "b8040d2b19a1c86672fe1b1be7fa1bf7f885aece79ded6ffa0cc670887b7136f", + "cmd/go-omni/migrate.go": "f1ce89fbe419205eb82172da2b90e8888905d2b85d1a9db2a72b568bcf849a70", + "cmd/go-omni/migrate_test.go": "15d9a4aaebf9f899e523e908f1fa3c2282eb1e6c8cadbefae6b68f1ae4e00e6e", + "cmd/go-omni/scaffold/addModule.go": "8f43bd68c2b3820cf1d036c4f2c5f9d06d1c7c8e5114f5081597d905f8d8cdcc", + "cmd/go-omni/scaffold/addModule_test.go": "667c45312f628f0fb5b8206ddef7ccc02a75f33b72f73fd6a17b03da0cb62e91", + "cmd/go-omni/scaffold/apiInfrastructure.go": "7542f6af48345249b84054411c449b950db85fa6c5c60e62620bf8864a1aaff3", + "cmd/go-omni/scaffold/moduleProfiles_test.go": "ae9790e5d3549d32fc72739a10a31ea858e5a8f8189cf1f5c1840bf80b3f8ad6", + "cmd/go-omni/scaffold/moduleTemplates/README.md.tmpl": "9d3bfc46cac5618bcff7d8a6b2cc9f123e818aeb3dee9bcdfe0f91db584994bd", + "cmd/go-omni/scaffold/moduleTemplates/handler-handler.go.tmpl": "729cd0f6faba53d5d2c1492935e12ac2adcefde4490748e25c18be7ccb3f7139", + "cmd/go-omni/scaffold/moduleTemplates/model-model.go.tmpl": "a005611f6b455d4d8469e316bc26fa571043b2da1834db2b3dd14a96d66221d7", + "cmd/go-omni/scaffold/moduleTemplates/module.go.tmpl": "fb0be7fd571ba2c0edb01d70d68cdbbfe53561caab98346bf1ac7cd8df07205a", + "cmd/go-omni/scaffold/moduleTemplates/repository-repository.go.tmpl": "902da3d9f12395fb226178c4428ff547e1c2fce3261882dba98d1400d8c90747", + "cmd/go-omni/scaffold/moduleTemplates/service-service.go.tmpl": "49d26044a69a43e983aa5ba4b5b2399b26b77898b23e230751ad1372c2af3178", + "cmd/go-omni/scaffold/scaffold.go": "71b3091b83c3e45ac1981164e8c819c997ad96607d535f1492333370f4872548", + "cmd/go-omni/scaffold/scaffold_test.go": "58aab9251b01e394bbc90aeb538ff035dd32c73c0d4815132ac6cbc086527cda", + "cmd/go-omni/scaffold/templates/README.md.tmpl": "8ec215775fcafd437480022307fffd2ffe43685bc2b5d328b0c1d58402986e0a", + "cmd/go-omni/scaffold/templates/api-context.go.tmpl": "d2d67c3d235c4b1c965364f40f49c441d92764ad7ab8fd6848f56b1028e93b1b", + "cmd/go-omni/scaffold/templates/cmd-server-main.go.tmpl": "07c4f81109ece43fcf15bb807ada5ae6dc7e732babc7293269514a21cd389955", + "cmd/go-omni/scaffold/templates/databaseIntegration_test.go.tmpl": "81c4f7fbc919d74b738d3a7338248cbd83cfa0803b2058e8ee184e3ce51e7e9b", + "cmd/go-omni/scaffold/templates/go.mod.tmpl": "7d3bb15d2292e36939cc1905258d2800aeaf0e37d7d041b2ff8cfdb6fc84c57a", + "cmd/go-omni/scaffold/templates/internal-handler-user.go.tmpl": "7f8dbcd2f9b0129a90160d6059e47d19d4758274060a1a3e5160055d0592d34f", + "cmd/go-omni/scaffold/templates/internal-model-errors.go.tmpl": "f123eace52952321b0665a92a2923514c52c8c3bf13df1065710e9675750f837", + "cmd/go-omni/scaffold/templates/internal-model-user.go.tmpl": "b391143e7386e169b53e7fed25de1146e8c52f3fa287b12728e8d22b33f386de", + "cmd/go-omni/scaffold/templates/internal-repository-memory.go.tmpl": "148abc273c1fab3f28536706792762ae41e8687dfe9986c374372f162600feac", + "cmd/go-omni/scaffold/templates/internal-repository-mysql.go.tmpl": "34bd333e29211a0025ff6ebaea18c7f142f18f1cc451d692f9d360e8c8f6f635", + "cmd/go-omni/scaffold/templates/internal-repository-users.go.tmpl": "6d7dd80f34a9c215daeb4a86bfbe11e845e7ff7e9864c8af43e1c2ec3a8b28ec", + "cmd/go-omni/scaffold/templates/internal-service-user.go.tmpl": "33fdee3a2c3d98e7ca6a323b5e888a73582f94e327b6c24a34c09bf180db7b3d", + "cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl": "de70ecc4194db8a2da5d65297f452a40b453652cb814cd228ce03d465761c856", + "cmd/go-omni/scaffold/templates/modules-README.md.tmpl": "4b84fbc2310b856d86c6fcd9a47d97d2261019c74ea8e0cdbc41229086aa3527", + "cmd/go-omni/scaffold/templates/modules-config.yaml.tmpl": "b8734080ae6d533a175bcad946ce2cb55d8bac1729918291753b9ca03064910f", + "cmd/go-omni/scaffold/templates/modules-filesHandler.go.tmpl": "426867fbef5b99a85394fa4457dd38b9a194adf4bdfb293384b2fbbc799091da", + "cmd/go-omni/scaffold/templates/modules-filesHandler_test.go.tmpl": "9571e39bd0a330739eb1fd7e636cd58961b41b44772b2b49162e41deda98547f", + "cmd/go-omni/scaffold/templates/modules-filesService.go.tmpl": "6460ee9a873af45ca6cec779bf53302acfce21499610425a4704c2f179eddb8f", + "cmd/go-omni/scaffold/templates/modules-gitignore.tmpl": "2579f2a63ff2962239b92dd841b0d0d26e783d85c9d0100011936c8bcbf5d0d3", + "cmd/go-omni/scaffold/templates/modules-main.go.tmpl": "cb8daddd4e12afd238540be7022187c74a548d9e409b3a7f1eaf84a9139218bf", + "cmd/go-omni/scaffold/templates/modules-settings.go.tmpl": "91084caaa1fd994dde714f52b242d23e41adb731383b12d74399247c87ac5597", + "cmd/go-omni/scaffold/templates/modules-settings_test.go.tmpl": "76e55f8954b7606d7146eb1e965065c521a2b81234a47742ed69a93aac300406", + "cmd/go-omni/scaffold/templates/modules-storage.go.tmpl": "a775acf2d2490288a8e5482bb0a8769be6b0a8c9cfcc9bb10b62aabbd05fa0c7", + "cmd/go-omni/scaffold/templates/modules-storage_test.go.tmpl": "dacca42d228bd9352948054eb2d071829a8a61d8c6fa24d95ff00fc39a74309e", + "cmd/go-omni/scaffold/templates/schema.sql.tmpl": "7c5108f18fc7d8ff75c97afd96fcaa9f8ed026c79c71c5612f55572d73354abb", + "cmd/go-omni/scaffold/version.go": "2ae216e676082a26e5778045e20964eb4f9e99a58d4f9a9ff4b559b286837d7a", + "database/configOverlay_test.go": "01c9b44fded226494ca12f7d0f41a9ff417d8135d263a5833727c1c028b6b8e3", + "database/database.go": "3d8615e1bf58ecdae21558591e745c241e499ba6e85fc07acfa2344e04ce95db", + "database/database_test.go": "18eac1e4c6dbd6a09216168839ba3511aac8a38c2ddf6fdd5267f35435b09f67", + "database/lifecycle_test.go": "e141b52edfb9867184dedac0e79feed56a5b137acbe64280ad69647653bbea21", + "database/migrate/migrate.go": "fd6cd27d2dce5b35ffe80eb5442c47c657b93ffc6545bbcc1ee4651a70a4f80b", + "database/migrate/migrate_test.go": "d7ba91eed8cfd97e4ef0741d582f493dd411b65ad3192f05b167f75d9b9dcf90", + "examples/userService/api/user.api": "cdbc7f356f336b6265e9c3fc9974952e05cf14001c58b330a826b4bb32f02712", + "examples/userService/cmd/server/main.go": "80ba4721f677947b7f2ece21d5f3e89ee0c4d2cc653412d83aef2b3a45bd3ccb", + "examples/userService/etc/config.yaml": "e86e4c92659bb5d9114c0428c6ed47e8d35d9f984091a4f399006a9c4de5b9a3", + "examples/userService/internal/handler/openapiGen.go": "8ddc60c338630db915b9f074ade776f526eabf60fc99b33e425359fd415c4926", + "examples/userService/internal/handler/routesGen.go": "7f45030e7eb9f055c4b4d2ff12ad855d7c726aad4aee63c5b3c3f97559a64257", + "examples/userService/internal/handler/user/handlersGen.go": "06f3c2798ff323812ff3eb2d8b933a569af3ce1b86a67b66036374add98c0d7e", + "examples/userService/internal/middleware/auth.go": "f12197d01da155ad56a8a44f951c83739cbfca48c3224bd52a7bc25357f05c9f", + "examples/userService/internal/middleware/auth_test.go": "6ffdd7c7b2ede7e5791eec0eb1d9105ea45b3a52cc3140fe7a94fdc9b6fb3ec6", + "examples/userService/internal/middleware/registryGen.go": "a6122a5453dedc6a4415860108814c50e155128cfe5412b39dd33ce2d01b4b00", + "examples/userService/internal/model/errors.go": "f123eace52952321b0665a92a2923514c52c8c3bf13df1065710e9675750f837", + "examples/userService/internal/model/user.go": "b391143e7386e169b53e7fed25de1146e8c52f3fa287b12728e8d22b33f386de", + "examples/userService/internal/repository/user/mysql.go": "924cae233a68e506e9b0b953529e0ccb13a0e71d24723309383c464e1852b833", + "examples/userService/internal/repository/user/repository.go": "2aca07788d8d59831b10ac665882ca8157b20f384115bb90742f235f8352a69d", + "examples/userService/internal/repository/user/store.go": "f776832e4f3b26fb889dad53b65742833ddb7f43b7dddc72915ad3fe47ec4d3e", + "examples/userService/internal/repository/user/transaction_test.go": "9af4cb3a03b71026b80aa5078ddda4e3c4f72a7ab14bed6c87603a4db2cc2a22", + "examples/userService/internal/service/user/createUser.go": "53d0ea7371e10a0bd3f32f57b9bff9b48b5f9a3315218f95a258fa45b81d9368", + "examples/userService/internal/service/user/deleteUser.go": "0cd0b038d5a11429de438311980ef19a8f6e28ea0d7cf5ebaaf78858daf3a124", + "examples/userService/internal/service/user/exportUser.go": "dac2adb0c9ea43a4af15511ae488afaa6d9c9b351e68e9b8a54f4014d29de14d", + "examples/userService/internal/service/user/getUser.go": "a307fa716c70286a2a6c5d5c0a6c35ec6d258c3278e2a7fe86b74b324d7b7283", + "examples/userService/internal/service/user/listUsers.go": "f0bf8b9c3a4c4a699112e3493b7fe611e2dd4f6e8a4437106396c47d484f1383", + "examples/userService/internal/service/user/service.go": "bf7f27f74b3f4e01877222f9eb6a5b2cc8e119dba8e438081689f18d7a116ae6", + "examples/userService/internal/service/user/updateUser.go": "9b0e3e4296c0e0df98956961a928612d9203c0fa09aa71bb0683e430e1133ff0", + "examples/userService/internal/servicecontext/context.go": "5527ca07049f5c5f5deffa65e5759494b2e0a77804b629b3ecb1705e7557ee25", + "examples/userService/internal/servicecontext/context_test.go": "04d008b6cf5b7894752bc5222abb8f4120f1ebe060034ac55ec9663c1f7bc613", + "examples/userService/internal/servicecontext/storage.go": "9402e1ee7a258b2db4b6dbf38fcb0674ad8907fb2a948d47dd8374f78bec444d", + "examples/userService/internal/settings/config.go": "721c3744a61a2d2807edcdfa190336901368966a0fb952561d0e00da5730cd3a", + "examples/userService/internal/types/typesGen.go": "3ab2ce9ffe467c3ea53ac3d1fec283ecebc87cef57fd592d08dca991fd964876", + "examples/userService/migrations/000001_users.down.sql": "64b47d3d2644745a2ba1197b73760e63fe0f2c1d12c193888c4417b3d5abb75e", + "examples/userService/migrations/000001_users.up.sql": "c6ea5cb92ee53fe20e43373d084ff360d49921b785602145e55380e4adbee1dc" + } +} diff --git a/performance/apiWorkflowTests.txt b/performance/apiWorkflowTests.txt new file mode 100644 index 0000000..09c60aa --- /dev/null +++ b/performance/apiWorkflowTests.txt @@ -0,0 +1,52 @@ +ok git.yuntuo.hk/k2safe/go-omni 1.368s +ok git.yuntuo.hk/k2safe/go-omni/app 2.293s +ok git.yuntuo.hk/k2safe/go-omni/audit 1.305s +ok git.yuntuo.hk/k2safe/go-omni/auth 1.329s +ok git.yuntuo.hk/k2safe/go-omni/cache 1.305s +ok git.yuntuo.hk/k2safe/go-omni/cache/redis 1.290s +ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni 1.734s +ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/apigen 25.935s +ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/scaffold 38.371s +ok git.yuntuo.hk/k2safe/go-omni/config 1.319s +ok git.yuntuo.hk/k2safe/go-omni/contract 1.313s +ok git.yuntuo.hk/k2safe/go-omni/database 1.291s +ok git.yuntuo.hk/k2safe/go-omni/database/migrate 1.308s +? git.yuntuo.hk/k2safe/go-omni/errors [no test files] +? git.yuntuo.hk/k2safe/go-omni/examples/basic [no test files] +ok git.yuntuo.hk/k2safe/go-omni/examples/modules 1.354s +ok git.yuntuo.hk/k2safe/go-omni/examples/platform 1.397s +ok git.yuntuo.hk/k2safe/go-omni/examples/reference 1.521s +? git.yuntuo.hk/k2safe/go-omni/examples/reference/handler [no test files] +? git.yuntuo.hk/k2safe/go-omni/examples/reference/model [no test files] +ok git.yuntuo.hk/k2safe/go-omni/examples/reference/repository 1.300s +? git.yuntuo.hk/k2safe/go-omni/examples/reference/service [no test files] +ok git.yuntuo.hk/k2safe/go-omni/health 1.336s +ok git.yuntuo.hk/k2safe/go-omni/httpc 1.815s +ok git.yuntuo.hk/k2safe/go-omni/httpx 1.318s +ok git.yuntuo.hk/k2safe/go-omni/internal/perfcheck 1.339s +? git.yuntuo.hk/k2safe/go-omni/internal/storagefixture [no test files] +? git.yuntuo.hk/k2safe/go-omni/internal/testutil [no test files] +ok git.yuntuo.hk/k2safe/go-omni/logx 1.609s +ok git.yuntuo.hk/k2safe/go-omni/metrics 1.307s +ok git.yuntuo.hk/k2safe/go-omni/middleware 1.337s +ok git.yuntuo.hk/k2safe/go-omni/middleware/redis 1.311s +ok git.yuntuo.hk/k2safe/go-omni/openapi 1.332s +ok git.yuntuo.hk/k2safe/go-omni/pagination 1.278s +? git.yuntuo.hk/k2safe/go-omni/queue [no test files] +ok git.yuntuo.hk/k2safe/go-omni/queue/redis 1.292s +ok git.yuntuo.hk/k2safe/go-omni/resilience 1.275s +? git.yuntuo.hk/k2safe/go-omni/response [no test files] +ok git.yuntuo.hk/k2safe/go-omni/storage 1.323s +ok git.yuntuo.hk/k2safe/go-omni/storage/local 1.297s +ok git.yuntuo.hk/k2safe/go-omni/storage/oss 1.454s +ok git.yuntuo.hk/k2safe/go-omni/storage/s3 9.482s +? git.yuntuo.hk/k2safe/go-omni/storage/storagetest [no test files] +ok git.yuntuo.hk/k2safe/go-omni/task 1.377s +ok git.yuntuo.hk/k2safe/go-omni/telemetry 1.330s +ok git.yuntuo.hk/k2safe/go-omni/transport 1.338s +ok git.yuntuo.hk/k2safe/go-omni/transport/http2 12.651s +? git.yuntuo.hk/k2safe/go-omni/transport/internal/routing [no test files] +ok git.yuntuo.hk/k2safe/go-omni/transport/native 2.017s +ok git.yuntuo.hk/k2safe/go-omni/transport/nethttp 1.325s +? git.yuntuo.hk/k2safe/go-omni/transport/transporttest [no test files] +? git.yuntuo.hk/k2safe/go-omni/validate [no test files] diff --git a/performance/loggingUploadsAcceptance.md b/performance/loggingUploadsAcceptance.md new file mode 100644 index 0000000..24bcd1e --- /dev/null +++ b/performance/loggingUploadsAcceptance.md @@ -0,0 +1,14 @@ +# 日志与上传扩展验收 + +2026-09-23，在 macOS arm64 / Go 1.27.1 开发环境完成；真实 S3 测试使用本机 Docker 内 Linux arm64 MinIO。此次未在业务服务器运行测试负载。 + +- 全库：`GOMAXPROCS=4 GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=2 -count=1 -timeout=240s ./...`，37 个有测试的包通过，包含脚手架集成和全部网络适配器。 +- 收尾新增业务错误堆栈、统一分片 MIME 签名及大小不符契约后，对 logx、middleware、storage 全部子包、config、app 重新运行 race，全部通过。 +- `go vet ./...`、`go build ./...`、`git diff --check` 通过。 +- 轮转：并发写入、备份份数、gzip 内容完整性、关闭后不重开；脱敏：嵌套 map/struct/group/LogValuer、循环截断、不修改调用者数据；HTTP：慢请求 WARN、5xx ERROR、panic 堆栈和已提交状态保持。 +- 上传：Local 签名篡改/超时参数/重复参数/类型伪造/实际大小不符/失败不覆盖；S3/OSS SDK 协议及分片分页；R2 签名参数。 +- 真实 MinIO：SDK/签名直传、两片合并、分页续传查询、重复取消、声明大小伪造、签名绑定大小和类型（篡改返回 403）、普通对象契约。测试环境最多一个容器，2 CPU、2 GiB、256 pids，network=none，无端口映射；退出后清理。 + +真实 S3 的运行命令、源码 SHA-256、镜像 ID、清理状态和日志摘要见 [uploadsLocal.json](uploadsLocal.json)，原始输出见 [uploadsLocal.txt](uploadsLocal.txt)。可用 `python3 scripts/verifyUploads.py` 复现（提前准备对应 MinIO 镜像）。 + +边界：OSS/R2 未使用云账号实测；HTTP fixture 不验证厂商签名。MinIO 的通过结果不能代替它们的实网验收。本地存储目前提供受限签名上传，不提供持久分片会话。分片实际内容校验、业务授权、对象归属、上传配额和过期分片清理由业务/部署配置。未做生产负载、磁盘耗尽、日志平台采集或云存储故障恢复验收。 diff --git a/performance/scaffoldModulesAcceptance.md b/performance/scaffoldModulesAcceptance.md new file mode 100644 index 0000000..4bc65bc --- /dev/null +++ b/performance/scaffoldModulesAcceptance.md @@ -0,0 +1,21 @@ +# 脚手架模块装配验收 + +2026-09-23，本机 macOS arm64 / Go 1.27.1，使用当前本地框架源码。新模板目标版本 v0.7.0-preview 尚未发布，验收使用显式本地 replace。 + +完成内容： + +- CLI 可选日志输出 stdout/stderr/file、JSON/text 格式、对象存储 none/local/s3/r2/oss，与 memory/mysql 数据库独立组合。 +- 生成严格的 YAML/JSON 配置、环境变量覆盖、凭据检查、专用客户端连接池和资源关闭代码。 +- 用户创建日志、服务端上传、直传链接、临时下载链接和错误处理示例；新增项目测试覆盖配置优先级、关闭幂等性/所有权、关闭后不可重新初始化，以及文件接口行为。 +- 生成器在写文件前验证参数和渲染所有模板；修复目标目录尚不存在时的 macOS /var 符号链接路径计算；拒绝将新模块模板与旧远程框架版本组合。强制重新生成时不自动删除旧代码，存在不兼容遗留文件会提前拒绝。 + +验收结果： + +1. `GO_OMNI_SCAFFOLD_INTEGRATION=1 go test -race -p=2 -count=1 -timeout=240s ./...` 全部通过（37 个有测试的框架包）。其中 10 种数据库/存储组合分别编译并执行生成项目的 race 测试；实际 MySQL 集成用例未提供数据库时跳过。原始输出见 [scaffoldModulesTests.txt](scaffoldModulesTests.txt)。 +2. `go vet ./...`、`go build ./...`、`git diff --check` 通过。 +3. `python3 scripts/verifyScaffoldModules.py` 验证四种真实服务组合：stderr/json/nethttp/SIGTERM、stdout/text/native/SIGINT、file/json/native/SIGTERM、file/text/nethttp/SIGINT。全部退出码 0、端口释放。每组验证上传/下载字节、直传、篡改签名 403、认证 401、大小 413、类型 415、非法 key 400、对象缺失 404，以及缺少示例 Token 时启动失败。 +4. 日志包含业务事件与 requestId，签名入口保留真实状态码，未记录示例 Token 或签名 URL。 + +运行结果、模板/生成器源码 SHA-256 和日志摘要见 [scaffoldModulesLocal.json](scaffoldModulesLocal.json)，生成项目的测试输出见 [scaffoldModulesLocal.txt](scaffoldModulesLocal.txt)。临时项目、二进制和测试进程已清理。 + +边界：本轮真实接口验收使用 Local。S3/R2/OSS 生成项目完成编译、配置/构造检查与基于 Local 的业务接口测试，没有访问云账号；MySQL 组合完成编译，本轮未再次运行真实数据库事务验收。示例共享 Token 授权整个文件模块，业务上线仍应替换认证和对象归属规则；这不构成固定用户/租户上下文。当前改动未推送或打标签。 diff --git a/performance/scaffoldModulesLocal.json b/performance/scaffoldModulesLocal.json new file mode 100644 index 0000000..fab36d9 --- /dev/null +++ b/performance/scaffoldModulesLocal.json @@ -0,0 +1,104 @@ +{ + "status": "passed", + "startedAt": "2026-09-23T09:31:52.278033+00:00", + "cases": [ + { + "logOutput": "stderr", + "logFormat": "json", + "transport": "nethttp", + "signal": "SIGTERM", + "status": "passed", + "exitCode": 0, + "portReleased": true, + "missingTokenRejected": true + }, + { + "logOutput": "stdout", + "logFormat": "text", + "transport": "native", + "signal": "SIGINT", + "status": "passed", + "exitCode": 0, + "portReleased": true, + "missingTokenRejected": true + }, + { + "logOutput": "file", + "logFormat": "json", + "transport": "native", + "signal": "SIGTERM", + "status": "passed", + "exitCode": 0, + "portReleased": true, + "missingTokenRejected": true + }, + { + "logOutput": "file", + "logFormat": "text", + "transport": "nethttp", + "signal": "SIGINT", + "status": "passed", + "exitCode": 0, + "portReleased": true, + "missingTokenRejected": true + } + ], + "sourceSha256": { + "cmd/go-omni/api.go": "59f3b8b45497a9f560d649ac5e7b622362f5f20edd81251d7dc76f21605fc90a", + "cmd/go-omni/api_test.go": "ee68b4b8b0155f4d123135b0706c6d17d8fc90e2bcfc1312aa4f9e7a9da2454b", + "cmd/go-omni/apigen/apigen_test.go": "8e7c94795ffbc3ee062756a77cfeb274f3cd1a78f1dbc3b51ec4ceb14f478518", + "cmd/go-omni/apigen/compile.go": "6709da0b69e8d10d49a874447e00c93d0a819d8367b2188e9626bd0c7b3baef2", + "cmd/go-omni/apigen/generate.go": "373ec9834aa4ed71a5311c1ef40b78dc6c303aa884d7bb16a680058ca751b2bc", + "cmd/go-omni/apigen/parse.go": "e85fe261fdec2e3487851e265babd7af8c7f6153694ad6fb73177aac33f8ef95", + "cmd/go-omni/apigen/profile.go": "877939398384c44316e5e052d92c4a93fb3b74a6742df2414a0054913f3fca27", + "cmd/go-omni/apigen/profile_test.go": "5f3c091da241fe0f8b99ac19e9efd2c3177aea72e63a346f99fecc0f548aad1a", + "cmd/go-omni/apigen/templates.go": "9303a2fafe44cb7e763a2c5beb193c16fa0dc0c2dbde4cb6e14f2ba533d3e56e", + "cmd/go-omni/apigen/write.go": "1ce3871fd92f638d097c576583c169daa0ce2fa8010dbcd34a8cd7faa314a1a2", + "cmd/go-omni/main.go": "6f851d21792a9eb9fcbc5095e8cacb3dcee1c715a3980650c75381f0e46f51a9", + "cmd/go-omni/main_test.go": "b8040d2b19a1c86672fe1b1be7fa1bf7f885aece79ded6ffa0cc670887b7136f", + "cmd/go-omni/migrate.go": "f1ce89fbe419205eb82172da2b90e8888905d2b85d1a9db2a72b568bcf849a70", + "cmd/go-omni/migrate_test.go": "15d9a4aaebf9f899e523e908f1fa3c2282eb1e6c8cadbefae6b68f1ae4e00e6e", + "cmd/go-omni/scaffold/addModule.go": "8f43bd68c2b3820cf1d036c4f2c5f9d06d1c7c8e5114f5081597d905f8d8cdcc", + "cmd/go-omni/scaffold/addModule_test.go": "667c45312f628f0fb5b8206ddef7ccc02a75f33b72f73fd6a17b03da0cb62e91", + "cmd/go-omni/scaffold/apiInfrastructure.go": "7542f6af48345249b84054411c449b950db85fa6c5c60e62620bf8864a1aaff3", + "cmd/go-omni/scaffold/moduleProfiles_test.go": "ae9790e5d3549d32fc72739a10a31ea858e5a8f8189cf1f5c1840bf80b3f8ad6", + "cmd/go-omni/scaffold/moduleTemplates/README.md.tmpl": "9d3bfc46cac5618bcff7d8a6b2cc9f123e818aeb3dee9bcdfe0f91db584994bd", + "cmd/go-omni/scaffold/moduleTemplates/handler-handler.go.tmpl": "729cd0f6faba53d5d2c1492935e12ac2adcefde4490748e25c18be7ccb3f7139", + "cmd/go-omni/scaffold/moduleTemplates/model-model.go.tmpl": "a005611f6b455d4d8469e316bc26fa571043b2da1834db2b3dd14a96d66221d7", + "cmd/go-omni/scaffold/moduleTemplates/module.go.tmpl": "fb0be7fd571ba2c0edb01d70d68cdbbfe53561caab98346bf1ac7cd8df07205a", + "cmd/go-omni/scaffold/moduleTemplates/repository-repository.go.tmpl": "902da3d9f12395fb226178c4428ff547e1c2fce3261882dba98d1400d8c90747", + "cmd/go-omni/scaffold/moduleTemplates/service-service.go.tmpl": "49d26044a69a43e983aa5ba4b5b2399b26b77898b23e230751ad1372c2af3178", + "cmd/go-omni/scaffold/scaffold.go": "71b3091b83c3e45ac1981164e8c819c997ad96607d535f1492333370f4872548", + "cmd/go-omni/scaffold/scaffold_test.go": "58aab9251b01e394bbc90aeb538ff035dd32c73c0d4815132ac6cbc086527cda", + "cmd/go-omni/scaffold/templates/README.md.tmpl": "8ec215775fcafd437480022307fffd2ffe43685bc2b5d328b0c1d58402986e0a", + "cmd/go-omni/scaffold/templates/api-context.go.tmpl": "d2d67c3d235c4b1c965364f40f49c441d92764ad7ab8fd6848f56b1028e93b1b", + "cmd/go-omni/scaffold/templates/cmd-server-main.go.tmpl": "07c4f81109ece43fcf15bb807ada5ae6dc7e732babc7293269514a21cd389955", + "cmd/go-omni/scaffold/templates/databaseIntegration_test.go.tmpl": "81c4f7fbc919d74b738d3a7338248cbd83cfa0803b2058e8ee184e3ce51e7e9b", + "cmd/go-omni/scaffold/templates/go.mod.tmpl": "7d3bb15d2292e36939cc1905258d2800aeaf0e37d7d041b2ff8cfdb6fc84c57a", + "cmd/go-omni/scaffold/templates/internal-handler-user.go.tmpl": "7f8dbcd2f9b0129a90160d6059e47d19d4758274060a1a3e5160055d0592d34f", + "cmd/go-omni/scaffold/templates/internal-model-errors.go.tmpl": "f123eace52952321b0665a92a2923514c52c8c3bf13df1065710e9675750f837", + "cmd/go-omni/scaffold/templates/internal-model-user.go.tmpl": "b391143e7386e169b53e7fed25de1146e8c52f3fa287b12728e8d22b33f386de", + "cmd/go-omni/scaffold/templates/internal-repository-memory.go.tmpl": "148abc273c1fab3f28536706792762ae41e8687dfe9986c374372f162600feac", + "cmd/go-omni/scaffold/templates/internal-repository-mysql.go.tmpl": "34bd333e29211a0025ff6ebaea18c7f142f18f1cc451d692f9d360e8c8f6f635", + "cmd/go-omni/scaffold/templates/internal-repository-users.go.tmpl": "6d7dd80f34a9c215daeb4a86bfbe11e845e7ff7e9864c8af43e1c2ec3a8b28ec", + "cmd/go-omni/scaffold/templates/internal-service-user.go.tmpl": "33fdee3a2c3d98e7ca6a323b5e888a73582f94e327b6c24a34c09bf180db7b3d", + "cmd/go-omni/scaffold/templates/internal-servicecontext-context.go.tmpl": "de70ecc4194db8a2da5d65297f452a40b453652cb814cd228ce03d465761c856", + "cmd/go-omni/scaffold/templates/modules-README.md.tmpl": "4b84fbc2310b856d86c6fcd9a47d97d2261019c74ea8e0cdbc41229086aa3527", + "cmd/go-omni/scaffold/templates/modules-config.yaml.tmpl": "b8734080ae6d533a175bcad946ce2cb55d8bac1729918291753b9ca03064910f", + "cmd/go-omni/scaffold/templates/modules-filesHandler.go.tmpl": "426867fbef5b99a85394fa4457dd38b9a194adf4bdfb293384b2fbbc799091da", + "cmd/go-omni/scaffold/templates/modules-filesHandler_test.go.tmpl": "9571e39bd0a330739eb1fd7e636cd58961b41b44772b2b49162e41deda98547f", + "cmd/go-omni/scaffold/templates/modules-filesService.go.tmpl": "6460ee9a873af45ca6cec779bf53302acfce21499610425a4704c2f179eddb8f", + "cmd/go-omni/scaffold/templates/modules-gitignore.tmpl": "2579f2a63ff2962239b92dd841b0d0d26e783d85c9d0100011936c8bcbf5d0d3", + "cmd/go-omni/scaffold/templates/modules-main.go.tmpl": "cb8daddd4e12afd238540be7022187c74a548d9e409b3a7f1eaf84a9139218bf", + "cmd/go-omni/scaffold/templates/modules-settings.go.tmpl": "91084caaa1fd994dde714f52b242d23e41adb731383b12d74399247c87ac5597", + "cmd/go-omni/scaffold/templates/modules-settings_test.go.tmpl": "76e55f8954b7606d7146eb1e965065c521a2b81234a47742ed69a93aac300406", + "cmd/go-omni/scaffold/templates/modules-storage.go.tmpl": "a775acf2d2490288a8e5482bb0a8769be6b0a8c9cfcc9bb10b62aabbd05fa0c7", + "cmd/go-omni/scaffold/templates/modules-storage_test.go.tmpl": "dacca42d228bd9352948054eb2d071829a8a61d8c6fa24d95ff00fc39a74309e", + "cmd/go-omni/scaffold/templates/schema.sql.tmpl": "7c5108f18fc7d8ff75c97afd96fcaa9f8ed026c79c71c5612f55572d73354abb", + "cmd/go-omni/scaffold/version.go": "2ae216e676082a26e5778045e20964eb4f9e99a58d4f9a9ff4b559b286837d7a", + "config/env.go": "ab01d21b088cfe497d9cadf28916b27f6cff8d788dd7661a9d93408892141b27", + "scripts/verifyScaffoldModules.py": "b73f3527e26126b8a7628922fdda0a3ac1cc96c708cc108bfb012c143b0427cb" + }, + "finishedAt": "2026-09-23T09:32:11.031629+00:00", + "logSha256": "e26649c11012f44bbbec004a7eb81430ad587ccb4e7817b098e2e09967fabbcc" +} diff --git a/performance/scaffoldModulesLocal.txt b/performance/scaffoldModulesLocal.txt new file mode 100644 index 0000000..fd7018c --- /dev/null +++ b/performance/scaffoldModulesLocal.txt @@ -0,0 +1,32 @@ +created GoOmni project /private/var/folders/wm/1sg73x8n7879sp3t6m4jpnrr0000gn/T/goOmniModuleAcceptanceio6u2uka/case0 +? example.com/moduleAcceptance/cmd/server [no test files] +ok example.com/moduleAcceptance/internal/handler 1.313s +? example.com/moduleAcceptance/internal/model [no test files] +? example.com/moduleAcceptance/internal/repository [no test files] +? example.com/moduleAcceptance/internal/service [no test files] +ok example.com/moduleAcceptance/internal/servicecontext 1.539s +ok example.com/moduleAcceptance/internal/settings 1.305s +created GoOmni project /private/var/folders/wm/1sg73x8n7879sp3t6m4jpnrr0000gn/T/goOmniModuleAcceptanceio6u2uka/case1 +? example.com/moduleAcceptance/cmd/server [no test files] +ok example.com/moduleAcceptance/internal/handler 1.371s +? example.com/moduleAcceptance/internal/model [no test files] +? example.com/moduleAcceptance/internal/repository [no test files] +? example.com/moduleAcceptance/internal/service [no test files] +ok example.com/moduleAcceptance/internal/servicecontext 1.621s +ok example.com/moduleAcceptance/internal/settings 1.298s +created GoOmni project /private/var/folders/wm/1sg73x8n7879sp3t6m4jpnrr0000gn/T/goOmniModuleAcceptanceio6u2uka/case2 +? example.com/moduleAcceptance/cmd/server [no test files] +ok example.com/moduleAcceptance/internal/handler 1.327s +? example.com/moduleAcceptance/internal/model [no test files] +? example.com/moduleAcceptance/internal/repository [no test files] +? example.com/moduleAcceptance/internal/service [no test files] +ok example.com/moduleAcceptance/internal/servicecontext 1.538s +ok example.com/moduleAcceptance/internal/settings 1.320s +created GoOmni project /private/var/folders/wm/1sg73x8n7879sp3t6m4jpnrr0000gn/T/goOmniModuleAcceptanceio6u2uka/case3 +? example.com/moduleAcceptance/cmd/server [no test files] +ok example.com/moduleAcceptance/internal/handler 1.318s +? example.com/moduleAcceptance/internal/model [no test files] +? example.com/moduleAcceptance/internal/repository [no test files] +? example.com/moduleAcceptance/internal/service [no test files] +ok example.com/moduleAcceptance/internal/servicecontext 1.554s +ok example.com/moduleAcceptance/internal/settings 1.312s diff --git a/performance/scaffoldModulesTests.txt b/performance/scaffoldModulesTests.txt new file mode 100644 index 0000000..17814cb --- /dev/null +++ b/performance/scaffoldModulesTests.txt @@ -0,0 +1,50 @@ +ok git.yuntuo.hk/k2safe/go-omni 1.486s +ok git.yuntuo.hk/k2safe/go-omni/app 2.299s +ok git.yuntuo.hk/k2safe/go-omni/audit 1.285s +ok git.yuntuo.hk/k2safe/go-omni/auth 1.332s +ok git.yuntuo.hk/k2safe/go-omni/cache 1.306s +ok git.yuntuo.hk/k2safe/go-omni/cache/redis 1.281s +ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni 1.486s +ok git.yuntuo.hk/k2safe/go-omni/cmd/go-omni/scaffold 38.201s +ok git.yuntuo.hk/k2safe/go-omni/config 1.548s +ok git.yuntuo.hk/k2safe/go-omni/database 1.281s +ok git.yuntuo.hk/k2safe/go-omni/database/migrate 1.677s +? git.yuntuo.hk/k2safe/go-omni/errors [no test files] +? git.yuntuo.hk/k2safe/go-omni/examples/basic [no test files] +ok git.yuntuo.hk/k2safe/go-omni/examples/modules 1.347s +ok git.yuntuo.hk/k2safe/go-omni/examples/platform 1.501s +ok git.yuntuo.hk/k2safe/go-omni/examples/reference 1.398s +? git.yuntuo.hk/k2safe/go-omni/examples/reference/handler [no test files] +? git.yuntuo.hk/k2safe/go-omni/examples/reference/model [no test files] +ok git.yuntuo.hk/k2safe/go-omni/examples/reference/repository 1.287s +? git.yuntuo.hk/k2safe/go-omni/examples/reference/service [no test files] +ok git.yuntuo.hk/k2safe/go-omni/health 1.326s +ok git.yuntuo.hk/k2safe/go-omni/httpc 1.399s +ok git.yuntuo.hk/k2safe/go-omni/httpx 1.430s +ok git.yuntuo.hk/k2safe/go-omni/internal/perfcheck 1.338s +? git.yuntuo.hk/k2safe/go-omni/internal/storagefixture [no test files] +? git.yuntuo.hk/k2safe/go-omni/internal/testutil [no test files] +ok git.yuntuo.hk/k2safe/go-omni/logx 1.840s +ok git.yuntuo.hk/k2safe/go-omni/metrics 1.301s +ok git.yuntuo.hk/k2safe/go-omni/middleware 1.357s +ok git.yuntuo.hk/k2safe/go-omni/middleware/redis 1.300s +ok git.yuntuo.hk/k2safe/go-omni/openapi 1.316s +ok git.yuntuo.hk/k2safe/go-omni/pagination 1.307s +? git.yuntuo.hk/k2safe/go-omni/queue [no test files] +ok git.yuntuo.hk/k2safe/go-omni/queue/redis 1.630s +ok git.yuntuo.hk/k2safe/go-omni/resilience 1.305s +? git.yuntuo.hk/k2safe/go-omni/response [no test files] +ok git.yuntuo.hk/k2safe/go-omni/storage 1.322s +ok git.yuntuo.hk/k2safe/go-omni/storage/local 1.351s +ok git.yuntuo.hk/k2safe/go-omni/storage/oss 1.439s +ok git.yuntuo.hk/k2safe/go-omni/storage/s3 9.596s +? git.yuntuo.hk/k2safe/go-omni/storage/storagetest [no test files] +ok git.yuntuo.hk/k2safe/go-omni/task 1.563s +ok git.yuntuo.hk/k2safe/go-omni/telemetry 1.311s +ok git.yuntuo.hk/k2safe/go-omni/transport 1.331s +ok git.yuntuo.hk/k2safe/go-omni/transport/http2 12.897s +? git.yuntuo.hk/k2safe/go-omni/transport/internal/routing [no test files] +ok git.yuntuo.hk/k2safe/go-omni/transport/native 2.001s +ok git.yuntuo.hk/k2safe/go-omni/transport/nethttp 1.318s +? git.yuntuo.hk/k2safe/go-omni/transport/transporttest [no test files] +? git.yuntuo.hk/k2safe/go-omni/validate [no test files] diff --git a/performance/uploadsLocal.json b/performance/uploadsLocal.json new file mode 100644 index 0000000..152cb72 --- /dev/null +++ b/performance/uploadsLocal.json @@ -0,0 +1,104 @@ +{ + "status": "passed", + "startedAt": "2026-09-23T07:35:47.522157+00:00", + "architecture": "arm64", + "goVersion": "go version go1.27.1 darwin/arm64", + "image": "quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z", + "imageId": "sha256:8f08aee614800a237906bd48114d733e5ac5bfac4ccdf731f141b0e880d7a253", + "sourceSha256": { + "go.mod": "2623f51aa8a352a8f4a8d42a7b2996d22f4bd8c1c76e432745bd9076413fa28c", + "go.sum": "9cdc3287be5a1ad128023b25f22d4f1dff968c06c72d5338b444f8140cd7f968", + "internal/storagefixture/multipart.go": "da9ec938b3bdb872e10e4f874b6a0224121c14a631dcf4e86d7ebbc1cd34a770", + "internal/storagefixture/server.go": "defacc905246a643faf5371e437f9ce458f05461e0a922766034d1f9ffdaf003", + "storage/local/local.go": "ce5fac2e54680b96db3e7dfd9a8d065888f246c5607882607d79c302ee3d231b", + "storage/local/local_test.go": "3a2241840968df8e37fd67169c3ef708228449e81f65aff908c8b60d49434dc0", + "storage/multipart.go": "3b718ebe6da681c89a3f797892c88e7acb15c295df2376d64ac35da0976ed69d", + "storage/multipart_test.go": "4053f680f7795db2a6b4aa23590b3669eec7a609dd92c4a1e20891a5299274ee", + "storage/oss/oss.go": "4216ce9f3fe3ac5418a5b809db3cef7e99dcca72a25a2a12301f5897cda85dd5", + "storage/oss/oss_test.go": "555a954010b1218804dc7951fc75e42eae24eba9a99c0871f5ba787f3fc87d45", + "storage/oss/upload.go": "556129611f9e5549cac8a36b4026bfb19c112a295b55192579f9ac116e20626f", + "storage/oss/upload_test.go": "08eb0ed157f3743288d781f61d4d2b7bfe77852a8259c5e598c719d35889409e", + "storage/s3/config.go": "626ec6a2c0b06c95ee7eb7aa0832b190cfcdeb8a127b6b89e99204040bc3b041", + "storage/s3/s3.go": "016bef5948e97f3d043d2e858716f1c02d9f27c4e02802bdc1be3860116f4b6c", + "storage/s3/s3_test.go": "f82af6bef236b293910c8d7504cb4e91717a729f5be92f089838ff791b28576c", + "storage/s3/upload.go": "3ce0b3626249ffe7a041606d6a9481743af8a0d4868fadea20ed3ed44ee0c765", + "storage/s3/upload_test.go": "a41fdbd100d9c79a3606c5942f98017afe26d841b251d96e979df87d0af93867", + "storage/signed.go": "aee2d98bd91fc8f55d2efd7bb40c57e3bcbc1f194cc59567699660cf954b986a", + "storage/signedUpload.go": "a8d31bcbbad64796f04611e69271e94d917c9bcd5aeabd7a97ea155b0093c814", + "storage/signed_test.go": "eaffd40894018814e976477f96cf0fbaecfa1987caad78830d9fc8e96995882f", + "storage/storage.go": "cb670872f01cf1d7b4ebf21cec9fc6f52ad93bfdf523b2f28bbb8d9c7fe39a6c", + "storage/storagetest/contract.go": "e7ad7242102437882269d4066462059d7974586b36a82d18639eca10fd932be9", + "storage/storagetest/uploads.go": "9b66f86dd94ccdf30c7d603d432bc2a55956608898e46c1274fa27d16fa1a969", + "storage/upload.go": "9ff2930ba389163acc8063b51107c09a60290609ba384de8004c498ce09f20a1", + "storage/upload_test.go": "12b60419807ed64b89e30861674e19abee7b0b7bb0e8ec9ac5b3aa2e0710a1da" + }, + "commands": [ + [ + "go", + "test", + "-p=2", + "-c", + "-o", + "/var/folders/wm/1sg73x8n7879sp3t6m4jpnrr0000gn/T/goOmniUploadq1j27rc4/s3.test", + "./storage/s3" + ], + [ + "docker", + "run", + "--rm", + "--name", + "go-omni-upload-647d8574e00d", + "--platform", + "linux/arm64", + "--cpus", + "2", + "--memory", + "2048m", + "--memory-swap", + "2048m", + "--pids-limit", + "256", + "--network", + "none", + "--read-only", + "--cap-drop", + "ALL", + "--security-opt", + "no-new-privileges", + "--user", + "1000:1000", + "--tmpfs", + "/tmp:rw,nosuid,noexec,size=64m,mode=1777", + "--tmpfs", + "/data:rw,nosuid,noexec,size=128m,mode=1777", + "-v", + "/var/folders/wm/1sg73x8n7879sp3t6m4jpnrr0000gn/T/goOmniUploadq1j27rc4:/test:ro", + "-e", + "GOMAXPROCS=2", + "-e", + "MINIO_ROOT_USER=goOmniTest", + "-e", + "MINIO_ROOT_PASSWORD=goOmniTestSecret", + "-e", + "GO_OMNI_S3_TEST_ADDR=127.0.0.1:9000", + "--entrypoint", + "sh", + "quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z", + "-c", + "set -eu\nminio --version\nminio server /data --address 127.0.0.1:9000 --console-address 127.0.0.1:9001 >/tmp/minio.log 2>&1 &\nserver=$!\ntrap 'kill \"$server\" 2>/dev/null || true; wait \"$server\" 2>/dev/null || true' EXIT\n/test/s3.test -test.v -test.run '^TestS3Integration$' -test.count=1 -test.timeout=60s || { cat /tmp/minio.log; exit 1; }\n" + ] + ], + "containerLimits": { + "cpus": 2, + "memoryBytes": 2147483648, + "pids": 256, + "network": "none" + }, + "notes": [ + "MinIO verifies S3 signatures; OSS/R2 real cloud accounts are not exercised." + ], + "preExistingContainersStillRunning": true, + "cleanedContainer": "go-omni-upload-647d8574e00d", + "finishedAt": "2026-09-23T07:35:58.127622+00:00", + "logSha256": "3cf85670fe5ad6de561d4f82049fd8a3f38c56aeb30cb4e868138fc3d886f40f" +} diff --git a/performance/uploadsLocal.txt b/performance/uploadsLocal.txt new file mode 100644 index 0000000..6c902e8 --- /dev/null +++ b/performance/uploadsLocal.txt @@ -0,0 +1,7 @@ +minio version RELEASE.2025-09-07T16-13-09Z (commit-id=07c3a429bfed433e49018cb0f78a52145d4bedeb) +Runtime: go1.24.6 linux/arm64 +License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html +Copyright: 2015-2025 MinIO, Inc. +=== RUN TestS3Integration +--- PASS: TestS3Integration (9.06s) +PASS diff --git a/scripts/verifyAPI.py b/scripts/verifyAPI.py new file mode 100644 index 0000000..30095c1 --- /dev/null +++ b/scripts/verifyAPI.py @@ -0,0 +1,233 @@ +#!/usr/bin/env python3 +"""Bounded local acceptance of API-generated projects, on both Omni transports. + +No remote hosts, cloud credentials or database are used. Child servers and +temporary modules are always cleaned up. Run from any directory with Python 3. +""" +import hashlib +import http.client +import json +import os +from pathlib import Path +import signal +import socket +import subprocess +import tempfile +import time + +ROOT = Path(__file__).resolve().parents[1] +REPORT = ROOT / "performance/apiGenerationLocal.json" +ENV = {k: v for k, v in os.environ.items() if not k.startswith("GO_OMNI_")} +ENV.update(GOWORK="off", GOFLAGS="") + + +def run(args, cwd=ROOT, ok=True): + result = subprocess.run([str(x) for x in args], cwd=cwd, env=ENV, + text=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, + timeout=90) + if ok and result.returncode: + raise AssertionError(f"{args}:\n{result.stdout}") + if not ok and not result.returncode: + raise AssertionError(f"command unexpectedly succeeded: {args}") + return result.stdout + + +def request(port, method, path, body=None, headers=None): + conn = http.client.HTTPConnection("127.0.0.1", port, timeout=3) + try: + if body is not None and len(body) > (1 << 20): + # Probe early Content-Length rejection without racing a 1 MiB + # client send against the server's intentional connection close. + conn.putrequest(method, path) + for key, value in (headers or {}).items(): + conn.putheader(key, value) + conn.putheader("Content-Length", str(len(body))) + conn.endheaders() + else: + conn.request(method, path, body=body, headers=headers or {}) + response = conn.getresponse() + return response.status, dict(response.getheaders()), response.read() + finally: + conn.close() + + +def write(path, text): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) + + +def exercise(binary, project, transport, starter): + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)) + port = sock.getsockname()[1] + environment = dict(ENV, GO_OMNI_HTTP_ADDR=f"127.0.0.1:{port}", + GO_OMNI_TRANSPORT=transport, GO_OMNI_LOG_OUTPUT="stderr", + GO_OMNI_LOG_FORMAT="json") + checks = [] + log_path = project / f"{transport}.log" + with log_path.open("w+") as log: + process = subprocess.Popen([str(binary)], cwd=project, env=environment, + stdout=log, stderr=log) + try: + deadline = time.monotonic() + 10 + while True: + if process.poll() is not None: + raise AssertionError(log_path.read_text()) + try: + if request(port, "GET", "/health/ready")[0] == 200: + break + except OSError: + pass + if time.monotonic() > deadline: + raise AssertionError("server startup timed out") + time.sleep(0.05) + checks.append("readiness") + status, headers, document = request(port, "GET", "/openapi.json") + assert status == 200 and document == (project / "docs/openapi.json").read_bytes() + checks.append("servedOpenApiMatchesFile") + if starter: + status, headers, data = request(port, "GET", "/api/greet/Alice") + assert status == 200 and json.loads(data)["data"]["message"] == "Hello Alice" + assert headers.get("X-Request-Id") or headers.get("X-Request-ID") + checks.extend(["greeting", "requestId"]) + else: + body = json.dumps({"name": "Alice", "active": False, "age": 0}) + base = {"Content-Type": "application/json", "X-Request-ID": "api-acceptance-42"} + for method, path, payload, expected in [ + ("PUT", "/api/v1/users/7?notify=true", body, 200), + ("PUT", "/api/v1/users/7", '{"name":"Alice"}', 400), + ("PUT", "/api/v1/users/7?notify=bad", body, 400), + ("PUT", "/api/v1/users/7?notify=true¬ify=false", body, 400), + ("PUT", "/api/v1/users/7", body[:-1] + ',"extra":1}', 400), + ("PUT", "/api/v1/users/7", body + "{}", 400), + ("PUT", "/api/v1/users/7", '{"name":"A","active":false,"age":151}', 400), + ("PUT", "/api/v1/users/0", body, 400), + ("PUT", "/api/v1/users/7", " " * ((1 << 20) + 1), 413), + ("GET", "/api/v1/users?page=2&pageSize=10&tag=a&tag=b", None, 200), + ("GET", "/api/v1/users?page=0", None, 400), + ("DELETE", "/admin/v1/users/7", None, 401), + ("POST", "/api/v1/newUsers", body, 201), + ("POST", "/api/v1/acceptedUsers", body, 202), + ]: + status, headers, data = request(port, method, path, payload, base) + assert status == expected, (method, path, status, data) + if path.startswith("/api/v1/users/7?") and status == 200: + assert json.loads(data)["data"] == {"userId": 7, "name": "Alice", "active": False} + assert json.loads(data)["requestId"] == "api-acceptance-42" + if path.startswith("/api/v1/users?page=2"): + assert json.loads(data)["data"]["page"] == 2 + checks.extend(["mixedSources", "zeroAndFalse", "pathRange", "defaultAndQuery", + "requiredFields", "unknownFields", "duplicateScalar", "trailingJson", + "rangeValidation", "bodyLimit413", "auth401", "status201", "status202"]) + assert request(port, "PUT", "/api/v1/users/7", body, + {"Content-Type": "text/plain"})[0] == 415 + status, headers, data = request(port, "DELETE", "/admin/v1/users/7", headers={"Authorization": "Bearer test-only"}) + assert status == 204 and data == b"" and headers.get("X-Audit") == "yes" + checks.extend(["contentType415", "middlewareComposition", "noContent204"]) + process.send_signal(signal.SIGTERM) + assert process.wait(timeout=12) == 0 + checks.append("gracefulShutdown") + try: + request(port, "GET", "/health/live") + raise AssertionError("listener survived shutdown") + except OSError: + checks.append("listenerReleased") + finally: + if process.poll() is None: + process.kill() + process.wait(timeout=5) + if not starter: + logs = [json.loads(line) for line in log_path.read_text().splitlines() if line.startswith("{")] + assert any(row.get("msg") == "user.updated" and row.get("requestId") == "api-acceptance-42" for row in logs) + checks.append("businessLogCorrelation") + return {"project": "starter" if starter else "contract", "transport": transport, + "checks": checks, "passed": True} + + +def main(): + started = time.monotonic() + results = [] + with tempfile.TemporaryDirectory(prefix="goOmniApiAcceptance") as temp: + temp = Path(temp) + cli = temp / "go-omni" + run(["go", "build", "-o", cli, "./cmd/go-omni"]) + for starter in (True, False): + project = temp / ("starter" if starter else "contract") + common = ["--module", "example.com/apiAcceptance", "--framework-dir", ROOT] + if starter: + run([cli, "api", "new", *common, project]) + entry = project / "api/service.api" + else: + run([cli, "new", "--api", ROOT / "examples/contracts/user.api", *common, project]) + entry = project / "api/user.api" + business = '''package user +import("context";"example.com/apiAcceptance/internal/types") +func(s *Service) UpdateUser(ctx context.Context,req *types.UpdateUserReq)(*types.UserResp,error){ + s.deps.Logger.InfoContext(ctx,"user.updated","userId",req.UserID) + return &types.UserResp{UserID:req.UserID,Name:req.Name,Active:req.Active},nil +} +''' + write(project / "internal/service/user/updateUser.go", business) + write(project / "internal/service/user/listUsers.go", '''package user +import("context";"example.com/apiAcceptance/internal/types") +func(s *Service) ListUsers(ctx context.Context,req *types.ListUsersReq)(*types.ListUsersResp,error){return &types.ListUsersResp{Items:[]types.UserResp{},Page:req.Page,Total:0},nil} +''') + write(project / "internal/service/admin/user/deleteUser.go", '''package user +import("context";"example.com/apiAcceptance/internal/types") +func(s *Service) DeleteUser(ctx context.Context,req *types.DeleteUserReq)error{return nil} +''') + write(project / "internal/middleware/auth.go", '''package middleware +import("git.yuntuo.hk/k2safe/go-omni/httpx";apierrors "git.yuntuo.hk/k2safe/go-omni/errors") +func Auth(next httpx.Handler)httpx.Handler{return func(c *httpx.Context)error{if c.Header("Authorization")!="Bearer test-only"{return apierrors.Unauthorized("UNAUTHORIZED","authentication required")};return next(c)}} +''') + write(project / "internal/middleware/audit.go", '''package middleware +import("git.yuntuo.hk/k2safe/go-omni/httpx") +func Audit(next httpx.Handler)httpx.Handler{return func(c *httpx.Context)error{c.SetResponseHeader("X-Audit","yes");return next(c)}} +''') + source = entry.read_text().replace(" UpdateUserReq {", ' CreateUserReq {\n Name string `json:"name"`\n Active bool `json:"active"`\n Age *int `json:"age,optional"`\n }\n UpdateUserReq {') + source += ''' +@server ( + group: user + prefix: /api/v1 +) +service user-api { + @handler createUser + @status 201 + post /newUsers (CreateUserReq) returns (UserResp) + @handler acceptUser + @status 202 + post /acceptedUsers (CreateUserReq) returns (UserResp) +} +''' + entry.write_text(source) + run([cli, "api", "generate", "--api", entry, "--dir", project, "--check"], ok=False) + run([cli, "api", "generate", "--api", entry, "--dir", project]) + assert (project / "internal/service/user/updateUser.go").read_text() == business + for operation in ("CreateUser", "AcceptUser"): + name = operation[0].lower() + operation[1:] + write(project / f"internal/service/user/{name}.go", f'''package user +import("context";"example.com/apiAcceptance/internal/types") +func(s *Service) {operation}(ctx context.Context,req *types.CreateUserReq)(*types.UserResp,error){{return &types.UserResp{{UserID:8,Name:req.Name,Active:req.Active}},nil}} +''') + run([cli, "api", "validate", "--api", entry]) + run([cli, "api", "generate", "--api", entry, "--dir", project, "--check"]) + binary = project / "server" + run(["go", "build", "-mod=mod", "-o", binary, "./cmd/server"], cwd=project) + for transport in ("nethttp", "native"): + result = exercise(binary, project, transport, starter) + results.append(result) + print(f"PASS {result['project']}/{transport}: {len(result['checks'])} checks", flush=True) + files = [] + for folder in ("cmd/go-omni/apigen", "contract", "openapi", "examples/contracts"): + files.extend(p for p in (ROOT / folder).rglob("*") if p.is_file()) + files.extend(ROOT / p for p in ("cmd/go-omni/api.go", "cmd/go-omni/main.go", "scripts/verifyAPI.py", + "transport/native/server.go", "transport/native/earlyReject_test.go")) + report = {"passed": True, "durationSeconds": round(time.monotonic() - started, 2), + "cases": results, + "sourceSha256": {str(p.relative_to(ROOT)): hashlib.sha256(p.read_bytes()).hexdigest() for p in sorted(files)}} + REPORT.write_text(json.dumps(report, ensure_ascii=False, indent=2) + "\n") + print(f"Saved {REPORT}", flush=True) + + +if __name__ == "__main__": + main() diff --git a/scripts/verifyAPIWorkflow.py b/scripts/verifyAPIWorkflow.py new file mode 100644 index 0000000..e2ed80e --- /dev/null +++ b/scripts/verifyAPIWorkflow.py @@ -0,0 +1,247 @@ +#!/usr/bin/env python3 +"""API + MySQL + storage business acceptance, using only disposable local resources. + +--version installs the real remote CLI and tests consumers with no local replace. +No deployment is performed. Secrets and signed URLs are excluded from reports. +""" +import argparse +import hashlib +import json +import os +from pathlib import Path +import secrets +import shutil +import signal +import socket +import subprocess +import tempfile +import time +import urllib.parse +import urllib.request +import urllib.error + +from verifyDatabase import MySQL, waitFor + +ROOT = Path(__file__).resolve().parents[1] +MODULE = "git.yuntuo.hk/k2safe/go-omni" + + +def run(args, cwd, env, timeout=180, expect_ok=True): + proc = subprocess.run([str(x) for x in args], cwd=cwd, env=env, text=True, + stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=timeout) + if (proc.returncode == 0) != expect_ok: + # Commands contain only paths and module versions, never credentials. + output = proc.stdout + for key, value in env.items(): + if value and any(word in key for word in ("TOKEN", "PASSWORD", "SECRET", "DSN", "SIGNING_KEY")): + output = output.replace(value, "[REDACTED]") + raise RuntimeError(f"{args[0]} failed ({proc.returncode}):\n{output}") + return proc.stdout + + +class LimitedMySQL(MySQL): + def start(self): + endpoint = self.env.get("DOCKER_HOST") or run( + ["docker", "context", "inspect", "--format", "{{.Endpoints.docker.Host}}"], self.directory, self.env).strip() + if not endpoint.startswith(("unix://", "npipe://")): + raise RuntimeError("a local Docker context is required") + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)); port = sock.getsockname()[1] + self.container = run(["docker", "run", "--detach", "--name", self.name, + "--label", "go-omni.test=apiWorkflow", "--cpus=1", "--memory=768m", "--pids-limit=256", + "--publish", f"127.0.0.1:{port}:3306", "--env", "MYSQL_ROOT_PASSWORD", + "--env", "MYSQL_PASSWORD", "--env", "MYSQL_DATABASE=omni_test", "--env", "MYSQL_USER=omni_test", + self.image], self.directory, self.env).strip() + self.wait() + self.dsn = f"omni_test:{self.password}@tcp(127.0.0.1:{port})/omni_test?parseTime=true&timeout=2s&readTimeout=2s&writeTimeout=2s" + + +def exercise(binary, project, env, db, transport): + with socket.socket() as sock: + sock.bind(("127.0.0.1", 0)); port = sock.getsockname()[1] + token, signing = secrets.token_hex(24), secrets.token_hex(24) + environment = dict(env, GO_OMNI_HTTP_ADDR=f"127.0.0.1:{port}", GO_OMNI_TRANSPORT=transport, + USER_SERVICE_API_TOKEN=token, GO_OMNI_STORAGE_SIGNING_KEY=signing, + GO_OMNI_STORAGE_ROOT=str(project / (transport + "Files")), + GO_OMNI_STORAGE_PUBLIC_URL=f"http://127.0.0.1:{port}/storage/object") + logs = project / "var/log/server.log" + logs.parent.mkdir(parents=True, exist_ok=True) + # Keep each transport's business log checks separate. + logs.write_text("") + checks = [] + client = urllib.request.build_opener(urllib.request.ProxyHandler({})) + def request(method, path, body=None, expected=200, auth=True): + headers = {"Content-Type": "application/json", "X-Request-ID": "apiWorkflow"} + if auth: headers["Authorization"] = "Bearer " + token + req = urllib.request.Request(f"http://127.0.0.1:{port}" + path, + data=None if body is None else json.dumps(body).encode(), headers=headers, method=method) + try: response = client.open(req, timeout=5) + except urllib.error.HTTPError as exc: response = exc + with response: + data = response.read() + if response.status != expected: raise AssertionError(f"{method} {path}: expected {expected}, got {response.status}") + if response.headers.get("X-Request-ID") != "apiWorkflow": raise AssertionError("requestId header absent") + if not data: return None + value = json.loads(data) + if path != "/openapi.json" and value.get("requestId") != "apiWorkflow": raise AssertionError("requestId body absent") + return value + def start(): + process = subprocess.Popen([str(binary)], cwd=project, env=environment, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + def ready(): + if process.poll() is not None: raise AssertionError("example failed startup") + try: return request("GET", "/health/ready")["code"] == "READY" + except (OSError, urllib.error.URLError): return False + try: waitFor(ready, "business service ready", 15) + except BaseException: + process.kill();process.wait(timeout=5);raise + return process + process = start() + try: + request("GET", "/api/v1/users", expected=401, auth=False); checks.append("authBeforeService") + empty = request("GET", "/api/v1/users")["data"] + assert empty["items"] == [] and empty["total"] == 0 + checks.append("emptyPageAndQueryDefaults") + request("POST", "/api/v1/users", {"name": "X", "email": "invalid"}, 400) + request("POST", "/api/v1/users", {"name": "X", "email": "x@example.com", "unknown": 1}, 400) + request("GET", "/api/v1/users?pageSize=101", expected=400) + request("GET", "/api/v1/users/0", expected=400); checks.append("contractValidation") + user = request("POST", "/api/v1/users", {"name": " Alice ", "email": "Alice@Example.com"}, 201)["data"] + assert user["name"] == "Alice" and user["email"] == "alice@example.com" + uid = user["id"]; path = f"/api/v1/users/{uid}" + assert request("GET", path)["data"] == user; checks.append("createReadNormalize") + assert request("POST", "/api/v1/users", {"name": "Other", "email": "alice@example.com"}, 409)["code"] == "EMAIL_TAKEN" + checks.append("uniqueEmailConflict") + user = request("PUT", path, {"name": "Updated", "email": "updated@example.com"})["data"] + assert user["name"] == "Updated" + page = request("GET", "/api/v1/users?page=1&pageSize=1")["data"] + assert page["total"] == 1 and page["items"][0]["id"] == uid + checks.append("updateAndPagination") + # Force the second SQL write to fail: create/update/delete must roll back. + db.sql("CREATE TRIGGER reject_audit BEFORE INSERT ON user_audit FOR EACH ROW SIGNAL SQLSTATE '45000' SET MESSAGE_TEXT='acceptance rollback';", "omni_test") + try: + request("POST", "/api/v1/users", {"name": "Rollback", "email": "rollback@example.com"}, 500) + request("PUT", path, {"name": "Rollback", "email": "rollback@example.com"}, 500) + request("DELETE", path, expected=500) + assert request("GET", path)["data"] == user + assert db.sql("SELECT COUNT(*) FROM users WHERE email='rollback@example.com'", "omni_test") == "0" + assert db.sql("SELECT COUNT(*) FROM user_audit", "omni_test") == "2" + finally: db.sql("DROP TRIGGER reject_audit", "omni_test") + checks.append("createUpdateDeleteRollbackOnAuditFailure") + link = request("POST", path + "/export")["data"]["downloadUrl"] + parsed = urllib.parse.urlparse(link) + assert parsed.hostname == "127.0.0.1" and parsed.port == port + with client.open(link, timeout=3) as response: exported = json.load(response) + assert exported == user; checks.append("storageUploadAndSignedDownload") + tampered = link.replace("signature=", "signature=bad") + if tampered == link: + query = urllib.parse.parse_qs(parsed.query) + key = next(k for k in query if k.lower() in ("sig", "signature")) + query[key] = ["bad"] + tampered = urllib.parse.urlunparse(parsed._replace(query=urllib.parse.urlencode(query, doseq=True))) + try: client.open(tampered, timeout=3); raise AssertionError("tampered signature accepted") + except urllib.error.HTTPError as exc: assert exc.code == 403 + checks.append("signedDownloadTamperRejected") + doc = request("GET", "/openapi.json") + assert doc == json.loads((project / "docs/openapi.json").read_text()); checks.append("openAPIMatchesContract") + run(["docker", "stop", "--time=3", db.container], project, db.env) + request("GET", "/health/ready", expected=503) + run(["docker", "start", db.container], project, db.env); db.wait() + waitFor(lambda: request("GET", "/health/ready")["code"] == "READY", "database recovery") + checks.append("readinessOutageAndRecovery") + process.send_signal(signal.SIGTERM); assert process.wait(timeout=12) == 0 + process = start() + assert request("GET", path)["data"] == user; checks.append("persistenceAfterServiceRestart") + request("DELETE", path, expected=204) + assert request("GET", path, expected=404)["code"] == "USER_NOT_FOUND" + assert request("GET", "/api/v1/users")["data"]["items"] == [] + assert db.sql("SELECT COUNT(*) FROM user_audit WHERE request_id='apiWorkflow'", "omni_test") == "3" + checks.append("deleteAndTransactionalAudit") + process.send_signal(signal.SIGTERM); assert process.wait(timeout=12) == 0 + with socket.socket() as sock: assert sock.connect_ex(("127.0.0.1", port)) != 0 + checks.append("gracefulStopAndListenerRelease") + text = logs.read_text() + entries = [json.loads(line) for line in text.splitlines() if line.startswith("{")] + assert any(e.get("msg") == "user.created" and e.get("requestId") == "apiWorkflow" for e in entries) + assert all(secret not in text for secret in (token, signing, env["GO_OMNI_DB_DSN"], link, "updated@example.com")) + checks.append("businessFileLogsAndSecretExclusion") + return {"transport": transport, "checks": checks} + finally: + if process.poll() is None: process.kill(); process.wait(timeout=5) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--version") + parser.add_argument("--report", type=Path, default=ROOT / "performance/apiWorkflowLocal.json") + args = parser.parse_args() + env = {k: v for k, v in os.environ.items() if not k.startswith(("GO_OMNI_", "USER_SERVICE_"))} + env.update(GOWORK="off", GOFLAGS="", GOPRIVATE="git.yuntuo.hk", GONOSUMDB="git.yuntuo.hk") + started = time.monotonic() + resolved = {} + with tempfile.TemporaryDirectory(prefix="goOmniAPIWorkflow-") as temp: + temp = Path(temp); cli = temp / "bin/go-omni"; cli.parent.mkdir() + if args.version: + env.update(GOBIN=str(cli.parent), GOMODCACHE=str(temp / "modcache")) + run(["go", "install", MODULE + "/cmd/go-omni@" + args.version], temp, env, timeout=300) + else: run(["go", "build", "-o", cli, "./cmd/go-omni"], ROOT, env) + project = temp / "userService" + source = ROOT / "examples/userService" + shutil.copytree(source, project, ignore=lambda directory, names: {"var", "server"} if Path(directory) == source else set()) + if args.version: + run(["go", "mod", "edit", "-dropreplace=" + MODULE, "-require=" + MODULE + "@" + args.version], project, env) + else: run(["go", "mod", "edit", "-replace=" + MODULE + "=" + str(ROOT)], project, env) + run(["go", "mod", "tidy"], project, env) + if args.version: + assert "replace " not in (project / "go.mod").read_text() + modules = run(["go", "list", "-m", "-json", MODULE], project, env) + resolved = json.loads(modules) + assert resolved["Version"] == args.version and "Replace" not in resolved + # Installed CLI must pin its own version without a source checkout. + fresh = temp / "fresh" + run([cli, "api", "new", "--module", "example.com/fresh", "--database", "mysql", "--storage", "local", fresh], temp, env) + assert args.version in (fresh / "go.mod").read_text() and "replace " not in (fresh / "go.mod").read_text() + run(["go", "test", "-mod=mod", "./..."], fresh, env) + # Regeneration must preserve all developer-owned code and config. + manifest = json.loads((project / ".goOmni/apiManifest.json").read_text()) + owned = set(manifest["generated"]) | {".goOmni/apiManifest.json"} + def hand_hashes(): + return {str(p.relative_to(project)): hashlib.sha256(p.read_bytes()).hexdigest() + for p in project.rglob("*") if p.is_file() and str(p.relative_to(project)) not in owned} + before = hand_hashes() + run([cli, "api", "generate", "--api", "api/user.api", "--dir", ".", "--check"], project, env) + run([cli, "api", "generate", "--api", "api/user.api", "--dir", "."], project, env) + assert before == hand_hashes() + db = LimitedMySQL(temp, env, "mysql:8.4") + try: + print("Starting bounded, isolated MySQL", flush=True); db.start() + env.update(GO_OMNI_DB_DSN=db.dsn, USER_SERVICE_TEST_DATABASE="1") + run([cli, "migrate", "--dir", "migrations", "--apply"], project, env) + run(["go", "test", "-race", "-count=1", "./..."], project, env) + run(["go", "vet", "./..."], project, env) + binary = project / "server";run(["go", "build", "-o", binary, "./cmd/server"], project, env) + groups = [] + for transport in ("nethttp", "native"): + db.sql("DELETE FROM user_audit; DELETE FROM users;", "omni_test") + groups.append(exercise(binary, project, env, db, transport)) + print("PASS business workflow: " + transport, flush=True) + finally: db.close() + version = args.version or "local" + report = {"passed": True, "frameworkVersion": version, "noLocalReplace": bool(args.version), + "durationSeconds": round(time.monotonic() - started, 2), + "regenerationPreservedHandwrittenFiles": len(before), "groups": groups, + "transactionTests": ["commit", "errorRollback", "panicRollback", "cancellationRollback"], + "startupTests": ["storageFailureClosesDatabase", "idempotentClose", "closedContextCannotReopen"], + "migrationCLI": True, + "frameworkSum": resolved.get("Sum"), + "sourceHashes": {str(p.relative_to(ROOT)): hashlib.sha256(p.read_bytes()).hexdigest() + for base in (ROOT / "cmd/go-omni", ROOT / "database", ROOT / "examples/userService") + for p in sorted(base.rglob("*")) if p.is_file() and p.suffix in (".go", ".tmpl", ".api", ".sql", ".yaml")}} + args.report.write_text(json.dumps(report, indent=2) + "\n") + print(f"PASS API workflow ({version}): {sum(len(g['checks']) for g in groups)} HTTP/lifecycle checks", flush=True) + + +if __name__ == "__main__": + def interrupt(signum, frame): + raise KeyboardInterrupt(f"signal {signum}") + for sig in (signal.SIGTERM, signal.SIGHUP): signal.signal(sig, interrupt) + main() diff --git a/scripts/verifyScaffoldModules.py b/scripts/verifyScaffoldModules.py new file mode 100644 index 0000000..3832d1c --- /dev/null +++ b/scripts/verifyScaffoldModules.py @@ -0,0 +1,178 @@ +#!/usr/bin/env python3 +"""Generate runnable logging/storage projects and verify their HTTP and shutdown paths.""" +import argparse +from datetime import datetime, timezone +import hashlib +import json +import os +from pathlib import Path +import secrets +import signal +import socket +import subprocess +import tempfile +import time +import urllib.error +import urllib.parse +import urllib.request + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--cli', type=Path) + parser.add_argument('--framework-dir', type=Path, default=Path(__file__).resolve().parent.parent) + parser.add_argument('--output', type=Path, default=Path('performance/scaffoldModulesLocal.json')) + args = parser.parse_args() + root = args.framework_dir.resolve() + output = args.output.resolve() + output.parent.mkdir(parents=True, exist_ok=True) + log_path = output.with_suffix('.txt') + env = {k: v for k, v in os.environ.items() if not k.startswith('GO_OMNI_')} + env.update(GOWORK='off', GOMAXPROCS='4') + report = {'status': 'running', 'startedAt': datetime.now(timezone.utc).isoformat(), 'cases': [], 'sourceSha256': {}} + sources = list((root / 'cmd/go-omni').rglob('*.go')) + list((root / 'cmd/go-omni/scaffold').rglob('*.tmpl')) + sources += [root / 'config/env.go', Path(__file__).resolve()] + for p in sorted(sources): + report['sourceSha256'][str(p.relative_to(root))] = hashlib.sha256(p.read_bytes()).hexdigest() + opener = urllib.request.build_opener(urllib.request.ProxyHandler({})) + + def request(method, url, body=None, headers=None): + req = urllib.request.Request(url, data=body, headers=headers or {}, method=method) + try: + response = opener.open(req, timeout=3) + except urllib.error.HTTPError as error: + response = error + with response: + return response.status, response.read(), dict(response.headers) + + def call(base, token, path, value=None, expected=200, method='POST'): + headers = {'Content-Type': 'application/json', 'Authorization': 'Bearer ' + token, 'X-Request-ID': 'scaffold-module-test'} + body = json.dumps(value).encode() if value is not None else None + status, data, _ = request(method, base + path, body, headers) + if status != expected: + raise AssertionError(f'{method} {path}: {status}, expected {expected}; {data[:512]!r}') + return json.loads(data) if data else None + + failure = None + with log_path.open('w') as log, tempfile.TemporaryDirectory(prefix='goOmniModuleAcceptance') as directory: + temp = Path(directory).resolve() + + def run(command, cwd): + subprocess.run(command, cwd=cwd, env=env, stdout=log, stderr=subprocess.STDOUT, check=True, timeout=180) + log.flush() + + try: + cli = args.cli.resolve() if args.cli else temp / 'go-omni' + if not args.cli: + run(['go', 'build', '-o', str(cli), './cmd/go-omni'], root) + profiles = [('stderr', 'json', 'nethttp', signal.SIGTERM), ('stdout', 'text', 'native', signal.SIGINT), + ('file', 'json', 'native', signal.SIGTERM), ('file', 'text', 'nethttp', signal.SIGINT)] + for index, (sink, fmt, transport, stop_signal) in enumerate(profiles): + print(f'Checking {sink}/{fmt}, {transport}, {stop_signal.name}', flush=True) + project = temp / ('case' + str(index)) + run([str(cli), 'new', '--module', 'example.com/moduleAcceptance', '--framework-dir', str(root), + '--storage', 'local', '--log-output', sink, '--log-format', fmt, str(project)], temp) + run(['go', 'test', '-mod=mod', '-race', '-p=2', './...'], project) + binary = project / 'server' + run(['go', 'build', '-o', str(binary), './cmd/server'], project) + with socket.socket() as reservation: + reservation.bind(('127.0.0.1', 0)) + port = reservation.getsockname()[1] + base = f'http://127.0.0.1:{port}' + token = secrets.token_hex(32) + runtime_env = dict(env, GO_OMNI_FILES_TOKEN=token, GO_OMNI_STORAGE_SIGNING_KEY=secrets.token_hex(32), + GO_OMNI_HTTP_ADDR=f'127.0.0.1:{port}', GO_OMNI_TRANSPORT=transport, + GO_OMNI_STORAGE_PUBLIC_URL=base + '/storage/object', GO_OMNI_STORAGE_MAX_BYTES='64') + process_log = project / 'process.log' + with process_log.open('w') as process_output: + process = subprocess.Popen([str(binary)], cwd=project, env=runtime_env, stdout=process_output, stderr=subprocess.STDOUT) + try: + deadline = time.monotonic() + 15 + while True: + if process.poll() is not None: + raise RuntimeError('startup failed: ' + process_log.read_text()) + try: + status, _, _ = request('GET', base + '/health/ready') + if status == 200: + break + except (OSError, urllib.error.URLError): + pass + if time.monotonic() > deadline: + raise TimeoutError('service readiness timeout') + time.sleep(0.05) + call(base, token, '/users', {'name': 'Example', 'email': 'module@example.com'}, 201) + data = b'hello go-omni' + upload_headers = {'Content-Type': 'text/plain', 'Authorization': 'Bearer ' + token, 'X-Request-ID': 'scaffold-module-test'} + status, raw, _ = request('POST', base + '/files', data, upload_headers) + if status != 201: + raise AssertionError(f'upload failed: {status}, {raw!r}') + key = json.loads(raw)['data']['key'] + download = call(base, token, '/files/downloadLink', {'key': key})['data']['url'] + status, raw, _ = request('GET', download) + if status != 200 or raw != data: + raise AssertionError('download content mismatch') + parts = urllib.parse.urlsplit(download) + query = urllib.parse.parse_qs(parts.query) + query['signature'] = ['tampered'] + altered = urllib.parse.urlunsplit(parts._replace(query=urllib.parse.urlencode(query, doseq=True))) + if request('GET', altered)[0] != 403: + raise AssertionError('tampered signature accepted') + direct = call(base, token, '/files/uploadLink', {'size': len(data), 'contentType': 'text/plain'})['data'] + signed = direct['request'] + headers = {k: ', '.join(v) for k, v in signed['headers'].items()} + if request(signed['method'], signed['url'], data, headers)[0] != 201: + raise AssertionError('signed upload failed') + if request('POST', base + '/files', data, {'Content-Type': 'text/plain'})[0] != 401: + raise AssertionError('missing authorization accepted') + if request('POST', base + '/files', b'x' * 65, upload_headers)[0] != 413: + raise AssertionError('oversized upload accepted') + wrong_headers = dict(upload_headers, **{'Content-Type': 'image/png'}) + if request('POST', base + '/files', data, wrong_headers)[0] != 415: + raise AssertionError('forged type accepted') + call(base, token, '/files/downloadLink', {'key': '../bad'}, 400) + call(base, token, '/files/downloadLink', {'key': 'missing'}, 404) + process.send_signal(stop_signal) + exit_code = process.wait(timeout=12) + if exit_code != 0: + raise AssertionError(f'{stop_signal.name}: exit {exit_code}') + finally: + if process.poll() is None: + process.kill() + process.wait(timeout=5) + sink_path = project / 'var/log/server.log' if sink == 'file' else process_log + text = sink_path.read_text() + if token in text or 'signature=' in text or '"url":' in text: + raise AssertionError('secret or signed URL appeared in logs') + for event in ('user.created', 'file.uploaded', 'file.downloadLink.issued', 'file.uploadLink.issued', 'http.request', 'scaffold-module-test'): + if event not in text: + raise AssertionError('missing log evidence: ' + event) + if fmt == 'json': + entries = [json.loads(line) for line in text.splitlines() if line.strip()] + if not any(e.get('status') == 403 and e.get('path') == '/storage/object' for e in entries): + raise AssertionError('signed endpoint status lost in access logs') + with socket.socket() as check: + check.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) + check.bind(('127.0.0.1', port)) + # Startup validation must fail instead of silently exposing control endpoints. + invalid_env = dict(runtime_env, GO_OMNI_FILES_TOKEN='') + invalid = subprocess.run([str(binary)], cwd=project, env=invalid_env, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, timeout=5) + if invalid.returncode == 0 or b'GO_OMNI_FILES_TOKEN' not in invalid.stdout: + raise AssertionError('missing token did not fail startup') + report['cases'].append({'logOutput': sink, 'logFormat': fmt, 'transport': transport, 'signal': stop_signal.name, + 'status': 'passed', 'exitCode': exit_code, 'portReleased': True, 'missingTokenRejected': True}) + report['status'] = 'passed' + except Exception as error: + failure = str(error) + report['status'], report['error'] = 'failed', failure + finally: + log.flush() + report['finishedAt'] = datetime.now(timezone.utc).isoformat() + report['logSha256'] = hashlib.sha256(log_path.read_bytes()).hexdigest() + output.write_text(json.dumps(report, indent=2) + '\n') + if failure: + raise SystemExit('FAIL: ' + failure + '; see ' + str(log_path)) + print('PASS generated project HTTP/logging/shutdown; report: ' + str(output), flush=True) + + +if __name__ == '__main__': + main() diff --git a/scripts/verifyUploads.py b/scripts/verifyUploads.py new file mode 100644 index 0000000..7d2fbb1 --- /dev/null +++ b/scripts/verifyUploads.py @@ -0,0 +1,95 @@ +#!/usr/bin/env python3 +"""Run real S3 uploads in one bounded, network-isolated local Docker container.""" +import argparse +from datetime import datetime, timezone +import hashlib +import json +import os +from pathlib import Path +import subprocess +import tempfile +import uuid + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--image', default='quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z') + parser.add_argument('--output', type=Path, default=Path('performance/uploadsLocal.json')) + args = parser.parse_args() + root = Path(__file__).resolve().parent.parent + output = args.output.resolve() + output.parent.mkdir(parents=True, exist_ok=True) + log_path = output.with_suffix('.txt') + name = 'go-omni-upload-' + uuid.uuid4().hex[:12] + + def capture(command): + return subprocess.check_output(command, text=True, timeout=15).strip() + + arch = capture(['docker', 'info', '--format', '{{.Architecture}}']) + arch = {'aarch64': 'arm64', 'x86_64': 'amd64'}.get(arch, arch) + if arch not in ('arm64', 'amd64'): + raise SystemExit('unsupported Docker architecture: ' + arch) + before = set(capture(['docker', 'ps', '-q']).split()) + report = { + 'status': 'running', 'startedAt': datetime.now(timezone.utc).isoformat(), + 'architecture': arch, 'goVersion': capture(['go', 'version']), + 'image': args.image, 'imageId': capture(['docker', 'image', 'inspect', args.image, '--format', '{{.Id}}']), + 'sourceSha256': {}, 'commands': [], + 'containerLimits': {'cpus': 2, 'memoryBytes': 2147483648, 'pids': 256, 'network': 'none'}, + 'notes': ['MinIO verifies S3 signatures; OSS/R2 real cloud accounts are not exercised.'], + } + for p in sorted(list((root / 'storage').rglob('*.go')) + list((root / 'internal/storagefixture').glob('*.go')) + [root / 'go.mod', root / 'go.sum']): + report['sourceSha256'][str(p.relative_to(root))] = hashlib.sha256(p.read_bytes()).hexdigest() + failure = None + with log_path.open('w') as log: + try: + with tempfile.TemporaryDirectory(prefix='goOmniUpload') as temporary: + Path(temporary).chmod(0o755) + env = dict(os.environ, GOOS='linux', GOARCH=arch, CGO_ENABLED='0', GOMAXPROCS='2', GOWORK='off') + build = ['go', 'test', '-p=2', '-c', '-o', temporary + '/s3.test', './storage/s3'] + report['commands'].append(build) + subprocess.run(build, cwd=root, env=env, stdout=log, stderr=subprocess.STDOUT, check=True, timeout=240) + check = '''set -eu +minio --version +minio server /data --address 127.0.0.1:9000 --console-address 127.0.0.1:9001 >/tmp/minio.log 2>&1 & +server=$! +trap 'kill "$server" 2>/dev/null || true; wait "$server" 2>/dev/null || true' EXIT +/test/s3.test -test.v -test.run '^TestS3Integration$' -test.count=1 -test.timeout=60s || { cat /tmp/minio.log; exit 1; } +''' + command = ['docker', 'run', '--rm', '--name', name, '--platform', 'linux/' + arch, + '--cpus', '2', '--memory', '2048m', '--memory-swap', '2048m', '--pids-limit', '256', + '--network', 'none', '--read-only', '--cap-drop', 'ALL', '--security-opt', 'no-new-privileges', + '--user', '1000:1000', '--tmpfs', '/tmp:rw,nosuid,noexec,size=64m,mode=1777', + '--tmpfs', '/data:rw,nosuid,noexec,size=128m,mode=1777', '-v', temporary + ':/test:ro', + '-e', 'GOMAXPROCS=2', '-e', 'MINIO_ROOT_USER=goOmniTest', '-e', 'MINIO_ROOT_PASSWORD=goOmniTestSecret', + '-e', 'GO_OMNI_S3_TEST_ADDR=127.0.0.1:9000', '--entrypoint', 'sh', args.image, '-c', check] + report['commands'].append(command) + subprocess.run(command, stdout=log, stderr=subprocess.STDOUT, check=True, timeout=90) + log.flush() + if '--- PASS: TestS3Integration' not in log_path.read_text(): + raise RuntimeError('missing integration pass evidence') + report['status'] = 'passed' + except Exception as error: + failure = str(error) + report['status'], report['error'] = 'failed', failure + finally: + remaining = subprocess.run(['docker', 'inspect', name], capture_output=True, timeout=15) + if remaining.returncode == 0: + subprocess.run(['docker', 'stop', '-t', '3', name], capture_output=True, timeout=15, check=True) + after = set(capture(['docker', 'ps', '-q']).split()) + report['preExistingContainersStillRunning'] = before.issubset(after) + if not before.issubset(after): + failure = 'a pre-existing container is no longer running' + report['status'], report['error'] = 'failed', failure + report['cleanedContainer'] = name + report['finishedAt'] = datetime.now(timezone.utc).isoformat() + log.flush() + report['logSha256'] = hashlib.sha256(log_path.read_bytes()).hexdigest() + output.write_text(json.dumps(report, indent=2) + '\n') + if failure: + raise SystemExit('FAIL: ' + failure + '; see ' + str(log_path)) + print('PASS real S3 direct/multipart uploads, pagination, abort, signed constraints; report: ' + str(output)) + + +if __name__ == '__main__': + main() diff --git a/storage/README.md b/storage/README.md new file mode 100644 index 0000000..6267702 --- /dev/null +++ b/storage/README.md @@ -0,0 +1,115 @@ +# 对象存储与上传 + +核心 `storage.Store` 保持 `Put/Get/Delete`，不会强迫每个适配器实现云厂商的所有能力。业务可按需依赖可选接口： + +| 能力 | 接口 | Local | S3 / R2 | OSS | +| --- | --- | --- | --- | --- | +| 读写删除 | Store | 支持 | 支持 | 支持 | +| 临时下载 | Presigner | NewSigner | 支持 | 支持 | +| 限定大小/类型的直传 | UploadPresigner | NewUploadSigner | 支持 | 支持，需要配置签名 Header | +| 创建、上传、查询、完成、取消分片 | MultipartStore | 当前不提供 | 支持 | 支持 | +| 分片直传签名 | PartPresigner | 当前不提供 | 支持 | 支持，需要配置签名 Header | + +没有固定业务字段、目录规则或身份模型。鉴权、对象归属、业务配额由应用定义。SDK 客户端和连接池由应用持有；适配器不会在一次调用后关闭客户端。 + +## 服务端上传限制 + +```go +uploader, err := storage.NewUploader(files, storage.UploadPolicy{ + MaxBytes: 32 << 20, + ContentTypes: []string{"image/png", "image/jpeg", "application/pdf"}, +}) +if err != nil { return err } +info, err := uploader.Put(ctx, key, reader, size, contentType) +``` + +`files` 是任意 Store。MaxBytes 必须为正数；ContentTypes 是精确 MIME 白名单，忽略参数，不接受通配符。空白名单允许所有合法声明类型。单次上传检查声明大小、流的实际长度；设置类型白名单时，再以头部最多 512 字节嗅探结果核对声明类型。适配器本身的大小限制仍然有效，建议与策略统一。这里只读取有限前缀，不把完整文件放进内存。 + +MIME 嗅探用于粗粒度识别，不验证完整文件结构或病毒；CSV/JSON 等类型可能被嗅探成 text/plain。需要复杂格式判断时，在业务校验流程中处理。必须传入可终止、正确返回 EOF 的 reader；ContextReader 不能中断任意用户 Reader 中已经阻塞的 Read。 + +使用 Local 时先写临时文件，校验失败不会替换旧对象。云端网络中断可能产生结果未知或遗留未完成上传；使用唯一暂存 key，并通过查询/对账确认后再更新业务记录，不要把上传 error 当成绝对未写入。 + +## 云端直传 + +```go +// files 为 S3/R2 或 OSS 的 Store；Uploader 统一执行策略检查。 +uploader, err := storage.NewUploader(files, storage.UploadPolicy{ + MaxBytes: 32 << 20, ContentTypes: []string{"image/png", "image/jpeg"}, +}) +if err != nil { return err } +// 业务先授权并生成暂存 key，随后签发。 +request, err := uploader.PresignPut(ctx, key, size, "image/png", 5*time.Minute) +``` + +返回 `method/url/headers/expiresAt`，有效期 1 秒到 1 小时。签名绑定 key、方法、确切 Content-Length 和声明 Content-Type，缺少这些签名约束时适配器报错。SDK 凭据提前失效时，链接也可能提前失效。 + +浏览器请求使用原始 File/Blob 作为 body，不能额外包 FormData；Content-Length 由浏览器按 body 长度生成，禁止手动设置。其余 Headers 按返回值原样发送： + +```js +const headers = new Headers(); +for (const [name, values] of Object.entries(request.headers)) { + if (name.toLowerCase() !== 'content-length') headers.set(name, values.join(', ')); +} +const response = await fetch(request.url, { + method: request.method, headers, body: file, +}); +if (!response.ok) throw new Error(`upload failed: ${response.status}`); +``` + +桶的 CORS 由部署方配置，允许业务域名、PUT 和返回的请求头；分片直传还需暴露 ETag 响应头。前端不能得到永久凭据。链接在到期前可能重复使用并覆盖同一个 key，不是一次性凭证。用随机暂存 key，业务确认前不要当作已发布文件。 + +**云端直传的类型限制针对声明 Header，无法由 API 服务实时检查文件字节。** 内容校验/扫描需要后处理；验证通过前不要公开文件。前端报告“上传完成”不等于文件可信。具体平台仍需实网验收，不能仅凭 S3 兼容声明判断一致。 + +OSS 使用官方 Go SDK v2： + +```go +client := ossSdk.NewClient(ossSdk.LoadDefaultConfig(). + WithRegion("cn-hangzhou"). + WithSignatureVersion(ossSdk.SignatureVersionV4). + WithAdditionalHeaders([]string{"content-length"}). + WithCredentialsProvider(credentialProvider)) +files, err := oss.New(client, bucket, 32<<20) +``` + +适配器不修改传入的 client；缺少签名大小约束会返回明确错误。SDK 的 presign 实现与返回 SignedHeaders 都经过检查。 + +## 本地签名上传 + +```go +files, err := local.New("./privateFiles", 32<<20) +if err != nil { return err } +signer, err := storage.NewUploadSigner(files, "https://files.example/upload", secret, + storage.UploadPolicy{MaxBytes: 32<<20, ContentTypes: []string{"image/png"}}) +if err != nil { return err } +// signer 实现 http.Handler，可挂到现有的 HTTP 服务；secret 至少 32 字节。 +request, err := signer.PresignPut(ctx, key, size, "image/png", time.Minute) +``` + +Handler 检查签名、有效期、单值参数、Content-Length/Content-Type，以及实际内容大小和嗅探类型，成功返回 201。本地存储只开放一个受控入口，不把磁盘目录映射成公共目录。原 `NewSigner` 继续只服务下载。请求超时、CORS、并发限制和速率限制由承载该 Handler 的服务配置。 + +## 分片与续传 + +S3/R2/OSS 均实现 `MultipartStore` 和 `PartPresigner`。Uploader 也暴露相同操作；不支持的底层返回 `ErrUnsupported`。 + +```go +session, err := uploader.CreateMultipart(ctx, key, "application/pdf") +if err != nil { return err } +// 业务持久保存 session，与已授权调用者关联，不能信任客户端任意传入的 key/uploadId。 +part, err := uploader.UploadPart(ctx, session, 1, reader, size) +// 或签发单个分片的 PUT 请求： +request, err := uploader.PresignPart(ctx, session, 1, size, 5*time.Minute) +// 重连后分页读取实际已上传的分片，NextMarker == 0 表示结束。 +page, err := uploader.ListParts(ctx, session, 0, 1000) +// 按 Number 升序从 1 连续提交，ETag 使用存储平台的返回值。 +info, err := uploader.CompleteMultipart(ctx, session, parts) +// 用户取消/任务过期时，用仍有效的 context 显式清理。 +err = uploader.AbortMultipart(ctx, session) +``` + +以上为互斥分支的 API 索引，实际调用需逐步处理 error。续传由业务保存 session，查询已有分片后重传缺失部分；框架不维护有身份语义的上传任务表，也不代替业务调度重试。 + +统一边界：每个对象最多 10,000 片、每片最多 5 GiB，除末片外至少 5 MiB；这是适配层为跨平台复用选取的共同约束。空文件走普通 Put。ListParts 每页最多 1,000 条。平台/桶限制可能更低。默认单对象仍为 32 MiB，需要大文件时显式提高适配器与 UploadPolicy 的限制；OSS 简单 Put 和云端单次签名 PUT 最大 5 GiB，更大使用分片。 + +Complete 会向平台分页核对全部分片的序号、ETag、真实大小和累计上限，前端填写的 `Part.Size` 不参与信任判断。完成时停止并发替换分片；Provider 仍会再次校验 ETag。返回 Info 的 Size 为已核对大小，ContentType 留空，完整对象元数据可通过 Get 获取。Multipart 的类型策略检查创建时的声明，实际内容应在暂存对象合并后校验。未完成的分片占用存储空间：应用需要超时取消或配置桶生命周期；URL 过期不会自动清理分片。 + +验收：Local 实际字节与原子性、S3/OSS SDK HTTP 协议测试、R2 签名配置测试、真实隔离 MinIO 验证。OSS/R2 尚未使用云账号实测。复现 MinIO 验证：先准备镜像，再运行 `python3 scripts/verifyUploads.py`；报告见 [uploadsLocal.json](../performance/uploadsLocal.json)。 diff --git a/storage/multipart.go b/storage/multipart.go new file mode 100644 index 0000000..31f3447 --- /dev/null +++ b/storage/multipart.go @@ -0,0 +1,173 @@ +package storage + +import ( + "context" + "errors" + "io" + "strings" + "time" + "unicode" +) + +// MultipartUpload is provider state, not authorization. Applications must bind +// it to the authenticated caller in their own durable session store. +type MultipartUpload struct { + Key string `json:"key"` + UploadID string `json:"uploadId"` +} +type Part struct { + Number int `json:"number"` + ETag string `json:"etag"` + Size int64 `json:"size"` +} +type PartsPage struct { + Parts []Part `json:"parts"` + NextMarker int `json:"nextMarker"` +} +type MultipartStore interface { + CreateMultipart(context.Context, string, string) (MultipartUpload, error) + UploadPart(context.Context, MultipartUpload, int, io.Reader, int64) (Part, error) + ListParts(context.Context, MultipartUpload, int, int) (PartsPage, error) + CompleteMultipart(context.Context, MultipartUpload, []Part) (Info, error) + AbortMultipart(context.Context, MultipartUpload) error +} +type PartPresigner interface { + PresignPart(context.Context, MultipartUpload, int, int64, time.Duration) (UploadRequest, error) +} + +func CheckMultipart(ctx context.Context, upload MultipartUpload) error { + if err := Check(ctx, upload.Key); err != nil { + return err + } + if upload.UploadID == "" || len(upload.UploadID) > 2048 || strings.IndexFunc(upload.UploadID, unicode.IsControl) >= 0 { + return errors.New("storage: invalid upload ID") + } + return nil +} +func CheckPart(ctx context.Context, upload MultipartUpload, number int, size, maxBytes int64) error { + if err := CheckMultipart(ctx, upload); err != nil { + return err + } + if number < 1 || number > 10000 || size <= 0 { + return errors.New("storage: part number must be 1..10000 and size positive") + } + if size > maxBytes || size > 5<<30 { + return ErrTooLarge + } + return nil +} +func CheckPartsPage(ctx context.Context, upload MultipartUpload, after, limit int) error { + if err := CheckMultipart(ctx, upload); err != nil { + return err + } + if after < 0 || after > 10000 || limit < 1 || limit > 1000 { + return errors.New("storage: invalid part pagination") + } + return nil +} + +// VerifyParts checks provider-reported sizes/ETags before completing. Caller +// sizes are not trusted. Do not upload/replace parts concurrently with completion. +// A common 5 MiB minimum (except last) keeps S3/OSS interchangeable. +func VerifyParts(ctx context.Context, store MultipartStore, upload MultipartUpload, parts []Part, maxBytes int64) (int64, error) { + if err := CheckMultipart(ctx, upload); err != nil { + return 0, err + } + if len(parts) == 0 || len(parts) > 10000 || maxBytes <= 0 { + return 0, errors.New("storage: invalid completion") + } + for i, p := range parts { + if p.Number != i+1 || p.ETag == "" || len(p.ETag) > 256 || strings.ContainsAny(p.ETag, "\r\n") { + return 0, errors.New("storage: ordered contiguous parts and ETags required") + } + } + var total int64 + marker, seen := 0, 0 + for { + page, err := store.ListParts(ctx, upload, marker, 1000) + if err != nil { + return 0, err + } + if len(page.Parts) > 1000 { + return 0, errors.New("storage: invalid provider page") + } + for _, actual := range page.Parts { + if seen >= len(parts) || actual.Number != parts[seen].Number || strings.Trim(actual.ETag, "\"") != strings.Trim(parts[seen].ETag, "\"") || actual.Size <= 0 { + return 0, errors.New("storage: parts changed or incomplete") + } + if actual.Size > 5<<30 || actual.Size > maxBytes-total { + return 0, ErrTooLarge + } + if seen < len(parts)-1 && actual.Size < 5<<20 { + return 0, errors.New("storage: non-final part smaller than 5 MiB") + } + total += actual.Size + seen++ + } + if page.NextMarker == 0 { + break + } + if len(page.Parts) == 0 || page.NextMarker <= marker || page.NextMarker > 10000 || page.NextMarker != page.Parts[len(page.Parts)-1].Number { + return 0, errors.New("storage: invalid provider pagination") + } + marker = page.NextMarker + } + if seen != len(parts) { + return 0, errors.New("storage: missing parts") + } + return total, nil +} +func (u *Uploader) CreateMultipart(ctx context.Context, key, contentType string) (MultipartUpload, error) { + if err := u.policy.Check(0, contentType); err != nil { + return MultipartUpload{}, err + } + s, ok := u.store.(MultipartStore) + if !ok { + return MultipartUpload{}, ErrUnsupported + } + return s.CreateMultipart(ctx, key, contentType) +} +func (u *Uploader) UploadPart(ctx context.Context, upload MultipartUpload, number int, body io.Reader, size int64) (Part, error) { + if err := CheckPart(ctx, upload, number, size, u.policy.MaxBytes); err != nil { + return Part{}, err + } + s, ok := u.store.(MultipartStore) + if !ok { + return Part{}, ErrUnsupported + } + return s.UploadPart(ctx, upload, number, body, size) +} +func (u *Uploader) PresignPart(ctx context.Context, upload MultipartUpload, number int, size int64, ttl time.Duration) (UploadRequest, error) { + if err := CheckPart(ctx, upload, number, size, u.policy.MaxBytes); err != nil { + return UploadRequest{}, err + } + s, ok := u.store.(PartPresigner) + if !ok { + return UploadRequest{}, ErrUnsupported + } + return s.PresignPart(ctx, upload, number, size, ttl) +} +func (u *Uploader) ListParts(ctx context.Context, upload MultipartUpload, after, limit int) (PartsPage, error) { + s, ok := u.store.(MultipartStore) + if !ok { + return PartsPage{}, ErrUnsupported + } + return s.ListParts(ctx, upload, after, limit) +} +func (u *Uploader) CompleteMultipart(ctx context.Context, upload MultipartUpload, parts []Part) (Info, error) { + s, ok := u.store.(MultipartStore) + if !ok { + return Info{}, ErrUnsupported + } + if _, err := VerifyParts(ctx, s, upload, parts, u.policy.MaxBytes); err != nil { + return Info{}, err + } + return s.CompleteMultipart(ctx, upload, parts) +} +func (u *Uploader) AbortMultipart(ctx context.Context, upload MultipartUpload) error { + s, ok := u.store.(MultipartStore) + if !ok { + return ErrUnsupported + } + return s.AbortMultipart(ctx, upload) +} diff --git a/storage/multipart_test.go b/storage/multipart_test.go new file mode 100644 index 0000000..c6219da --- /dev/null +++ b/storage/multipart_test.go @@ -0,0 +1,60 @@ +package storage + +import ( + "context" + "errors" + "testing" +) + +type listedParts struct { + MultipartStore + parts []Part + calls int + loop bool +} + +func (s *listedParts) ListParts(ctx context.Context, u MultipartUpload, marker, limit int) (PartsPage, error) { + s.calls++ + if s.loop { + return PartsPage{Parts: []Part{{Number: 1, ETag: "a", Size: 5 << 20}}, NextMarker: marker}, nil + } + end := min(marker+limit, len(s.parts)) + next := 0 + if end < len(s.parts) { + next = end + } + return PartsPage{Parts: s.parts[marker:end], NextMarker: next}, nil +} +func TestVerifyPartsUsesProviderSizes(t *testing.T) { + upload := MultipartUpload{Key: "file", UploadID: "id"} + first := Part{Number: 1, ETag: "a", Size: 5 << 20} + last := Part{Number: 2, ETag: "b", Size: 3} + supplied := []Part{{Number: 1, ETag: "a", Size: 1}, {Number: 2, ETag: "b", Size: 1}} + s := &listedParts{parts: []Part{first, last}} + n, err := VerifyParts(context.Background(), s, upload, supplied, 6<<20) + if err != nil || n != (5<<20)+3 { + t.Fatal(n, err) + } + if _, err = VerifyParts(context.Background(), s, upload, supplied, 5<<20); !errors.Is(err, ErrTooLarge) { + t.Fatal(err) + } + for _, parts := range [][]Part{nil, {last, first}, {first, first}, {{Number: 1, ETag: "changed"}, last}, {first}} { + if _, err := VerifyParts(context.Background(), s, upload, parts, 6<<20); err == nil { + t.Fatal("bad completion accepted", parts) + } + } + s.parts[0].Size = 1 + if _, err = VerifyParts(context.Background(), s, upload, supplied, 6<<20); err == nil { + t.Fatal("small non-final part accepted") + } +} +func TestVerifyPartsPagination(t *testing.T) { + s := &listedParts{} + for i := 1; i <= 1001; i++ { + s.parts = append(s.parts, Part{Number: i, ETag: "etag", Size: 5 << 20}) + } + n, err := VerifyParts(context.Background(), s, MultipartUpload{Key: "file", UploadID: "id"}, s.parts, 10<<30) + if err != nil || s.calls != 2 || n != 1001*(5<<20) { + t.Fatal(n, s.calls, err) + } +} diff --git a/storage/oss/oss.go b/storage/oss/oss.go index 57904dd..9c8d9fb 100644 --- a/storage/oss/oss.go +++ b/storage/oss/oss.go @@ -19,10 +19,10 @@ type Store struct { maxBytes int64 } -// New borrows a configured SDK client. This adapter uses simple uploads; large -// multipart uploads are a separate provider capability, not buffered in memory. +// New borrows a configured SDK client. Put uses simple uploads (up to 5 GiB). +// Larger objects use the optional MultipartStore capability. func New(client *sdk.Client, bucket string, maxBytes int64) (*Store, error) { - if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<30 { + if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<40 { return nil, errors.New("storage/oss: invalid client, bucket or size limit") } if maxBytes == 0 { @@ -41,7 +41,14 @@ func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int6 if size > s.maxBytes { return storage.Info{}, storage.ErrTooLarge } - _, err := s.client.PutObject(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType), Body: storage.ContextReader{Context: ctx, Reader: reader}}) + if size > 5<<30 { + return storage.Info{}, storage.ErrTooLarge + } + checked, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, reader, size, contentType) + if err != nil { + return storage.Info{}, err + } + _, err = s.client.PutObject(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType), Body: checked}) if err != nil { return storage.Info{}, mapError(err) } diff --git a/storage/oss/upload.go b/storage/oss/upload.go new file mode 100644 index 0000000..da28294 --- /dev/null +++ b/storage/oss/upload.go @@ -0,0 +1,133 @@ +package oss + +import ( + "context" + "errors" + "io" + "net/http" + "strconv" + "time" + + "git.yuntuo.hk/k2safe/go-omni/storage" + sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss" +) + +// PresignPut requires an OSS V4 client configured with +// WithAdditionalHeaders([]string{"content-length"}). Missing signed constraints +// fail closed; the borrowed client is never mutated by this adapter. +func (s *Store) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (storage.UploadRequest, error) { + if err := storage.CheckUpload(ctx, key, size, s.maxBytes, contentType, ttl); err != nil { + return storage.UploadRequest{}, err + } + if size > 5<<30 { + return storage.UploadRequest{}, storage.ErrTooLarge + } + if contentType == "" { + contentType = "application/octet-stream" + } + r, err := s.client.Presign(ctx, &sdk.PutObjectRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentLength: sdk.Ptr(size), ContentType: sdk.Ptr(contentType)}, sdk.PresignExpires(ttl)) + if err != nil { + return storage.UploadRequest{}, err + } + return uploadRequest(r, size, contentType) +} +func uploadRequest(r *sdk.PresignResult, size int64, contentType string) (storage.UploadRequest, error) { + headers := make(http.Header) + for k, v := range r.SignedHeaders { + headers.Set(k, v) + } + if headers.Get("Content-Length") != strconv.FormatInt(size, 10) || (contentType != "" && headers.Get("Content-Type") != contentType) { + return storage.UploadRequest{}, errors.New("storage/oss: upload requires signed Content-Length and Content-Type; configure V4 and WithAdditionalHeaders(content-length)") + } + return storage.UploadRequest{Method: r.Method, URL: r.URL, Headers: headers, ExpiresAt: r.Expiration}, nil +} +func (s *Store) CreateMultipart(ctx context.Context, key, contentType string) (storage.MultipartUpload, error) { + if err := storage.Check(ctx, key); err != nil { + return storage.MultipartUpload{}, err + } + if _, err := storage.MediaType(contentType); err != nil { + return storage.MultipartUpload{}, err + } + if contentType == "" { + contentType = "application/octet-stream" + } + r, err := s.client.InitiateMultipartUpload(ctx, &sdk.InitiateMultipartUploadRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(key), ContentType: sdk.Ptr(contentType)}) + if err != nil { + return storage.MultipartUpload{}, err + } + return storage.MultipartUpload{Key: key, UploadID: sdk.ToString(r.UploadId)}, nil +} +func (s *Store) UploadPart(ctx context.Context, upload storage.MultipartUpload, number int, body io.Reader, size int64) (storage.Part, error) { + if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil { + return storage.Part{}, err + } + reader, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, body, size, "") + if err != nil { + return storage.Part{}, err + } + r, err := s.client.UploadPart(ctx, &sdk.UploadPartRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), PartNumber: int32(number), ContentLength: sdk.Ptr(size), Body: reader}) + if err != nil { + return storage.Part{}, mapError(err) + } + return storage.Part{Number: number, ETag: sdk.ToString(r.ETag), Size: size}, nil +} +func (s *Store) PresignPart(ctx context.Context, upload storage.MultipartUpload, number int, size int64, ttl time.Duration) (storage.UploadRequest, error) { + if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil { + return storage.UploadRequest{}, err + } + if err := storage.CheckUpload(ctx, upload.Key, size, s.maxBytes, "", ttl); err != nil { + return storage.UploadRequest{}, err + } + r, err := s.client.Presign(ctx, &sdk.UploadPartRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), PartNumber: int32(number), ContentLength: sdk.Ptr(size), RequestCommon: sdk.RequestCommon{Headers: map[string]string{"Content-Type": "application/octet-stream"}}}, sdk.PresignExpires(ttl)) + if err != nil { + return storage.UploadRequest{}, err + } + return uploadRequest(r, size, "application/octet-stream") +} +func (s *Store) ListParts(ctx context.Context, upload storage.MultipartUpload, after, limit int) (storage.PartsPage, error) { + if err := storage.CheckPartsPage(ctx, upload, after, limit); err != nil { + return storage.PartsPage{}, err + } + r, err := s.client.ListParts(ctx, &sdk.ListPartsRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), PartNumberMarker: int32(after), MaxParts: int32(limit)}) + if err != nil { + return storage.PartsPage{}, err + } + page := storage.PartsPage{Parts: make([]storage.Part, 0, len(r.Parts))} + for _, p := range r.Parts { + page.Parts = append(page.Parts, storage.Part{Number: int(p.PartNumber), ETag: sdk.ToString(p.ETag), Size: p.Size}) + } + if r.IsTruncated { + page.NextMarker = int(r.NextPartNumberMarker) + } + return page, nil +} +func (s *Store) CompleteMultipart(ctx context.Context, upload storage.MultipartUpload, parts []storage.Part) (storage.Info, error) { + size, err := storage.VerifyParts(ctx, s, upload, parts, s.maxBytes) + if err != nil { + return storage.Info{}, err + } + completed := make([]sdk.UploadPart, len(parts)) + for i, p := range parts { + completed[i] = sdk.UploadPart{PartNumber: int32(p.Number), ETag: sdk.Ptr(p.ETag)} + } + _, err = s.client.CompleteMultipartUpload(ctx, &sdk.CompleteMultipartUploadRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID), CompleteMultipartUpload: &sdk.CompleteMultipartUpload{Parts: completed}}) + if err != nil { + return storage.Info{}, err + } + return storage.Info{Key: upload.Key, Size: size, ModifiedAt: time.Now().UTC()}, nil +} +func (s *Store) AbortMultipart(ctx context.Context, upload storage.MultipartUpload) error { + if err := storage.CheckMultipart(ctx, upload); err != nil { + return err + } + _, err := s.client.AbortMultipartUpload(ctx, &sdk.AbortMultipartUploadRequest{Bucket: sdk.Ptr(s.bucket), Key: sdk.Ptr(upload.Key), UploadId: sdk.Ptr(upload.UploadID)}) + var serviceError *sdk.ServiceError + if errors.As(err, &serviceError) && serviceError.Code == "NoSuchUpload" { + return nil + } + return err +} + +var _ storage.UploadPresigner = (*Store)(nil) +var _ storage.MultipartStore = (*Store)(nil) +var _ storage.PartPresigner = (*Store)(nil) diff --git a/storage/oss/upload_test.go b/storage/oss/upload_test.go new file mode 100644 index 0000000..687944c --- /dev/null +++ b/storage/oss/upload_test.go @@ -0,0 +1,29 @@ +package oss + +import ( + "context" + "net/http/httptest" + "testing" + "time" + + "git.yuntuo.hk/k2safe/go-omni/internal/storagefixture" + "git.yuntuo.hk/k2safe/go-omni/storage/storagetest" + sdk "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss" + "github.com/aliyun/alibabacloud-oss-go-sdk-v2/oss/credentials" +) + +func TestUploadProtocol(t *testing.T) { + server := httptest.NewTLSServer(storagefixture.MultipartHandler()) + defer server.Close() + cfg := sdk.LoadDefaultConfig().WithRegion("cn-hangzhou").WithEndpoint(server.URL).WithUsePathStyle(true).WithCredentialsProvider(credentials.NewStaticCredentialsProvider("fixtureAccess", "fixtureSecret")).WithHttpClient(server.Client()).WithSignatureVersion(sdk.SignatureVersionV4) + missing, _ := New(sdk.NewClient(cfg), "testbucket", 6<<20) + if _, err := missing.PresignPut(context.Background(), "file", 5, "text/plain", time.Minute); err == nil { + t.Fatal("unsigned length accepted") + } + cfg.WithAdditionalHeaders([]string{"content-length"}) + s, err := New(sdk.NewClient(cfg), "testbucket", 6<<20) + if err != nil { + t.Fatal(err) + } + storagetest.RunUploads(t, s, server.Client()) +} diff --git a/storage/s3/s3.go b/storage/s3/s3.go index 11a58a6..2a62560 100644 --- a/storage/s3/s3.go +++ b/storage/s3/s3.go @@ -19,7 +19,7 @@ type Store struct { } func New(client *minio.Client, bucket string, maxBytes int64) (*Store, error) { - if client == nil || bucket == "" || maxBytes < 0 { + if client == nil || bucket == "" || maxBytes < 0 || maxBytes > 5<<40 { return nil, errors.New("storage/s3: client, bucket and nonnegative limit required") } if maxBytes == 0 { @@ -37,7 +37,11 @@ func (s *Store) Put(ctx context.Context, key string, reader io.Reader, size int6 if size > s.maxBytes { return storage.Info{}, storage.ErrTooLarge } - result, err := s.client.PutObject(ctx, s.bucket, key, storage.ContextReader{Context: ctx, Reader: reader}, size, minio.PutObjectOptions{ContentType: contentType}) + checked, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, reader, size, contentType) + if err != nil { + return storage.Info{}, err + } + result, err := s.client.PutObject(ctx, s.bucket, key, checked, size, minio.PutObjectOptions{ContentType: contentType}) if err != nil { return storage.Info{}, err } diff --git a/storage/s3/s3_test.go b/storage/s3/s3_test.go index 05fb010..6c8a84b 100644 --- a/storage/s3/s3_test.go +++ b/storage/s3/s3_test.go @@ -73,6 +73,26 @@ func TestS3Integration(t *testing.T) { t.Fatal(err) } defer client.RemoveBucket(context.Background(), bucket) + uploads, _ := New(client, bucket, 6<<20) + httpClient := &http.Client{Timeout: 10 * time.Second} + storagetest.RunUploads(t, uploads, httpClient) + link, err := uploads.PresignPut(ctx, "constrained", 5, "text/plain", time.Minute) + if err != nil { + t.Fatal(err) + } + for _, tc := range []struct{ body, contentType string }{{"longer", "text/plain"}, {"hello", "image/png"}} { + req, _ := http.NewRequestWithContext(ctx, link.Method, link.URL, strings.NewReader(tc.body)) + req.Header = link.Headers.Clone() + req.Header.Set("Content-Type", tc.contentType) + response, requestErr := httpClient.Do(req) + if requestErr != nil { + t.Fatal(requestErr) + } + response.Body.Close() + if response.StatusCode != 403 { + t.Fatalf("tampered upload constraints accepted: %d", response.StatusCode) + } + } s, _ := New(client, bucket, 64) storagetest.Run(t, s) s.Put(ctx, "signed", strings.NewReader("hello"), 5, "text/plain") diff --git a/storage/s3/upload.go b/storage/s3/upload.go new file mode 100644 index 0000000..4b6795d --- /dev/null +++ b/storage/s3/upload.go @@ -0,0 +1,128 @@ +package s3 + +import ( + "context" + "errors" + "io" + "net/http" + "net/url" + "strconv" + "strings" + "time" + + "git.yuntuo.hk/k2safe/go-omni/storage" + "github.com/minio/minio-go/v7" +) + +func (s *Store) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (storage.UploadRequest, error) { + if err := storage.CheckUpload(ctx, key, size, s.maxBytes, contentType, ttl); err != nil { + return storage.UploadRequest{}, err + } + if size > 5<<30 { + return storage.UploadRequest{}, storage.ErrTooLarge + } + if contentType == "" { + contentType = "application/octet-stream" + } + headers := http.Header{"Content-Length": {strconv.FormatInt(size, 10)}, "Content-Type": {contentType}} + return s.signUpload(ctx, key, nil, headers, ttl) +} +func (s *Store) signUpload(ctx context.Context, key string, query url.Values, headers http.Header, ttl time.Duration) (storage.UploadRequest, error) { + expires := time.Now().UTC().Add(ttl) + u, err := s.client.PresignHeader(ctx, http.MethodPut, s.bucket, key, ttl, query, headers) + if err != nil { + return storage.UploadRequest{}, err + } + signed := ";" + u.Query().Get("X-Amz-SignedHeaders") + ";" + for name := range headers { + if !strings.Contains(signed, ";"+strings.ToLower(name)+";") { + return storage.UploadRequest{}, errors.New("storage/s3: upload constraints were not signed (SigV4 required)") + } + } + return storage.UploadRequest{Method: http.MethodPut, URL: u.String(), Headers: headers, ExpiresAt: expires}, nil +} +func (s *Store) CreateMultipart(ctx context.Context, key, contentType string) (storage.MultipartUpload, error) { + if err := storage.Check(ctx, key); err != nil { + return storage.MultipartUpload{}, err + } + if _, err := storage.MediaType(contentType); err != nil { + return storage.MultipartUpload{}, err + } + if contentType == "" { + contentType = "application/octet-stream" + } + id, err := (minio.Core{Client: s.client}).NewMultipartUpload(ctx, s.bucket, key, minio.PutObjectOptions{ContentType: contentType}) + if err != nil { + return storage.MultipartUpload{}, err + } + return storage.MultipartUpload{Key: key, UploadID: id}, nil +} +func (s *Store) UploadPart(ctx context.Context, upload storage.MultipartUpload, number int, body io.Reader, size int64) (storage.Part, error) { + if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil { + return storage.Part{}, err + } + reader, err := (storage.UploadPolicy{MaxBytes: s.maxBytes}).PrepareUpload(ctx, body, size, "") + if err != nil { + return storage.Part{}, err + } + part, err := (minio.Core{Client: s.client}).PutObjectPart(ctx, s.bucket, upload.Key, upload.UploadID, number, reader, size, minio.PutObjectPartOptions{}) + if err != nil { + return storage.Part{}, err + } + return storage.Part{Number: number, ETag: part.ETag, Size: size}, nil +} +func (s *Store) PresignPart(ctx context.Context, upload storage.MultipartUpload, number int, size int64, ttl time.Duration) (storage.UploadRequest, error) { + if err := storage.CheckPart(ctx, upload, number, size, s.maxBytes); err != nil { + return storage.UploadRequest{}, err + } + if err := storage.CheckUpload(ctx, upload.Key, size, s.maxBytes, "", ttl); err != nil { + return storage.UploadRequest{}, err + } + return s.signUpload(ctx, upload.Key, url.Values{"uploadId": {upload.UploadID}, "partNumber": {strconv.Itoa(number)}}, http.Header{"Content-Length": {strconv.FormatInt(size, 10)}, "Content-Type": {"application/octet-stream"}}, ttl) +} +func (s *Store) ListParts(ctx context.Context, upload storage.MultipartUpload, after, limit int) (storage.PartsPage, error) { + if err := storage.CheckPartsPage(ctx, upload, after, limit); err != nil { + return storage.PartsPage{}, err + } + r, err := (minio.Core{Client: s.client}).ListObjectParts(ctx, s.bucket, upload.Key, upload.UploadID, after, limit) + if err != nil { + return storage.PartsPage{}, err + } + page := storage.PartsPage{Parts: make([]storage.Part, 0, len(r.ObjectParts))} + for _, p := range r.ObjectParts { + page.Parts = append(page.Parts, storage.Part{Number: p.PartNumber, ETag: p.ETag, Size: p.Size}) + } + if r.IsTruncated { + page.NextMarker = r.NextPartNumberMarker + } + return page, nil +} +func (s *Store) CompleteMultipart(ctx context.Context, upload storage.MultipartUpload, parts []storage.Part) (storage.Info, error) { + size, err := storage.VerifyParts(ctx, s, upload, parts, s.maxBytes) + if err != nil { + return storage.Info{}, err + } + completed := make([]minio.CompletePart, len(parts)) + for i, p := range parts { + completed[i] = minio.CompletePart{PartNumber: p.Number, ETag: strings.Trim(p.ETag, "\"")} + } + _, err = (minio.Core{Client: s.client}).CompleteMultipartUpload(ctx, s.bucket, upload.Key, upload.UploadID, completed, minio.PutObjectOptions{}) + if err != nil { + return storage.Info{}, err + } + return storage.Info{Key: upload.Key, Size: size, ModifiedAt: time.Now().UTC()}, nil +} +func (s *Store) AbortMultipart(ctx context.Context, upload storage.MultipartUpload) error { + if err := storage.CheckMultipart(ctx, upload); err != nil { + return err + } + err := (minio.Core{Client: s.client}).AbortMultipartUpload(ctx, s.bucket, upload.Key, upload.UploadID) + if minio.ToErrorResponse(err).Code == "NoSuchUpload" { + return nil + } + return err +} + +var _ storage.UploadPresigner = (*Store)(nil) +var _ storage.MultipartStore = (*Store)(nil) +var _ storage.PartPresigner = (*Store)(nil) diff --git a/storage/s3/upload_test.go b/storage/s3/upload_test.go new file mode 100644 index 0000000..f7a7c5a --- /dev/null +++ b/storage/s3/upload_test.go @@ -0,0 +1,34 @@ +package s3 + +import ( + "context" + "net/http/httptest" + "strings" + "testing" + "time" + + "git.yuntuo.hk/k2safe/go-omni/internal/storagefixture" + "git.yuntuo.hk/k2safe/go-omni/storage/storagetest" + "github.com/minio/minio-go/v7/pkg/credentials" +) + +func TestUploadProtocol(t *testing.T) { + server := httptest.NewTLSServer(storagefixture.MultipartHandler()) + defer server.Close() + s, err := Open(Config{Endpoint: strings.TrimPrefix(server.URL, "https://"), Region: "us-east-1", Bucket: "testbucket", Secure: true, MaxBytes: 6 << 20, Credentials: credentials.NewStaticV4("fixtureAccess", "fixtureSecret", ""), Transport: server.Client().Transport}) + if err != nil { + t.Fatal(err) + } + storagetest.RunUploads(t, s, server.Client()) +} +func TestR2UploadSigning(t *testing.T) { + cfg, _ := R2Config(strings.Repeat("a", 32), "testbucket", credentials.NewStaticV4("fixtureAccess", "fixtureSecret", "")) + s, err := Open(cfg) + if err != nil { + t.Fatal(err) + } + r, err := s.PresignPut(context.Background(), "file", 5, "text/plain", time.Minute) + if err != nil || !strings.Contains(r.URL, "/auto/s3/") && !strings.Contains(r.URL, "%2Fauto%2Fs3%2F") { + t.Fatal("R2 upload signing", err) + } +} diff --git a/storage/signed.go b/storage/signed.go index d26d977..a3f02e7 100644 --- a/storage/signed.go +++ b/storage/signed.go @@ -14,10 +14,11 @@ import ( ) type Signer struct { - store Store - base *url.URL - secret []byte - now func() time.Time + store Store + base *url.URL + secret []byte + now func() time.Time + uploadPolicy *UploadPolicy } func NewSigner(store Store, baseURL string, secret []byte) (*Signer, error) { @@ -51,6 +52,10 @@ func (s *Signer) PresignGet(ctx context.Context, key string, ttl time.Duration) // ServeHTTP serves signed downloads as attachments. Issue links only after // authorization; possession of an unexpired URL grants access to its object. func (s *Signer) ServeHTTP(w http.ResponseWriter, r *http.Request) { + if r.Method == http.MethodPut && s.uploadPolicy != nil { + s.serveUpload(w, r) + return + } if r.Method != "GET" && r.Method != "HEAD" { w.Header().Set("Allow", "GET, HEAD") http.Error(w, "method not allowed", 405) diff --git a/storage/signedUpload.go b/storage/signedUpload.go new file mode 100644 index 0000000..b049758 --- /dev/null +++ b/storage/signedUpload.go @@ -0,0 +1,98 @@ +package storage + +import ( + "context" + "crypto/hmac" + "crypto/sha256" + "encoding/base64" + "errors" + "io" + "net/http" + "net/url" + "strconv" + "time" +) + +// NewUploadSigner extends the local download handler with signed PUT requests. +// No global CORS policy is installed; applications choose their allowed origins. +// URLs are reusable until expiry. Use random staging keys and authorize issuance. +func NewUploadSigner(store Store, baseURL string, secret []byte, policy UploadPolicy) (*Signer, error) { + if err := policy.Validate(); err != nil { + return nil, err + } + s, err := NewSigner(store, baseURL, secret) + if err != nil { + return nil, err + } + policy.ContentTypes = append([]string(nil), policy.ContentTypes...) + s.uploadPolicy = &policy + return s, nil +} +func (s *Signer) uploadSignature(key, expiry, size, contentType string) string { + mac := hmac.New(sha256.New, s.secret) + io.WriteString(mac, "PUT\n"+key+"\n"+expiry+"\n"+size+"\n"+contentType) + return base64.RawURLEncoding.EncodeToString(mac.Sum(nil)) +} +func (s *Signer) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (UploadRequest, error) { + if s.uploadPolicy == nil { + return UploadRequest{}, ErrUnsupported + } + if contentType == "" { + contentType = "application/octet-stream" + } + if err := CheckUpload(ctx, key, size, s.uploadPolicy.MaxBytes, contentType, ttl); err != nil { + return UploadRequest{}, err + } + if err := s.uploadPolicy.Check(size, contentType); err != nil { + return UploadRequest{}, err + } + expires := s.now().Add(ttl).Truncate(time.Second) + expiry, n := strconv.FormatInt(expires.Unix(), 10), strconv.FormatInt(size, 10) + u := *s.base + u.RawQuery = url.Values{"key": {key}, "expires": {expiry}, "size": {n}, "contentType": {contentType}, "signature": {s.uploadSignature(key, expiry, n, contentType)}}.Encode() + return UploadRequest{Method: http.MethodPut, URL: u.String(), Headers: http.Header{"Content-Length": {n}, "Content-Type": {contentType}}, ExpiresAt: expires}, nil +} +func (s *Signer) serveUpload(w http.ResponseWriter, r *http.Request) { + w.Header().Set("Cache-Control", "no-store") + q, err := url.ParseQuery(r.URL.RawQuery) + valid := err == nil && len(q) == 5 + for _, key := range []string{"key", "expires", "size", "contentType", "signature"} { + valid = valid && len(q[key]) == 1 + } + if !valid { + http.Error(w, "invalid link", 403) + return + } + key, expiry, n, contentType := q.Get("key"), q.Get("expires"), q.Get("size"), q.Get("contentType") + expires, err := strconv.ParseInt(expiry, 10, 64) + if err != nil || expires <= s.now().Unix() || expires > s.now().Unix()+3600 || Check(r.Context(), key) != nil || !hmac.Equal([]byte(q.Get("signature")), []byte(s.uploadSignature(key, expiry, n, contentType))) { + http.Error(w, "invalid or expired link", 403) + return + } + size, err := strconv.ParseInt(n, 10, 64) + if err != nil || s.uploadPolicy.Check(size, contentType) != nil || r.ContentLength != size || len(r.Header.Values("Content-Type")) != 1 || r.Header.Get("Content-Type") != contentType { + http.Error(w, "upload constraints mismatch", 400) + return + } + body := http.MaxBytesReader(w, r.Body, size) + defer body.Close() + uploader, _ := NewUploader(s.store, *s.uploadPolicy) + _, err = uploader.Put(r.Context(), key, body, size, contentType) + if err != nil { + status := 500 + var limit *http.MaxBytesError + switch { + case errors.Is(err, ErrTooLarge), errors.As(err, &limit): + status = 413 + case errors.Is(err, ErrContentType): + status = 415 + case errors.Is(err, ErrSizeMismatch): + status = 400 + } + http.Error(w, "upload rejected", status) + return + } + w.WriteHeader(http.StatusCreated) +} + +var _ UploadPresigner = (*Signer)(nil) diff --git a/storage/storagetest/contract.go b/storage/storagetest/contract.go index 63d16c7..9ff15f6 100644 --- a/storage/storagetest/contract.go +++ b/storage/storagetest/contract.go @@ -9,11 +9,13 @@ import ( "io" "strings" "testing" + "time" ) func Run(t *testing.T, store storage.Store) { t.Helper() - ctx := context.Background() + ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second) + defer cancel() key := "contract/文件 +%.txt" defer store.Delete(ctx, key) if _, err := store.Get(ctx, key); !errors.Is(err, storage.ErrNotFound) { @@ -37,13 +39,19 @@ func Run(t *testing.T, store storage.Store) { if _, err := store.Put(ctx, "tooBig", strings.NewReader(strings.Repeat("a", 65)), 65, "text/plain"); !errors.Is(err, storage.ErrTooLarge) { t.Fatal(err) } + for _, size := range []int64{0, 2, 4} { + if _, err := store.Put(ctx, "sizeMismatch", strings.NewReader("abc"), size, "text/plain"); err == nil { + t.Fatalf("declared size %d accepted for 3 bytes", size) + } + } + defer store.Delete(context.Background(), "sizeMismatch") for _, key := range []string{"../bad", "a/../b", "/absolute", "a//b", "a\\b"} { if _, err := store.Put(ctx, key, strings.NewReader("a"), 1, "text/plain"); err == nil { t.Fatal(key) } } - canceled, cancel := context.WithCancel(ctx) - cancel() + canceled, cancelRequest := context.WithCancel(ctx) + cancelRequest() if _, err := store.Get(canceled, key); !errors.Is(err, context.Canceled) { t.Fatal(err) } diff --git a/storage/storagetest/uploads.go b/storage/storagetest/uploads.go new file mode 100644 index 0000000..2bb8965 --- /dev/null +++ b/storage/storagetest/uploads.go @@ -0,0 +1,127 @@ +package storagetest + +import ( + "context" + "io" + "net/http" + "strconv" + "strings" + "testing" + "time" + + "git.yuntuo.hk/k2safe/go-omni/storage" +) + +type UploadStore interface { + storage.Store + storage.UploadPresigner + storage.MultipartStore + storage.PartPresigner +} + +func sendUpload(t *testing.T, client *http.Client, link storage.UploadRequest, body string) *http.Response { + t.Helper() + req, err := http.NewRequest(link.Method, link.URL, strings.NewReader(body)) + if err != nil { + t.Fatal(err) + } + req.Header = link.Headers.Clone() + resp, err := client.Do(req) + if err != nil { + t.Fatal(err) + } + if resp.StatusCode < 200 || resp.StatusCode >= 300 { + raw, _ := io.ReadAll(io.LimitReader(resp.Body, 4096)) + resp.Body.Close() + t.Fatalf("upload rejected: %d %s", resp.StatusCode, raw) + } + return resp +} + +// RunUploads exercises both SDK upload and direct HTTP requests against a +// caller-owned server with maxBytes >=6 MiB. The caller cleans its bucket. +func RunUploads(t *testing.T, s UploadStore, client *http.Client) { + t.Helper() + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + defer cancel() + link, err := s.PresignPut(ctx, "direct", 5, "text/plain", time.Minute) + if err != nil { + t.Fatal(err) + } + if link.Method != "PUT" || link.Headers.Get("Content-Length") != "5" || link.Headers.Get("Content-Type") != "text/plain" { + t.Fatal("missing constraints", link.Headers) + } + resp := sendUpload(t, client, link, "hello") + resp.Body.Close() + defer s.Delete(context.Background(), "direct") + object, err := s.Get(ctx, "direct") + if err != nil { + t.Fatal(err) + } + raw, _ := io.ReadAll(object.Body) + object.Body.Close() + if string(raw) != "hello" { + t.Fatal("bad direct body") + } + if _, err = s.PresignPut(ctx, "direct", -1, "text/plain", time.Minute); err == nil { + t.Fatal("negative size accepted") + } + if _, err = s.PresignPut(ctx, "direct", 1, "text/plain", 2*time.Hour); err == nil { + t.Fatal("long TTL accepted") + } + upload, err := s.CreateMultipart(ctx, "multipart", "text/plain") + if err != nil { + t.Fatal(err) + } + defer s.AbortMultipart(context.Background(), upload) + data := strings.Repeat("a", 5<<20) + first, err := s.UploadPart(ctx, upload, 1, strings.NewReader(data), int64(len(data))) + if err != nil { + t.Fatal(err) + } + link, err = s.PresignPart(ctx, upload, 2, 5, time.Minute) + if err != nil { + t.Fatal(err) + } + resp = sendUpload(t, client, link, "world") + etag := resp.Header.Get("ETag") + resp.Body.Close() + if etag == "" { + t.Fatal("no part ETag") + } + page, err := s.ListParts(ctx, upload, 0, 1) + if err != nil || len(page.Parts) != 1 || page.NextMarker != 1 { + t.Fatal(page, err) + } + next, err := s.ListParts(ctx, upload, page.NextMarker, 1) + if err != nil || len(next.Parts) != 1 || next.Parts[0].Number != 2 || next.Parts[0].Size != 5 || next.NextMarker != 0 { + t.Fatal(next, err) + } + first.Size = 1 // The provider's size must prevail over caller metadata. + info, err := s.CompleteMultipart(ctx, upload, []storage.Part{first, {Number: 2, ETag: etag, Size: 1}}) + if err != nil || info.Size != int64(len(data)+5) { + t.Fatal(info, err) + } + defer s.Delete(context.Background(), "multipart") + object, err = s.Get(ctx, "multipart") + if err != nil { + t.Fatal(err) + } + raw, err = io.ReadAll(object.Body) + object.Body.Close() + if err != nil || string(raw) != data+"world" { + t.Fatal("multipart bytes mismatch", len(raw), err) + } + canceled, err := s.CreateMultipart(ctx, "canceled", "text/plain") + if err != nil { + t.Fatal(err) + } + for i := 0; i < 2; i++ { + if err = s.AbortMultipart(ctx, canceled); err != nil { + t.Fatal("abort "+strconv.Itoa(i), err) + } + } + if _, err = s.ListParts(ctx, canceled, 0, 1); err == nil { + t.Fatal("aborted upload remains") + } +} diff --git a/storage/upload.go b/storage/upload.go new file mode 100644 index 0000000..0f177a0 --- /dev/null +++ b/storage/upload.go @@ -0,0 +1,219 @@ +package storage + +import ( + "bytes" + "context" + "errors" + "io" + "mime" + "net/http" + "strings" + "time" +) + +var ErrUnsupported = errors.New("storage: capability not supported") +var ErrContentType = errors.New("storage: content type not allowed") +var ErrSizeMismatch = errors.New("storage: declared size mismatch") + +// UploadRequest is a short-lived bearer capability; never log its URL. Headers +// must accompany the request. Browsers derive Content-Length from the Blob body. +type UploadRequest struct { + Method string `json:"method"` + URL string `json:"url"` + Headers http.Header `json:"headers"` + ExpiresAt time.Time `json:"expiresAt"` +} + +type UploadPresigner interface { + PresignPut(context.Context, string, int64, string, time.Duration) (UploadRequest, error) +} + +// UploadPolicy contains only transport-neutral constraints. ContentTypes is an +// exact MIME allowlist (parameters are ignored). Empty permits any valid type. +type UploadPolicy struct { + MaxBytes int64 `json:"maxBytes"` + ContentTypes []string `json:"contentTypes"` +} + +func (p UploadPolicy) Validate() error { + if p.MaxBytes <= 0 || p.MaxBytes > 5<<40 { + return errors.New("storage: maxBytes must be 1..5 TiB") + } + for _, value := range p.ContentTypes { + t, err := MediaType(value) + if err != nil || strings.Contains(t, "*") { + return ErrContentType + } + } + return nil +} +func MediaType(value string) (string, error) { + if value == "" { + return "application/octet-stream", nil + } + if len(value) > 256 || strings.ContainsAny(value, "\r\n") { + return "", ErrContentType + } + t, _, err := mime.ParseMediaType(value) + if err != nil || !strings.Contains(t, "/") || strings.Contains(t, "*") { + return "", ErrContentType + } + return strings.ToLower(t), nil +} +func (p UploadPolicy) Check(size int64, contentType string) error { + if err := p.Validate(); err != nil { + return err + } + if size < 0 { + return ErrSizeMismatch + } + if size > p.MaxBytes { + return ErrTooLarge + } + t, err := MediaType(contentType) + if err != nil { + return err + } + if len(p.ContentTypes) == 0 { + return nil + } + for _, allowed := range p.ContentTypes { + a, _ := MediaType(allowed) + if a == t { + return nil + } + } + return ErrContentType +} +func CheckUpload(ctx context.Context, key string, size, maxBytes int64, contentType string, ttl time.Duration) error { + if err := Check(ctx, key); err != nil { + return err + } + if err := (UploadPolicy{MaxBytes: maxBytes}).Check(size, contentType); err != nil { + return err + } + if ttl < time.Second || ttl > time.Hour { + return errors.New("storage: TTL must be 1s..1h") + } + return nil +} + +// PrepareUpload checks the first 512 bytes against the declared type when an +// allowlist is set, then streams the body with exact-length checking. It does +// not close the caller's reader. MIME sniffing is not antivirus/file validation. +func (p UploadPolicy) PrepareUpload(ctx context.Context, reader io.Reader, size int64, contentType string) (io.Reader, error) { + if ctx == nil || reader == nil { + return nil, errors.New("storage: context and reader required") + } + if err := ctx.Err(); err != nil { + return nil, err + } + if err := p.Check(size, contentType); err != nil { + return nil, err + } + var r io.Reader = ContextReader{Context: ctx, Reader: reader} + if len(p.ContentTypes) > 0 && size > 0 { + prefix := make([]byte, min(size, 512)) + if _, err := io.ReadFull(r, prefix); err != nil { + return nil, errors.Join(ErrSizeMismatch, err) + } + detected, _ := MediaType(http.DetectContentType(prefix)) + declared, _ := MediaType(contentType) + if detected != declared { + return nil, ErrContentType + } + r = io.MultiReader(bytes.NewReader(prefix), r) + } + if size == 0 { + var b [1]byte + n, err := io.ReadFull(r, b[:]) + if n > 0 { + return nil, ErrSizeMismatch + } + if err != io.EOF { + return nil, err + } + return bytes.NewReader(nil), nil + } + return &exactReader{reader: r, remaining: size}, nil +} + +type exactReader struct { + reader io.Reader + remaining int64 + done bool +} + +func (r *exactReader) Read(p []byte) (int, error) { + if len(p) == 0 { + return 0, nil + } + if r.done { + return 0, io.EOF + } + if int64(len(p)) > r.remaining { + p = p[:r.remaining] + } + n, err := r.reader.Read(p) + r.remaining -= int64(n) + if r.remaining == 0 { + r.done = true + // Probe before handing the last byte to a known-length SDK, which might + // otherwise stop reading without observing a trailing error. + var b [1]byte + extra, probeErr := io.ReadFull(r.reader, b[:]) + if extra > 0 { + return max(0, n-1), ErrSizeMismatch + } + if probeErr != io.EOF { + return max(0, n-1), probeErr + } + if err != nil && err != io.EOF { + return max(0, n-1), err + } + return n, io.EOF + } + if err == io.EOF { + r.done = true + return n, ErrSizeMismatch + } + return n, err +} + +// Uploader applies a policy without coupling the framework to business fields. +// Store remains unchanged; optional operations return ErrUnsupported when absent. +type Uploader struct { + store Store + policy UploadPolicy +} + +func NewUploader(store Store, policy UploadPolicy) (*Uploader, error) { + if store == nil { + return nil, errors.New("storage: store required") + } + if err := policy.Validate(); err != nil { + return nil, err + } + policy.ContentTypes = append([]string(nil), policy.ContentTypes...) + return &Uploader{store: store, policy: policy}, nil +} +func (u *Uploader) Put(ctx context.Context, key string, body io.Reader, size int64, contentType string) (Info, error) { + if err := Check(ctx, key); err != nil { + return Info{}, err + } + reader, err := u.policy.PrepareUpload(ctx, body, size, contentType) + if err != nil { + return Info{}, err + } + return u.store.Put(ctx, key, reader, size, contentType) +} +func (u *Uploader) PresignPut(ctx context.Context, key string, size int64, contentType string, ttl time.Duration) (UploadRequest, error) { + if err := u.policy.Check(size, contentType); err != nil { + return UploadRequest{}, err + } + signer, ok := u.store.(UploadPresigner) + if !ok { + return UploadRequest{}, ErrUnsupported + } + return signer.PresignPut(ctx, key, size, contentType, ttl) +} diff --git a/storage/upload_test.go b/storage/upload_test.go new file mode 100644 index 0000000..f987d17 --- /dev/null +++ b/storage/upload_test.go @@ -0,0 +1,128 @@ +package storage_test + +import ( + "bytes" + "context" + "errors" + "io" + "net/http" + "net/http/httptest" + "net/url" + "strings" + "testing" + "time" + + "git.yuntuo.hk/k2safe/go-omni/storage" + "git.yuntuo.hk/k2safe/go-omni/storage/local" +) + +func TestUploadPolicyActualBytes(t *testing.T) { + ctx := context.Background() + files, _ := local.New(t.TempDir(), 1024) + uploader, err := storage.NewUploader(files, storage.UploadPolicy{MaxBytes: 64, ContentTypes: []string{"text/plain"}}) + if err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + name, body, contentType string + size int64 + want error + }{ + {"valid", "hello", "text/plain; charset=utf-8", 5, nil}, + {"tooLong", "hello!", "text/plain", 5, storage.ErrSizeMismatch}, + {"tooShort", "hello", "text/plain", 6, storage.ErrSizeMismatch}, + {"emptyMismatch", "x", "text/plain", 0, storage.ErrSizeMismatch}, + {"overLimit", "hello", "text/plain", 65, storage.ErrTooLarge}, + {"declaredType", "hello", "image/png", 5, storage.ErrContentType}, + {"forgedType", "hello", "text/plain", 18, storage.ErrContentType}, + } { + t.Run(tc.name, func(t *testing.T) { + _, err := uploader.Put(ctx, tc.name, strings.NewReader(tc.body), tc.size, tc.contentType) + if !errors.Is(err, tc.want) { + t.Fatalf("%v want %v", err, tc.want) + } + object, getErr := files.Get(ctx, tc.name) + if getErr == nil { + object.Body.Close() + } + if tc.want != nil && !errors.Is(getErr, storage.ErrNotFound) { + t.Fatal("rejected object committed", getErr) + } + }) + } + canceled, cancel := context.WithCancel(ctx) + cancel() + if _, err := uploader.Put(canceled, "cancel", strings.NewReader("x"), 1, "text/plain"); !errors.Is(err, context.Canceled) { + t.Fatal(err) + } + if _, err := uploader.CreateMultipart(ctx, "local", "text/plain"); !errors.Is(err, storage.ErrUnsupported) { + t.Fatal(err) + } + if _, err := uploader.PresignPut(ctx, "local", 1, "text/plain", time.Minute); !errors.Is(err, storage.ErrUnsupported) { + t.Fatal(err) + } +} +func TestSignedUploadConstraintsAndAtomicity(t *testing.T) { + files, _ := local.New(t.TempDir(), 1024) + signer, err := storage.NewUploadSigner(files, "https://files.example/upload", bytes.Repeat([]byte{1}, 32), storage.UploadPolicy{MaxBytes: 64, ContentTypes: []string{"text/plain"}}) + if err != nil { + t.Fatal(err) + } + link, err := signer.PresignPut(context.Background(), "file", 5, "text/plain", time.Minute) + if err != nil { + t.Fatal(err) + } + for _, tc := range []struct { + name, body string + mutate func(*http.Request) + status int + }{ + {"success", "hello", nil, 201}, + {"key", "hello", func(r *http.Request) { q := r.URL.Query(); q.Set("key", "other"); r.URL.RawQuery = q.Encode() }, 403}, + {"expiry", "hello", func(r *http.Request) { q := r.URL.Query(); q.Set("expires", "1"); r.URL.RawQuery = q.Encode() }, 403}, + {"type", "hello", func(r *http.Request) { r.Header.Set("Content-Type", "image/png") }, 400}, + {"length", "hello", func(r *http.Request) { r.ContentLength = 6 }, 400}, + {"extraBytes", "longer", func(r *http.Request) { r.ContentLength = 5 }, 413}, + {"shortBody", "no", func(r *http.Request) { r.ContentLength = 5 }, 400}, + {"duplicateKey", "hello", func(r *http.Request) { q := r.URL.Query(); q.Add("key", "file"); r.URL.RawQuery = q.Encode() }, 403}, + } { + t.Run(tc.name, func(t *testing.T) { + req := httptest.NewRequest("PUT", link.URL, strings.NewReader(tc.body)) + req.Header = link.Headers.Clone() + if tc.mutate != nil { + tc.mutate(req) + } + w := httptest.NewRecorder() + signer.ServeHTTP(w, req) + if w.Code != tc.status { + t.Fatalf("status=%d want %d, %s", w.Code, tc.status, w.Body.String()) + } + }) + } + object, err := files.Get(context.Background(), "file") + if err != nil { + t.Fatal(err) + } + data, _ := io.ReadAll(object.Body) + object.Body.Close() + if string(data) != "hello" { + t.Fatal("failed upload overwrote good object") + } + download, _ := signer.PresignGet(context.Background(), "file", time.Minute) + req := httptest.NewRequest("PUT", download.String(), strings.NewReader("hello")) + req.Header = link.Headers.Clone() + w := httptest.NewRecorder() + signer.ServeHTTP(w, req) + if w.Code != 403 { + t.Fatal("GET signature reused for PUT") + } + // Original read-only constructor must never accept signed uploads. + readOnly, _ := storage.NewSigner(files, "https://files.example/upload", bytes.Repeat([]byte{1}, 32)) + if _, err = readOnly.PresignPut(context.Background(), "file", 1, "text/plain", time.Minute); !errors.Is(err, storage.ErrUnsupported) { + t.Fatal(err) + } + u, _ := url.Parse(link.URL) + if u.Query().Get("signature") == "" { + t.Fatal("unsigned URL") + } +} diff --git a/transport/native/earlyReject_test.go b/transport/native/earlyReject_test.go new file mode 100644 index 0000000..dd59eed --- /dev/null +++ b/transport/native/earlyReject_test.go @@ -0,0 +1,60 @@ +package native_test + +import ( + "bufio" + "context" + "fmt" + "io" + "net" + "net/http" + "sync/atomic" + "testing" + "time" + + "git.yuntuo.hk/k2safe/go-omni/transport/native" +) + +func TestApplicationRejectsUnreadBodyPromptly(t *testing.T) { + for _, framing := range []string{"Content-Length: 1048577\r\n", "Transfer-Encoding: chunked\r\n"} { + t.Run(framing[:8], func(t *testing.T) { + var calls atomic.Int32 + server := native.New(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + calls.Add(1) + w.Header().Set("Content-Type", "application/json") + w.WriteHeader(http.StatusRequestEntityTooLarge) + _, _ = io.WriteString(w, `{"code":"BODY_TOO_LARGE"}`) + })) + // Much longer than the client's deadline: a draining implementation fails. + server.ReadTimeout = 10 * time.Second + listener, err := net.Listen("tcp", "127.0.0.1:0") + if err != nil { + t.Fatal(err) + } + done := make(chan error, 1) + go func() { done <- server.Serve(context.Background(), listener) }() + defer stopServer(t, server, listener, done) + conn, err := net.DialTimeout("tcp", listener.Addr().String(), time.Second) + if err != nil { + t.Fatal(err) + } + defer conn.Close() + _ = conn.SetDeadline(time.Now().Add(time.Second)) + if _, err = fmt.Fprintf(conn, "POST /upload HTTP/1.1\r\nHost: local\r\n%s\r\n", framing); err != nil { + t.Fatal(err) + } + response, err := http.ReadResponse(bufio.NewReader(conn), &http.Request{Method: "POST"}) + if err != nil { + t.Fatal(err) + } + body, err := io.ReadAll(response.Body) + _ = response.Body.Close() + if err != nil || response.StatusCode != 413 || !response.Close || string(body) != `{"code":"BODY_TOO_LARGE"}` { + t.Fatalf("%+v %s %v", response, body, err) + } + requirePeerClose(t, conn) + if calls.Load() != 1 { + t.Fatalf("handler calls=%d", calls.Load()) + } + }) + } +} diff --git a/transport/native/server.go b/transport/native/server.go index a2c76c0..e5e784c 100644 --- a/transport/native/server.go +++ b/transport/native/server.go @@ -481,7 +481,17 @@ func (s *Server) serveConnection(baseCtx context.Context, state *connection) { disconnectWatch = newClientDisconnectWatch(reader, state.conn, cancel) } handler.ServeHTTP(writer, request) - bodyErr := request.Body.Close() + var bodyErr error + if body, ok := request.Body.(*requestBody); ok && !body.done && body.err == nil && writer.status == http.StatusRequestEntityTooLarge { + // An application can reject size from the headers alone when its + // body limit is below the transport limit. Waiting to + // drain a slow or absent body would withhold the rejection response. + // Abandon it and close; unread bytes can never become a next request. + body.closed = true + closeAfter = true + } else { + bodyErr = request.Body.Close() + } if disconnectWatch != nil { disconnectWatch.stop() }